Computing apparatus and methods for image pixel processing networks

Abstract

The application provides a computing device and method for an image pixel processing network, the device comprising an external storage, a bus controller, an address controller, an input buffer, a weight buffer, a bias buffer, an output buffer, and q block computing units. The application proposes a novel hardware architecture which can convert various convolutions into specific deformable convolutions for calculation to flexibly support multiple types of convolution layers. The application also designs corresponding computing units and a brand-new decoupled computing process to unify diverse memory access modes and ensure the consistency of input / output layout formats.

Description

Technical Field

[0001] This invention relates to a computing device and method for image pixel processing networks. Background Technology

[0002] Deep learning algorithms are rapidly evolving and have been applied in numerous vision applications. In particular, Convolutional Neural Networks (CNNs), Deformable Convolutional Networks (DCNs), and Generative Adversarial Networks (GANs) have demonstrated unprecedented performance in image pixel processing tasks such as super-resolution (SR), low-light enhancement (LLE), denoising, and painting. Table 1 shows four main operation types widely used in image pixel processing networks: convolution (Conv), deconvolution (Deconv), deformable convolution (Dfconv), and dilated convolution (Diconv). Compared to traditional deep neural networks, which primarily target high-level vision tasks such as object detection and image classification, these image pixel processing networks aim to transform input images into enhanced images, involving more types of downsampling / upsampling functions and operations of various kernel sizes. This urgently requires the development of a flexible hardware accelerator that provides high efficiency for a variety of low-level vision tasks. Table 1 lists the types of convolution computations in mainstream image pixel processing networks.

[0003] Table 1

[0004]

[0005] In Table 1, Reference 1 is C. Dong, CC Loy, and X. Tang, “Accelerating the super-resolution convolutional neural network,” in European conference on computer vision. Springer, 2016, pp. 391–407;

[0006] Document 2 is S.Zhang, W.Mao, and Z.Wang, "An efficient accelerator ofdeformable 3dconvolutional network for video super-resolution," in 2022IEEEComputer Society Annual Symposium on VLSI(ISVLSI).IEEE, 2022, pp.110–115;

[0007] Reference 3 is K. Zhang, W. Zuo, S. Gu, and L. Zhang, “Learning deep cnn denoiser prior for image restoration,” in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 2808–2817;

[0008] Reference 4 is J. Guan, R. Lai, Y. Lu, Y. Li, H. Li, L. Feng, Y. Yang, and L. Gu,

[0009] “Memory-efficient deformable convolution based joint denoising and demosaicing for uhd images,” IEEE Transactions on Circuits and Systems for Video Technology, pp. 1–1, 2022;

[0010] Reference 5 is D. Pathak, P. Krahenbühl, J. Donahue, T. Darrell, and A. A. Efros,

[0011] “Context encoders: Feature learning by inpainting,” in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 2536–​​​​​​​​Document 7 is E.de Stoutz, A.Ignatov, N.Kobyshev, R.Timofte, and L.Van Gool, "Fast perceptual image enhancement," in Proceedings of the European Conference on Computer Vision (ECCV) Workshops, 2018, pp.0–0;

[0015] To effectively deploy image pixel processing networks to edge devices, some researchers have proposed dedicated accelerators for super-resolution tasks (see J. Lee, J. Lee, and H.-J. Yoo, “Srnpu: An energy-efficient cnn-based super-resolution processor with tile-based selective super-resolution in mobile devices,” IEEE Journal on Emerging and Selected Topics in Circuits and Systems, vol. 10, no. 3, pp. 320–334, 2020). However, these accelerators are not flexible enough and cannot support a variety of operations in different image pixel processing networks, thus limiting their scalability to real-world scenarios. Then, the paper S. Li, S. Cao, L. Hui, Z. Jiang, Y. Sun, and S. Xu, “A computational efficient deformable convolution network accelerator via hardware and algorithm co-optimization,” in 2022 IEEE Workshop on Signal Processing Systems (SiPS), 2022, pp. 1–6, proposed a framework for co-optimization of algorithm and hardware for deformable convolutional networks to perform image denoising and demosaic tasks, but it only supports standard convolution and simplified deconvolution.

[0016] Developing a flexible accelerator for various operations of image pixel processing networks presents three main challenges. First, deformable convolutions add overhead to bilinear interpolation and deformable offsets, leading to dynamic and irregular input access patterns. Compared to standard convolutions, they have higher computational complexity and limited opportunities for data reuse of input features, posing a significant challenge to hardware acceleration. Furthermore, the completely different computational and memory access patterns result in high hardware costs for supporting both convolutions and other types of convolutions simultaneously. Second, convolutions with different kernel sizes have different memory and multiplexer requirements. For example, the overlap area of ​​adjacent convolutional windows varies due to different kernel sizes, requiring additional memory and multiplexers to handle the most demanding cases. Third, input and output data may have significant differences in data layout format, especially in deconvolutions, necessitating data rearrangement to unify them before computing the next layer, thus increasing system overhead.

[0017] The article X. Wu, Y. Ma, M. Wang and Z. Wang, "A Flexible and Efficient FPGA Accelerator for Various Large-Scale and Lightweight CNNs," in IEEE Transactions on Circuits and Systems I: Regular Papers, vol. 69, no. 3, pp. 1185-1198, March 2022, doi:10.1109 / TCSI.2021.3131581, also supports various convolution operations, including standard convolution, deconvolution, and dilated convolution. Its approach is to adopt a common block-based data storage method. Different optimization schemes are used for different operators, and different hardware modules are designed to implement them. Finally, all modules are integrated into a single hardware architecture. This architecture shares computing units when computing different operators, but each uses its own dedicated optimization module to achieve the effect of supporting multiple convolutions.

[0018] Data storage method: Off-chip storage employs a layer-by-layer approach. The movement of data from off-chip to on-chip memory undergoes specific transformations depending on the corresponding computing mode, thus implementing the appropriate storage model. On-chip storage also includes dedicated modules to handle serial-to-parallel conversion.

[0019] Specifically, optimization is achieved by employing a zero-transfer and skipping method for deconvolution. Directly incorporating deconvolution into the computational units of standard convolution leads to an unbalanced computational load. By systematically reorganizing the input feature map according to the positions of the corresponding weight elements in the kernel graph, a transformed feature map can be obtained. In this way, the deconvolution computation can be reconstructed into a fused convolution with a stride of 2. The fused deconvolution consists of four parse convolutions with regularly positioned non-zero weights, thus resolving the load imbalance problem.

[0020] For dilated convolutions, a sparsity-like processing approach is employed for optimization. Dilated convolutional layers perform multiplication and accumulation operations on the enlarged kernel map by inserting zero elements to maintain the feature map size while increasing the receptive field. It can be viewed as a form of weighted sparse convolution with regular sparsity, depending on its dilation rate. This approach uses a sparse method to process dilated convolutions, concatenating data streams that need to be computed together and designing corresponding concatenation strategies. Summary of the Invention

[0021] Purpose of the Invention: Deep neural networks, including convolutional neural networks, deformable convolutional networks, and generative adversarial networks, have shown great potential in image pixel processing tasks, such as image super-resolution and denoising. These different networks bring different memory access methods and complex computational patterns, posing significant challenges to their hardware deployment. To address these complex computational patterns, this invention specifically provides a computing device for image pixel processing networks, including external storage, a bus controller, an address controller, an input buffer, a weight buffer, a bias buffer, an output buffer, and q block computing units, where q represents the number of block computing units.

[0022] Each block computing unit includes n normalized computing modules, where n represents the number of normalized computing modules in each block computing unit; each normalized computing module includes m multipliers, an adder array, and an accumulator; m represents the number of multipliers in each normalized computing module; q, n, and m are all natural numbers greater than 0.

[0023] The external storage is used to store processed pixel information;

[0024] The bus controller is used to exchange pixel information stored externally with the input buffer, weight buffer, bias buffer, and output buffer.

[0025] The address controller is used to provide data addresses and control signals to the bus controller, input buffer, weight buffer, bias buffer, and output buffer, thereby performing address control.

[0026] The block computing unit is used to calculate the segmented portion of the image. One block computing unit is composed of n normalization computing modules.

[0027] For a block computation unit, the input activation is broadcast from the input buffer to n normalized computation modules, and the weights of the n output channels are respectively input from the weight buffer to the n normalized computation modules to compute the n output channels simultaneously; the input activation is processed pixel information.

[0028] The normalization calculation module is used to calculate the pixel result of one output channel of the result image. The simultaneous calculation of n output channels is achieved by combining n normalization calculation modules.

[0029] For a normalized computation module, the input activation and weight activation are transferred from the input buffer and weight buffer to m multipliers for computation. The result of the multiplier is processed by the adder array and accumulator for output. Due to the computational characteristics of neural networks, the result of the accumulator computation requires more than two cycles to obtain an output result. The output result is transferred to the output buffer when each data channel level loop is completed.

[0030] When a block computation unit processes an input block, the address controller generates a sampled address for deformable inputs or weights. Then, the corresponding activations of m input channels with the same height and width dimensions Phw are read from the input buffer and broadcast to n normalized computation modules. The corresponding activations of the m input channels with the same height and width dimensions Phw are represented as A0, B0, ..., H0.

[0031] The letters in A0, B0, ..., H0 represent the channel encoding of the image patch, and the numbers represent the position encoding of the image patch. Simultaneously, the weights of the m input channels in an output channel are sent to a normalization calculation module. After successive multiplication, the result is sent to an adder array for channel-wise summation. Due to limited hardware resources, it is difficult to simultaneously load all feature maps used in the image pixel processing network onto the chip. Therefore, the feature maps are segmented into two or more image patches as input blocks. Since the task is related to neural networks, there is a channel dimension, which is a new dimension brought about by multiple images side-by-side. Therefore, this image expands from a two-dimensional concept to a three-dimensional structure, called a block. The input buffer stores the data of the input blocks.

[0032] After the normalization module completes one cycle of computation, its result, i.e., the partial sum, is sent to the accumulator for sequential kernel-level and channel-level accumulation loops. Within a block computation unit, each height and width dimension Phw is stored at an independent address, thus decoupling the positional dependencies within the deformable receiver field. "Partial sum" is a technical term used to describe the intermediate result obtained before the final result is obtained during the accumulation process.

[0033] To improve the throughput of the hardware design, the device processes two or more input blocks simultaneously.

[0034] The device includes q input buffers and bias buffers, which are used to store the pixels and bias values ​​of q input blocks, respectively. The output results of the q block calculation units are sent to the corresponding libraries of the output buffers. There is one weight buffer, which is used to broadcast weights in the q input blocks.

[0035] The device executes a field unification algorithm, unifying the calculation modes of standard convolution, dilated convolution, and deconvolution into the calculation mode of deformable convolution. Deformable convolution, standard convolution, dilated convolution, and deconvolution are all common calculation methods in image pixel processing networks. Specifically, it includes the following steps:

[0036] Step a1: Standard convolution divides the input image into parts the same size as the convolution kernel and then performs the convolution operation. The position of each part on the input image is fixed. Deformable convolution adds a direction parameter to the selection of pixels in the input image, allowing for a wider range of pixels to be selected as activation points compared to standard convolution. The calculation process of deformable convolution is as follows:

[0037]

[0038] Where Ω represents the receptive field of the input feature map, p0 is the center position of the standard convolutional window, and p n Δ represents the two-dimensional coordinates of the original sample position in Ω. pn p represents the original sample location n The two-dimensional deformable offset. w and u represent the weight and activation, respectively; y(p0) represents the output pixel at position P0;

[0039] Step a2, based on the calculation process of deformable convolution and the description of standard convolution in step a1, when Δ in the deformable convolution calculation formula... pn When the value is 0, deformable convolution is transformed into standard convolution. Therefore, standard convolution is a special case of deformable convolution.

[0040] Step a3: Compared to standard convolution, dilated convolution introduces a scaling parameter, causing the input image pixels to be selected according to the scaling ratio during the convolution operation, and the convolution operation is performed using the expanded pixels; according to the calculation formula for deformable convolution in step a1, when Δ pn When the two-dimensional deformable offset represented by it satisfies the variation law of dilated convolution, dilated convolution is regarded as a special case of deformable convolution.

[0041] Step a4: Compared to standard convolution, deconvolution inserts zero values ​​between the pixels of the input image in standard convolution to form a completely new input image, and then performs calculations according to the standard convolution calculation process. In fact, deconvolution also shifts the pixels of the input image in the convolution operation. According to the calculation formula of deformable convolution in step a1, when the two-dimensional deformable offset represented by Δpn satisfies the change law of deconvolution, deconvolution is regarded as a special case of deformable convolution.

[0042] Step a5: Based on the descriptions in steps a1, a2, a3, and a4, standard convolution, dilated convolution, and deconvolution can all be regarded as special cases of deformable convolution. Thus, the calculation modes of standard convolution, dilated convolution, and deconvolution are unified into the calculation mode of deformable convolution, which is the field unification algorithm.

[0043] This invention also provides a computational method for image pixel processing networks. It employs a novel image storage method and a field unification algorithm to unify the computational steps of standard convolution, dilated convolution, and deconvolution into the computational steps of deformable convolution. Furthermore, it distinguishes different computational modes using data stored in an bias buffer. Specifically, it includes the following steps:

[0044] Step 1: Since the storage method of the computing device for images is different from that of conventional systems, the computing device adopts the following new image storage method: storing pixels at the same position on different channels of the image into the same storage address of the corresponding storage unit. The computing device first stores the pixel values, weight values, and offset values ​​stored according to the storage method in external storage.

[0045] Step 2: The bus controller transmits the required data from external storage to q input buffers, one weight buffer, and q bias buffers. Since deformable convolutions have irregular offsets at the pixel level, the values ​​stored in the bias buffers are the corresponding offset values. Dilated convolutions and deconvolutions can be regarded as deformable convolutions that meet specific constraints, so the values ​​stored in the bias buffers conform to the offset rules of their respective calculation modes. Standard convolutions can be regarded as deformable convolutions that do not have offsets, so the value stored in the bias buffers of standard convolutions is zero.

[0046] Step 3: The address controller generates the corresponding addresses of the data that the input buffer, weight buffer, and offset buffer need to obtain. During the address generation process, the address generated by the input buffer also needs to be updated according to the data obtained from the corresponding address in the offset buffer.

[0047] Step 4: After obtaining the data corresponding to the input buffer and the weight buffer, the data corresponding to the q input buffers are sent to the corresponding q block computing units. At the same time, the data corresponding to the address of one weight buffer is broadcast to the q block computing units.

[0048] Step 5: In each block computing unit, after receiving the input data and weight data from Step 4, the input data is broadcast to n normalization computing modules. Since one address stores the weight data of two or more output channels, the weight data is divided into n parts according to the channel and sent to the corresponding n normalization computing modules.

[0049] Step 6: The data received by each normalization calculation module is transmitted to m multipliers for product calculation. The results are then processed by adders and passed to an accumulator for accumulation. The calculation process described in step 6 is completed within one cycle.

[0050] Step 7: The calculations completed in Step 6 are performed within one cycle. Due to the complexity of pixel processing tasks, an output result may require more than two cycles to be calculated. Therefore, it is necessary to plan the data transmission loop: Since the proposed computing device uses a new data storage method, the corresponding data transmission loop will also change. The loop hierarchy is divided from high to low as follows: image block level, output channel level, weight window level, weight row level, and input channel level.

[0051] The calculations completed in one cycle as described in step 6 belong to the weight row-level calculation loop; after completing the weight row-level calculation loop, the weight window moves to the next position and selects a new input pixel for calculation; when the weight window moves to the last position, the calculation of the output channel is completed; after completing the output channel-level loop, the data of the new weight window is passed in to start a new loop, and finally, there is the image block-level loop.

[0052] In step 7, the input data and weight data are expanded and looped in three dimensions to achieve parallel computing. The three dimensions include the input channel level, the output channel level, and the image patch level.

[0053] Step 8: When the loop at the output channel level is completed, the accumulators of the n normalization calculation modules will output n valid output values ​​corresponding to the n output channels, thus simultaneously obtaining the output results of the same pixel position of the n output channels; the data storage format of the output buffer is the same as that of the input buffer, effectively solving the data storage format problem brought about by the previous system.

[0054] The present invention also provides a storage medium storing a computer program or instructions, which, when executed, implement the aforementioned computational method for an image pixel processing network.

[0055] Beneficial effects: This invention proposes a novel hardware architecture that can convert various convolutions into specific deformable convolutions for computation to flexibly support multiple types of convolutional layers. At the same time, it designs corresponding computation units and a brand-new decoupled computation process, thereby unifying diverse memory access modes and ensuring the consistency of input / output layout formats. Attached Figure Description

[0056] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments, and the advantages of the present invention in the above and / or other aspects will become clearer.

[0057] Figure 1a This is a diagram illustrating the on-chip data storage method.

[0058] Figure 1b This is the architecture diagram of the address controller.

[0059] Figure 1c This is the architecture diagram of the normalization calculation module.

[0060] Figure 1d This is an architecture diagram of a block processing unit used in the system.

[0061] Figure 2 This is the overall structure of the device of the present invention. Detailed Implementation

[0062] like Figure 2 As shown, the present invention specifically provides a computing device for an image pixel processing network, including external storage, a bus controller, an address controller, an input buffer, a weight buffer, a bias buffer, an output buffer, and q block computing units, where q represents the number of block computing units;

[0063] Each block computing unit includes n normalized computing modules, where n represents the number of normalized computing modules in each block computing unit; each normalized computing module includes m multipliers, an adder array, and an accumulator; m represents the number of multipliers in each normalized computing module; q, n, and m are all natural numbers greater than 0.

[0064] The external storage is used to store processed pixel information;

[0065] The bus controller is used to exchange pixel information stored externally with the input buffer, weight buffer, bias buffer, and output buffer.

[0066] The address controller is used to provide data addresses and control signals to the bus controller, input buffer, weight buffer, bias buffer, and output buffer, thereby performing address control.

[0067] The block computing unit is used to calculate the segmented portion of the image. One block computing unit is composed of n normalization computing modules.

[0068] For a block computation unit, the input activation is broadcast from the input buffer to n normalized computation modules, and the weights of the n output channels are respectively input from the weight buffer to the n normalized computation modules to compute the n output channels simultaneously; the input activation is processed pixel information.

[0069] The normalization calculation module is used to calculate the pixel result of one output channel of the result image. The simultaneous calculation of n output channels is achieved by combining n normalization calculation modules.

[0070] For a normalized computation module, the input activation and weight activation are transferred from the input buffer and weight buffer to m multipliers for computation. The result of the multiplier is processed by the adder array and accumulator for output. Due to the computational characteristics of neural networks, the result of the accumulator computation requires more than two cycles to obtain an output result. The output result is transferred to the output buffer when each data channel level loop is completed.

[0071] When a block computation unit processes an input block, the address controller generates a sampled address for deformable inputs or weights. Then, the corresponding activations of m input channels with the same height and width dimensions Phw are read from the input buffer and broadcast to n normalized computation modules. The corresponding activations of the m input channels with the same height and width dimensions Phw are represented as A0, B0, ..., H0.

[0072] The letters in A0, B0, ..., H0 represent the channel encoding of the image patch, and the numbers represent the position encoding of the image patch. Simultaneously, the weights of the m input channels in an output channel are sent to a normalization calculation module. After successive multiplication, the result is sent to an adder array for channel-wise summation. Due to limited hardware resources, it is difficult to simultaneously load all feature maps used in the image pixel processing network onto the chip. Therefore, the feature maps are segmented into two or more image patches as input blocks. Since the task is related to neural networks, there is a channel dimension, which is a new dimension brought about by multiple images side-by-side. Therefore, this image expands from a two-dimensional concept to a three-dimensional structure, called a block. The input buffer stores the data of the input blocks.

[0073] After the normalization module completes one cycle of computation, its result, i.e., the partial sum, is sent to the accumulator for sequential kernel-level and channel-level accumulation loops. Within a block computation unit, each height and width dimension Phw is stored at an independent address, thus decoupling the positional dependencies within the deformable receiver field. "Partial sum" is a technical term used to describe the intermediate result obtained before the final result is obtained during the accumulation process.

[0074] To improve the throughput of the hardware design, the device processes two or more input blocks simultaneously.

[0075] The device includes q input buffers and bias buffers, which are used to store the pixels and bias values ​​of q input blocks, respectively. The output results of the q block calculation units are sent to the corresponding libraries of the output buffers. There is one weight buffer, which is used to broadcast weights in the q input blocks.

[0076] The device executes a field unification algorithm, unifying the calculation modes of standard convolution, dilated convolution, and deconvolution into the calculation mode of deformable convolution. Deformable convolution, standard convolution, dilated convolution, and deconvolution are all common calculation methods in image pixel processing networks. Specifically, it includes the following steps:

[0077] Step a1: Standard convolution divides the input image into parts the same size as the convolution kernel and then performs the convolution operation. The position of each part on the input image is fixed. Deformable convolution adds a direction parameter to the selection of pixels in the input image, allowing for a wider range of pixels to be selected as activation points compared to standard convolution. The calculation process of deformable convolution is as follows:

[0078]

[0079] Where Ω represents the receptive field of the input feature map, p0 is the center position of the standard convolutional window, and pn Δ represents the two-dimensional coordinates of the original sample position in Ω. pn p represents the original sample location n The two-dimensional deformable offset. w and u represent the weight and activation, respectively; y(p0) represents the output pixel at position P0;

[0080] Step a2, based on the calculation process of deformable convolution and the description of standard convolution in step a1, when Δ in the deformable convolution calculation formula... pn When the value is 0, deformable convolution is transformed into standard convolution. Therefore, standard convolution is a special case of deformable convolution.

[0081] Step a3: Compared to standard convolution, dilated convolution introduces a scaling parameter, causing the input image pixels to be selected according to the scaling ratio during the convolution operation, and the convolution operation is performed using the expanded pixels; according to the calculation formula for deformable convolution in step a1, when Δ pn When the two-dimensional deformable offset represented by it satisfies the variation law of dilated convolution, dilated convolution is regarded as a special case of deformable convolution.

[0082] Step a4: Compared to standard convolution, deconvolution inserts zero values ​​between the pixels of the input image in standard convolution to form a completely new input image, and then performs calculations according to the standard convolution calculation process. In fact, deconvolution also shifts the pixels of the input image in the convolution operation. According to the calculation formula of deformable convolution in step a1, when the two-dimensional deformable offset represented by Δpn satisfies the change law of deconvolution, deconvolution is regarded as a special case of deformable convolution.

[0083] Step a5: Based on the descriptions in steps a1, a2, a3, and a4, standard convolution, dilated convolution, and deconvolution can all be regarded as special cases of deformable convolution. Thus, the calculation modes of standard convolution, dilated convolution, and deconvolution are unified into the calculation mode of deformable convolution, which is the field unification algorithm.

[0084] This invention also provides a computational method for image pixel processing networks. It employs a novel image storage method and a field unification algorithm to unify the computational steps of standard convolution, dilated convolution, and deconvolution into the computational steps of deformable convolution. Furthermore, it distinguishes different computational modes using data stored in an bias buffer. Specifically, it includes the following steps:

[0085] Step 1: Since the storage method of the computing device for images is different from that of conventional systems, the computing device adopts the following new image storage method: storing pixels at the same position on different channels of the image into the same storage address of the corresponding storage unit. The computing device first stores the pixel values, weight values, and offset values ​​stored according to the storage method in external storage.

[0086] Step 2: The bus controller transmits the required data from external storage to q input buffers, one weight buffer, and q bias buffers. Since deformable convolutions have irregular offsets at the pixel level, the values ​​stored in the bias buffers are the corresponding offset values. Dilated convolutions and deconvolutions can be regarded as deformable convolutions that meet specific constraints, so the values ​​stored in the bias buffers conform to the offset rules of their respective calculation modes. Standard convolutions can be regarded as deformable convolutions that do not have offsets, so the value stored in the bias buffers of standard convolutions is zero.

[0087] Step 3: The address controller generates the corresponding addresses of the data that the input buffer, weight buffer, and offset buffer need to obtain. During the address generation process, the address generated by the input buffer also needs to be updated according to the data obtained from the corresponding address in the offset buffer.

[0088] Step 4: After obtaining the data corresponding to the input buffer and the weight buffer, the data corresponding to the q input buffers are sent to the corresponding q block computing units. At the same time, the data corresponding to the address of one weight buffer is broadcast to the q block computing units.

[0089] Step 5: In each block computing unit, after receiving the input data and weight data from Step 4, the input data is broadcast to n normalization computing modules. Since one address stores the weight data of two or more output channels, the weight data is divided into n parts according to the channel and sent to the corresponding n normalization computing modules.

[0090] Step 6: The data received by each normalization calculation module is transmitted to m multipliers for product calculation. The results are then processed by adders and passed to an accumulator for accumulation. The calculation process described in step 6 is completed within one cycle.

[0091] Step 7: The calculations completed in Step 6 are performed within one cycle. Due to the complexity of pixel processing tasks, an output result may require more than two cycles to be calculated. Therefore, it is necessary to plan the data transmission loop: Since the proposed computing device uses a new data storage method, the corresponding data transmission loop will also change. The loop hierarchy is divided from high to low as follows: image block level, output channel level, weight window level, weight row level, and input channel level.

[0092] The calculations completed in one cycle as described in step 6 belong to the weight row-level calculation loop; after completing the weight row-level calculation loop, the weight window moves to the next position and selects a new input pixel for calculation; when the weight window moves to the last position, the calculation of the output channel is completed; after completing the output channel-level loop, the data of the new weight window is passed in to start a new loop, and finally, there is the image block-level loop.

[0093] In step 7, the input data and weight data are expanded and looped in three dimensions to achieve parallel computing. The three dimensions include the input channel level, the output channel level, and the image patch level.

[0094] Step 8: When the loop at the output channel level is completed, the accumulators of the n normalization calculation modules will output n valid output values ​​corresponding to the n output channels, thus simultaneously obtaining the output results of the same pixel position of the n output channels; the data storage format of the output buffer is the same as that of the input buffer, effectively solving the data storage format problem brought about by the previous system.

[0095] The present invention also provides a storage medium storing a computer program or instructions, which, when executed, implement the aforementioned computational method for an image pixel processing network.

[0096] Example

[0097] To reduce computational complexity, this embodiment rounds the offset in deformable convolution to an integer to avoid bilinear interpolation operations. The literature Q. Huang, D. Wang, Z. Dong, Y. Gao, Y. Cai, T. Li, B. Wu, K. Keutzer, and J. Wawrzynek, “Codenet: Efficient deployment of input-adaptive object detection on embedded fpgas,” in The 2021 ACM / SIGDA International Symposium on Field-Programmable Gate Arrays, 2021, pp. 206–216, and J. Guan, R. Lai, Y. Lu, Y. Li, H. Li, L. Feng, Y. Yang, and L. Gu, “Memory-efficient deformable convolution based joint denoising and demosaicing for uhd images,” IEEE Transactions on Circuits and Systems for Video Technology, pp. 1–1, 2022, has demonstrated that the accuracy reduction brought about by this modification is negligible. The calculation process of the modified deformable convolution is as follows:

[0098]

[0099] Where Ω represents the receptive field of the input feature map, p0 is the center position of the standard convolutional window, and p n Δ represents the two-dimensional coordinates of the original sample position in Ω. pn p represents the original sample location n The two-dimensional deformable offset. w and u represent the weight and activation, respectively. y(p0) represents the output pixel corresponding to position P0.

[0100] Based on the analysis of the calculation process of standard convolution, deconvolution, dilated convolution, and deformable convolution, their differences mainly stem from different input patterns or receptive fields. These differences can be represented by offsets; therefore, this invention proposes a field-unified algorithm (FUA) to unify the above calculation process using offsets.

[0101] (1) Standard convolution: Standard convolution can be regarded as a special case of deformable convolution.

[0102] (2) Dilated Convolution: The characteristic of dilated convolution is that it forms an expanded receptive field by regularly inserting zeros into the kernel. Therefore, the expansion of the weights can be converted into the offset of the corresponding static input rules.

[0103] (3) Deconvolution: Deconvolution is achieved by inserting zeros into the input activations and then performing a standard convolution on the expanded input feature map. Unlike dilated convolution, inserting zeros reduces the effective receptive field because the original receptive field contains some redundant zeros. The weights in the reduced receptive field can be sampled by shifting the weights.

[0104] Through the proposed field unification algorithm, the above convolution operation is transformed into a computation method that is consistent with the deformable convolution budget.

[0105] Analysis of the image pixel processing network reveals that it establishes relationships between pixels between the input and output images. The small blocks can be processed individually, and the corresponding output blocks can be sorted to obtain the final image. Therefore, based on the proposed algorithm, this invention designs a block computation unit and a corresponding position-decoupled computation process to achieve multiple types of convolution processing on an input block while maintaining the consistency of the arrangement between the input data.

[0106] (1) Data storage scheme: such as Figure 1a As shown, the Input feature map represents the input feature map, Ci represents the i input channels, H represents the height of the input feature map, W represents the width of the input feature map, A0 to Ai represent the encoding of different pixel positions on one channel, A0 to P0 represent the pixels of different channels at the same pixel position, the Input Buffer represents the input buffer, Bank 0 of the Input Buffer represents an input buffer, addr0 to addrj of the Input Buffer represent j+1 storage addresses, A0 to H0 of the Input Buffer represent the storage positions of pixels A0 to H0 in the input buffer, A1 to H1 of the Input Buffer represent the storage positions of pixels A1 to H1 in the input buffer, the Output Buffer represents the output buffer, Bank 0 of the Output Buffer represents an output buffer, addr0 to addrj of the Output Buffer represent j+1 storage addresses, A0 to N0 of the Output Buffer represent the storage positions of pixels A0 to N0 in the output buffer, and the Output Buffer represents the output buffer. The A1 to N1 values ​​in the Buffer represent the storage locations of pixels A1 to N1 in the output buffer.

[0107] The activation order is defined based on their position in the height and width dimensions (Phw), such as A0, A1, A2, ..., Ai, where different letters (e.g., A, B) represent different input channels. The activations of the m input channels are then concatenated and stored in an address in the bank, since there are no overlapping convolutional regions between adjacent channels. At the next address, the next m input channels are stored, and so on for the remaining channels, until all input channels at this address are stored in consecutive dCi / m addresses, where Ci represents the number of input channels. Similarly, other pixels are stored in addresses according to their height / width order. After computation by the block processing unit, n output channels are obtained. These n results are concatenated and stored in an address in the output buffer. The results obtained in subsequent loops are written into the output buffer in the order of their height and width dimensions (Phw). This storage scheme maintains a consistent data layout between the input and output buffers.

[0108] (2) Address Generation Module. The input offset set ΨuΔ is loaded into the offset buffer. A two-dimensional offset (xij, yij) is concatenated and stored in an address, where xj, yij, ... ij and y ij These represent the horizontal and vertical offsets of the j-th activated ith weight, respectively. The weight offset set ΨwΔ is generated by the controller. Figure 1a As shown, an address generation module (AGM) was developed to calculate the address of each point within the deformable receptive field, so that the sampled activations or weights can be accessed through the generated addresses.

[0109] Block computing units and computing processes: such as Figure 1b As shown, the Offset Buffer represents the bias buffer. Bank 0 of the Offset Buffer represents the bias buffer required by an address generator. Addr0 to Addri of the Offset Buffer represent the addresses of the bias buffers. The x... 00 To x ij The y-axis value of the offset buffer represents the offset value of the input address stored in the offset buffer along the horizontal axis. 00 to y ij This represents the bias value of the input address stored in the bias buffer along the vertical axis. The Address Generating Module (AGM) represents the address controller. ij The y-axis represents the bias value of the input address stored in the bias buffer along the horizontal axis. ijThis represents the bias value of the input address stored in the bias buffer on the vertical axis. *Htile*ci / m represents the control change of the input address. Controller represents the controller. Initial addri represents the initial input address. Initial addrw represents the initial output address. Input addr represents the input address. Weight addr represents the output address. A unified computation module (UM) was developed, containing m multipliers, an adder array, and an accumulator to perform multiplication-accumulation computations for various operations.

[0110] Furthermore, a block computing unit can be developed by stacking n UMs, such as Figure 1d As shown. AGM represents the address controller, Input offset represents the bias data of the input address, Offset Buffer represents the bias buffer, Bank 0 of Offset Buffer indicates that one bias buffer is required for one block processing unit, Input addr represents the input address, InputBuffer represents the input buffer, Bank 0 of Input Buffer indicates that one input buffer is required for one block processing unit, Weight addr represents the weight address, Weight Buffer represents the input buffer, Bank 0 of Weight Buffer indicates that one input buffer is required for one block processing unit, Tile Engine (TE) represents the block processing unit, Weight oc[0] to Weight oc[n-1] represent the weight data corresponding to n output channels, and m indicates that each normalized weight data corresponds to m values. UM[0] to UM[N-1] represent n normalization calculation modules, Input boardcast represents the output data being broadcast to different normalization calculation modules, Ai to Ni represent the output data of N output channels, OutputBuffer represents the output memory, and Bank 0 of Output Buffer represents that one block processing unit needs one output memory.

[0111] Each block computing unit includes n normalization computing modules. Figure 1c This is the architecture diagram of the normalization computing module. The Uniform Computing Module (UM) represents the normalization computing module, from w0 to w... m-1 Represents m weighted data points, from u0 to u m-1 This represents m input data, where Adder array represents an array of adders, and Accelerator represents an accumulator;

[0112] For a block computation unit, the input activation is broadcast to n computation modules UM, and the weights of the n output channels are respectively input to the n computation modules UM to compute the n output channels simultaneously.

[0113] When processing an input block, sampled addresses of deformable inputs or weights are generated, followed by the corresponding activations of m input channels with the same height and width dimensions Phw, such as... Figure 1d A0, B0, ..., H0 in the input buffer are read from the input buffer and processed through ports u0 to u00. m-1 The inputs are fed into n computation modules UM. Simultaneously, the corresponding weights of the m input channels in an output channel are sent to a UM via ports w0 to wm-1. After successive multiplication, the results are sent to an adder array for channel-by-channel summation.

[0114] Once each channel-wise loop is completed, the partial sums are fed into the accumulator, and kernel-level and channel-level accumulation loops are performed sequentially. Within a block computation unit, each Phw is stored at an independent address; therefore, positional dependencies within the deformable receiver field are decoupled, thus avoiding memory access conflicts and additional overlapping memory. Regarding the overall hardware architecture, by utilizing large feature maps in the image pixel processing network, multiple input blocks can be processed simultaneously. The image processing core contains q sets of input and offset buffers, storing the pixels and offsets of q blocks respectively. Similarly, their outputs are sent to the corresponding libraries in the output buffers. There is one set of weight buffers that broadcast weights across the q input blocks. The proposed dataflow is looped in three dimensions to achieve parallel computation: input channel level, output channel level, and block level. Figure 2The overall structure of the device of the present invention is described, including external storage, bus controller, address controller, input buffer, weight buffer, bias buffer, and q block computing units (q represents the number of block computing units); Memory (DDR4) indicates the use of external storage, On-chip Logic indicates on-chip logic, BUS indicates bus, Control Bus indicates control bus, Data Bus indicates data bus, DMA indicates direct memory access, Data Scatter indicates data distribution, Data Gather indicates data collection, AGM group indicates address controller, Offset Buffer indicates bias buffer, Bank0 to q-1 under Offset Buffer indicates q groups under bias buffer, Input Buffer indicates input buffer, Bank0 to q-1 under Input Buffer indicates q groups under input buffer, Weight Buffer indicates weight buffer, UCA indicates a group of block processing units, TE[0] to TE[q-1] under UCA indicates q block processing units, Result Post-process Module indicates output post-processing unit, Output Buffer indicates output buffer, Bank0 to q-1 under Output Buffer indicates q groups under output buffer.

[0115] Although both the present invention and existing solutions can support multiple types of convolution, they differ in their implementation approaches.

[0116] This invention employs a pixel-level storage approach, storing data from the same location across different input channels at the same on-chip address. This allows direct access to the corresponding pixel for processing, rather than using block storage. This storage design maximizes the flexibility of data processing.

[0117] The idea behind this invention is to transform convolutions of various cases into deformable convolutions for computation, rather than designing dedicated modules to optimize convolutions for different cases. This formal uniformity simplifies the design architecture and allows different convolution cases to achieve the same data flow on hardware.

[0118] This invention designs a location-decoupled computation process for storage solutions. This computation process transforms various cases of different types of convolution operations, different weight kernels, and different timing lengths into address control situations for stored data, thereby effectively supporting multiple convolution operations. This is significantly different from the computation process of existing solutions.

[0119] In its specific implementation, this application provides a computer storage medium and a corresponding data processing unit. The computer storage medium is capable of storing a computer program, which, when executed by the data processing unit, can run the invention's content regarding a calculation method for an image pixel processing network, as well as some or all of the steps in various embodiments. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0120] Those skilled in the art will clearly understand that the technical solutions in the embodiments of the present invention can be implemented using computer programs and their corresponding general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of computer programs, i.e., software products. These computer program software products can be stored in a storage medium and include several instructions to cause a device containing a data processing unit (which may be a personal computer, server, microcontroller, MUU, or network device, etc.) to execute the methods described in various embodiments or certain parts of the embodiments of the present invention.

[0121] This invention provides a computing device and method for image pixel processing networks. Many methods and approaches exist for implementing this technical solution; the above description is merely a preferred embodiment of the invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.

Claims

1. A computing device for an image pixel processing network, the computing device comprising: The external storage, the bus controller, the address controller, the input buffer, the weight buffer, the bias buffer, the output buffer, and q block calculation units, q representing the number of the block calculation units; Each block calculation unit comprises n normalization calculation modules, n representing the number of the normalization calculation modules in each block calculation unit; each normalization calculation module comprises m multipliers, an adder array, and an accumulator; m representing the number of the multipliers in each normalization calculation module; q, n, and m are natural numbers greater than 0; The external storage is configured to store the processed pixel information; The bus controller is configured to exchange the pixel information between the external storage and the input buffer, the weight buffer, the bias buffer, and the output buffer; The address controller is configured to provide data addresses and control signals to the bus controller, the input buffer, the weight buffer, the bias buffer, and the output buffer, thereby controlling the addresses; The block calculation unit is configured to calculate a segmented partial image in the image, and one block calculation unit is combined by n normalization calculation modules; For one block calculation unit, input activations are broadcast from the input buffer to the n normalization calculation modules, and the weights of n output channels are input from the weight buffer to the n normalization calculation modules, so as to simultaneously calculate the n output channels; the input activations are the processed pixel information; The normalization calculation module is configured to calculate the pixel results of one output channel of the result image, and the simultaneous calculation of the n output channels is realized by the combination of the n normalization calculation modules; For one normalization calculation module, input activations and weight activations are transmitted from the input buffer and the weight buffer to the m multipliers for calculation, the results of the multipliers are output after adder array and accumulator calculation, and the result of the accumulator calculation needs more than two periods to obtain one output result; the output result is transmitted to the output buffer when each data channel level cycle is completed; When the block calculation unit processes an input block, the address controller generates sampling addresses of deformable input or weight, then the corresponding activations of m input channels with the same height and width dimensions Phw are read from the input buffer and broadcast to the n normalization calculation modules, and the corresponding activations of m input channels with the same height and width dimensions Phw are represented as A0, B0,..., H0; The letters in A0, B0,..., H0 represent the channel codes of the image blocks, and the numerical parts represent the position codes of the image blocks; meanwhile, the corresponding weights of m input channels in one output channel are sent to one normalization calculation module; after the element-by-element multiplication, the result is sent to an adder array for channel-by-channel summation; the feature map is cut into more than two image blocks as input blocks by cutting processing, and the input buffer stores the data of the input blocks. The operation result, i.e., partial sum, is sent to an accumulator after the normalization calculation module completes a cycle of operation, and then kernel-level and channel-level accumulation loops are performed in turn; in a block calculation unit, each height and width dimension Phw is stored in a separate address, so that the position dependence in the deformable receptive field is decoupled; The device simultaneously processes two or more input blocks; The device includes q input buffers and a bias buffer, the q input buffers and the bias buffer are used to store the pixels of the q input blocks and the bias values respectively, and the output results of the q block calculation units are sent to the corresponding banks of the output buffer; the weight buffer is one, which is used to broadcast the weights in the q input blocks; The device performs a field unification algorithm to unify the calculation modes of standard convolution, dilated convolution and deconvolution into the calculation mode of deformable convolution, and specifically includes the following steps: Step a1, the standard convolution operation is to divide the input image into parts with the same size as the convolution kernel, and then perform convolution operation, the position of each part on the input image is fixed; the deformable convolution increases the direction parameter in the selection of the pixel points of the input image, so that the selection range of the pixel points of the input image as the activation is wider than that of the standard convolution; the calculation process of the deformable convolution is as follows: ， where Ω represents the receptive field of the input feature map, p0is the center position of the standard convolution window, p n is a two-dimensional coordinate representing the original sample position in Ω, represents a two-dimensional deformable offset of the original sample position p n ; w and u represent weights and activations, respectively; y(p0) represents the output pixel corresponding to the p0position. Step a2, when the value of in the formula of deformable convolution is 0, the deformable convolution is transformed into the standard convolution, thus, the standard convolution is a special case of the deformable convolution. Step a2, when the value of in the formula of deformable convolution is 0, the deformable convolution is transformed into the standard convolution, thus, the standard convolution is a special case of the deformable convolution. Step a3, compared with the standard convolution, the dilated convolution introduces the parameter of the expansion ratio, so that the pixel points of the input image are selected according to the expansion ratio in the convolution operation, and the convolution operation is performed on the expanded pixel points; according to the calculation formula of the deformable convolution in step a1, when the two-dimensional variable offset represented by the formula satisfies the change rule of the dilated convolution, the dilated convolution is regarded as a special case of the deformable convolution. When the two-dimensional variable offset represented by the formula satisfies the change rule of the dilated convolution, the dilated convolution is regarded as a special case of the deformable convolution. Step a4, deconvolution compared with standard convolution, deconvolution inserts 0 value between the pixel points of the input image of the standard convolution to form a new input image, and then calculates according to the calculation process of the standard convolution. Deconvolution actually also offsets the pixel points of the input image in the convolution operation; according to the calculation formula of the deformable convolution in step a1, when the two-dimensional variable shift represented by the formula meets the change rule of deconvolution, the deconvolution is regarded as a special case of the deformable convolution. When the two-dimensional variable shift represented by the formula meets the change rule of deconvolution, the deconvolution is regarded as a special case of the deformable convolution. Step a5, according to the descriptions of steps a1, a2, a3 and a4, the standard convolution, the dilated convolution and the deconvolution can be regarded as special cases in the deformable convolution, so the calculation modes of the standard convolution, the dilated convolution and the deconvolution are unified into the calculation mode of the deformable convolution, that is, the field unification algorithm.

2. A computing method for an image pixel processing network based on the computing device of claim 1, characterized in that, The new image storage method and the field unification algorithm are adopted to unify the calculation steps of the standard convolution, the dilated convolution and the deconvolution into the calculation steps of the deformable convolution, and the data stored in the bias buffer is used to distinguish different calculation modes, and specifically includes the following steps: Step 1, the calculation device adopts the following new image storage method: storing the pixels at the same position on different channels of the image into the same storage address of the corresponding storage unit, and first storing the pixel values, weight values and offset values stored according to the storage method in the external storage; Step 2, the bus controller transmits the required data from the external storage to the q input buffers, one weight buffer and q bias buffers, wherein, due to irregular offset on the pixel level, the values stored in the bias buffer are the corresponding offset values, the dilated convolution and the deconvolution are regarded as deformable convolution conforming to specific restrictions, so the values stored in the bias buffer conform to the offset rules of the respective calculation modes, and the standard convolution is regarded as deformable convolution without offset, so the value stored in the bias buffer of the standard convolution is zero; Step 3, the address controller generates the corresponding addresses required to obtain data from the input buffer, the weight buffer and the offset buffer, and in the address generation process, the address generated by the input buffer also needs to update the address of the input buffer according to the data obtained from the corresponding address of the offset buffer; Step 4, after obtaining the data corresponding to the input buffer and the weight buffer, the data corresponding to the q input buffers are sent to the corresponding q block calculation units, and the data corresponding to one weight buffer address is broadcast to the q block calculation units; Step 5, in each block calculation unit, after receiving the input data and the weight data in step 4, the input data is broadcast to n normalization calculation modules; since one address stores the weight data of two or more output channels, the weight data is divided into n parts according to the channels and sent to the corresponding n normalization calculation modules; Step 6, the data received by each normalization calculation module is transmitted to m multipliers for product calculation, and the results are transmitted to accumulators after addition calculation by an adder array for accumulation, and the calculation process described in step 6 is completed in one period; Step 7, the calculation completed in step 6 is a calculation step completed in one period, and one output result needs to be calculated for two or more periods, so the transmission cycle of the data needs to be planned: according to the cycle level from high to low, it is divided into an image block level, an output channel level, a weight window level, a weight row level, and an input channel level; The calculation completed in one period described in step 6 belongs to the calculation cycle of the weight row level; after completing the calculation cycle of the weight row level, the weight window moves to the next position and selects a new input pixel for calculation; when the weight window moves to the last position, the calculation of the output channel is completed; after completing the cycle of the output channel level, the data of the new weight window is transmitted to start a new cycle, and finally the cycle of the image block level is completed; In step 7, the input data and the weight data are unfolded in three dimensions to realize parallel calculation, and the three dimensions include the input channel level, the output channel level, and the image block level; Step 8, when the cycle of the output channel level is completed, the accumulators of the n normalization calculation modules output n valid output values corresponding to n output channels, so that the output results of n output channels at the same pixel position are obtained at the same time; the data storage form of the output buffer is the same as that of the input buffer.

3. A storage medium, characterized by The computer program or instructions stored therein, when executed, implement the method as claimed in claim 2.

Images