Chip computing architecture and design method for pixel-wise differential convolutional neural network

Abstract

This application relates to a chip computing architecture and design method for pixel-differential convolutional neural networks, belonging to the field of chip design technology. This application employs a multiply-accumulate array architecture, where the same multiply-accumulate column in the array shares the same weight data. Under the control of the activation loading enable signal set for each multiply-accumulate column, the same multiply-accumulate row in the array shares the same feature image activation data or loads different feature image activation data, realizing standard convolution, pointwise convolution, depthwise convolution, and dilated convolution in pixel-differential convolutional neural networks. This application enables the deployment of pixel-differential convolutional neural networks built based on differential convolution operators to low-power chips for efficient inference.

Description

Technical Field

[0001] This application relates to the field of chip design technology, and in particular to a chip computing architecture and design method for pixel-differential convolutional neural networks. Background Technology

[0002] Compared to traditional neural networks that only use standard convolutions, pixel-difference convolutional neural networks (PDNNs) achieve significant improvements in real-time inference speed on GPUs (Graphics Processing Units) and CPUs (Central Processing Units). However, when deploying this network in an edge acceleration chip with a multiply-accumulate array at its core, the computational methods of depthwise separable convolutions (including depthwise and pointwise convolutions) and dilated convolutions in the network are unique, while existing edge chips are typically designed for standard convolutions. This results in severely insufficient utilization of computational resources, limiting the practical application value of PDNNs.

[0003] Existing solutions typically require adding dedicated circuit modules to the chip to implement depthwise separable convolutions and dilated convolutions. However, these additional modules often cannot run standard convolutions, resulting in increased chip area and wasted computing resources. Summary of the Invention

[0004] Therefore, it is necessary to provide a chip computing architecture and design method for pixel-differential convolutional neural networks that is compatible with standard convolution, depthwise separable convolution, and dilated convolution, in order to address the above-mentioned technical problems, achieve efficient utilization of computing resources, and thus improve inference computing efficiency.

[0005] A chip computing architecture for pixel-differential convolutional neural networks includes: M activation loading queues, M weight loading queues, and a multiply-accumulate array composed of M×M multiply-accumulate units; wherein each multiply-accumulate column in the multiply-accumulate array is configured with an activation loading enable signal. The value of M is a positive integer power of 2.

[0006] The activation loading queue is used to load feature image activation data into the multiply-accumulate array in the horizontal direction, and the weight loading queue is used to load weight data into the multiply-accumulate array in the vertical direction.

[0007] In a multiply-accumulate array, the same multiply-accumulate column shares the same weight data, and the activation load enable signal is activated. Under the control of [the system], the same multiply-accumulate row can share the same feature image activation data or load different feature image activation data to achieve standard convolution, pointwise convolution, depthwise convolution and dilated convolution in pixel difference convolutional neural networks.

[0008] In one embodiment, each multiply-accumulate unit includes a multiplier, a multiplexer, an adder, and an output feature image buffer. The multiplier is used to perform the multiplication operation between the feature image activation data and the weight data. The multiplexer is used to select the result calculated by the local multiplier, the multiplier in the previous row multiply-accumulate unit, or the multiplier in the next row multiply-accumulate unit. The adder is used to accumulate the data output by the multiplexer with the partial sum stored in the output feature image buffer, and then write it back into the output feature image buffer.

[0009] In one embodiment, two constraints must be satisfied when loading feature image activation data and weight data into the multiply-accumulate array:

[0010] Constraint 1: All loaded feature image activation data and weight data have the same input channel number;

[0011] Constraint 2: The weight data loaded into different multiply-accumulate columns belong to different convolution kernels, that is, they have different output channel numbers.

[0012] In one embodiment, the multiply-accumulate array implements standard convolution and pointwise convolution, including:

[0013] Load the activation enable signal for each multiply-accumulate column in the multiply-accumulate array. All values ​​are set to 1. Within any clock cycle, the same multiply-accumulate column shares the same weight data, and the same multiply-accumulate row shares the same feature image activation data, thus realizing standard convolution and pointwise convolution in pixel difference convolutional neural networks.

[0014] In one embodiment, the multiply-accumulate array implements the standard convolution data loading method, including:

[0015] First, feature image activation data and weight data for any input channel number are loaded into the multiply-accumulate array. The feature image activation data is loaded into the corresponding multiply-accumulate row according to the remainder when its column coordinate number is divided by M, while the weight data is loaded into the corresponding multiply-accumulate column according to the remainder when its output channel number is divided by M. The result of multiplying the feature image activation data and the weight data is temporarily stored in the output feature image cache in the multiply-accumulate unit.

[0016] Secondly, depending on the needs, choose to update the weight data or the feature image activation data first;

[0017] In one implementation, the feature image activation data is kept unchanged, and all weight data belonging to the same input channel are loaded and traversed. The calculation result is temporarily stored in the output feature image buffer according to its coordinates in the output feature image. Then, a set of feature image activation data is updated, and the weight data is loaded again. The multiplication calculation result is accumulated with the previously temporarily stored result in the output feature image buffer according to the convolution operation rules. The above process is repeated until the convolution operation of the feature image activation data and weight data in the current input channel is completed. Then, the system switches to the next input channel and completes the standard convolution operation in the same way.

[0018] In another implementation, the weight data is kept unchanged, and the feature image activation data within the same input channel is updated traversally. Then the weight data is updated, and the feature image activation data is loaded traversally until all standard convolution operations are completed.

[0019] In one embodiment, the multiply-accumulate array implements depthwise convolution, including:

[0020] When M is greater than 9, the multiply-accumulate array uses at most 9 columns out of M multiply-accumulate columns to achieve depthwise convolution. Without loss of generality, the number of multiply-accumulate columns used in the multiply-accumulate array is set to N, where N≤9, that is, the actual size of the multiply-accumulate array used is M×N, and when M is less than 9, N=M.

[0021] The method for implementing depthwise convolution using an M×N multiply-accumulate array is as follows: during the 0th to N-1th period, the activation enable signals corresponding to the 0th to N-1th multiply-accumulate columns are loaded. The sequence is cyclically set to 1, and the remaining columns are multiplied by the corresponding activation and load enable signals. Setting it to 0 ensures that, in each clock cycle from cycle 0 to cycle (N-1), the feature image activation data of each input channel read from the activation loading queue will only be loaded onto the current activation loading enable signal. The multiply-accumulate column is set to 1; simultaneously, in each clock cycle from cycle 0 to cycle N-1, each activation load enable signal is applied. For the multiply-accumulate column where the historical value is 1 and the current value is 0, the loaded feature image activation data remains unchanged, and the weight data is updated to the new weight data in the historically loaded output channel. The multiplication operation of the activation data and weight data is performed within each multiply-accumulate unit. Specifically, the first multiplication operation is completed in the (N-1)th multiply-accumulate column during the (N-1)th cycle. In the Nth cycle, the activation enable signal for the 0th multiply-accumulate column is reloaded. Set to 1, update the feature image activation data of the 0th multiply-accumulated column to the new feature image activation data loaded in the input channel in cycle 0, and restore the weight data to the weight data loaded in cycle 0 of the 0th multiply-accumulated column, completing the multiplication operation. At the same time, synchronously update the weight data of the 1st to N-1th multiply-accumulated columns to the new weight data in the output channel historically loaded for each multiply-accumulated column; in the following N+1 to 2N-1 cycles, sequentially load the activation enable signal of the 1st to N-1th multiply-accumulated columns. Set to 1, update the feature image activation data of each multiply-accumulate column to the new feature image activation data loaded in the input channels from the 1st cycle to the (N-1)th cycle, and restore the weight data to the weight data loaded in each multiply-accumulate column from the 1st cycle to the (N-1)th cycle, complete the multiplication operation, and simultaneously update the weight data of the remaining multiply-accumulate columns to the new weight data loaded in the output channels of each multiply-accumulate column in the previous period; update the feature image activation data and weight data in the above manner every N clock cycles until the depthwise convolution calculation of N channels is completed, and then carry out the depthwise convolution calculation of the next N channels in the same manner until the calculation of the current convolutional layer is completed.

[0022] In one embodiment, when the multiply-accumulate array performs depthwise convolution, it further includes:

[0023] When the actual number of columns M of the multiply-accumulate array is greater than N, the activation enable signal of the (N+i)th multiply-accumulate column can also be applied in the (N+i)th cycle. Set to 1, load the feature image activation data and weight data of the N+i input channel into the N+i multiply-accumulate column. At the same time, consider that the feature image activation data of the 0th to the ith multiply-accumulate columns have not been updated at this time, and reloading the weight data of the 0th to the ith multiply-accumulate columns will not produce meaningful new calculations. Therefore, the multiply-accumulate units of the 0th to the ith multiply-accumulate columns need to be turned off to save power consumption; where 0≤i<MN.

[0024] In one embodiment, when the multiply-accumulate array implements depthwise convolution, in each clock cycle following the Nth cycle, there are M×N sets of multiplication operations performed simultaneously in the multiply-accumulate array until all depthwise convolution operations in the pixel difference convolutional neural network are completed, and the computational resource utilization is at most N / M.

[0025] In one embodiment, the multiply-accumulate array implements dilated convolution, including:

[0026] First, the dilated convolution is degenerated into a standard convolution by rearranging the feature images in the pixel difference convolutional neural network; then, the activation corresponding to each multiply-accumulate column in the multiply-accumulate array is loaded with the enable signal. All values ​​are set to 1. Within any clock cycle, the same multiply-accumulate column shares the same weight data, and the same multiply-accumulate row shares the same feature image activation data, thus realizing the standard convolution in the pixel difference convolutional neural network.

[0027] The feature images are rearranged as follows:

[0028] First, set the dilation coefficient to D, the height of the feature image to H, and the width to W. Let h be the row number and w be the column number of the feature image activation data in each input channel. Then, reassemble the feature image activation data into D×D sub-feature images based on the remainders of row number h divided by D and column number w divided by D. Each sub-feature image has a height of H / D rounded up and a width of W / D rounded up. If the amount of data in the original feature image is insufficient to completely fill a sub-feature image, zero values ​​are used for padding. The D×D sub-feature images are then reassembled into a large feature image in a matrix manner. During the assembly, adjacent sub-feature images in the horizontal and vertical directions are padded with D-1 zeros to prevent mutual interference during convolution calculations. The size of the reassembled feature image is... ;

[0029] Then, zero values ​​in the dilated convolution weight matrix are removed, degenerating it to the standard convolution size. The re-concatenated feature image and the degenerated standard convolution weight matrix are then calculated in a multiply-accumulate array using standard convolution. The resulting output feature image has the same size as the input feature image. The calculation results at the positions corresponding to the zero-filling positions are invalid data;

[0030] Finally, the calculated output feature image data, after removing invalid data, is rearranged in the reverse manner of the feature image rearrangement process to obtain the correct dilated convolution result.

[0031] A design method for a chip computing architecture oriented towards pixel-difference convolutional neural networks, the method comprising:

[0032] Design a chip computing architecture for pixel-differential convolutional neural networks, including M activation loading queues, M weight loading queues, and a multiply-accumulate array consisting of M×M multiply-accumulate units; wherein each multiply-accumulate column in the multiply-accumulate array is configured with an activation loading enable signal. The value of M is a positive integer power of 2.

[0033] The activation data of the feature image is loaded into the multiply-accumulate array in the horizontal direction using an activation loading queue, and the weight loading queue is used to load the weight data into the multiply-accumulate array in the vertical direction.

[0034] By having the same weight data shared within the multiply-accumulate array, and by activating the load enable signal... Under the control of [the system], the same multiply-accumulate row can share the same feature image activation data or load different feature image activation data to achieve standard convolution, pointwise convolution, depthwise convolution and dilated convolution in pixel difference convolutional neural networks.

[0035] The chip computing architecture and design method described above for pixel-differential convolutional neural networks have the following advantages compared to existing technologies:

[0036] This application employs a multiply-accumulate array architecture, where the same multiply-accumulate column in the array shares the same weight data, and an activation loading enable signal is set for each multiply-accumulate column of the array. Under the control of [the system], the same multiply-accumulate row in the multiply-accumulate array shares the same feature image activation data or loads different feature image activation data, thereby realizing standard convolution, pointwise convolution, depthwise convolution, and dilated convolution in pixel-difference convolutional neural networks. This can effectively support the inference acceleration of pixel-difference convolutional neural networks and various lightweight neural networks, avoiding the introduction of dedicated modules to implement depthwise convolution, pointwise convolution, and dilated convolution. This is beneficial to reducing the accelerator chip area and avoiding the waste of computing resources. Ultimately, it enables the deployment of pixel-difference convolutional neural networks built based on differential convolution operators to low-power chips for efficient inference. Attached Figure Description

[0037] Figure 1 This is a schematic diagram of a standard convolution operator in one embodiment;

[0038] Figure 2 This is a schematic diagram of a depthwise convolution operator in one embodiment;

[0039] Figure 3 This is a schematic diagram of dilated convolution in one embodiment; wherein, Figure 3 (a) is a dilated convolution with an inflation rate of 1. Figure 3 (b) is a dilated convolution with a dilation rate of 2. Figure 3 (c) is a dilated convolution with an inflation rate of 3;

[0040] Figure 4 This is a schematic diagram illustrating the calculation of three pixel difference convolution operators in one embodiment;

[0041] Figure 5 This is a schematic diagram of a pixel-differential convolutional neural network in one embodiment;

[0042] Figure 6 This is a schematic diagram of a chip computing architecture for a pixel-differential convolutional neural network in one embodiment;

[0043] Figure 7 This is a schematic diagram illustrating the data loading method for implementing standard convolution using a multiply-accumulate array in one embodiment;

[0044] Figure 8 This is a schematic diagram illustrating the data loading method for depthwise convolution implemented using a multiply-accumulate array in one embodiment;

[0045] Figure 9 This is a schematic diagram of the input feature image data rearrangement method when the inflation rate is 2 in one embodiment. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0047] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0048] 1. Standard Convolution Operator: A standard convolution calculation process is as follows: Figure 1 As shown. The input feature image (IFM) is a graph of size [size missing]. A 3D matrix, that is, containing There are 1 input channel, each input channel containing a height of 1. and width are The feature image layer; the convolutional filter bank consists of multiple convolutional kernels, which can be regarded as a feature image layer with a size of 1. A 4D matrix, i.e., the number of convolution kernels is The number of channels in each convolution kernel is The height and width of each channel are respectively and ,generally The output feature image (OFM) can be viewed as a shape of size 1. A 3D matrix, that is, containing There are 1 output channel, and each output channel contains a channel with a height of 1. and width are The feature image layer is used. During convolution calculation, each convolution kernel performs sliding convolution with the IFM in a sliding window manner. Within each sliding window, only the weights of the convolution kernels belonging to the same input channel are multiplied bit-by-bit with the IFM image data. Then, all the product results are summed to obtain one element in one output channel of OFM. After a convolution kernel completes convolution with IFM, the image data of one output channel of OFM is obtained. After all convolution kernels complete convolution operations with IFM, the complete OFM data is obtained.

[0049] 2. Pointwise Convolution: The implementation of pointwise convolution is basically the same as the standard convolution operator, the only difference being that the height and width of each channel are both 1. .

[0050] 3. Depthwise Convolution: The implementation of depthwise convolution is as follows... Figure 2 As shown. The number of convolutional kernels is the same as the number of IFM channels, both being [missing information]. Each convolutional kernel has only one channel, and its size is typically [missing information]. Furthermore, each convolution kernel is convolved with only one channel in the IFM to calculate one channel in the OFM.

[0051] 4. Dilated Convolution: Dilated convolution is implemented similarly to standard convolution, but... Figure 3 As shown, depending on the dilation coefficient, there is a certain interval between the IFM data participating in the calculation within each convolution window.

[0052] 5. Pixel Difference Convolution (PDC) operator: Compared to ordinary convolution, PDC can better focus on the multi-directional gradient information of the image, thereby improving the accuracy of lightweight models in tasks such as detection and recognition. The specific calculation method is as follows:

[0053] ;

[0054] ;

[0055] in, To output the value of a local image region in the feature map, and These represent the local image region and the convolution kernel weights, respectively. This is the convolution operation function. and These are all elements within that local image region. For the first Each convolutional kernel weight, The size of the convolution kernel. For a set of pixel pairs, The number of pixel pairs depends on the specific type of pixel difference convolution operator. The pixels they represent are slightly different; for The local difference.

[0056] Based on the sampling strategy for candidate pixel pairs, the PDC operator has three forms, such as... Figure 4 As shown, Figure 4 From top to bottom:

[0057] Center pixel difference convolution operator: Performs center difference on neighborhood features.

[0058] Angular pixel difference convolution operator: Performs pairwise differences on the neighborhood in a clockwise direction.

[0059] Radial pixel difference convolution operator: for larger receptive fields The outer ring and inner ring of the neighborhood are differentiated.

[0060] Experiments show that, compared with the standard convolution operator, the pixel difference convolution operator described above can extract the edge information of the target to be identified in the feature image better, thereby significantly reducing the size of the neural network.

[0061] 6. Pixel-Differential Convolutional Neural Network: Pixel-differential convolutional neural networks combine pixel-differential convolution operators with depthwise separable convolutional architectures, such as... Figure 5 As shown, in addition to standard convolution operators, this network also includes various lightweight convolution operators, such as depthwise convolution, pointwise convolution, and dilated convolution. Among them, the pixel difference convolution operator is fused with depthwise convolution, called depthwise with PDC. Depthwise convolution and pointwise convolution are collectively referred to as depth-separable convolution.

[0062] The entire network consists of four stages (Stage 1-Stage 4) connected by max pooling layers. Each stage comprises a residual network-based convolutional block, a Compact Dilation Convolution Based Module (CDCM), and a Compact Spatial Attention Module (CSAM). The CDCM includes four parallel dilated convolution branches, each using a 3×3 kernel with dilation rates of 5, 7, 9, and 11. In each stage, the input first passes through three blocks for feature extraction, then undergoes spatial resolution reduction via a 2×2 pooling layer before being input to the next stage. The output of the last block in each stage is processed sequentially by the CDCM and CSAM. The CSAM output features are first reduced to one channel by a one-dimensional convolution (1 Conv), then activated by the Sigmoid activation function to generate the initial edge prediction map for that stage. The EdgeMap outputs from each stage are concatenated along the channel dimension to obtain a 4-channel feature map. The concatenated features are then fused and dimensionality reduced by another one-dimensional convolution, and finally activated by Sigmoid to output the final edge prediction map (Final EdgeMap). The loss function is calculated with the Ground Truth (real edge label) to guide network training.

[0063] In the inference computation process, pixel-difference depthwise separable convolution can degenerate into a single standard 3D convolution operator by reparameterizing each layer of the network's difference convolution operator. A depthwise separable convolution operator of size 3. Wherein, for each weighted kernel... Difference convolution operator i ,Will Reparameterized to To convert the differential convolution operator into a standard convolution:

[0064] ;

[0065] in, In the formula, " The set of coefficients for "". Then, all parameters in the current layer can be... Reparameterization Converted to a single-branch depthwise separable convolution operator version:

[0066] .

[0067] In one embodiment, such as Figure 6 As shown, a chip computing architecture for pixel-differential convolutional neural networks is provided, including M activation loading queues, M weight loading queues, and a multiply-accumulate array composed of M×M multiply-accumulate units; wherein, each multiply-accumulate column in the multiply-accumulate array is configured with an activation loading enable signal. M takes the value of a positive integer power of 2; the activation loading queue is used to load feature image activation data into the multiply-accumulate array in the horizontal direction, and the weight loading queue is used to load weight data into the multiply-accumulate array in the vertical direction; in the multiply-accumulate array, the same multiply-accumulate column shares the same weight data, and the activation loading enable signal is activated. Under the control of [the system], the same multiply-accumulate row can share the same feature image activation data or load different feature image activation data to achieve standard convolution, pointwise convolution, depthwise convolution and dilated convolution in pixel difference convolutional neural networks.

[0068] The aforementioned chip computing architecture for pixel-differential convolutional neural networks employs a multiply-accumulate array architecture. The same multiply-accumulate column in the array shares the same weight data, and an activation loading enable signal is set for each multiply-accumulate column. Under the control of [the system], the same multiply-accumulate row in the multiply-accumulate array shares the same feature image activation data or loads different feature image activation data, thereby realizing standard convolution, pointwise convolution, depthwise convolution, and dilated convolution in pixel-difference convolutional neural networks. This can effectively support the inference acceleration of pixel-difference convolutional neural networks and various lightweight neural networks, avoiding the introduction of dedicated modules to implement depthwise convolution, pointwise convolution, and dilated convolution. This is beneficial to reducing the accelerator chip area and avoiding the waste of computing resources. Ultimately, it enables the deployment of pixel-difference convolutional neural networks built based on differential convolution operators to low-power chips for efficient inference.

[0069] In one embodiment, each multiply-accumulate unit includes a multiplier, a multiplexer (MUX), an adder, and an output feature image buffer (Ofm_buf). The multiplier performs the multiplication of the feature image activation data and weight data. The multiplexer selects the result calculated by the local multiplier, the multiplier in the previous row's multiply-accumulate unit, or the multiplier in the next row's multiply-accumulate unit. The adder accumulates the data output by the multiplexer with a partial sum stored in the output feature image buffer, and then writes it back into the output feature image buffer. Figure 6 As shown, when loading feature image activation data, the activation loading queue writes the data to a register under the control of an enable switch, and then inputs the feature image activation data into the multiplier. Similarly, when loading weight data, the weight loading queue first passes the data through a register, and then inputs the weight data into the multiplier.

[0070] In one embodiment, two constraints must be satisfied when loading feature image activation data and weight data into the multiply-accumulate array:

[0071] Constraint 1: All loaded feature image activation data and weight data have the same input channel number;

[0072] Constraint 2: The weight data loaded into different multiply-accumulate columns belong to different convolution kernels, that is, they have different output channel numbers.

[0073] The only difference between the pointwise convolution operator and the standard convolution operator is that the former has a height and width of 1 for each channel, i.e. Based on this, in one embodiment, the multiply-accumulate array implements standard convolution and pointwise convolution in the same way, including:

[0074] Load the activation enable signal for each multiply-accumulate column in the multiply-accumulate array. All values ​​are set to 1. Within any clock cycle, the same multiply-accumulate column shares the same weight data, and the same multiply-accumulate row shares the same feature image activation data, thus realizing standard convolution and pointwise convolution in pixel difference convolutional neural networks.

[0075] In one embodiment, the multiply-accumulate array implements the standard convolution data loading method, including:

[0076] First, feature image activation data and weight data for any input channel number are loaded into the multiply-accumulate array. The feature image activation data is loaded into the corresponding multiply-accumulate row according to the remainder when its column coordinate number is divided by M, while the weight data is loaded into the corresponding multiply-accumulate column according to the remainder when its output channel number is divided by M. The result of multiplying the feature image activation data and the weight data is temporarily stored in the output feature image cache in the multiply-accumulate unit.

[0077] Secondly, depending on the needs, choose to update the weight data or the feature image activation data first;

[0078] In one implementation, the feature image activation data is kept unchanged, and all weight data belonging to the same input channel are loaded and traversed. The calculation result is temporarily stored in the output feature image buffer according to its coordinates in the output feature image. Then, a set of feature image activation data is updated, and the weight data is loaded again. The multiplication calculation result is accumulated with the previously temporarily stored result in the output feature image buffer according to the convolution operation rules. The above process is repeated until the convolution operation of the feature image activation data and weight data in the current input channel is completed. Then, the system switches to the next input channel and completes the standard convolution operation in the same way.

[0079] In another implementation, the weight data is kept unchanged, and the feature image activation data within the same input channel is updated traversally. Then the weight data is updated, and the feature image activation data is loaded traversally until all standard convolution operations are completed.

[0080] Figure 7 This demonstrates the data loading method for standard convolutions with an 8×8 multiply-accumulate array. Figure 7 In the example shown, the feature image activation data and weight data with input channel number 0 are first loaded into the multiply-accumulate array. Specifically, the feature image activation data is loaded into the corresponding row of the multiply-accumulate array according to the remainder when its column coordinate number is divided by 8, while the weight data is loaded into the corresponding column of the multiply-accumulate array according to the remainder when its output channel number is divided by 8. Figure 7 The right side of the arrow shows the calculations completed in cycle 0 (Cycle0), and the result of the multiplication will be temporarily stored in ofm_buf. Figure 7 In the calculation process shown, the multiply-accumulate computational resources are fully utilized within each clock cycle, and 64 multiply-accumulate results can be obtained for every 8 feature image activation data and 8 weight data loaded, thus exhibiting very high computational efficiency. It should be noted that the size of the multiply-accumulate array can be larger, such as 16×16 or 32×32.

[0081] In depthwise convolution operators, each convolution kernel has only one channel, resulting in low utilization of multiply-accumulate resources during computation. This is because, in standard convolution operators, each input channel in an IFM (Integrated Feature Model) needs to be convolved with weight data from multiple convolution kernels that share the same input channel. Therefore, in multiply-accumulate arrays, each loaded feature image activation data can be multiplied with multiple weight data simultaneously, achieving data reuse. In depthwise convolution, each feature image activation data only needs to be multiplied with the weight data in one convolution kernel. Figure 7 Only one column of the multiply-accumulate unit can be used.

[0082] To address the aforementioned issues, in one embodiment, depthwise convolution is implemented using a multiply-accumulate array, including:

[0083] When M is greater than 9, the multiply-accumulate array uses at most 9 columns out of M multiply-accumulate columns to achieve depthwise convolution. Without loss of generality, the number of multiply-accumulate columns used in the multiply-accumulate array is set to N, where N≤9, that is, the actual size of the multiply-accumulate array used is M×N, and when M is less than 9, N=M.

[0084] The method for implementing depthwise convolution using an M×N multiply-accumulate array is as follows: during the 0th to N-1th period, the activation enable signals corresponding to the 0th to N-1th multiply-accumulate columns are loaded. The sequence is cyclically set to 1, and the remaining columns are multiplied by the corresponding activation and load enable signals. Setting it to 0 ensures that, in each clock cycle from cycle 0 to cycle (N-1), the feature image activation data of each input channel read from the activation loading queue will only be loaded onto the current activation loading enable signal. The multiply-accumulate column is set to 1; simultaneously, in each clock cycle from cycle 0 to cycle N-1, each activation load enable signal is applied. For the multiply-accumulate column where the historical value is 1 and the current value is 0, the loaded feature image activation data remains unchanged, and the weight data is updated to the new weight data in the historically loaded output channel. The multiplication operation of the activation data and weight data is performed within each multiply-accumulate unit. Specifically, the first multiplication operation is completed in the (N-1)th multiply-accumulate column during the (N-1)th cycle. In the Nth cycle, the activation enable signal for the 0th multiply-accumulate column is reloaded. Set to 1, update the feature image activation data of the 0th multiply-accumulated column to the new feature image activation data loaded in the input channel in cycle 0, and restore the weight data to the weight data loaded in cycle 0 of the 0th multiply-accumulated column, completing the multiplication operation. At the same time, synchronously update the weight data of the 1st to N-1th multiply-accumulated columns to the new weight data in the output channel historically loaded for each multiply-accumulated column; in the following N+1 to 2N-1 cycles, sequentially load the activation enable signal of the 1st to N-1th multiply-accumulated columns. Set to 1, update the feature image activation data of each multiply-accumulate column to the new feature image activation data loaded in the input channels from the 1st cycle to the (N-1)th cycle, and restore the weight data to the weight data loaded in each multiply-accumulate column from the 1st cycle to the (N-1)th cycle, complete the multiplication operation, and simultaneously update the weight data of the remaining multiply-accumulate columns to the new weight data loaded in the output channels of each multiply-accumulate column in the previous period; update the feature image activation data and weight data in the above manner every N clock cycles until the depthwise convolution calculation of N channels is completed, and then carry out the depthwise convolution calculation of the next N channels in the same manner until the calculation of the current convolutional layer is completed.

[0085] Furthermore, when implementing depthwise convolution using a multiply-accumulate array, it also includes:

[0086] When the actual number of columns M of the multiply-accumulate array is greater than N, the activation enable signal of the (N+i)th multiply-accumulate column can also be applied in the (N+i)th cycle. Set to 1, load the feature image activation data and weight data of the N+i input channel into the N+i multiply-accumulate column. At the same time, consider that the feature image activation data of the 0th to the ith multiply-accumulate columns have not been updated at this time, and reloading the weight data of the 0th to the ith multiply-accumulate columns will not produce meaningful new calculations. Therefore, the multiply-accumulate units of the 0th to the ith multiply-accumulate columns need to be turned off to save power consumption; where 0≤i<MN.

[0087] Furthermore, when the multiply-accumulate array implements depthwise convolution, in each clock cycle following the Nth cycle, there are M×N sets of multiplication operations performed simultaneously in the multiply-accumulate array until all depthwise convolution operations in the pixel difference convolutional neural network are completed, and the computational resource utilization is at most N / M.

[0088] Figure 8 This demonstrates the data loading method for depthwise convolution when the parallel size of the multiply-accumulate array is 8×9. Figure 8 In the context of image activation data, the superscript indicates the input channel to which it belongs, while the superscript of weight data indicates the output channel to which it belongs.

[0089] exist Figure 8In the example shown, the feature image activation data register and weight data register of all multiply-accumulate units are initialized to 0. In cycle 0, the activation enable signal for the 0th multiply-accumulate column is loaded. The activation enable signal for the remaining multiply-accumulate columns is set to 1, and the activation enable signal for the other multiply-accumulate columns is set to 0. This ensures that the feature image activation data of input channel 0 read from the activation loading queue is only loaded into the 0th multiply-accumulate column. Simultaneously, in the 0th cycle, the corresponding weight data for output channel 0 is loaded into the 0th multiply-accumulate column. And complete the multiplication operation; in the first cycle, the activation enable signal of the first multiplication accumulation column is loaded. With the activation enable signal set to 1, the activation enable signals for the remaining multiply-accumulate columns are all set to 0. This ensures that the feature image activation data read from the activation loading queue of input channel 1 is only loaded into the first multiply-accumulate column, while the feature image activation data loaded into the 0th multiply-accumulate column remains unchanged. The loaded weight data is then updated to the values ​​in output channel 0. At the same time, the first weight data of the corresponding output channel 1 is loaded into the first multiply-accumulate column. Over the next 7 clock cycles, the feature image activation data from input channels 2 to 8 are loaded into the 2nd to 8th multiply-accumulate columns. Simultaneously, the weight data in each multiply-accumulate column is updated with the next weight data for the corresponding output channel each clock cycle. By the 8th cycle, the 0th multiply-accumulate column has been loaded with the last weight data for output channel 0. The 8th multiply-accumulate column loads the first weight data for output channel 8. In the 9th cycle, the activation data of the 8 feature images following input channel 0 can be loaded into the 0th multiply-accumulate column, and then reloaded from... Start loading weight data and synchronously update the weight data of columns 1 through 8 of the multiply-accumulate array. Furthermore, when the actual number of columns M in the multiply-accumulate array is greater than 9, in addition to... Figure 8 As shown, in the 9th cycle, in addition to loading the 8 feature image activation data following input channel 0 to the 0th multiply-accumulate column, in the 9th cycle (i.e., the N+i cycle, N=9 and i=0), feature image activation data and weight data can also be optionally loaded to the 9th multiply-accumulate column. Since the feature image activation data on the 0th multiply-accumulate column has not been updated at this time, reloading the weight data will not produce meaningful calculations. Specifically, at the end of the 8th cycle, all feature image activation data has been multiplied with all weight data. If the feature image activation data of the 0th multiply-accumulate column is not updated in the 9th cycle, but instead loaded into the 9th multiply-accumulate column, then updating the weight data of the 0th multiply-accumulate column later is meaningless because the same calculation has already been completed. Therefore, the multiply-accumulate unit of the 0th multiply-accumulate column can be turned off to save power.

[0090] Starting from the 9th cycle, each subsequent clock cycle involves 8×9 sets of multiplication operations simultaneously in the multiply-accumulate array, until all depthwise convolutions in the pixel difference convolutional neural network are completed, achieving the highest computational resource utilization. .

[0091] It should be understood that Figure 8 In the example shown, when implementing depthwise convolution using a multiply-accumulate array, by using different activation loading enable signals for each multiply-accumulate column, and supplementing this with an appropriate loading order of feature image activation data and weight data, all nine columns of the multiply-accumulate array can be used in parallel during depthwise convolution. In contrast, the traditional method of sharing the same feature image activation data among multiply-accumulate units in the same row only allows the use of one column in the multiply-accumulate array during depthwise convolution. Therefore, the architecture proposed in this application has higher computational resource utilization. Furthermore, in the computational architecture proposed in this application, only one weight data point needs to be updated per column and one feature image activation data point per row per cycle during depthwise convolution. In other accelerators that employ multiply-accumulate arrays, to improve computational resource utilization, multiple data points need to be loaded into the same column or row per cycle, requiring more readout ports in the activation loading queue and weight loading queue, thus increasing design complexity. Therefore, this application simplifies the loading queue design, thereby reducing design difficulty.

[0092] In one embodiment, the multiply-accumulate array implements dilated convolution, including:

[0093] First, the dilated convolution is degenerated into a standard convolution by rearranging the feature images in the pixel difference convolutional neural network; then, the activation corresponding to each multiply-accumulate column in the multiply-accumulate array is loaded with the enable signal. All values ​​are set to 1. Within any clock cycle, the same multiply-accumulate column shares the same weight data, and the same multiply-accumulate row shares the same feature image activation data, thus realizing the standard convolution in the pixel difference convolutional neural network.

[0094] The feature images are rearranged as follows:

[0095] First, set the dilation coefficient to D, the height of the feature image to H, and the width to W. Let h be the row number and w be the column number of the feature image activation data in each input channel. Then, reassemble the feature image activation data into D×D sub-feature images based on the remainders of row number h divided by D and column number w divided by D. Each sub-feature image has a height of H / D rounded up and a width of W / D rounded up. If the amount of data in the original feature image is insufficient to completely fill a sub-feature image, zero values ​​are used for padding. The D×D sub-feature images are then reassembled into a large feature image in a matrix manner. During the assembly, adjacent sub-feature images in the horizontal and vertical directions are padded with D-1 zeros to prevent mutual interference during convolution calculations. The size of the reassembled feature image is... ;

[0096] Then, zero values ​​in the dilated convolution weight matrix are removed, degenerating it to the standard convolution size. The re-concatenated feature image and the degenerated standard convolution weight matrix are then calculated in a multiply-accumulate array using standard convolution. The resulting output feature image has the same size as the input feature image. The calculation results at the positions corresponding to the zero-filling positions are invalid data;

[0097] Finally, the calculated output feature image data, after removing invalid data, is rearranged in the reverse manner of the feature image rearrangement process to obtain the correct dilated convolution result.

[0098] Specifically, when the dilation rate is 2, the input feature image data rearrangement method is as follows: Figure 9 As shown, when the expansion rate is 2, for Figure 9 Performing dilated convolution on the left side of the IFM is equivalent to performing 3×3 convolution on elements numbered 1 to 4 respectively. Therefore, the IFM can be divided into... Figure 9 The pixels are rearranged as shown on the right side of the arrow, and the same number of zeros as the dilation rate are inserted between pixels with different numbers for isolation. Then, a 3×3 convolution kernel is used to perform the operation in a standard convolution manner. Finally, the results are rearranged in reverse order to restore the correct feature matrix.

[0099] In one embodiment, a design method for a chip computing architecture for pixel-differential convolutional neural networks is provided, including:

[0100] Design a chip computing architecture for pixel-differential convolutional neural networks, including M activation loading queues, M weight loading queues, and a multiply-accumulate array consisting of M×M multiply-accumulate units; wherein each multiply-accumulate column in the multiply-accumulate array is configured with an activation loading enable signal. The value of M is a positive integer power of 2.

[0101] The activation data of the feature image is loaded into the multiply-accumulate array in the horizontal direction using an activation loading queue, and the weight loading queue is used to load the weight data into the multiply-accumulate array in the vertical direction.

[0102] By having the same weight data shared within the multiply-accumulate array, and by activating the load enable signal... Under the control of [the system], the same multiply-accumulate row can share the same feature image activation data or load different feature image activation data to achieve standard convolution, pointwise convolution, depthwise convolution and dilated convolution in pixel difference convolutional neural networks.

[0103] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0104] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application.

Claims

1. A chip computing architecture for pixel-differential convolutional neural networks, characterized in that, include: There are M activation loading queues, M weight loading queues, and a multiply-accumulate array consisting of M×M multiply-accumulate units; wherein, each multiply-accumulate column in the multiply-accumulate array is provided with an activation loading enable signal. The value of M is a positive integer power of 2. The activation loading queue is used to load feature image activation data into the multiply-accumulate array in the horizontal direction, and the weight loading queue is used to load weight data into the multiply-accumulate array in the vertical direction. In the multiply-accumulate array, the same multiply-accumulate column shares the same weight data, and the activation load enable signal is activated. Under the control of , the same multiply-accumulate row can share the same feature image activation data or load different feature image activation data to realize standard convolution, pointwise convolution, depthwise convolution and dilated convolution in pixel difference convolutional neural networks; Each multiply-accumulate unit includes a multiplier, a multiplexer, an adder, and an output feature image buffer. The multiplier performs the multiplication of feature image activation data and weight data. The multiplexer selects the result calculated by the local multiplier, the multiplier in the previous row's multiply-accumulate unit, or the multiplier in the next row's multiply-accumulate unit. The adder accumulates the data output by the multiplexer with the partial sum stored in the output feature image buffer, and then writes it back into the output feature image buffer. When loading feature image activation data, the activation loading queue writes the data to a register under the control of an enable switch, and then inputs the feature image activation data into the multiplier. When loading weight data, the weight loading queue first passes the weight data through a register, and then inputs the weight data into the multiplier. When the feature image activation data and the weight data are loaded into the multiply-accumulate array, two constraints must be satisfied: Constraint 1: All loaded feature image activation data and weight data have the same input channel number; Constraint 2: The weight data loaded into different multiply-accumulate columns belong to different convolution kernels, that is, they have different output channel numbers; The multiply-accumulate array implements dilated convolution, including: First, the dilated convolution is degenerated into a standard convolution by rearranging the feature images in the pixel difference convolutional neural network; then, the activation enable signal corresponding to each multiply-accumulate column in the multiply-accumulate array is loaded. All values ​​are set to 1. Within any clock cycle, the same multiply-accumulate column shares the same weight data, and the same multiply-accumulate row shares the same feature image activation data, thus realizing the standard convolution in the pixel difference convolutional neural network. The feature images are rearranged as follows: First, set the dilation factor to D, the height of the feature image to H, and the width to W. For each input channel of the feature image, the row number of the activation data is h, and the column number is w. The activation data is then reassembled into D×D sub-feature images based on the remainders of h divided by D and w divided by D, respectively. When the data volume is insufficient to completely fill a sub-feature image, zero values ​​are used for padding. All sub-feature images are then concatenated into a large feature image in a matrix manner, with D-1 zeros between adjacent sub-feature images in the horizontal and vertical directions during concatenation. The size of the reassembled feature image is... Next, the zero values ​​in the dilated convolution weight matrix are removed, resulting in a standard convolution weight matrix. This matrix is ​​then used in the multiply-accumulate array to perform calculations on the reassembled feature image. The calculated output feature image has the same size as the input feature image, where the calculation results at the positions corresponding to the zeros are invalid data. After removing invalid data, the output feature image data is rearranged in the reverse manner of the feature image rearrangement process to obtain the correct dilated convolution result.

2. The chip computing architecture for pixel-differential convolutional neural networks according to claim 1, characterized in that, The multiply-accumulate array implements standard convolution and pointwise convolution, including: The activation enable signal corresponding to each multiply-accumulate column in the multiply-accumulate array is loaded. All values ​​are set to 1. Within any clock cycle, the same multiply-accumulate column shares the same weight data, and the same multiply-accumulate row shares the same feature image activation data, thus realizing standard convolution and pointwise convolution in pixel difference convolutional neural networks.

3. The chip computing architecture for pixel-differential convolutional neural networks according to claim 2, characterized in that, The multiply-accumulate array implements the data loading method for standard convolution, including: First, feature image activation data and weight data for any input channel number are loaded into the multiply-accumulate array. The feature image activation data is loaded into the corresponding multiply-accumulate row according to the remainder when its column coordinate number is divided by M, while the weight data is loaded into the corresponding multiply-accumulate column according to the remainder when its output channel number is divided by M. The result of multiplying the feature image activation data and the weight data is temporarily stored in the output feature image cache within the multiply-accumulate unit. Secondly, depending on the needs, choose to update the weight data or the feature image activation data first; In one implementation, the feature image activation data is kept unchanged, and all weight data belonging to the same input channel are loaded and traversed. The calculation result is temporarily stored in the output feature image buffer according to its coordinates in the output feature image. Then, a set of feature image activation data is updated, and the weight data is loaded again. The multiplication calculation result is accumulated with the previously temporarily stored result in the output feature image buffer according to the convolution operation rules. The above process is repeated until the convolution operation of the feature image activation data and weight data in the current input channel is completed. Then, the system switches to the next input channel and completes the standard convolution operation in the same way. In another implementation, the weight data is kept unchanged, and the feature image activation data within the same input channel is updated traversally. Then the weight data is updated, and the feature image activation data is loaded traversally until all standard convolution operations are completed.

4. The chip computing architecture for pixel-differential convolutional neural networks according to claim 1, characterized in that, The multiply-accumulate array implements depthwise convolution, including: When M is greater than 9, the multiply-accumulate array uses at most 9 columns out of M multiply-accumulate columns to achieve depthwise convolution; without loss of generality, the number of multiply-accumulate columns used in the multiply-accumulate array is set to N, where N≤9, that is, the actual size of the multiply-accumulate array used is M×N, and when M is less than 9, N=M; The method for implementing depthwise convolution using an M×N multiply-accumulate array is as follows: during the 0th to N-1th period, the activation enable signals corresponding to the 0th to N-1th multiply-accumulate columns are loaded. The sequence is cyclically set to 1, and the remaining columns are multiplied by the corresponding activation and load enable signals. Setting it to 0 ensures that, in each clock cycle from cycle 0 to cycle (N-1), the feature image activation data of each input channel read from the activation loading queue will only be loaded onto the current activation loading enable signal. The multiply-accumulate column is set to 1; simultaneously, in each clock cycle from cycle 0 to cycle N-1, each activation load enable signal is applied. For the multiply-accumulate column where the historical value is 1 and the current value is 0, the loaded feature image activation data remains unchanged, and the weight data is updated to the new weight data in the historically loaded output channel. The multiplication operation of the activation data and weight data is performed within each multiply-accumulate unit. Specifically, the first multiplication operation is completed in the (N-1)th multiply-accumulate column during the (N-1)th cycle. In the Nth cycle, the activation enable signal for the 0th multiply-accumulate column is reloaded. Set to 1, update the feature image activation data of the 0th multiply-accumulated column to the new feature image activation data loaded in the input channel in cycle 0, and restore the weight data to the weight data loaded in cycle 0 of the 0th multiply-accumulated column, completing the multiplication operation. At the same time, synchronously update the weight data of the 1st to N-1th multiply-accumulated columns to the new weight data in the output channel historically loaded for each multiply-accumulated column; in the following N+1 to 2N-1 cycles, sequentially load the activation enable signal of the 1st to N-1th multiply-accumulated columns. Set to 1, update the feature image activation data of each multiply-accumulate column to the new feature image activation data loaded in the input channels from the 1st cycle to the (N-1)th cycle, and restore the weight data to the weight data loaded in each multiply-accumulate column from the 1st cycle to the (N-1)th cycle, complete the multiplication operation, and simultaneously update the weight data of the remaining multiply-accumulate columns to the new weight data loaded in the output channels of each multiply-accumulate column in the previous period; update the feature image activation data and weight data in the above manner every N clock cycles until the depthwise convolution calculation of N channels is completed, and then carry out the depthwise convolution calculation of the next N channels in the same manner until the calculation of the current convolutional layer is completed.

5. The chip computing architecture for pixel-differential convolutional neural networks according to claim 4, characterized in that, When the multiply-accumulate array implements depthwise convolution, it also includes: When the actual number of columns M of the multiply-accumulate array is greater than N, the activation enable signal of the (N+i)th multiply-accumulate column can also be applied in the (N+i)th cycle. Set to 1, load the feature image activation data and weight data of the N+i input channel into the N+i multiply-accumulate column. At the same time, consider that the feature image activation data of the 0th to the ith multiply-accumulate columns have not been updated at this time, and reloading the weight data of the 0th to the ith multiply-accumulate columns will not produce meaningful new calculations. Therefore, the multiply-accumulate units of the 0th to the ith multiply-accumulate columns need to be turned off to save power consumption; where 0≤i<MN.

6. The chip computing architecture for pixel-differential convolutional neural networks according to claim 4 or 5, characterized in that, When the multiply-accumulate array implements depthwise convolution, in each clock cycle following the Nth cycle, there are M×N groups of multiplication operations performed simultaneously in the multiply-accumulate array until all depthwise convolution operations in the pixel difference convolutional neural network are completed, and the highest computational resource utilization is N / M.

7. A design method for a chip computing architecture oriented towards pixel-difference convolutional neural networks, characterized in that, The method includes: Design a chip computing architecture for pixel-differential convolutional neural networks, including M activation loading queues, M weight loading queues, and a multiply-accumulate array composed of M×M multiply-accumulate units; wherein, each multiply-accumulate column in the multiply-accumulate array is configured with an activation loading enable signal. The value of M is a positive integer power of 2. The activation loading queue is used to load feature image activation data into the multiply-accumulate array in the horizontal direction, and the weight loading queue is used to load weight data into the multiply-accumulate array in the vertical direction. By sharing the same weight data in the same multiply-accumulate column within the multiply-accumulate array, and in the activation load enable signal... Under the control of , the same multiply-accumulate row can share the same feature image activation data or load different feature image activation data to realize standard convolution, pointwise convolution, depthwise convolution and dilated convolution in pixel difference convolutional neural networks; Each multiply-accumulate unit includes a multiplier, a multiplexer, an adder, and an output feature image buffer. The multiplier performs the multiplication of feature image activation data and weight data. The multiplexer selects the result calculated by the local multiplier, the multiplier in the previous row's multiply-accumulate unit, or the multiplier in the next row's multiply-accumulate unit. The adder accumulates the data output by the multiplexer with the partial sum stored in the output feature image buffer, and then writes it back into the output feature image buffer. When loading feature image activation data, the activation loading queue writes the data to a register under the control of an enable switch, and then inputs the feature image activation data into the multiplier. When loading weight data, the weight loading queue first passes the weight data through a register, and then inputs the weight data into the multiplier. When the feature image activation data and the weight data are loaded into the multiply-accumulate array, two constraints must be satisfied: Constraint 1: All loaded feature image activation data and weight data have the same input channel number; Constraint 2: The weight data loaded into different multiply-accumulate columns belong to different convolution kernels, that is, they have different output channel numbers; The multiply-accumulate array implements dilated convolution, including: First, the dilated convolution is degenerated into a standard convolution by rearranging the feature images in the pixel difference convolutional neural network; then, the activation enable signal corresponding to each multiply-accumulate column in the multiply-accumulate array is loaded. All values ​​are set to 1. Within any clock cycle, the same multiply-accumulate column shares the same weight data, and the same multiply-accumulate row shares the same feature image activation data, thus realizing the standard convolution in the pixel difference convolutional neural network. The feature images are rearranged as follows: First, set the dilation factor to D, the height of the feature image to H, and the width to W. For each input channel of the feature image, the row number of the activation data is h, and the column number is w. The activation data is then reassembled into D×D sub-feature images based on the remainders of h divided by D and w divided by D, respectively. When the data volume is insufficient to completely fill a sub-feature image, zero values ​​are used for padding. All sub-feature images are then concatenated into a large feature image in a matrix manner, with D-1 zeros between adjacent sub-feature images in the horizontal and vertical directions during concatenation. The size of the reassembled feature image is... Next, the zero values ​​in the dilated convolution weight matrix are removed, resulting in a standard convolution weight matrix. This matrix is ​​then used in the multiply-accumulate array to perform calculations on the reassembled feature image. The calculated output feature image has the same size as the input feature image, where the calculation results at the positions corresponding to the zeros are invalid data. After removing invalid data, the output feature image data is rearranged in the reverse manner of the feature image rearrangement process to obtain the correct dilated convolution result.

Images