Convolutional neural network acceleration method and system based on pulsation data flow and sparse perception

Abstract

The invention discloses a convolutional neural network acceleration method and system based on pulsation data flow and sparse perception, and relates to the technical field of neural networks. The method comprises the following steps: constructing a convolutional layer acceleration device of a sparse convolutional neural network; and performing acceleration processing on the convolutional layer in the sparse convolutional neural network by using the convolutional layer acceleration device. According to the method, invalid calculation of sparse convolution can be effectively reduced, convolution layer calculation of the sparse convolutional neural network is accelerated, and the recognition speed of the sparse convolutional neural network in the field of image recognition is accelerated.

Description

Technical Field

[0001] The present invention relates to the field of neural network technology, and in particular to a convolutional neural network acceleration method and system based on pulsating data flow and sparse perception. Background Art

[0002] Convolutional neural networks have been widely used in various fields, such as speech recognition, image recognition, automatic modulation recognition, autonomous driving, etc., among which image recognition is the most typical application scenario. Figure 1 It is a typical convolution processing diagram in the prior art. With the development of artificial intelligence, recognition tasks have become more and more complex, and the depth of convolutional neural networks has also increased. This has led to an increasing amount of computation for convolutional neural networks, making it increasingly difficult to deploy on mobile terminals with limited computing resources. Therefore, the lightweighting of convolutional neural networks has become particularly important. Common lightweighting methods include pruning and quantization. Pruning will cause the weights of the network to be sparse. On the other hand, the commonly used activation function of convolutional neural networks is the Relu function, which directly assigns activation values ​​less than 0 to 0, resulting in sparse feature maps. Therefore, sparse convolutional neural networks have bilateral sparsity.

[0003] The advantages of sparse convolutional neural networks include: reducing data storage, reducing invalid calculations, and reducing data movement, thereby reducing power consumption. At the same time, there are two key challenges in the processing of sparse convolution: one is the problem of internal connection of data during sparse convolution operations, and the other is the load imbalance problem caused by the different sparsity of each layer of the network. The first problem is that after the sparse network is stored in a compressed manner, it is impossible to use a regular matrix-vector multiplication form during the convolution operation. Only a coordinated traversal method can be designed to multiply and accumulate the corresponding non-zero values. The second problem is caused by the network's activation function Relu. Different layers will cause different degrees of sparsity after activation.

[0004] Currently, neural networks are usually processed using GPUs or customized ASIC accelerators, among which sparse convolutional neural networks are more often accelerated using ASICs. However, existing accelerators have great limitations when processing sparse convolutions. Some accelerators only handle unilateral sparsity; some accelerators cannot handle the internal connection problem of down-convolution calculations with multiple step sizes; some accelerators do not solve the problem of load imbalance when processing sparse convolutions at different layers, all of which will result in a waste of computing resources. Summary of the invention

[0005] The technical problem to be solved by the present invention is how to provide a convolution acceleration method and system based on pulsating data flow and sparse perception, which can effectively reduce the invalid calculation of sparse convolution and reduce the running time of sparse convolution.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is: a convolutional neural network acceleration method based on pulsating data flow and sparse perception, comprising the following steps: Construct a convolutional layer accelerator for sparse convolutional neural networks; The convolutional layer acceleration device is used to accelerate the convolutional layer in the sparse convolutional neural network.

[0007] A further technical solution is that the convolutional layer acceleration device is constructed by the following method: S1: Use the convolution kernel as an anchor point and design all the data required to perform multiplication at a single position of the convolution kernel into a channel; S2: The input feature map data of the channel is transmitted in a pulsating manner within the channel according to the sliding characteristics of the convolution, completing the full reuse of the input feature map and convolution kernel weight data; S3: Input the input data in the channel to the non-zero judgment unit with the corresponding weight register in sequence. If the weight is 0, the channel is closed and the data is not passed to the subsequent calculation unit. If the weight is not 0, determine whether the input data is 0. If the input data is not 0, the non-zero weight and input are passed to the subsequent multiplication and accumulation calculation unit. S4: Perform multiplication and accumulation calculation on the non-zero data passed; S5: Perform nonlinear activation processing on the result of the multiplication and accumulation calculation to obtain the output, completing the construction of the convolutional layer acceleration device.

[0008] Correspondingly, the present invention also discloses a convolutional neural network acceleration system based on pulsating data flow and sparse perception, comprising: Convolutional layer accelerator building module, used to build a convolutional layer accelerator for sparse convolutional neural networks; The convolutional neural network acceleration module is used to accelerate the convolutional layer in the sparse convolutional neural network using the convolutional layer acceleration device.

[0009] The convolutional layer acceleration device building module includes: The channel synthesis module is used to design all the data required for multiplication at a single position of the convolution kernel into a channel and output all non-zero operands; A cross array module is used to output all non-zero operands obtained in sequence to subsequent multiplication and accumulation modules; The multiplication-accumulation module is used to perform multiplication-accumulation calculations on the non-zero data passed in, including a multiplication tree and an addition tree. Based on the principle of sparseness, the number of multipliers in the multiplication tree is less than the number of channels. Based on the multi-clock domain design, once the input operand multiplication tree cannot be fully calculated in one cycle, a higher clock is used; The nonlinear activation unit is used to perform nonlinear activation on the result of multiplication and accumulation and output the result.

[0010] The beneficial effects of the above technical solution are: the method avoids the data compression and decompression process in the sparse convolution process, and the sparse perception greatly utilizes the bilateral sparse characteristics of weights and input data in the convolution process, reduces unnecessary calculations, and improves the utilization rate of computing resources. In addition, a new pulsating data stream is used to realize all data reuse in the convolution process, reduce the frequent interaction between data and memory, reduce the power consumption of convolution processing, and accelerate the convolution processing process, thereby accelerating the processing of sparse convolution neural networks, thereby improving the speed of image recognition. In addition, the method of the present invention is also applicable to three-dimensional convolution and multi-convolution kernel convolution. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0012] Figure 1 It is a typical convolution processing schematic diagram in the prior art; Figure 2 is a main flow chart of the method described in the embodiment of the present invention; Figure 3 is a flow chart of constructing a convolutional layer acceleration device in the method described in an embodiment of the present invention; Figure 4 is a convolution processing flow chart of the method described in an embodiment of the present invention; Figure 5 is a schematic diagram of the halo of a block feature map in an embodiment of the present invention; Figure 6 is a specific processing flow chart of a convolution in an embodiment of the present invention; Figure 7 is a principle block diagram of the system according to an embodiment of the present invention; Figure 8 It is a principle block diagram of a building block of a convolutional layer acceleration device in the system described in an embodiment of the present invention; Fig. 9 Schematic diagram of the architecture of the building block of the convolutional layer acceleration device in an embodiment of the present invention; Fig.10 It is a structural diagram of a non-zero judgment unit in an embodiment of the present invention. DETAILED DESCRIPTION

[0013] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0014] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0015] Overall, such as Figure 2 As shown, the embodiment of the present invention discloses a convolutional neural network acceleration method based on pulsating data flow and sparse perception, comprising the following steps: Construct a convolutional layer accelerator for sparse convolutional neural networks; The convolutional layer acceleration device is used to accelerate the convolutional layer in the sparse convolutional neural network.

[0016] The above method completes the acceleration of the sparse convolutional neural network, and the accelerated sparse convolutional neural network can be used to process images to complete image recognition.

[0017] Further, such as Figure 3-Figure 4 As shown, the convolutional layer acceleration device is constructed by the following method: S1: Use the convolution kernel as an anchor point and design all the data required to perform multiplication at a single position of the convolution kernel into a channel, where the number of channels is the size of the convolution kernel and the depth of the channel is the width of the output feature map.

[0018] S2: The input feature map data of the channel is transmitted in a pulsating manner within the channel according to the sliding characteristics of the convolution, completing the full multiplexing of the input feature map and weight data.

[0019] S3: Input the input data in the channel into the non-zero decision unit with the corresponding weight register in sequence. If the weight is 0, the channel is closed and the data is not passed to the subsequent calculation unit, because 0 in the convolution multiplication and accumulation process will not have any effect on the final result. If the weight is not 0, determine whether the input data is 0. If the input data is not 0, the non-zero weight and input are passed to the subsequent multiplication and accumulation calculation unit.

[0020] S4: The multiplication-accumulation calculation unit performs multiplication-accumulation calculation on the non-zero data transmitted.

[0021] S5: Activate the result of the multiplication and accumulation calculation to obtain the output, completing the construction of the convolutional layer acceleration device.

[0022] In the present application, steps S2 to S5 are performed in a pipeline, and a column of data organized in the channel is transmitted to a non-zero judgment unit with a register for subsequent operations, wherein S2 specifically includes the following steps: S21: Channel initialization, use the feature map data corresponding to the size of 1 convolution kernel to assign the value of the first column of the channel.

[0023] S22: The convolution kernel slides to the right for the first time, storing the data to be reused in the second column of the channel, and putting the newly retrieved data page from the memory into the second column of the channel.

[0024] S23: Repeat step S22 until the first horizontal sliding is completed.

[0025] S24: The convolution kernel starts the first vertical sliding, pulsating the data to be multiplexed to the adjacent channel, and taking new data from the memory and putting it into the corresponding position of the first column of the channel.

[0026] S25: The convolution kernel starts to slide to the right, the multiplexed data pulsates to the adjacent channel, and the newly obtained data is placed in the corresponding position of the second column of the channel.

[0027] S26: Repeat step S25 until the sliding is completed.

[0028] S27: Repeat steps S24 to S26 until the convolution of one feature map is completed.

[0029] When the feature map is too large, a block processing method is used to deal with the halo generated by the block (a halo structure such as Figure 5 As shown in the figure, the data of the connecting parts of adjacent blocks will be reused by two blocks at the same time, and the size of the halo is related to the size of the convolution kernel and the step size of the convolution. In order to solve the problem, the present invention uses additional storage space for storage.

[0030] The present invention takes two-dimensional convolution as an example. A 2×2 convolution kernel convolves a 6×6 input feature map with a step size of 1. The data of each convolution kernel corresponds to a separate input feature map channel, which has achieved all the reuse of weights. The pulsating data stream mainly completes the reuse of feature maps. The convolution operation generally slides to the right first, then slides down, and then slides to the right again until the convolution completes the input feature map. Figure 6 As shown, the designed data flow can be described as follows.

[0031] (1) Internal data organization of the channel. When the convolution kernel performs the first convolution, Figure 6As shown in Cycle0, according to the design concept of the accelerator, the convolution kernel corresponding to each channel is in the same position. First, the input feature map data corresponding to the first convolution of the convolution kernel is taken out and put into the first column of the channel.

[0032] (2) The data in the first column of the channel is passed to the non-zero decision device with the corresponding weight. First, it is determined whether the weight is 0. If it is 0, the channel is closed. In this embodiment, the channel corresponding to the fourth weight value is closed. Then the input data passed by the channel is determined. If it is 0, no subsequent calculation is performed. If it is not 0, the two non-zero values ​​are passed to the subsequent calculation module for calculation.

[0033] (3) The convolution kernel starts to slide to the right, and the second column of feature maps is reused. The corresponding data is moved upward from the last two rows of the first column of the channel to the first two rows of the second column of the channel. The newly taken third column of feature map data is placed in the last two rows of the second column of the channel, as shown in Figure 6 As shown in Cycle1.

[0034] (4) The data in the second column of the channel is passed to the non-zero decision device with the corresponding weight. At this time, it is only necessary to determine whether the input data passed through the channel that is not closed is 0. If it is 0, no subsequent calculation is performed. If it is not 0, the two non-zero values ​​are passed to the subsequent calculation module for calculation.

[0035] (5) Repeat (3)-(4) until the last column of data in the first two rows of the input feature map is retrieved, and the horizontal sliding ends. At this point, the channel depth is 5. Figure 6 Cycle2-Cycle4 as shown.

[0036] (6) The convolution kernel then returns to the leftmost side of the input feature map and slides down one step, as shown in Figure 6 As shown in Cycle5, the data in the third row is newly added data, and the data in the second row is reused. The data of channel 0 and channel 2 are all discarded, and the data of the remaining channels are shifted to the previous channel. For example, the data of channel 1 is pulsed to channel 0, and then the two newly added data after this pulse are written to the first column of channel 1 and channel 3.

[0037] (7) The data in the first column of the channel at this time is passed to the non-zero decision device with the corresponding weight. If the input is 0, no subsequent calculation is performed. If it is not 0, the two non-zero values ​​are passed to the subsequent calculation module for calculation.

[0038] (8) Then the convolution kernel continues to slide to the right, as Figure 6 As shown in Cycle6, at this time, the data in the first row remains unchanged, the corresponding data of channel 0 and channel 2 remain unchanged, the data in the first column of channel 3 is put into the second column of channel 1, and channel 3 reads a new number from the memory and puts it into the second column.

[0039] (9) The data in the second column of the channel is passed to the non-zero decision device with the corresponding weight. If it is 0, no subsequent calculation is performed. If it is not 0, the two non-zero values ​​are passed to the subsequent calculation module for calculation.

[0040] (10) Repeat the operations (8)-(9) until the horizontal convolution slide is completed. Figure 5 Cycle7-Cycle9 as shown.

[0041] (11) Repeat steps (6) to (10) until the entire convolution is completed.

[0042] Figure 6 The red data in the middle are the newly retrieved data from the memory, the black data are the pulsating data in the channel, and the blue data are the data that remain stationary in the channel; the gray at the bottom represents a closed channel, and the yellow part at the bottom is the data currently to be passed to the non-zero judgment unit.

[0043] The calculations of the subsequent calculation modules in the above steps include: Different clock domains are used to perform multiplication and accumulation operations according to the amount of data passed into the multiplication and accumulation unit.

[0044] The multiplication and accumulation results are then passed to the activation unit for activation operation.

[0045] Finally, the activated result is passed to the global cache of the feature map.

[0046] Correspondingly, such as Figure 7 As shown, the embodiment of the present invention also discloses a convolutional neural network acceleration system based on pulsating data flow and sparse perception, including: Convolutional layer accelerator building module, used to build a convolutional layer accelerator for sparse convolutional neural networks; The convolutional neural network acceleration module is used to accelerate the convolutional layer in the sparse convolutional neural network using the convolutional layer acceleration device.

[0047] It should be noted that the system can add pooling layer and fully connected layer modules to realize the accelerated processing of the image by the entire sparse convolutional neural network and complete the recognition of the image.

[0048] like Figure 8-Figure 9 As shown, the convolutional layer acceleration device construction module specifically includes: The channel synthesis module, the cache channel of the input feature map and the convolution kernel weight, designs all the data required to perform multiplication at a single position of the convolution kernel into a channel. The channel consists of two parts, one is a non-zero decision unit with a weight register, and the other part stores the input feature map data to be calculated with the weight.

[0049] A cross array module is used to output all non-zero operands obtained in sequence to subsequent multiplication and accumulation modules; The multiplication-accumulation module is used to perform multiplication-accumulation calculations on the non-zero data passed in, including a multiplication tree and an addition tree. Based on the principle of sparseness, the number of multipliers in the multiplication tree is less than the number of channels. Based on the multi-clock domain design, once the input operand multiplication tree cannot be fully calculated in one cycle, a higher clock is used; The nonlinear activation unit is used to perform nonlinear activation on the result of multiplication and accumulation and output the result.

[0050] The channel synthesized by the channel synthesis module includes two parts, one part is a non-zero decision unit with a weight register, and the other part stores the input feature map data to be calculated with the convolution kernel weight. Fig.10 As shown in the figure, the non-zero decision unit with weight register includes a non-zero decision device and a weight register. The weight register stores the weight data of the convolution kernel. The decision device executes the non-zero decision logic, that is, if the weight value is 0, the channel is directly closed and no multiplication operation is performed with all subsequent input values; if the input activation value is 0, the input activation is skipped and no subsequent calculation is performed.

[0051] Global cache module, used to store feature maps and convolution kernel weights.

[0052] There may be multiple non-zero decision units with weight registers inside each channel in the system to support convolution operations of multiple convolution kernels on the same input feature map.

[0053] The method of the present invention uses sparse perception to process bilateral sparsity in the sparse convolution process, without the need for additional data compression and decompression operations. A new data stream is also designed in a pulsating manner to achieve multiplexing of all data in the convolution process, which can effectively reduce the invalid calculation of sparse convolution and reduce the running time of sparse convolution. In turn, the processing of sparse convolutional neural networks can be accelerated, thereby improving the speed of image recognition.

Claims

1. A convolutional neural network acceleration method based on pulsating data flow and sparse perception, characterized in that The steps include: Construct a convolutional layer accelerator for sparse convolutional neural networks; The convolutional layer acceleration device is used to accelerate the convolutional layer in the sparse convolutional neural network.

2. The convolutional neural network acceleration method based on pulsating data flow and sparse perception as claimed in claim 1, characterized in that The convolutional layer acceleration device is constructed by the following method: S1: Use the convolution kernel as an anchor point and design all the data required to perform multiplication at a single position of the convolution kernel into a channel; S2: The input feature map data of the channel is transmitted in a pulsating manner within the channel according to the sliding characteristics of the convolution, completing the full reuse of the input feature map and convolution kernel weight data; S3: Input the input data in the channel to the non-zero judgment unit with the corresponding weight register in sequence. If the weight is 0, the channel is closed and the data is not passed to the subsequent calculation unit. If the weight is not 0, determine whether the input data is 0. If the input data is not 0, the non-zero weight and input are passed to the subsequent multiplication and accumulation calculation unit. S4: Perform multiplication and accumulation calculation on the non-zero data passed; S5: Perform nonlinear activation processing on the result of the multiplication and accumulation calculation to obtain the output, completing the construction of the convolutional layer acceleration device.

3. The convolutional neural network acceleration method based on pulsating data flow and sparse perception as claimed in claim 2, characterized in that: The S2 specifically includes the following steps: S21: Initialize the channel, use the feature map data corresponding to the size of one convolution kernel to assign the value of the first column of the channel; S22: The convolution kernel slides to the right for the first time, storing the data to be reused in the second column of the channel, and the newly retrieved data page from the memory is placed in the second column of the channel; S23: repeat step S22 until the first horizontal sliding is completed; S24: The convolution kernel starts the first vertical sliding, pulsating the data to be multiplexed to the adjacent channel, and taking new data from the memory and putting it into the corresponding position of the first column of the channel; S25: The convolution kernel starts to slide to the right, the multiplexed data pulsates to the adjacent channel, and the newly obtained data is placed in the corresponding position of the second column of the channel; S26: Repeat step S25 until the sliding is completed; S27: Repeat steps S24 to S26 until the convolution of one feature map is completed.

4. The convolutional neural network acceleration method based on pulsating data flow and sparse perception as claimed in claim 2, characterized in that: When the feature map is too large, it is processed in a block-by-block manner, and the halo generated by the block is stored in an additional storage space.

5. The convolutional neural network acceleration method based on pulsating data flow and sparse perception as claimed in claim 2, characterized in that: In step S1: the number of channels is the size of the convolution kernel, and the depth of the channel is the width of the output feature map.

6. The convolutional neural network acceleration method based on pulsating data flow and sparse perception as claimed in claim 1, characterized in that: The method further includes: before step S1, the step of inputting the feature map and the convolution kernel weight into the cache channel for storage.

7. A convolutional neural network acceleration system based on pulsating data flow and sparse perception, characterized in that include: Convolutional layer accelerator building module, used to build a convolutional layer accelerator for sparse convolutional neural networks; The convolutional neural network acceleration module is used to accelerate the convolutional layer in the sparse convolutional neural network using the convolutional layer acceleration device.

8. The convolutional neural network acceleration system based on pulsating data flow and sparse perception as claimed in claim 7, characterized in that: The convolutional layer acceleration device building module includes: The channel synthesis module is used to design all the data required for multiplication at a single position of the convolution kernel into a channel and output all non-zero operands; A cross array module is used to output all non-zero operands obtained in sequence to subsequent multiplication and accumulation modules; The multiplication-accumulation module is used to perform multiplication-accumulation calculations on the non-zero data passed in, including a multiplication tree and an addition tree. Based on the principle of sparseness, the number of multipliers in the multiplication tree is less than the number of channels. Based on the multi-clock domain design, once the input operand multiplication tree cannot be fully calculated in one cycle, a higher clock is used; The nonlinear activation unit is used to perform nonlinear activation on the result of multiplication and accumulation and output the result.

9. The convolutional neural network acceleration system based on pulsating data flow and sparse perception as claimed in claim 7, characterized in that: The system also includes a cache module for caching input feature maps and convolution kernel weights.

10. The convolutional neural network acceleration system based on pulsating data flow and sparse perception as claimed in claim 7, characterized in that: The channel synthesized by the channel synthesis module includes two parts, one part is a non-0 decision unit with a weight register, and the other part stores the input feature map data to be calculated with the convolution kernel weight; the non-0 decision unit includes a non-0 decision device and a weight register, the weight register is used to store the weight data of the convolution kernel, the non-0 decision device executes non-0 decision logic, if the weight value is 0, the channel is directly closed, and no multiplication operation is performed with all subsequent input values; if the input activation value is 0, the input activation is skipped, and no subsequent calculation is performed.

Images