Configurable sparse convolution hardware acceleration method and system based on channel fusion

Abstract

The present invention provides a configurable sparse convolution hardware acceleration method and system based on channel fusion, including: step 1: obtaining the offset addresses of all non-zero valid activation value data and corresponding convolution kernel weight data; step 2: storing the offset addresses of the valid activation value data and the corresponding convolution kernel weight data, and performing multiplication and accumulation operations on the corresponding valid value data pairs in sequence; step 3: setting a data selector and a data splitter in the multiplication and accumulation queue, and re-decoupling the data results of the fused convolution kernel to the corresponding output channel for accumulation; step 4: reallocating the data of different channels, and re-sending the fused corresponding output channel data to the position of the output channel before fusion. The present invention performs channel fusion preprocessing on the sparse convolution kernel, so that the amount of valid data between the fused output channels is as balanced as possible, thereby bringing higher hardware utilization and higher sparse convolution acceleration efficiency.

Description

Technical Field

[0001] The present invention relates to the technical field of convolutional neural networks, and in particular to a configurable sparse convolution hardware acceleration method and system based on channel fusion. Background Art

[0002] In recent years, with the advancement of algorithms and the improvement of computing power, the field of artificial intelligence has developed greatly and has affected all industries. However, large convolutional neural networks have excellent performance but also place extremely high demands on computing resources and storage resources. Especially in some resource-constrained scenarios, some methods are often needed to reduce the cost of computing and storage resources, that is, lightweight models.

[0003] The nonlinear activation layers (ReLU, etc.) in convolutional neural networks make the activation values ​​quite sparse. The activation values ​​in a typical neural network are about 70% sparse. L1 and L2 regularization accelerates training and avoids overfitting while also bringing more zero values ​​to the weights. Furthermore, Han et al. proposed that after pruning, dropout, and quantization of convolutional neural networks, the effective neuron connections can be compressed to 1 / 9 to 1 / 13 of the original while maintaining comparable accuracy, and the weight data can be reduced to about 1 / 10 of the original.

[0004] Therefore, a feasible way is to make full use of the sparsity of convolutional neural networks to reduce resource requirements. That is, since there are many invalid zero-value data in convolutional neural networks, these data do not need to be stored or calculated. By skipping these invalid data in a reasonable way, a lot of storage space and computing resources can be saved.

[0005] However, factors that cause convolution sparsity due to pruning, Dropout, ReLU, regularization, etc. are often irregular, and the resulting unstructured irregular sparsity makes it difficult for general hardware to benefit from its sparsity. Structured pruning often has a large performance gap compared to unstructured pruning. When calculating convolution, CPUs and GPUs often convert it into matrix multiplication for calculation. Irregular sparse matrix multiplication may even bring more additional overhead than benefits due to branch jumps and uneven loads between different threads. Therefore, customized hardware is required to accelerate unstructured pruning.

[0006] Patent document CN107341544B (application number: CN201710524017.0) discloses a reconfigurable accelerator based on a divisible array and an implementation method thereof, wherein the reconfigurable accelerator includes: a scratch pad memory cache area for realizing data reuse of convolution calculation and sparse fully connected calculation; a divisible calculation array, including multiple reconfigurable calculation units, the divisible calculation array is divided into a convolution calculation array and a sparse fully connected calculation array; a register cache area, a storage area composed of multiple registers, providing input data, weight data and corresponding output results for convolution calculation and sparse fully connected calculation; the input data and weight data of the convolution calculation are respectively input into the convolution calculation array, and the convolution calculation result is output; the input data and weight data of the sparse fully connected calculation are respectively input into the sparse fully connected calculation array, and the sparse fully connected calculation result is output.

[0007] Regarding the hardware acceleration design of sparse convolutional neural networks, similar inventions either skip sparse convolution kernels, or skip sparse weight values, or skip both at the same time. These designs have good sparse computing acceleration efficiency at low parallelism, but when the operator parallelism increases, these designs will be affected by low hardware utilization caused by load imbalance due to the huge difference in the amount of effective data between each parallel. Based on this, the present invention proposes a configurable sparse convolution hardware acceleration design scheme based on channel fusion, which skips coefficient activation values ​​while performing channel fusion preprocessing on sparse weight data of different output channels. While skipping invalid weight calculations, the channel fusion algorithm makes the effective data between each output channel as balanced as possible, so that a high sparse acceleration ratio and hardware utilization can still be maintained at a large operator parallelism. Summary of the invention

[0008] In view of the defects in the prior art, the object of the present invention is to provide a configurable sparse convolution hardware acceleration method and system based on channel fusion.

[0009] The configurable sparse convolution hardware acceleration method based on channel fusion provided by the present invention includes:

[0010] Step 1: Use the bitmap sliding window method to obtain the offset addresses of all non-zero valid activation value data and corresponding convolution kernel weight data;

[0011] Step 2: Store the effective activation value data and the offset address of the corresponding convolution kernel weight data, and perform multiplication and accumulation operations on the corresponding effective value data pairs in sequence;

[0012] Step 3: Set the data selector and data splitter in the multiplication and accumulation queue to decouple the data results of the fused convolution kernel to the corresponding output channel for accumulation;

[0013] Step 4: Redistribute the data of different channels and send the data of the corresponding output channel after fusion back to the position of the output channel before fusion.

[0014] Preferably, for the case where the convolution kernel is sparsely distributed and uneven, spatial redundancy is set for the position where the effective value of the hardware operator 3×3 convolution kernel appears most frequently, so that the hardware convolution operator and the corresponding sliding window size become 3×3+1.

[0015] Preferably, for channel fusion:

[0016] First, the corresponding K groups of output channels are divided into C / C_in_parallel groups of convolution kernels according to the input parallelism C_in_parallel of the operation unit. Each group of convolution kernels contains K groups of output channels and C_in_parallel groups of input channels.

[0017] Next, for each group of convolution kernels, count and sort the amount of valid data contained in all C_in_parallel group input channels under each group of output channels;

[0018] Finally, all output channel groups are matched in pairs according to the smallest and largest valid data amounts. The rule for successful matching is that the data at all corresponding positions are not valid values ​​at the same time.

[0019] All successfully matched channels are merged into one channel, and the channels that are not successfully merged continue to merge with the next channel in order until all remaining unmatched channels are traversed.

[0020] Preferably, a bitmap is used to store sparse activation values. If the final activation value is stored with an n-bit width, the bitmap storage stores whether all data are valid values ​​and all truly valid data at a cost of 1 / n.

[0021] Preferably, for sparse weight convolution kernels, by performing channel fusion on the sparse weight data on the server side, the storage overhead of the weight data is reduced, invalid zero value calculations are skipped, and the effective calculation amount between different operation units is balanced.

[0022] The configurable sparse convolution hardware acceleration system based on channel fusion provided by the present invention includes:

[0023] Module M1: Use a bitmap sliding window to obtain the offset addresses of all non-zero valid activation value data and corresponding convolution kernel weight data;

[0024] Module M2: Store the effective activation value data and the offset address of the corresponding convolution kernel weight data, and perform multiplication and accumulation operations on the corresponding effective value data pairs in sequence;

[0025] Module M3: Set the data selector and data splitter in the multiplication and accumulation queue to decouple the data results of the fused convolution kernel to the corresponding output channel for accumulation;

[0026] Module M4: Redistribute the data of different channels and send the data of the corresponding output channel after fusion back to the position of the output channel before fusion.

[0027] Preferably, for the case where the convolution kernel is sparsely distributed and uneven, spatial redundancy is set for the position where the effective value of the hardware operator 3×3 convolution kernel appears most frequently, so that the hardware convolution operator and the corresponding sliding window size become 3×3+1.

[0028] Preferably, for channel fusion:

[0029] First, the corresponding K groups of output channels are divided into C / C_in_parallel groups of convolution kernels according to the input parallelism C_in_parallel of the operation unit. Each group of convolution kernels contains K groups of output channels and C_in_parallel groups of input channels.

[0030] Next, for each group of convolution kernels, count and sort the amount of valid data contained in all C_in_parallel group input channels under each group of output channels;

[0031] Finally, all output channel groups are matched in pairs according to the smallest and largest valid data amounts. The rule for successful matching is that the data at all corresponding positions are not valid values ​​at the same time.

[0032] All successfully matched channels are merged into one channel, and the channels that are not successfully merged continue to merge with the next channel in order until all remaining unmatched channels are traversed.

[0033] Preferably, a bitmap is used to store sparse activation values. If the final activation value is stored with an n-bit width, the bitmap storage stores whether all data are valid values ​​and all truly valid data at a cost of 1 / n.

[0034] Preferably, for sparse weight convolution kernels, by performing channel fusion on the sparse weight data on the server side, the storage overhead of the weight data is reduced, invalid zero value calculations are skipped, and the effective calculation amount between different operation units is balanced.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] While skipping the sparse activation values, the present invention performs channel fusion preprocessing on the sparse convolution kernels, so that the amount of effective data between the fused output channels is as balanced as possible, thereby bringing higher hardware utilization and higher sparse convolution acceleration efficiency; for unevenly distributed convolution kernels, the channel fusion efficiency can be improved when the sparse data of the convolution kernel is unevenly distributed through the spatial redundancy design of the convolution kernel unit. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0038] Figure 1 A schematic diagram of a common convolution operator channel fusion algorithm provided by the present invention;

[0039] Figure 2 A schematic diagram of the failure of channel fusion of the common convolution operator provided by the present invention;

[0040] Figure 3 A schematic diagram of a successful channel fusion of a customized convolution operator provided by the present invention;

[0041] Figure 4 A schematic diagram of a zero-jumping calculation scheme for a sparse convolution operator based on channel fusion provided by the present invention;

[0042] Figure 5 Schematic diagram of the two-dimensional sparse convolution operator array provided by the present invention. DETAILED DESCRIPTION

[0043] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0044] Example:

[0045] The present invention compresses the storage of activation values ​​and weights of sparse convolution and skips invalid calculations. At the same time, it can balance the amount of effective data between different output channels as much as possible through channel fusion preprocessing, so as to achieve high hardware utilization and acceleration efficiency when the parallelism of the computing unit is expanded. At the same time, for the case of sparse and unevenly distributed convolution kernels, customized hardware operators can also provide good channel fusion and acceleration support.

[0046] The sparse convolution calculation of the present invention is divided into two parts: sparse activation value and sparse weight kernel:

[0047] 1. For sparse activation values, the present invention adopts bitmap storage. Assuming that the final activation value is stored with a bit width of n-bit, the bitmap storage only needs to store whether all data are valid values ​​and all truly valid data at a cost of 1 / n.

[0048] 2. For sparse weight convolution kernels, the present invention performs channel fusion on the sparse weight data on the server side to reduce the storage overhead of the weight data, skip invalid zero value calculations, and balance the effective calculation amount between different operation units.

[0049] Specifically, for the channel fusion algorithm:

[0050] First, the corresponding K groups of output channels are divided into C / C_in_parallel groups of convolution kernels according to the input parallelism C_in_parallel of the operation unit. Each group of convolution kernels contains K groups of output channels and C_in_parallel groups of input channels. Then, for each group of convolution kernels, the amount of valid data contained in all C_in_parallel groups of input channels under each group of output channels is counted and sorted. Finally, all output channel groups are matched in pairs according to the minimum and maximum valid data amounts. The rule for successful matching is that the data at all corresponding positions are not valid values ​​at the same time. All successfully matched channels are fused into one channel, and the channels that have not been successfully fused continue to be combined with the next channel in order until all remaining unmatched channels are traversed. This channel fusion algorithm aims to minimize the number of valid channels after fusion, and adopts a nearly traversal algorithm. Although the complexity of the algorithm is relatively high, because the weights will not change after the model is trained, it only needs to be run once, and the high complexity is acceptable.

[0051] The present invention performs channel fusion on sparse convolution kernels and performs bitmap sparse storage of activation values. At the same time, for sparse calculations, it also considers zero-jump acceleration for invalid calculations.

[0052] Specifically, for the zero-skipping calculation of sparse convolution:

[0053] For sparse activation values, the present invention adopts a sliding window method for the bitmap, and sends it to the first non-zero detection unit. The first non-zero detection unit then sends the relative offset addresses of all non-zero values ​​to the corresponding valid activation value storage unit and the fused convolution kernel weight storage unit in turn. The above storage unit then sends the corresponding valid activation value and weight data to the multiplication unit and the addition unit to perform multiplication and accumulation operations. The corresponding multiplication and accumulation queue has a data selector and a data splitter to re-decouple the data results of the fused convolution kernel to the corresponding output channel for accumulation and finally output. For the output parallel units of different channels, they are finally sent to the data redistribution unit to re-send the data of the corresponding output channel after fusion to the position of the output channel before fusion.

[0054] In particular, when the sparse distribution of the convolution kernel is uneven, spatial redundancy design is implemented for the position where the effective value of the hardware operator 3×3 convolution kernel appears most frequently, so that the size of the hardware convolution operator and the corresponding sliding window becomes 3×3+1, so as to avoid the impact of the decrease in channel fusion efficiency caused by the uneven distribution of sparse convolution kernel data.

[0055] The configurable sparse convolution hardware acceleration system based on channel fusion provided by the present invention includes: module M1: using a bitmap sliding window method to obtain the offset addresses of all non-zero valid activation value data and corresponding convolution kernel weight data; module M2: storing the offset addresses of the valid activation value data and the corresponding convolution kernel weight data, and performing multiplication and accumulation operations on the corresponding valid value data pairs in sequence; module M3: setting a data selector and a data splitter in the multiplication and accumulation queue, and re-decoupling the data results of the fused convolution kernel to the corresponding output channel for accumulation; module M4: reallocating the data of different channels, and re-sending the data of the corresponding output channel after fusion to the position of the output channel before fusion.

[0056] For the case of uneven sparse distribution of convolution kernels, spatial redundancy is set for the position where the effective value of the hardware operator 3×3 convolution kernel appears most frequently, so that the size of the hardware convolution operator and the corresponding sliding window becomes 3×3+1.

[0057] For channel fusion: First, the corresponding K groups of output channels are divided into C / C_in_parallel groups of convolution kernels according to the input parallelism C_in_parallel of the operation unit. Each group of convolution kernels contains K groups of output channels and C_in_parallel groups of input channels. Then, for each group of convolution kernels, the amount of valid data contained in all C_in_parallel groups of input channels under each group of output channels is counted and sorted. Finally, all output channel groups are matched in pairs according to the smallest and largest amounts of valid data. The rule for successful matching is that the data at all corresponding positions are not valid values ​​at the same time. All successfully matched channels are fused into one channel, and the channels that have not been successfully fused continue to be combined with the next channel in order until all remaining unmatched channels are traversed.

[0058] The sparse activation values ​​are stored in a bitmap format. If the final activation value is stored with an n-bit width, the bitmap format stores all valid values ​​and all truly valid data at a cost of 1 / n. For sparse weight convolution kernels, the sparse weight data is channel-fused on the server side to reduce the storage overhead of the weight data, skip invalid zero value calculations, and balance the effective calculation amount between different computing units.

[0059] Figure 1 The figure shows a schematic diagram of the channel fusion algorithm for sparse weight convolution kernels proposed by the present invention, where the two convolution kernels of each group do not have positions where they are valid values ​​at the same time. Therefore, after the channel fusion algorithm, the sparse convolution kernels of the four different output channels are fused into dense convolution kernels of two channels. It can be seen that the channel fusion algorithm compresses the storage of sparse weights and skips invalid calculations, and can balance the amount of valid data between different output channels as much as possible through channel fusion preprocessing, so that when the parallelism of the computing unit is expanded, it can still have a high hardware utilization and acceleration efficiency.

[0060] Figure 2 The figure shows an example of the failure of the general convolution operator to support the channel fusion algorithm when the sparse weights are unevenly distributed. It can be seen that, for example, in the RepVGG-style network, the middle of the convolution kernel is often a valid value. Under such sparse distribution, it is difficult for general operators to support channel fusion of such sparsely distributed weight data.

[0061] Figure 3The figure shows a successful example of the channel fusion algorithm for sparsely distributed weighted convolution kernels of the configurable custom operator proposed by the present invention. When the sparse distribution of the convolution kernel is uneven, the present invention implements spatial redundancy design for the position where the effective value of the hardware operator 3×3 convolution kernel appears most frequently, so that the hardware convolution operator and the corresponding sliding window size become 3×3+1, so as to avoid the influence of the uneven distribution of sparse convolution kernel data on the channel fusion efficiency. It can be seen that in the case where the data in the middle of the 3×3 convolution kernel as shown in the figure is often a valid value, the present invention greatly improves the efficiency of the channel fusion algorithm at the cost of a smaller storage space by performing spatial redundancy design on the data in the middle of the convolution kernel.

[0062] For the case of custom operators, the channel fusion algorithm can be described by the following pseudo code. This channel fusion algorithm aims to minimize the number of effective channels after fusion.

[0063] Customized channel fusion algorithm:

[0064] 1:for(c=1;c<=C;c+=I_parallel){

[0065] 2:for(k=1;k<=K;k+=O_parallel){

[0066] 3:kernel_cnt[c,k]=sum(kernel[1:3][1:3][c][k]); / / Get the number of valid weight values ​​for different input and output channel groups

[0067] 4:}}

[0068] 5:for(c=1;c<=C;c+=I_parallel){

[0069] 6:for(n=0;n <I_parallel;n++){

[0070] 7:for(k=1;k<=K;k++){tmp_kernel_cnt[k]+=kernel_cnt[c+n][k]; / / tmp_kernel_cnt[k] represents the amount of valid data contained in the kth output channel of the current I_parallel group input channel; this will be used as a reference for pairing

[0071] 8:}}

[0072] 9: channel_index = sort (tmp_kernel_cnt); / / Sort tmp_kernel_cnt in ascending order, and obtain the corresponding ascending subscript value

[0073] 10:for(k=1;k<=K;k++){ / / Start from the one with the smallest amount of valid data

[0074] 11:if(tmp_kernel_cnt(k)!=0){ / / All zero channels directly skip calculation

[0075] 12:for(m=K;m>k;m--){ / / Attempt to merge the channel with the largest effective data volume

[0076] 13:tmp_flag = True; / / Flag used to determine whether the two output channels can be fused

[0077] 14:for(n=0;n <I_parallel;n++){

[0078] 15:if(FuseKernel(kernel[1:3][1:3][c+n][channel_index(k)],kernel[1:3][1:3][c+n][channel_index(m)]) == Fail){ / / It is necessary to fuse all I_parallel parallel input channels

[0079] 16:tmp_flag=Fasle;

[0080] 17:}}

[0081] 18:if(tmp_flag == True){

[0082] 19:FuseChannel(channel_index(k),channel_index(m)); / / If the flag is still true, fuse the two channels

[0083] 20:}

[0084] twenty one:}}}

[0085] Figure 4 The present invention is based on Figure 3Schematic diagram of customized sparse convolution operator under weight distribution. First, for the sparse activation value bitmap, the present invention first transforms its bitmap, and then uses a sliding window method for the transformed bitmap to send it to the first non-zero detection unit, and the first non-zero detection unit then sends the relative offset addresses of all non-zero values ​​to the corresponding valid activation value storage unit and the fused convolution kernel weight storage unit in turn. The above storage unit then sends the corresponding valid activation value and weight data to the multiplication unit and the addition unit for multiplication and accumulation operations. The corresponding multiplication and accumulation queue has a data selector and a data splitter to re-decouple the data results of the fused convolution kernel to the corresponding output channel for accumulation and finally output.

[0086] Figure 5 The figure shows the two-dimensional operator parallel design of the configurable sparse convolution hardware acceleration design based on channel fusion proposed by the present invention. The left side shows the effective activation value data output by different input channels, and the upper side shows the different convolution kernel data corresponding to different output channels after fusion. After feeding these data into the two-dimensional sparse convolution operator array, the array sends the obtained data of different output channels to the channel reallocation unit below to re-send the data of the corresponding output channel after fusion to the position of the output channel before fusion.

[0087] Those skilled in the art know that, in addition to implementing the system, device and its various modules provided by the present invention in a purely computer-readable program code, it is entirely possible to implement the same program in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers and embedded microcontrollers by logically programming the method steps. Therefore, the system, device and its various modules provided by the present invention can be considered as a hardware component, and the modules included therein for implementing various programs can also be considered as structures within the hardware component; the modules for implementing various functions can also be considered as both software programs for implementing the method and structures within the hardware component.

[0088] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A configurable sparse convolution hardware acceleration method based on channel fusion, characterized in that: include: Step 1: Use the bitmap sliding window method to obtain the offset addresses of all non-zero valid activation value data and corresponding convolution kernel weight data; Step 2: Store the effective activation value data and the offset address of the corresponding convolution kernel weight data, and perform multiplication and accumulation operations on the corresponding effective value data pairs in sequence; Step 3: Set the data selector and data splitter in the multiplication and accumulation queue to decouple the data results of the fused convolution kernel to the corresponding output channel for accumulation; Step 4: Redistribute the data of different channels, and send the data of the corresponding output channel after fusion back to the position of the output channel before fusion; For the case of uneven sparse distribution of convolution kernels, spatial redundancy is set for the position with the highest frequency of effective value of the hardware operator 3×3 convolution kernel, so that the size of the hardware convolution operator and the corresponding sliding window becomes 3×3+1; For channel fusion: First, the corresponding K groups of output channels are divided into C / C_in_parallel groups of convolution kernels according to the input parallelism C_in_parallel of the operation unit. Each group of convolution kernels contains K groups of output channels and C_in_parallel groups of input channels. Next, for each group of convolution kernels, count and sort the amount of valid data contained in all C_in_parallel group input channels under each group of output channels; Finally, all output channel groups are matched in pairs according to the smallest and largest valid data amounts. The rule for successful matching is that the data at all corresponding positions are not valid values ​​at the same time. All successfully matched channels are merged into one channel, and the channels that are not successfully merged continue to merge with the next channel in order until all remaining unmatched channels are traversed.

2. The configurable sparse convolution hardware acceleration method based on channel fusion according to claim 1, characterized in that: The sparse activation values ​​are stored in a bitmap format. If the final activation value is stored with an n-bit width, the bitmap format stores whether all data are valid values ​​and all truly valid data at a cost of 1 / n.

3. The configurable sparse convolution hardware acceleration method based on channel fusion according to claim 1, characterized in that: For sparse weight convolution kernels, the sparse weight data is channel-fused on the server side to reduce the storage overhead of the weight data, skip invalid zero value calculations, and balance the effective computing amount between different computing units.

4. A configurable sparse convolution hardware acceleration system based on channel fusion, characterized in that: include: Module M1: Use a bitmap sliding window to obtain the offset addresses of all non-zero valid activation value data and corresponding convolution kernel weight data; Module M2: Store the effective activation value data and the offset address of the corresponding convolution kernel weight data, and perform multiplication and accumulation operations on the corresponding effective value data pairs in sequence; Module M3: Set the data selector and data splitter in the multiplication and accumulation queue to decouple the data results of the fused convolution kernel to the corresponding output channel for accumulation; Module M4: reallocates the data of different channels, and sends the data of the corresponding output channel after fusion back to the position of the output channel before fusion; For the case of uneven sparse distribution of convolution kernels, spatial redundancy is set for the position with the highest frequency of effective value of the hardware operator 3×3 convolution kernel, so that the size of the hardware convolution operator and the corresponding sliding window becomes 3×3+1; For channel fusion: First, the corresponding K groups of output channels are divided into C / C_in_parallel groups of convolution kernels according to the input parallelism C_in_parallel of the operation unit. Each group of convolution kernels contains K groups of output channels and C_in_parallel groups of input channels. Next, for each group of convolution kernels, count and sort the amount of valid data contained in all C_in_parallel group input channels under each group of output channels; Finally, all output channel groups are matched in pairs according to the smallest and largest valid data amounts. The rule for successful matching is that the data at all corresponding positions are not valid values ​​at the same time. All successfully matched channels are merged into one channel, and the channels that are not successfully merged continue to merge with the next channel in order until all remaining unmatched channels are traversed.

5. The configurable sparse convolution hardware acceleration system based on channel fusion according to claim 4, characterized in that: The sparse activation values ​​are stored in a bitmap format. If the final activation value is stored with an n-bit width, the bitmap format stores whether all data are valid values ​​and all truly valid data at a cost of 1 / n.

6. The configurable sparse convolution hardware acceleration system based on channel fusion according to claim 4, characterized in that: For sparse weight convolution kernels, the sparse weight data is channel-fused on the server side to reduce the storage overhead of the weight data, skip invalid zero value calculations, and balance the effective computing amount between different computing units.

Images