An accelerator and acceleration method for sparse convolutional neural networks

Abstract

This invention provides an acceleration method for sparse convolutional neural networks, comprising: S1, aggregating and rearranging the non-zero weights of the sparse convolutional kernel according to their importance to obtain a rearranged convolutional kernel; S2, expanding the input block according to the indices of the non-zero weights in the rearranged convolutional kernel to obtain an expanded input block; S3, accumulating the feature map sub-blocks in the expanded input block to obtain an accumulated block, performing a convolution operation with the rearranged convolutional kernel on the accumulated block and taking the average to obtain the convolution result; or, filtering the feature map sub-blocks according to their importance to obtain candidate blocks, performing a convolution operation with the rearranged convolutional kernel on the candidate blocks to obtain the convolution result. This invention significantly reduces redundant computation in sparse convolutional neural network models while improving the network structure.

Description

Technical Field

[0001] This invention relates to convolutional neural network accelerators, and more specifically to an accelerator and acceleration method for sparse convolutional neural networks. Background Technology

[0002] A Convolutional Neural Network (CNN) is a deep neural network with convolutional structures. Unlike ordinary artificial neural networks, CNNs possess local receptive fields and effectively reduce the number of parameters through weight sharing and pooling operations, mitigating overfitting. Furthermore, due to their invariance to transformations such as translation, rotation, and scaling, CNNs demonstrate significant advantages and potential in tasks such as object recognition and classification, and are therefore widely used in image processing, speech recognition, autonomous driving, and natural language processing.

[0003] With the widespread application of CNNs, customizing hardware accelerators to speed up their inference process has become a key research focus. However, as CNNs are used to perform increasingly complex tasks, their network layer count and size are growing rapidly, resulting in significant storage and computational overhead for convolutional neural network accelerators. Against this backdrop, neural network pruning techniques remove a large number of redundant parameters from the network model to obtain sparse CNN models, achieving a substantial reduction in parameter and computational costs without significantly impacting neural network performance. Therefore, customizing hardware accelerators for sparse CNNs and leveraging their sparsity to improve accelerator performance has become one of the mainstream directions in current CNN acceleration research.

[0004] Based on the sparsity of CNNs, sparse CNN accelerators can be divided into those for structured sparse CNNs and those for unstructured sparse CNNs. Since structured sparse neural network models have regular network structures, hardware accelerators for structured sparse CNNs often have simple and regular hardware architectures, such as the MPBF accelerator proposed by Harbin Institute of Technology and the hardware accelerator with a VGM module proposed by Zhejiang University. Unstructured sparse CNNs, due to their flexible pruning methods, can achieve higher sparsity while maintaining the same model performance. Therefore, many researchers seek higher acceleration performance by customizing hardware accelerators for unstructured sparse CNNs. Related work includes the STICKER processor proposed by Tsinghua University and the APA accelerator jointly proposed by the University of Zaragoza and others.

[0005] Existing hardware accelerators still have the following problems:

[0006] 1. Accelerators designed for structured sparsity have relatively simple and regular hardware structures. However, their performance is limited by the low sparsity caused by the structured sparsity.

[0007] 2. While accelerators designed for unstructured sparsity can take advantage of their high sparsity, the resulting irregular network structure can negatively impact accelerator performance.

[0008] 3. Existing research usually focuses on optimizing the operations within the convolutional layer, neglecting to reduce redundant operations in the convolutional layer from the perspective of combining the convolutional layer with the pooling layer. Summary of the Invention

[0009] To address the problems of the prior art, this invention provides an accelerator and acceleration method for sparse convolutional neural networks, which can improve the network structure while significantly reducing redundant computations in the model.

[0010] This invention is achieved through the following technical solution:

[0011] An acceleration method for sparse convolutional neural networks includes:

[0012] S1. Based on the importance of the non-zero weights in the sparse convolution kernel in the sparse convolutional neural network, the non-zero weights in the sparse convolution kernel are aggregated and rearranged to obtain the rearranged convolution kernel.

[0013] S2, expand the input block according to the index of the non-zero weight in the rearranged convolution kernel to obtain the expanded input block;

[0014] S3, accumulate the feature map sub-blocks in the expanded input block to obtain the accumulated block, perform convolution operation with the rearranged convolution kernel and take the average value to obtain the convolution result; or, filter the feature map sub-blocks according to the importance of the feature map sub-blocks in the expanded input block to obtain the candidate block, perform convolution operation with the rearranged convolution kernel to obtain the convolution result.

[0015] Preferably, S1 specifically involves finding the input sub-block corresponding to the non-zero weights based on the indices of the non-zero weights in the sparse convolution kernel, specifically according to formula (1):

[0016]

[0017] Where i and j represent the i-th weight in the sparse convolution kernel and the j-th pixel in the input sub-block corresponding to the i-th weight, respectively; IC i and IR i Then, r represents the column index and row index of the i-th weight in the sparse convolution kernel, respectively; p Indicates the pooling size; r o Indicates the sparse convolution kernel size;

[0018] Then, the pixel values ​​of the input sub-blocks are accumulated, and the importance of the non-zero weights is calculated using the accumulated result and the weight values ​​of the non-zero weights. The pixel value of the highest importance r is then selected. r 2 Each non-zero weight is sequentially filled into the rearranged convolution kernel; where r r This represents the size of the rearranged convolution kernel.

[0019] Preferably, in S2, the input block is expanded according to the index of the non-zero weight in the rearranged convolution kernel to obtain an expanded input block. The pixel value addresses in the expanded input block are specifically determined according to formula (2):

[0020]

[0021] Where i and j represent the i-th weight in the sparse convolution kernel and the j-th pixel in the input sub-block corresponding to the i-th weight, respectively; r p Indicates the pooling size; r r This represents the size of the rearranged convolution kernel.

[0022] Preferably, in S3, the feature map sub-blocks in the expanded input block are accumulated to obtain the accumulated block, specifically according to formula (3):

[0023]

[0024] Among them, SF n (x,y) represents the pixel value at point (x,y) in the nth feature map sub-block; Acc(x,y) represents the pixel value at point (x,y) in the accumulation block; r p Indicates the pooling size; n = 1, 2, 3, ..., r p 2 .

[0025] Furthermore, in S3, the accumulated block is convolved with the rearranged convolution kernel, and the average value is taken to obtain the convolution result, specifically according to formula (4):

[0026]

[0027] Where Filter(x,y) represents the rearranged convolution kernel, Conv(x,y) represents the pixel value at point (x,y) in the convolution result, Acc(x,y) represents the pixel value at point (x,y) in the accumulated block, and r p Indicates the size of the pool.

[0028] Preferably, in S3, the feature map sub-blocks in the expanded input block are screened according to their importance to obtain candidate blocks. Specifically, the importance of each feature map sub-block in the expanded input block is evaluated based on the pixel values ​​of the feature map sub-blocks and the weights of the corresponding rearranged convolution kernels to obtain the importance of each feature map sub-block. The importance evaluation is specifically performed according to formula (5):

[0029]

[0030] Among them, Im k The importance of the k-th feature map sub-block; Filter(x,y) Quantified The quantized weight values; SF k (x,y) represents the pixel value in the k-th feature map sub-block; InC represents the number of input channels; r p Indicates the pooling size; r r This refers to the rearranged convolution kernel size;

[0031] Then, the feature map sub-blocks are sorted according to their importance, and the feature map sub-block with the highest importance is selected as the candidate block.

[0032] An accelerator for sparse convolutional neural networks, comprising:

[0033] The kernel rearrangement module is used to aggregate and rearrange the non-zero weights in the sparse convolutional kernel according to their importance in the sparse convolutional neural network, to obtain rearranged convolutional kernels; and to expand the input block according to the index of the non-zero weights in the rearranged convolutional kernels, to obtain expanded input blocks.

[0034] The convolutional pooling module is used to accumulate feature map sub-blocks in the expanded input block to obtain an accumulated block, or to filter feature map sub-blocks according to their importance to obtain candidate blocks;

[0035] The multiplication module is used to perform convolution operations on the accumulated block and the rearranged convolution kernel and take the average value to obtain the convolution result, or to perform convolution operations on the candidate block and the rearranged convolution kernel to obtain the convolution result.

[0036] Preferably, the convolution kernel rearrangement module includes:

[0037] The weight importance evaluation module is used to evaluate the importance of the non-zero weights of the sparse convolutional kernel in a sparse convolutional neural network.

[0038] The weight sorting module is used to sort non-zero weights according to their importance, placing the highest-important weights (r) as the top-ranked weights. r 2 The weights are sequentially filled into the rearranged convolution kernel to obtain the rearranged convolution kernel; where r rThis refers to the rearranged convolution kernel size;

[0039] The input block expansion module is used to fill the expanded input block with the input sub-block corresponding to each weight in the rearranged convolution kernel, thus obtaining the expanded input block.

[0040] Furthermore, the weight importance evaluation module includes a bit separation module, an address generator, a non-zero detection module, a first adder tree, and a shifter;

[0041] The bit separation module is used to obtain the weight index and weight value based on the non-zero weights;

[0042] The address generator is used to calculate the address of the input sub-block corresponding to the non-zero weight based on the weight index, and to obtain the input sub-block based on the address of the input sub-block;

[0043] The first addition tree is used to accumulate the input sub-blocks to obtain the cumulative sum;

[0044] The non-zero detection module is used to perform accumulation and quantization;

[0045] A shifter is used to calculate the importance of non-zero weights by combining the quantized sum with the weight values.

[0046] Preferably, the convolutional pooling combined module includes:

[0047] Importance assessment module: This module assesses the importance of each feature map sub-block in the expanded input block based on the pixel values ​​of the feature map sub-blocks and the weights of the corresponding rearranged convolutional kernels, thus obtaining the importance of the feature map sub-blocks.

[0048] Partial summation module: When a sparse convolutional neural network contains multiple input channels, it sums the importance of feature map sub-blocks of all input channels to obtain the final importance of the feature map sub-blocks.

[0049] The block predictor sorts feature map sub-blocks according to their importance and selects the feature map sub-block with the highest importance as the candidate block.

[0050] Alternatively, convolutional pooling combined with modules may include:

[0051] The second addition tree is used to accumulate the pixel values ​​at the same position in different feature map sub-blocks in the expanded input block to obtain an accumulation block.

[0052] Compared with the prior art, the present invention has the following beneficial effects:

[0053] This invention provides an acceleration method for sparse convolutional neural networks. It rearranges the convolutional kernels in unstructured sparse models and removes relatively less important weights, resulting in rearranged convolutional kernels of uniform size. This combines the advantages of structured and unstructured sparsity, improving network structure while fully utilizing the high sparsity of unstructured sparsity, thus significantly reducing redundant computation in sparse convolutional neural network models. Furthermore, based on the characteristics of convolution and pooling operations, this invention merges convolutional and pooling layer operations, further reducing ineffective computations in convolutional layers by leveraging the data extraction characteristics of pooling layers. In the convolution-pooling combination scheme for average pooling in this invention, unlike traditional algorithms, the feature map sub-blocks in the expanded input block that need to be convolved with the rearranged convolutional kernel are first accumulated to obtain an accumulated block. Then, the accumulated block is convolved with the rearranged convolutional kernel. By exchanging the order of multiplication and accumulation operations, the rearranged convolutional kernel is extracted as a common factor, thereby reducing the amount of multiplication operations in the convolution process.

[0054] This invention constructs an accelerator for sparse CNNs. A convolutional kernel rearrangement module rearranges convolutional kernels, combining the advantages of structured and unstructured sparsity. This significantly reduces redundant computations in sparse convolutional neural network models while improving network structure. A convolution-pooling combination module merges convolutional and pooling layer operations, further reducing ineffective computations in convolutional layers by leveraging the data extraction characteristics of pooling layers. In this invention's convolution-pooling combination scheme for average pooling, the feature map sub-blocks in the expanded input block that need to be convolved with the rearranged convolutional kernels are first accumulated to obtain an accumulated block. Then, the accumulated block is convolved with the rearranged convolutional kernels. By exchanging the order of multiplication and accumulation operations, the rearranged convolutional kernels are extracted as common factors, thereby reducing the amount of multiplication operations during convolution. This invention can eliminate redundant computations in sparse CNNs. Attached Figure Description

[0055] Figure 1 This method combines kernel rearrangement and convolutional pooling.

[0056] Figure 2 Examples of kernel rearrangement for sparse convolutional neural networks: (a) weight importance evaluation methods; (b) weight sorting and rearrangement process; (c) input block expansion process;

[0057] Figure 3 A convolutional pooling combination scheme oriented towards average pooling;

[0058] Figure 4 A convolutional pooling scheme for max pooling is proposed: (a) a method for evaluating the importance of feature map sub-blocks; (b) sorting and filtering of feature map sub-blocks.

[0059] Figure 5For high-performance hardware architectures for sparse CNNs;

[0060] Figure 6 Hardware implementation of convolution kernel rearrangement;

[0061] Figure 7 This is a hardware implementation of convolutional pooling. Detailed Implementation

[0062] To further understand the present invention, the present invention will be described below with reference to embodiments. These descriptions are only for further explaining the features and advantages of the present invention and are not intended to limit the claims of the present invention.

[0063] This invention proposes an acceleration method for sparse convolutional neural networks, such as... Figure 1 .

[0064] Figure 1 This explanation uses the convolutional kernels and input blocks typical of sparse CNNs. For ease of hardware processing, the original input feature map needs to be divided into blocks to obtain input blocks. First, the convolutional kernels (sparse kernels, original kernels) in the sparse CNN are rearranged, that is, the non-zero weights in the sparse kernels are aggregated and rearranged to obtain dense rearranged kernels. If the number of non-zero weights exceeds the capacity of the rearranged kernels, some of the less important non-zero weights need to be removed. Simultaneously, the input blocks are expanded according to the indices of the non-zero weights in the rearranged kernels to obtain expanded input blocks. Then, the feature map sub-blocks in the expanded input blocks are accumulated or filtered using a convolutional pooling algorithm to obtain accumulated blocks (candidate blocks). Finally, the accumulated blocks (candidate blocks) are convolved with the rearranged kernels.

[0065] The specific process of convolution kernel rearrangement and input block expansion proposed in this invention is as follows: Figure 2 .

[0066] Before rearranging convolutional kernels, the importance of non-zero weights in the kernels needs to be evaluated. Therefore, as... Figure 2 As shown in (a), in this invention, the importance of each non-zero weight is first evaluated using the weights of the non-zero weights and the pixel values ​​of their corresponding input sub-blocks. Taking the first non-zero weight "2" as an example, according to its index (0, 0), its corresponding input sub-block (containing four pixel values ​​"2", "1", "0" and "4") can be found. The addresses of the pixel values ​​in the input sub-block are shown in formula (1), where i and j represent the i-th weight in the sparse convolution kernel and the j-th pixel in the input sub-block corresponding to the i-th weight, respectively; IC i and IR i Then, r represents the column index and row index of the i-th weight in the sparse convolution kernel, respectively; p Indicates the pooling size; ro This represents the sparse convolution kernel size. Then, based on the sum of the four pixel values ​​and the weight value, its importance is calculated to be "8" (for ease of hardware calculation, the pixel values ​​of the input sub-blocks are uniformly quantized to powers of 2 in this invention). Subsequently, in Figure 2 In (b), the non-zero weights are sorted from highest to lowest importance, and the weights with the highest importance r are selected. r 2 (r r (The weights are arranged to be the size of the rearranged convolution kernel) and then sequentially filled into the rearranged convolution kernel. At the same time, the input sub-blocks corresponding to the above weights are passed to the expanded input block. Figure 2 (c)). In this process, since weight "3" is the most important, it is sent to the rearranged convolution kernel first, and its corresponding input sub-block is also passed to the corresponding position in the expanded input block. The process will continue in a similar manner until the rearranged convolution kernel is filled. The pixel value addresses in the expanded input block are shown in Equation (2).

[0067]

[0068]

[0069] The convolutional pooling combination algorithm proposed in this invention has two types: one is a convolutional pooling combination scheme oriented towards average pooling, such as... Figure 3 .

[0070] Average pooling requires averaging the convolution outputs of the convolution kernel and the feature map blocks within the same pooling window. Unlike traditional algorithms that first perform dot multiplication and accumulation between the convolution kernel and the feature map blocks to obtain the convolution result, and then average the convolution result, this invention first accumulates the feature map sub-blocks in the expanded input block that need to be convolved with the rearranged convolution kernel to obtain the accumulated block, and then performs convolution operation with the rearranged convolution kernel. By changing the order of multiplication and accumulation operations, the rearranged convolution kernel is extracted as a common factor, thereby reducing the amount of multiplication operations in the convolution process. The calculation of the accumulated block and the convolution result is shown in formulas (3) and (4). Wherein, SF n (x,y), Acc(x,y), Filter(x,y), and Conv(x,y) represent the pixel values ​​at the (x,y)th point (x and y represent the horizontal and vertical coordinates) in the nth feature map sub-block, the accumulated block, the rearranged convolution kernel, and the convolution result, respectively, where n = 1, 2, 3, ..., r. p 2 ;by Figure 3For example, the pixel values ​​"2", "2", "0" and "31" in the top left corner of the four feature map sub-blocks are summed to get 35. Using the above summation method, the pixel values ​​of the other 8 pixels in the summation block can be obtained. Then, the summation block is multiplied and summed with the rearranged convolution kernel, and finally divided by 4 to get the average pooling result.

[0071]

[0072]

[0073] Another convolutional pooling combination algorithm proposed in this invention is a convolutional pooling combination strategy based on max pooling, such as... Figure 4 .

[0074] The convolutional pooling strategy based on max pooling can be divided into two steps: feature map sub-block importance evaluation and feature map sub-block sorting and filtering. First, in order to predict the maximum convolution output within the pooling window, the feature map sub-blocks that produce convolution outputs within the pooling window are evaluated for importance based on their pixel values ​​and the weights of the rearranged convolutional kernels. The importance of a feature map sub-block is determined by the importance of each pixel value, which is derived from the pixel value and the weight of its corresponding weight. To facilitate hardware computation, the weights in the rearranged convolutional kernels are quantized to powers of 2. k The importance of the k-th feature map sub-block; Filter(x,y) Quantified The quantized weight values; SF k (x,y) represents the pixel value in the k-th feature map sub-block; InC represents the number of input channels; r p Indicates the pooling size; r r This refers to the size of the rearranged convolutional kernel. It's important to note that when a CNN contains multiple input channels, the importance of the corresponding feature map sub-blocks for each input channel needs to be summed to obtain the final importance. Subsequently, the importance of each feature map sub-block will be ranked, and the block with the highest importance will be used as a candidate block for convolution with the rearranged convolutional kernel. Figure 4For example, firstly, the importance of the four feature map sub-blocks in the expanded input block is evaluated based on the weight values ​​in the rearranged convolution kernel and the pixel values ​​within the feature map sub-blocks. The importance of one feature map sub-block in the image is calculated to be 85, while the importance of the other three feature map sub-blocks are 81, 45, and 129, respectively. Then, the calculated importance is sorted, revealing that the feature map sub-block with the highest importance of 129 (the lower right corner of the expanded input block) is followed by the feature map sub-blocks with the weight values ​​in the rearranged convolution kernel, thus obtaining the max pooling result.

[0075]

[0076] The high-performance hardware accelerator implemented in this invention, such as... Figure 5 .

[0077] The accelerator mainly consists of five parts:

[0078] 1. Controller: Provides configuration information based on the operations of the current layer.

[0079] 2. Storage System: This system comprises two parts: external memory and on-chip memory. External memory stores activations and weights that cannot be stored on-chip. On-chip memory is divided into an input data buffer, a weight buffer, and a rearranged convolutional kernel data register / dilated input block buffer. The input data buffer and weight buffer store the feature maps and sparse CNN model loaded onto the chip, while the rearranged convolutional kernel / dilated input block buffer stores the rearranged convolutional kernels and dilated input blocks generated before the candidate block number calculation is completed.

[0080] 3. Convolution kernel rearrangement module: used to implement Figure 2 The convolution kernel rearrangement algorithm.

[0081] 4. Convolutional pooling combined module: used to implement Figure 3 , Figure 4 The convolutional pooling combined algorithm.

[0082] 5. Computation Module: This module consists of three parts: a PE array, a partial and cumulative module, and a full cumulative module. This module is primarily used for convolution calculations.

[0083] The convolution kernel rearrangement module implemented in this invention, such as Figure 6 .

[0084] This module can be divided into the following three modules:

[0085] 1. Weight Importance Evaluation Module: This module includes a bit separation module, an address generator, a non-zero detection module, a first adder tree, and a shifter. This module is used to implement... Figure 2 Weight importance evaluation in (a). Non-zero weights retrieved from the convolution kernel data register are processed by a bit separation module to obtain weight indices and weight values. The weight indices are used by an address generator to calculate the address of the corresponding input sub-block, thereby retrieving the input sub-block from input block register 2. The input sub-block is then processed by a first addition tree to obtain an accumulated sum. The accumulated sum, quantized by the non-zero detection module, is combined with the weight values ​​and processed by a shifter to calculate the weight importance.

[0086] 2. Weighted sorting module, this module is used to implement... Figure 2 (b) Weight sorting. The weight sorting module sorts non-zero weights according to the weight importance output by the weight importance evaluation module. This module is actually a hardware implementation of the parallel full sorting algorithm. This algorithm can complete the data sorting function within 3 clock cycles.

[0087] 3. Input block expansion module, this module is used to implement Figure 2 (c) Input block expansion. In this module, the address of the pixel value in the input sub-block corresponding to each weight in the rearranged convolution kernel is first calculated by the address generator in the expanded input block. Then, the input sub-block obtained from the weight importance evaluation module is sent to the expanded input block register according to the address. At this point, the convolution kernel rearrangement and input block expansion process is completed.

[0088] The convolutional pooling combined module implemented in this invention, such as Figure 7 .

[0089] This module consists of three parts.

[0090] 1. Importance Assessment Module: Used to implement... Figure 4(a) The importance of each feature map sub-block (single channel) is evaluated. This consists of three parts: a non-zero detector, a shift array, and a second addition tree. The non-zero detector quantizes the weights in the rearranged convolutional kernel to powers of 2, thus enabling the subsequent shifter to perform the importance evaluation step (for max pooling). This module takes the rearranged convolutional kernel in the weight buffer as input, finds the highest non-zero bit in each weight of the rearranged convolutional kernel through a priority 8-3 encoder, and outputs it as the quantization result of the weights. The shift array performs pixel value importance calculation. Based on the quantization result of the weights, it shifts the pixel values ​​in the feature map sub-block from the feature map buffer to obtain the importance of each pixel value. Then, the pixel value importances are accumulated through the second addition tree to obtain the importance of the feature map sub-block. The function of the second addition tree can be divided into two parts. For average pooling, the second addition tree can accumulate the pixel values ​​at the same position in different feature map sub-blocks in the pooling window to obtain the accumulation block. For max pooling, the second addition tree accumulates the importance of individual feature map sub-blocks in the feature map obtained by the shift array to calculate the importance of the feature map sub-block.

[0091] 2. Partial Sum Accumulation Module: This module is used when the CNN has multiple input channels. It needs to accumulate the importance of feature map sub-blocks from all input channels to obtain the final importance. Therefore, it requires storing the importance of some channel feature map sub-blocks before accumulation. This module includes an adder and a partial sum register.

[0092] 3. Block predictor, used to implement Figure 4 (b) shows the importance ranking of feature map sub-blocks. This block predictor is a hardware implementation of a parallel full ranking algorithm, capable of ranking multiple data within 3 clock cycles. The module takes the importance of feature map sub-blocks as input and outputs the addresses of candidate blocks after ranking.

[0093] The performance comparison between the present invention and existing methods is as follows:

[0094] exist Figures 5 to 7 The structural differences between the proposed convolutional kernel rearrangement method and existing sparse convolutional neural network acceleration methods are illustrated in Table 1 below.

[0095] Table 1. Performance comparison between the acceleration method of the present invention and existing sparse convolutional neural network acceleration methods.

[0096]

[0097] As can be seen from the above comparison, the high-performance hardware accelerator architecture implemented in this invention can achieve a performance improvement of up to 15.32 times and a DSP efficiency improvement of 7.73 times.

Claims

1. An accelerator for sparse convolutional neural networks, characterized in that, The application relates to a sparse convolutional neural network (CNN) accelerator, which comprises the following parts: a convolution kernel rearrangement module, which is used for gathering and rearranging non-zero weights in a sparse convolution kernel in a sparse CNN according to the importance of the non-zero weights, so as to obtain a rearranged convolution kernel; and expanding an input block according to the indexes of the non-zero weights in the rearranged convolution kernel, so as to obtain an expanded input block; a convolution pooling combination module, which is used for accumulating feature map subblocks in the expanded input block to obtain an accumulated block, or screening the feature map subblocks according to the importance of the feature map subblocks to obtain a candidate block; a multiplication operation module, which comprises a PE array, a partial sum accumulation module and a full accumulation module, and is used for performing convolution operation on the accumulated block and the rearranged convolution kernel and taking an average value to obtain a convolution result, or performing convolution operation on the candidate block and the rearranged convolution kernel to obtain a convolution result; a controller, which provides configuration information according to the operation of a current layer; a storage system, which comprises an external storage and an on-chip memory, wherein the external storage is used for storing activations and weights that cannot be borne by the on-chip memory; the on-chip memory is divided into an input data buffer, a weight buffer and a rearranged convolution kernel data register / expanded input block buffer; the input data buffer and the weight buffer are used for storing feature maps and a sparse CNN model loaded to the on-chip memory; and the rearranged convolution kernel / expanded input block buffer is used for storing the rearranged convolution kernel and the expanded input block generated before the candidate block number calculation is completed. The convolution kernel rearrangement module comprises: a weight importance evaluation module, which is used for evaluating the importance of the non-zero weights in the sparse convolution kernel in the sparse CNN; The weight sorting module is configured to sort the non-zero weights according to the importance of the non-zero weights, and sequentially fill the weights with the highest importance into the rearranged convolution kernel to obtain the rearranged convolution kernel. The size of the rearranged convolution kernel is denoted as (M, N). The size of the rearranged convolution kernel is denoted as (M, N). an input block expansion module, which is used for filling an input subblock corresponding to each weight in the rearranged convolution kernel into an expanded input block to obtain the expanded input block. The weight importance evaluation module comprises a bit separation module, an address generator, a non-zero detection module, a first addition tree and a shifter. The bit separation module is used for obtaining a weight index and a weight value according to a non-zero weight. The address generator is used for calculating an address of an input subblock corresponding to the non-zero weight according to the weight index, and obtaining the input subblock according to the address of the input subblock. The first addition tree is used for accumulating the input subblock to obtain an accumulated sum. The non-zero detection module is used for quantizing the accumulated sum. The shifter is used for calculating the importance of the non-zero weight by using the quantized accumulated sum and the weight value.

2. The accelerator for sparse convolutional neural networks of claim 1, wherein, The convolution pooling combination module comprises: an importance evaluation module, which is used for evaluating the importance of each feature map subblock in the expanded input block according to pixel values of the feature map subblocks and weights of the rearranged convolution kernel corresponding to the feature map subblocks, so as to obtain the importance of the feature map subblocks; a partial sum accumulation module, which is used for accumulating the importance of the feature map subblocks of all input channels when the sparse CNN comprises multiple input channels, so as to obtain the final importance of the feature map subblocks; a block predictor, which is used for sorting the feature map subblocks according to the importance, and taking the feature map subblock with the highest importance as a candidate block. Or, the convolution pooling combination module comprises: a second addition tree, which is used for accumulating pixel values of different feature map subblocks at the same position in the expanded input block to obtain an accumulated block.

3. An acceleration method for a sparse convolutional neural network, characterized in that, The sparse convolutional neural network oriented accelerator based on claim 1 comprises: S1, the convolution kernel rearrangement module rearranges the non-zero weights in the sparse convolution kernel according to the importance of the non-zero weights in the sparse convolution kernel in the sparse convolutional neural network, and obtains a rearranged convolution kernel by clustering and rearranging the non-zero weights in the sparse convolution kernel; S2, the convolution kernel rearrangement module expands the input block according to the index of the non-zero weights in the rearranged convolution kernel, and obtains an expanded input block; S3, the convolution pooling combination module accumulates the feature map subblocks in the expanded input block to obtain an accumulated block, and the multiplication operation module performs convolution operation on the accumulated block and the rearranged convolution kernel and takes the average value to obtain a convolution result; or, the convolution pooling combination module filters the feature map subblocks according to the importance of the feature map subblocks in the expanded input block to obtain a candidate block, and the multiplication operation module performs convolution operation on the candidate block and the rearranged convolution kernel to obtain a convolution result.

4. The acceleration method for sparse convolutional neural network according to claim 3, characterized in that, S1 is specifically: according to the index of the non-zero weights in the sparse convolution kernel, the input subblock corresponding to the non-zero weights is found, and the formula (1) is specifically used: (1) wherein, respectively represent the first weight in the sparse convolution kernel and the first weight in the sparse convolution kernel corresponding to the first pixel in the input sub-block; and respectively represent the column index and the row index of the first weight in the sparse convolution kernel; represents the scale of the pooling; represents the sparse convolution kernel size; Then, the pixel values of the input sub-block are accumulated, and the importance of the non-zero weights is calculated by using the accumulated result in combination with the weight values of the non-zero weights. The non-zero weights with the highest importance are sequentially filled into the rearranged convolution kernel. , wherein, is the size of the rearranged convolution kernel.

5. The acceleration method for sparse convolutional neural network according to claim 3, wherein, In S2, the input block is expanded according to the index of the non-zero weights in the rearranged convolution kernel to obtain an expanded input block, and the pixel value address in the expanded input block is specifically according to formula (2): (2) wherein, respectively represent the first weight in the sparse convolution kernel and the first weight in the sparse convolution kernel correspond to the first pixel in the input sub-block; denotes the size of the pooling; is the rearranged convolution kernel size.

6. The acceleration method for sparse convolutional neural network according to claim 3, wherein, In S3, the feature map subblocks in the expanded input block are accumulated to obtain an accumulated block, which is specifically according to formula (3): (3) wherein, represents the pixel value of the point in the n-th feature map sub-block; represents the pixel value of the point in the accumulated block; represents the scale of the pooling; n = 1, 2, 3, …, .

7. The sparse convolutional neural network oriented acceleration method according to claim 6, characterized in that, In S3, the accumulated block and the rearranged convolution kernel are convolved and the average value is taken to obtain a convolution result, which is specifically according to formula (4): (4) wherein, represents a rearranged convolution kernel, represents a pixel value of the point in the convolution result; represents a pixel value of the point in the accumulation block; represents a scale of the pooling.

8. The acceleration method for sparse convolutional neural network according to claim 3, wherein, In S3, the feature map subblocks in the expanded input block are filtered according to the importance of the feature map subblocks to obtain a candidate block, which is specifically: the importance of each feature map subblock is evaluated according to the pixel value of the feature map subblock and the weight of the corresponding rearranged convolution kernel, and the importance of each feature map subblock is obtained; The importance evaluation is specifically according to formula (5): (5) wherein, is the importance of the kth feature map sub-block, ; is the quantized weight value; is the pixel value in the kth feature map sub-block; is the number of input channels; denotes the scale of the pooling; is the rearranged convolution kernel size; Then, each feature map subblock is sorted according to the importance, and the feature map subblock with the highest importance is taken as the candidate block.

Images