A compression method, system and acceleration device for convolutional neural network

Abstract

The present invention provides a compression method, system, and acceleration device for a convolutional neural network. The compression method includes grouping the convolution kernels of each output channel in a convolutional layer; using inter-kernel pruning to prune the same number of convolution kernels from each group of convolution kernels; grouping the weights of the unpruned convolution kernels in each group of convolution kernels by row; using intra-kernel pruning to prune the convolution kernels corresponding to the row with the smallest weights from each group of convolution kernels; presetting a pruning threshold in a fully connected layer; pruning the convolution kernels corresponding to weights in the fully connected layer whose absolute values are less than the pruning threshold; quantizing all non-zero weights in the convolution neural network to a target accuracy; and retraining the convolution neural network until convergence. The present invention obtains a regular compression model through block-wise inter-kernel pruning and intra-kernel pruning that can be efficiently calculated by an acceleration device; the weights in the fully connected layer are pruned using a threshold to eliminate as many redundant parameters as possible, so that the network can be compressed to a smaller scale.

Description

Technical Field

[0001] The present invention belongs to the technical field of lightweight deep neural networks, and in particular relates to a compression method, system and acceleration device for convolutional neural networks. Background Art

[0002] The rapid development of convolutional neural networks in object detection and recognition has made it possible to significantly improve image recognition accuracy. However, to achieve even better object detection and recognition performance, convolutional neural networks are becoming increasingly deep, leading to a rapid increase in computational complexity and expansion of model size. Therefore, convolutional neural networks require parallel devices to accelerate them, such as high-power GPUs, to reduce training time or meet the requirements of real-time object detection. To enable the deployment of deep convolutional neural networks on low-power embedded devices, FPGA-based convolutional neural network acceleration design has become a research focus in both academia and industry.

[0003] In recent years, neural network model compression algorithms have emerged in large numbers, among which weight pruning and quantization methods are widely used in model compression. For a convolutional neural network, the vast majority of computation is concentrated in the convolutional layers, while the fully connected layers require less computation. Therefore, convolutional layers are more suitable for acceleration, while fully connected layers are more suitable for compression. However, most current pruning methods apply a single method to convolutional neural networks. Due to the different characteristics of convolutional and fully connected layers, using a single pruning method to compress the network is difficult to strike a balance between network compression and acceleration. For example, threshold pruning can effectively eliminate many parameters in the network, but it also leads to irregularities in the compressed model, resulting in low parallel computing efficiency of the network. Structured pruning is a hardware-friendly pruning method, but it generally does not achieve a high compression rate. Summary of the Invention

[0004] In view of the deficiencies in the prior art, the present invention provides a compression method, system and acceleration device for a convolutional neural network.

[0005] In a first aspect, the present invention provides a method for compressing a convolutional neural network, wherein the convolutional neural network includes a convolutional layer and a convolutional layer, including:

[0006] Group the convolution kernels of each output channel in the convolution layer to obtain multiple groups of convolution kernels;

[0007] Inter-core pruning is used to prune the same number of convolution kernels from each group of convolution kernels;

[0008] Group the weights of the unpruned convolution kernels in each convolution kernel group by rows;

[0009] Use intra-core pruning to prune the convolution kernel corresponding to the row with the smallest weight in each group of convolution kernels;

[0010] Pre-set pruning thresholds in fully connected layers;

[0011] Prune the convolution kernels corresponding to the weights in the fully connected layer whose absolute values are less than the pruning threshold;

[0012] Quantize all non-zero weights in the convolutional neural network to the target accuracy;

[0013] Retrain the convolutional neural network until it converges.

[0014] The inter-core pruning is used to prune the same number of convolution kernels from each group of convolution kernels, including:

[0015] Set the initial pruning rate P0;

[0016] Prune the convolution kernel of the convolution layer to obtain the first convolutional neural network;

[0017] Train the first convolutional neural network to the first target number of times, increase the pruning rate to P1, and obtain the second convolutional neural network;

[0018] Train the second convolutional neural network to the second target number of times, increase the pruning rate to P2, and obtain the third convolutional neural network;

[0019] The third convolutional neural network is trained repeatedly for the first target number of times and the second target number of times until a preset pruning rate P is met, thereby obtaining a fourth convolutional neural network, where P>P2>P1>P0.

[0020] Furthermore, pruning the convolution kernels corresponding to the weights in the fully connected layer whose absolute values are less than the pruning threshold includes:

[0021] Set the initial pruning threshold W0;

[0022] Prune the convolution kernels corresponding to the weights whose absolute values are less than W0 in the fully connected layer of the fourth convolutional neural network to obtain the fifth convolutional neural network;

[0023] Train the fifth convolutional neural network to the third target number of times, increase the pruning threshold to W1, and obtain the sixth convolutional neural network;

[0024] Prune the convolution kernels corresponding to the weights whose absolute values are less than W1 in the fully connected layer of the sixth convolutional neural network to obtain the seventh convolutional neural network;

[0025] The seventh convolutional neural network is trained to the fourth target number of times, and the pruning threshold is increased to W2 to obtain the eighth convolutional neural network;

[0026] The eighth convolutional neural network is repeatedly trained for the third target number of times and the fourth target number of times until a preset pruning threshold W is met, thereby obtaining a ninth convolutional neural network, where W>W2>W1>W0.

[0027] In a second aspect, the present invention provides a compression system for a convolutional neural network, wherein the convolutional neural network includes a convolutional layer and a convolutional layer, including:

[0028] The first grouping module is used to group the convolution kernels of each output channel in the convolution layer to obtain multiple groups of convolution kernels;

[0029] The first pruning module is used to prune the same number of convolution kernels from each group of convolution kernels using inter-kernel pruning;

[0030] The second grouping module is used to group the weights of the unpruned convolution kernels in each convolution kernel group by row;

[0031] The second pruning module is used to prune the convolution kernel corresponding to the row with the smallest weight in each group of convolution kernels by using intra-kernel pruning;

[0032] The pruning threshold setting module is used to pre-set the pruning threshold in the fully connected layer;

[0033] The third pruning module is used to prune the convolution kernels corresponding to the weights in the fully connected layer whose absolute values are less than the pruning threshold;

[0034] The weight quantization module is used to quantize all non-zero weights in the convolutional neural network to the target accuracy;

[0035] The retraining module is used to retrain the convolutional neural network until the convolutional neural network converges.

[0036] Furthermore, the first pruning module includes:

[0037] A first setting unit is used to set an initial pruning rate P0;

[0038] A first pruning unit, used to prune the convolution kernel of the convolution layer to obtain a first convolutional neural network;

[0039] A first training unit is used to train the first convolutional neural network to a first target number of times, increase the pruning rate to P1, and obtain a second convolutional neural network;

[0040] A second training unit is used to train the second convolutional neural network to a second target number of times, increase the pruning rate to P2, and obtain a third convolutional neural network;

[0041] The third training unit is used to repeatedly train the third convolutional neural network for the first target number of times and the second target number of times until a preset pruning rate P is met, thereby obtaining a fourth convolutional neural network, wherein P>P2>P1>P0.

[0042] Furthermore, the third pruning module includes:

[0043] A second setting unit is used to set an initial pruning threshold W0;

[0044] A second pruning unit is used to prune the convolution kernels corresponding to the weights whose absolute values are less than W0 in the fully connected layer of the fourth convolutional neural network to obtain a fifth convolutional neural network;

[0045] A fourth training unit is used to train the fifth convolutional neural network to a third target number of times, increase the pruning threshold to W1, and obtain a sixth convolutional neural network;

[0046] A third pruning unit is used to prune the convolution kernels corresponding to the weights whose absolute values are less than W1 in the fully connected layer of the sixth convolutional neural network to obtain a seventh convolutional neural network;

[0047] a fifth training unit, configured to train the seventh convolutional neural network to a fourth target number of times, increase the pruning threshold to W2, and obtain an eighth convolutional neural network;

[0048] The sixth training unit is used to repeatedly train the eighth convolutional neural network for the third target number of times and the fourth target number of times until a preset pruning threshold W is met, thereby obtaining a ninth convolutional neural network, wherein W>W2>W1>W0.

[0049] In a third aspect, the present invention provides a convolutional neural network acceleration device for accelerating a convolutional neural network compressed by the compression method according to the first aspect, the acceleration device comprising: a plurality of index-weight RAMs, a plurality of feature map RAMs, a plurality of convolution processing modules, a plurality of accumulators, a plurality of post-processing modules, a data reorganization module, and a control module;

[0050] Each convolution processing module includes multiple dual convolution processing units for parallel calculation of multiple input channels to generate the output of one channel;

[0051] The index-weight RAM and feature map RAM provide the data required for calculation; the control module selects the feature map for each dual convolution processing unit according to the index calculated during pruning; the output of each convolution processing module is sent to the corresponding connected accumulator for accumulation, and the intermediate data is stored in the cache, waiting to be accumulated with the next batch of data until the feature maps of all input channels are processed and the feature map data of an output channel is generated; the output feature map data is sent to the post-processing module for pooling or ReLu operation, and the feature map data after pooling or ReLu operation is sent to the data reconstruction module for processing and stored in the feature map RAM as the activation data of the next layer.

[0052] Furthermore, the dual convolution processing unit includes two three-to-two data selectors MUX1 and MUX2, and each three-to-two data selector is connected to a PE unit;

[0053] The PE unit includes two three-tap FIR filters for calculating a two-dimensional convolution; the two PE units respectively calculate two adjacent convolution operations in the vertical direction.

[0054] The present invention provides a compression method, system, and acceleration device for a convolutional neural network, wherein the compression method includes grouping the convolution kernels of each output channel in a convolutional layer to obtain multiple convolution kernel groups; using inter-kernel pruning to prune the same number of convolution kernels from each convolution kernel group; grouping the weights of the unpruned convolution kernels in each convolution kernel group by row; using intra-kernel pruning to prune the convolution kernels corresponding to the row with the smallest weights from each convolution kernel group; presetting a pruning threshold in a fully connected layer; pruning the convolution kernels corresponding to the weights in the fully connected layer whose absolute values are less than the pruning threshold; quantizing all non-zero weights in the convolution neural network to a target accuracy; and retraining the convolution neural network until the convolution neural network converges. The present invention adopts the above scheme, and through block-wise inter-kernel pruning and intra-kernel pruning, a regular compression model is obtained, which can be efficiently calculated by an acceleration device; the weights in the fully connected layer are pruned by threshold to eliminate as many redundant parameters as possible, so that the network can be compressed to a smaller scale. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0056] Figure 1 A schematic diagram of a flow chart of a convolutional neural network compression method provided by an embodiment of the present invention;

[0057] Figure 2 A schematic diagram of the structure of a convolutional neural network compression method provided by an embodiment of the present invention;

[0058] Figure 3 A schematic diagram of pruning a single output channel of a convolutional layer provided in an embodiment of the present invention;

[0059] Figure 4 A schematic diagram of the structure of a convolutional neural network compression system provided by an embodiment of the present invention;

[0060] Figure 5 A schematic diagram of the structure of a convolutional neural network acceleration device provided by an embodiment of the present invention;

[0061] Figure 6 A schematic diagram of the structure of a dual convolution processing unit in a convolutional neural network acceleration device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0062] The following will provide a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0063] As described in the background of this application, in the prior art, most convolutional neural networks use a single pruning method. Due to the different characteristics of convolutional layers and fully connected layers, using a single pruning method to compress the network can make it difficult to strike a balance between network compression and acceleration. For example, threshold pruning can effectively eliminate many parameters in the network, but it can also lead to irregularities in the compression model, resulting in low parallel computing efficiency of the network. Structured pruning is a hardware-friendly pruning method, but it generally cannot achieve a high compression rate.

[0064] Therefore, in order to solve the above problems, the embodiment of the present invention partially provides a compression method of a convolutional neural network, wherein the convolutional neural network includes a convolutional layer and a convolutional layer, see Figure 2 , Figure 2 It is a structural diagram of the compression method of convolutional neural network.

[0065] For details, see Figure 1 , the compression method comprises the following steps:

[0066] Step S101: group the convolution kernels of each output channel in the convolution layer to obtain multiple groups of convolution kernels.

[0067] In step S102, inter-core pruning is performed to prune the same number of convolution kernels from each group of convolution kernels.

[0068] In this step, an iterative pruning method is used to avoid significant accuracy degradation. An initial pruning rate, P0, is set. The convolutional kernels of the convolutional layer are pruned to obtain a first convolutional neural network. The first convolutional neural network is trained to a first target number of iterations, such as 10,000 iterations, and the pruning rate is increased to P1 to obtain a second convolutional neural network. The second convolutional neural network is trained to a second target number of iterations, such as 20,000 iterations, and the pruning rate is increased to P2 to obtain a third convolutional neural network. The third convolutional neural network is trained repeatedly for the first and second target numbers of iterations until the pre-set pruning rate, P, is met, resulting in a fourth convolutional neural network, where P>P2>P1>P0. The shape of each convolutional kernel after pruning is recorded by its corresponding 3-bit kernel-level index. The highest bit indicates whether the kernel has been pruned, with 0 representing pruned and 1 representing unpruned. The lower 2-bit index represents the kernel weight distribution: 00, 01, 10, and 11 represent the four cases in which all weights in the kernel are pruned, the first row of weights are pruned, the second row of weights are pruned, and the third row of weights are pruned, respectively.

[0069] Step S103: Group the weights of the unpruned convolution kernels in each convolution kernel group by row.

[0070] In step S104, intra-core pruning is used to prune the convolution kernels corresponding to the row with the smallest weight in each group of convolution kernels.

[0071] Step S105: pre-set a pruning threshold in the fully connected layer.

[0072] Step S106: prune the convolution kernels corresponding to the weights in the fully connected layer whose absolute values are less than the pruning threshold.

[0073] In this step, an initial pruning threshold W0 is set; the convolution kernels corresponding to the weights whose absolute values are less than W0 in the fully connected layer of the fourth convolutional neural network are pruned; a fifth convolutional neural network is obtained; the fifth convolutional neural network is trained to the third target number of times, and the pruning threshold is increased to W1 to obtain a sixth convolutional neural network; the convolution kernels corresponding to the weights whose absolute values are less than W1 in the fully connected layer of the sixth convolutional neural network are pruned; a seventh convolutional neural network is obtained; the seventh convolutional neural network is trained to the fourth target number of times, and the pruning threshold is increased to W2 to obtain an eighth convolutional neural network; the eighth convolutional neural network is repeatedly trained for the third target number of times and the fourth target number of times until the pre-set pruning threshold W is met, and a ninth convolutional neural network is obtained, wherein W>W2>W1>W0.

[0074] Step S107: quantize all non-zero weights in the convolutional neural network to the target accuracy.

[0075] Step S108: retrain the convolutional neural network until the convolutional neural network converges.

[0076] In steps S107-S108, non-zero weights are quantized from 32-bit floating-point operations to low-precision fixed-point numbers, generally with 16-bit, 8-bit, or 4-bit precision. The weights are quantized to an appropriate precision within the allowable range of network error. The convolutional layer has a greater impact on the calculation results than the fully connected layer. Generally, the weights in the convolutional layer can be quantized to a higher precision, while the weights in the fully connected layer can be quantized to a lower precision. For example, the weights in the convolutional layer can be quantized to 8 bits, and the weights in the fully connected layer can be quantized to 4 bits. Retrain the convolutional neural network until the convolutional neural network converges.

[0077] The following takes the convolutional neural network model VGG-16 as an example to further illustrate the specific implementation process of the present invention:

[0078] like Figure 2 As shown in , the convolutional neural network is pruned. The pruning method in the convolutional layer is as follows Figure 3 As shown in the figure, for each output channel of the convolutional layer, the convolution kernels are divided into blocks. The importance evaluation module evaluates the importance of the convolution kernels in each block and selects the less important convolution kernels for pruning. Intra-core pruning groups the weights of the unpruned convolution kernels by row and prunes the convolution kernels corresponding to the row with the lowest importance. Pruning in fully connected layers is threshold pruning, retaining weights with absolute values greater than the pruning threshold and pruning weights with absolute values less than the pruning threshold.

[0079] After pruning, the weights in the convolutional neural network are quantized, where the weights in the convolutional layer are linearly quantized to 8 bits, and the weights in the fully connected layer are quantized to ±2 n The activations are quantized to 16 bits in the form of 4-bit fixed-point numbers. The accuracy loss caused by pruning and quantization of convolutional neural networks is recovered through retraining.

[0080] like Figure 4 As shown, the embodiment of the present invention partially provides a compression system for a convolutional neural network, including:

[0081] The first grouping module 10 is used to group the convolution kernels of each output channel in the convolution layer to obtain multiple groups of convolution kernels.

[0082] The first pruning module 20 is used to prune the same number of convolution kernels from each group of convolution kernels by using inter-kernel pruning.

[0083] The second grouping module 30 is used to group the weights of the unpruned convolution kernels in each convolution kernel group by rows.

[0084] The second pruning module 40 is used to prune the convolution kernels corresponding to the row with the smallest weight in each group of convolution kernels by using intra-kernel pruning.

[0085] The pruning threshold setting module 50 is used to pre-set the pruning threshold in the fully connected layer.

[0086] The third pruning module 60 is used to prune the convolution kernels corresponding to the weights in the fully connected layer whose absolute values are less than the pruning threshold.

[0087] The weight quantization module 70 is used to quantize all non-zero weights in the convolutional neural network to the target accuracy.

[0088] The retraining module 80 is used to retrain the convolutional neural network until the convolutional neural network converges.

[0089] Optionally, the first pruning module includes:

[0090] A first setting unit is used to set an initial pruning rate P0;

[0091] A first pruning unit, used to prune the convolution kernel of the convolution layer to obtain a first convolutional neural network;

[0092] A first training unit is used to train the first convolutional neural network to a first target number of times, increase the pruning rate to P1, and obtain a second convolutional neural network;

[0093] A second training unit is used to train the second convolutional neural network to a second target number of times, increase the pruning rate to P2, and obtain a third convolutional neural network;

[0094] The third training unit is used to repeatedly train the third convolutional neural network for the first target number of times and the second target number of times until a preset pruning rate P is met, thereby obtaining a fourth convolutional neural network, wherein P>P2>P1>P0.

[0095] Optionally, the third pruning module includes:

[0096] A second setting unit is used to set an initial pruning threshold W0;

[0097] A second pruning unit is used to prune the convolution kernels corresponding to the weights whose absolute values are less than W0 in the fully connected layer of the fourth convolutional neural network to obtain a fifth convolutional neural network;

[0098] A fourth training unit is used to train the fifth convolutional neural network to a third target number of times, increase the pruning threshold to W1, and obtain a sixth convolutional neural network;

[0099] A third pruning unit is used to prune the convolution kernels corresponding to the weights whose absolute values are less than W1 in the fully connected layer of the sixth convolutional neural network to obtain a seventh convolutional neural network;

[0100] a fifth training unit, configured to train the seventh convolutional neural network to a fourth target number of times, increase the pruning threshold to W2, and obtain an eighth convolutional neural network;

[0101] The sixth training unit is used to repeatedly train the eighth convolutional neural network for the third target number of times and the fourth target number of times until a preset pruning threshold W is met, thereby obtaining a ninth convolutional neural network, wherein W>W2>W1>W0.

[0102] like Figure 5 As shown, the embodiment of the present invention partially provides a convolutional neural network acceleration device, which is used to accelerate the convolutional neural network compressed by the compression method. The acceleration device includes: multiple index-weight RAMs, multiple feature map RAMs, multiple convolution processing modules, multiple accumulators, multiple post-processing modules, a data reorganization module, and a control module;

[0103] Each convolution processing module includes multiple dual convolution processing units for parallel calculation of multiple input channels to generate the output of one channel;

[0104] The index-weight RAM and feature map RAM provide the data required for calculation; the control module selects the feature map for each dual convolution processing unit according to the index calculated during pruning; the output of each convolution processing module is sent to the corresponding connected accumulator for accumulation, and the intermediate data is stored in the cache, waiting to be accumulated with the next batch of data until the feature maps of all input channels are processed and the feature map data of an output channel is generated; the output feature map data is sent to the post-processing module for pooling or ReLu operation, and the feature map data after pooling or ReLu operation is sent to the data reconstruction module for processing and stored in the feature map RAM as the activation data of the next layer.

[0105] Taking the convolutional layers of VGG-16 as an example, except for the first layer, the greatest common divisor of the channels in all other layers is 64. Considering the hardware resources and computational efficiency on the FPGA, the value of n can be set to 8, 16, or 32. The value of m is related to the block size during pruning.

[0106] When the calculation of this layer begins, the n index-weight RAMs load the weights and indices into the 32 convolution processing modules through the cache. The data bit width of the index-weight RAM is 52 bits, of which the first 4 bits are the two-level index and the last 48 bits are the 6 non-zero weights in a single convolution kernel. The upper two bits in the index-weight RAM are the first-level index, which is used to represent the distance between two non-zero convolution kernels. The first-level index is sent to the control module to decode the compressed output channel, and the corresponding feature map is selected for each convolution processing unit in the 32 feature map RAMs for calculation. The bit width of m feature map RAMs is 64 bits, which can output 4m feature map data at one time.

[0107] The convolution processing module receives the index, weight and feature map data from the cache for calculation. The number of dual convolution processing units contained in each convolution processing module is equal to the number of convolution kernels remaining in each block after block pruning, and multiple convolution operations can be processed in parallel. Figure 6 As shown in the figure, each dual convolution processing module consists of two PE units and two MUX units. Each PE unit contains two three-tap FIR filters. During a cycle, the dual convolution processing unit receives four rows of activation data, A0-A3. Under the control of the index, MUX1 and MUX2 select appropriate weights for PE1 and PE2. For example, when the index is 01, A1 and A2 are selected by MUX1 and sent to PE1 for calculation. A2 and A3 are selected by MUX2 and sent to PE2 for calculation. PE1 and PE2 each calculate the output feature map data for two adjacent rows.

[0108] The results of the n convolution processing modules are fed into the subsequent accumulator module for addition. The intermediate data is stored in a buffer, awaiting accumulation with the next round of intermediate data to produce the final output feature map. The feature map data output by the accumulator is fed into the subsequent post-processing module for pooling or ReLU operations. The data processed by the post-processing module is then fed into the data reconstruction module, which integrates it into the input feature map format for the next layer and stores it in the feature map RAM, awaiting calculation for the next layer.

[0109] like Figure 6 As shown, the dual convolution processing unit includes two three-to-two data selectors MUX1 and MUX2, and each three-to-two data selector is connected to a PE unit.

[0110] The PE unit includes two three-tap FIR filters for calculating a two-dimensional convolution; the two PE units each calculate two adjacent convolution operations in the vertical direction. When a convolution kernel is calculated with an input feature map, the six non-zero weights W0-W5 in the convolution kernel are input to the PE unit. In each cycle, the four adjacent activations A0-A3 in the vertical direction from the feature map are input to the dual convolution processing unit. MUX1 selects two corresponding activation data from A0-A2 based on the 2-bit index and sends them to the subsequent PE1 for calculation to obtain the output data Y1. The same is true for MUX2, which selects two activation data from A1-A3 based on the index and sends them to PE2 to calculate Y2.

[0111] In this specification, the same or similar parts between the various embodiments can be referred to each other. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the description in the method embodiment.

[0112] The present invention has been described in detail above with reference to specific embodiments and exemplary examples. However, these descriptions should not be construed as limiting the present invention. Those skilled in the art will appreciate that various equivalent substitutions, modifications, or improvements may be made to the technical solutions and implementations of the present invention without departing from the spirit and scope of the present invention, all of which fall within the scope of the present invention. The scope of protection of the present invention shall be determined by the appended claims.

Claims

1. A convolutional neural network acceleration device, characterized in that: Used to accelerate a convolutional neural network compressed by the following compression method, wherein the convolutional neural network includes convolutional layers and convolutional layers, and the compression method of the convolutional neural network includes: Group the convolution kernels of each output channel in the convolution layer to obtain multiple groups of convolution kernels; Inter-core pruning is used to prune the same number of convolution kernels from each group of convolution kernels; Group the weights of the unpruned convolution kernels in each convolution kernel group by rows; Use intra-core pruning to prune the convolution kernel corresponding to the row with the smallest weight in each group of convolution kernels; Pre-set pruning thresholds in fully connected layers; Prune the convolution kernels corresponding to the weights in the fully connected layer whose absolute values are less than the pruning threshold; Quantize all non-zero weights in the convolutional neural network to the target accuracy; Retrain the convolutional neural network until it converges; The inter-core pruning is used to prune the same number of convolution kernels from each group of convolution kernels, including: Set the initial pruning rate P0; Prune the convolution kernel of the convolution layer to obtain the first convolutional neural network; Train the first convolutional neural network to the first target number of times, increase the pruning rate to P1, and obtain the second convolutional neural network; Train the second convolutional neural network to the second target number of times, increase the pruning rate to P2, and obtain the third convolutional neural network; Repeat the training of the third convolutional neural network for the first target number of times and the second target number of times until the preset pruning rate P is met, thereby obtaining a fourth convolutional neural network, where P>P2>P1>P0; The acceleration device includes: multiple index-weight RAMs, multiple feature map RAMs, multiple convolution processing modules, multiple accumulators, multiple post-processing modules, a data reorganization module, and a control module; Each convolution processing module includes multiple dual convolution processing units for parallel calculation of multiple input channels to generate the output of one channel; The index-weight RAM and feature map RAM provide the data required for calculation; the control module selects the feature map for each dual convolution processing unit according to the index calculated during pruning; the output of each convolution processing module is sent to the corresponding connected accumulator for accumulation, and the intermediate data is stored in the cache, waiting to be accumulated with the next batch of data until the feature maps of all input channels are processed and the feature map data of an output channel is generated; the output feature map data is sent to the post-processing module for pooling or ReLu operation, and the feature map data after pooling or ReLu operation is sent to the data reconstruction module for processing and stored in the feature map RAM as the activation data of the next layer.

2. The convolutional neural network acceleration device according to claim 1, characterized in that The step of pruning the convolution kernels corresponding to the weights in the fully connected layer whose absolute values are less than the pruning threshold comprises: Set the initial pruning threshold W0; Prune the convolution kernels corresponding to the weights whose absolute values are less than W0 in the fully connected layer of the fourth convolutional neural network to obtain the fifth convolutional neural network; Train the fifth convolutional neural network to the third target number of times, increase the pruning threshold to W1, and obtain the sixth convolutional neural network; Prune the convolution kernels corresponding to the weights whose absolute values are less than W1 in the fully connected layer of the sixth convolutional neural network to obtain the seventh convolutional neural network; The seventh convolutional neural network is trained to the fourth target number of times, and the pruning threshold is increased to W2 to obtain the eighth convolutional neural network; The eighth convolutional neural network is repeatedly trained for the third target number of times and the fourth target number of times until a preset pruning threshold W is met, thereby obtaining a ninth convolutional neural network, where W>W2>W1>W0.

3. The convolutional neural network acceleration device according to claim 1, characterized in that The dual convolution processing unit includes two three-to-two data selectors MUX1 and MUX2, each of which is connected to a PE unit; The PE unit includes two three-tap FIR filters for calculating a two-dimensional convolution; the two PE units respectively calculate two adjacent convolution operations in the vertical direction.

Images