FPGA-based convolutional neural network block pruning method

Abstract

This invention discloses a block pruning method for convolutional neural networks based on FPGA, comprising the following steps: training the network or using a pre-trained network and recording the network accuracy; grouping continuous weights into blocks of a predetermined size and determining the average value of each block; iteratively pruning from the block with the lowest average value according to a pruning percentage p; recording the accuracy of the pruned network, and when the difference between the accuracy of the original network and the accuracy of the pruned network is less than a threshold, iteratively increasing the pruning percentage p and continuing pruning; after determining a suitable pruning percentage p, performing 8-bit quantization on the data; fine-tuning the network, outputting the network parameters, and performing image recognition through the neural network. The method of this invention has a simple calculation process and high inference performance.

Description

Technical Field

[0001] This invention belongs to the field of deep neural network compression technology, and specifically relates to a block pruning method for convolutional neural networks based on FPGA. Background Technology

[0002] Convolutional Neural Networks (CNNs) are deep neural networks used for image classification and object recognition. In recent years, with the continuous development of CNNs and the gradual expansion of their application areas, more and more people have begun to focus on research into CNN-related technologies. However, while CNNs are becoming increasingly powerful, their number of parameters and computational cost are also increasing dramatically. Reducing the number of parameters and computational cost without compromising network efficiency is a key research focus. At the same time, running network models on edge devices close to the data source is also attracting considerable attention. This is because it avoids the high-latency data communication required in cloud processing and allows for real-time decision-making at the edge.

[0003] Pruning is a well-known model optimization method. Neural network pruning first filters out unimportant neurons and weights from a large network, then removes them from the network while preserving network performance as much as possible. This method is very effective, but it introduces sparsity into the weight matrix, which reduces the computational efficiency of regular pipelined structures that compute data paths.

[0004] Deploying CNNs in edge computing platforms follows two main research directions: model optimization and novel computational architectures and platforms. Model optimization can proportionally reduce the computational complexity and storage requirements of the target computing platform; on the other hand, dedicated architectures can improve the performance, energy efficiency, and cost of the computing platform. In both dimensions, most previous research has not followed a co-design approach. Model optimization does not consider the target platform; the computational architecture is designed for a specific model.

[0005] The problem with existing technologies is that conventional pruning methods reduce computational efficiency, and model optimization and computational architecture design are not synergistic. Summary of the Invention

[0006] The purpose of this invention is to provide a block pruning method for convolutional neural networks based on FPGA, which overcomes the sparsity problem caused by conventional pruning methods, improves computational efficiency, and enables high-performance inference of convolutional neural networks based on FPGA.

[0007] The technical solution to achieve the purpose of this invention is as follows:

[0008] A block pruning method for convolutional neural networks based on FPGA, comprising the following steps:

[0009] S10, perform network training or use a pre-trained network, and record the network accuracy;

[0010] S20, group the continuous weights into blocks of a predetermined size and determine the average value of each block;

[0011] S30, iteratively prune from the block with the lowest average value according to the pruning percentage p;

[0012] S40, record the precision of the pruned network. When the difference between the precision of the original network and the precision of the pruned network is lower than the threshold, iteratively increase the pruning percentage p and continue pruning.

[0013] S50, after determining the appropriate pruning percentage p, perform 8-bit quantization on the data;

[0014] S60, after fine-tuning the network, outputs the network parameters and performs image recognition through the neural network.

[0015] In step S40, after pruning according to the pruning percentage p, the accuracy of the pruned network is compared with the accuracy of the original network. If the interpolation between the two is lower than a preset threshold, the pruning percentage is adjusted, and then pruning and training continue until the expected goal is achieved.

[0016] In step S60, the output network parameters are stored as a sparse vector, where each position contains a weight block and an index of the next block position. Each index has four bits and represents the offset relative to the previous index.

[0017] The FPGA architecture described above is used to accelerate the operation of convolutional neural networks. Based on the different characteristics of different types of layers, hardware modules for convolutional layers and hardware modules for fully connected layers are designed, along with various control modules. Specifically, the FPGA architecture includes an ARM processor, controller, external memory, input / output feature map memory, weight memory, configuration instruction control unit, activation and pooling units, PE array, and index address generator; wherein:

[0018] The ARM processor, controller, and configuration instruction control unit are used to process control instructions and perform operations such as loading and updating information at each layer.

[0019] The input / output feature map memory and weight memory are used to store data read from external memory, and also to store the calculated activation data;

[0020] After the activation and pooling unit completes the calculation of the output channel results of the convolutional layer or fully connected layer of the convolutional neural network, it retrieves the results from the partial sum accumulator of the sparse computing acceleration module, performs ReLU operation or pooling operation, and then writes them into the output feature map memory.

[0021] The PE array is used to perform convolution and dot product operations in the convolutional neural network;

[0022] The index address generator is used to store the index and generate the read address of the input / output feature map memory, and its internal structure is modified according to the size of the pruned block.

[0023] Furthermore, the workflow for implementing the method is as follows:

[0024] (10) Each module is configured according to the layer characteristics;

[0025] (20) Load the image or input feature map into on-chip memory;

[0026] (30) Load the weights into local memory;

[0027] (40) The processing unit performs convolution and dot product operations;

[0028] (50) Send the results to the pooling and activation module;

[0029] (60) The new output feature map is stored in external memory;

[0030] (70) If more weights need to be loaded, repeat step (30);

[0031] (80) Repeat the entire process for each layer.

[0032] Compared with the prior art, the significant advantages of the present invention are:

[0033] 1. This invention allows for the pruning of weighted blocks, not just individual weights, reducing index overhead and the number of memory ports required;

[0034] 2. The size of the pruning block in this invention can be adjusted according to the actual situation. Through the FPGA architecture, it can be matched with the convolutional neural network to achieve high computing density and high energy efficiency. Attached Figure Description

[0035] The present invention will now be described in further detail with reference to the accompanying drawings.

[0036] Figure 1 This is a flowchart of the convolutional neural network block pruning method of the present invention.

[0037] Figure 2 This is a schematic diagram of the pruning method for 4-weighted blocks.

[0038] Figure 3 It is a diagram of the weight storage structure in memory.

[0039] Figure 4 This is a diagram of a convolutional neural network FPGA inference architecture based on block pruning.

[0040] Figure 5 yes Figure 4 The PE array structure diagram in the image.

[0041] Figure 6 yes Figure 5 Architecture diagram of the memory address generator. Detailed Implementation

[0042] like Figure 1 As shown, the convolutional neural network block pruning method of the present invention is used to improve the hardware implementation and performance of pruned networks, and includes the following steps:

[0043] (10) Training to obtain the original convolutional neural network model: Here, Caffe is used as the main framework for training, and the accuracy of the trained network is recorded as a0.

[0044] (20) Block partitioning and average calculation: Expand the convolutional kernel weights or fully connected layer weights into a one-dimensional vector, divide the continuous weights into blocks of a predetermined size, and calculate the average value of each block;

[0045] The weight block partitioning method in step (20) is as follows: Figure 2 As shown in the figure, the continuous weights are divided into several blocks with a size of 4, and the average value of the weights in each block is calculated.

[0046] (30) Iterative pruning according to the pruning percentage p: After the pruning blocks are divided, they are sorted from low to high according to the average value, and pruned sequentially according to the pruning percentage p; the remaining blocks after pruning are stored as sparse vectors, where each position contains the weight block and the index of the next block;

[0047] The pruning method in step (30) is as follows: Figure 2 As shown, the blocks are sorted from lowest to highest weighted average, with 1.25 < 2 < 4 < 5. When the pruning percentage is 25%, the block with an average of 1.25 is pruned, and the weight of the entire block is filled with 0; when the pruning percentage is 50%, the blocks with averages of 1.25 and 2 are pruned, and the weight of the entire block is filled with 0; and so on.

[0048] (40) Record the precision of the pruned network. When the difference between the precision of the original network and the precision of the pruned network is lower than the threshold, iteratively increase the percentage p of pruning and continue pruning.

[0049] The purpose of step (40) is to prune the network as much as possible within the allowable range of accuracy error, so as to reduce the number of network parameters and computational load, and accelerate the network inference speed.

[0050] (50) After determining the appropriate pruning percentage, the remaining network weights are quantized. In order to further reduce the performance degradation caused by quantization, this problem is solved by retraining. Here, 8-bit data quantization is used.

[0051] (60) Fine-tune the network and then output the network parameters to realize the recognition of the image through the current network.

[0052] The network parameters output in step (60) are stored as a sparse vector, where each position contains a weight block and an index of the next block position. Each index has four bits and represents the offset relative to the previous index. Its organization in memory is as follows: Figure 3 As shown. The number of indices depends on the size of the pruned block and the pruning percentage, as shown in the following formula:

[0053]

[0054] In the formula N i N is the number of indices. w S is the number of weights, S is the size of the pruning block, and p is the pruning percentage; for example, a kernel with 8192 weights, pruned 90% using blocks with 8 weights, requires... The more indexes are needed, the smaller the block and pruning percentage.

[0055] This technique allows for the pruning of weight blocks, rather than just individual weights, reducing index overhead and the number of memory ports required.

[0056] like Figure 4 As shown, the convolutional neural network method is based on an FPGA architecture. This FPGA architecture is used to accelerate the pruning method and the pruned convolutional neural network. The FPGA architecture includes an ARM processor, a controller, external memory, input / output feature map memory, weight memory, a configuration instruction control unit, activation and pooling units, a PE array, and an index address generator; wherein:

[0057] The ARM processor, controller, and configuration instruction control unit are used to process control instructions and perform operations such as loading and updating information at each layer.

[0058] The input / output feature map memory and weight memory are used to store data read from external memory, and also to store the calculated activation data;

[0059] After the activation and pooling unit completes the calculation of the output channel results of the convolutional layer or fully connected layer of the convolutional neural network, it retrieves the results from the partial sum accumulator of the sparse computing acceleration module, performs ReLU operation or pooling operation, and then writes them into the output feature map memory.

[0060] The PE array is used to perform convolution and dot product operations in the convolutional neural network;

[0061] The index address generator is used to store the index and generate the read address of the input / output feature map memory, and its internal structure is modified according to the size of the pruned block.

[0062] The process of implementing the pruning method and the pruned convolutional neural network using the FPGA architecture is as follows:

[0063] Each module is configured based on layer characteristics;

[0064] Each module in the architecture is independent of the others, and one layer is executed at a time. Before running a layer, the specific features of that layer are configured through the configuration instruction controller, such as the number of convolution kernels, the size of the convolution kernels, the addresses of feature maps and weights in the chip memory, the optional execution of pooling after the convolutional layer, and the fixed-point format that may be different for different layers.

[0065] Loading the image or input feature map into on-chip memory: This step is accomplished by accessing external memory via Direct Memory Access Block (DMA). DMA reads the feature map and weights from external memory and can also write the feature map back to external memory.

[0066] Load the weights into the local memory of the PE array;

[0067] The FPGA architecture in this invention treats 3D convolutions with activations and weights as long vectors and computes the convolutions independently of the convolution window size. Each element of the output feature map is obtained by the dot product between the 3D convolution kernel xk×yk×zk and the corresponding element of the input feature map of size xp×yp×zp, where zp is the number of input feature maps. Formally, the dot product calculation is given by the following formula:

[0068]

[0069] Where startAddr is the address of the first element of the input feature map being convolved. This operation is used to convolve the kernel with the input feature map, causing the 3D convolution kernel to slide along the feature map.

[0070] PE arrays perform convolution and dot product calculations;

[0071] The PE array structure diagram is as follows: Figure 5 As shown, the PE array, together with the index memory address generator, feature map memory, weight memory, and activation pooling integration module, constitutes a hardware functional module. The data bit width of the weight memory is 64 bits, allowing eight 8-bit weights to be read in parallel.

[0072] The functional module includes an index memory address generator, which stores the index and generates the read address for the feature map memory. This module can be modified according to the pruning block size of 1, 2, 4, or 8. Figure 6 The architecture of an index memory address generator with a block size of 8 is described;

[0073] Combination Figure 6 Sixteen indices are read from a 64-bit DMA, followed by 16 weight blocks, each with 8 weights (16×64). The sixteen indices are serialized and written to a FIFO. Each distinct kernel has an associated FIFO. During inner product computation, indices are read and sequentially added to the initial address of the feature map memory to determine the read addresses of the eight active blocks. The feature map memory is a dual-port memory, with one port for each kernel in a row. All feature map memories receive the same address pairs. To compute more kernels in parallel, the architecture shown in the diagram is replicated, including a batch memory with replicated contents to provide additional read ports.

[0074] When the size of the weighted blocks changes, the frame must be adjusted, as shown below:

[0075] Block size = 4: The indices will be stored in pairs in parallel in the FIFO because two weight blocks (a total of 8 weights) will be read in parallel from the weight memory during reads. Two read addresses are generated from the two parallel indices, and each dual-port feature map memory provides two quadruple-activated blocks for the same core in a row. In this case, each core in each layer requires one dual-port memory;

[0076] Block size = 2: Four indices are stored in parallel in the FIFO because four blocks will be read in parallel from the weight memory, for a total of eight weights. Generating four read addresses from the four parallel indices requires copying the dual-port feature map memory to provide four ports for the same feature map memory mapping. Therefore, each core in each layer requires two dual-port memories;

[0077] Block size = 1: This is normal pruning. Eight indices are stored in parallel in the FIFO. Eight read addresses are generated from these eight parallel indices, requiring duplex feature map memory to be copied to provide eight ports of the same feature map memory mapping. Therefore, each core in each layer requires four duplex memories.

[0078] Send the results to the pooling and activation modules;

[0079] The new output feature map is stored in external memory;

[0080] If more weights need to be loaded, repeat the loading of weights into local storage;

[0081] Repeat the entire process on each layer.

[0082] Using the method of this invention, convolutional neural network inference experiments were conducted on a Zynq-7000 series FPGA. It was observed that the accuracy difference between the convolutional neural network with 70% block pruning and a block size of 8 and the unpruned network was only 1.8%, which is 18% higher than the performance of similar architectures.

[0083] Example

[0084] This invention has achieved excellent results in the field of radar anti-chaff interference. After the convolutional neural network model of this invention is constructed, the echo data of chaff and targets are input into the model in the form of 1024 points per group. After processing by the neural network after chaff trimming, the type of echo data is output, completing the classification and identification of chaff and targets, thereby achieving the purpose of anti-chaff interference. On Zynq-7000 series FPGA, the processing time of the network based on the ordinary pruning method for chaff target data is more than 1.2ms, while the processing time of the network of this invention for chaff target data is 1ms.

[0085] In summary, this invention improves the computational efficiency of convolutional neural networks by optimizing pruning methods, proposes a configurable block pruning method, and constructs an FPGA inference architecture that supports block pruning networks, achieving excellent accuracy under high image processing throughput.

Claims

1. A block pruning method for convolutional neural networks based on FPGA, comprising the following steps: S10, perform convolutional neural network training or use a pre-trained network, and obtain network accuracy; S20, group the continuous weights into blocks of a predetermined size and determine the average value of each block; S30, iteratively prune from the block with the lowest average value according to the pruning percentage p, and store the remaining blocks after pruning as a sparse vector, where each position contains the weight block and the index of the next block; S40, obtain the accuracy of the pruned network. When the difference between the accuracy of the original network and the accuracy of the pruned network is lower than the threshold, iteratively increase the pruning percentage p and continue to repeat step S30 to prune until the requirements are met. S50, obtain the required pruning percentage p, and quantify the network weights; S60: Obtain the parameters of the convolutional neural network and input the image features into the convolutional neural network for image recognition; The FPGA architecture includes an ARM processor, a controller, external memory, input / output feature map memory, weight memory, configuration instruction control unit, activation and pooling unit, PE array, and index address generator; wherein: The ARM processor, controller, and configuration instruction control unit are used to process control instructions and perform operations such as loading and updating information at each layer. The input / output feature map memory and weight memory are used to store data read from external memory, and also to store the calculated activation data; After the activation and pooling unit completes the calculation of the output channel results of the convolutional layer or fully connected layer of the convolutional neural network, it retrieves the results from the partial sum accumulator of the sparse computing acceleration module, performs ReLU operation or pooling operation, and then writes them into the output feature map memory. The PE array is used to perform convolution and dot product operations in the convolutional neural network; The index address generator is used to store the index and generate the read address of the input / output feature map memory, and its internal structure is modified according to the size of the pruned block; The index address generator modifies its internal structure according to the size of the pruned block, specifically including: Block size = 4: The indexes will be stored in pairs in parallel in the FIFO. When reading, two weight blocks will be read in parallel from the weight memory, and two read addresses will be generated from the two parallel indexes. Each dual-port feature map memory provides two quadruple activation blocks for the same core in a row. In this case, each core in each layer uses a dual-port memory. Block size = 2: Four indices are stored in parallel in FIFO. The four blocks will be read in parallel from the weight memory. There are a total of eight weights. Four read addresses are generated from the four parallel indices. The input / output feature map memory memory must be copied to provide four ports of the same feature map memory mapping. Each core in each layer uses two dual-port memory. Block size = 1: Normal pruning, eight indices are stored in parallel in a FIFO, eight read addresses are generated from the eight parallel indices, the input / output feature map memory memory must be copied to provide eight identical input / output feature map memory mapping ports, and each core in each layer uses four dual-port memory.

2. The convolutional neural network block pruning method according to claim 1, characterized in that, In step S20, the size of the pruned blocks is divided into 1, 2, 4, and 8 according to actual needs, and then the average value of all blocks is calculated.

3. The convolutional neural network block pruning method according to claim 1, characterized in that, Step S30 specifically involves: sorting the blocks from low to high according to their average value, pruning them sequentially according to the pruning percentage p, and filling the pruned weighted blocks with 0.

4. The convolutional neural network block pruning method according to claim 1, characterized in that, The index has four bits and represents the offset relative to the previous index.

5. The convolutional neural network block pruning method according to claim 4, characterized in that, The number of indexes depends on the size of the pruned block and the pruning percentage, specifically: ； In the formula For the number of indexes, denoted by weight, S represents the size of the pruning block, and p represents the pruning percentage.

6. The convolutional neural network block pruning method according to claim 1, characterized in that, In step S50, the network weights are quantized using an 8-bit data quantization method.

7. The convolutional neural network block pruning method according to claim 1, characterized in that, The weight memory has a data width of 64 bits, allowing eight 8-bit weights to be read in parallel.

8. The convolutional neural network block pruning method according to claim 1, characterized in that, The input / output feature map memory uses dual-port memory, with one port for each kernel in a row; the input / output feature map memory receives the same address pairs.

Images