Neural network target detection method based on FPGA acceleration

Through the reparameterized channel pruning and quantization perception training of the YOLOv3-Tiny network model, combined with efficient data transmission and convolutional computing architecture, the problems of reduced accuracy and high latency of the YOLO network in the FPGA accelerator are solved, and efficient Object detection performance.

CN118941770AActive Publication Date: 2024-11-12HARBIN INST OF TECH

Patent Information

Application Number
CN202410957820.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-17
Publication Date
2024-11-12
Estimated Expiration
2044-07-17

AI Technical Summary

Technical Problem

The existing FPGA-based convolutional neural network accelerator design has significantly reduced the accuracy of networks such as YOLO and has high latency problems.

Method used

The YOLOv3-Tiny network model is compressed by reparameterized channel pruning and quantization perception training, and data transmission is carried out through AXIDMA and MIG, and the convolution calculation is performed using double signed multiplication encapsulation and row convolution calculation water flow architecture, combined with the maximum pooling The parameterized design of the Concat layer, as well as the NMS non-maximum suppression module of the FIFO structure for post-processing.

Benefits of technology

It significantly reduces the number of parameters and calculations of the model, improves the operation efficiency of the model, reduces the data transmission delay, improves the efficiency of convolutional calculation and the accuracy of the target detection results, and is suitable for resource-constrained FPGA implementation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118941770A_ABST
    Figure CN118941770A_ABST
Patent Text Reader

Abstract

The invention discloses a neural network target detection method based on FPGA acceleration, and relates to the technical field of convolutional neural network accelerators. In order to solve the technical problems that in the design of an existing FPGA-based convolutional neural network accelerator in the prior art, the accuracy of a YOLO network is remarkably reduced, and the delay is high, the technical scheme provided by the invention is as follows: the method for constructing the neural network target detection system based on FPGA acceleration comprises the following steps: obtaining a compressed network model; carrying out data processing and transmission to obtain a preprocessed feature map and weight data; performing convolution calculation on the preprocessed feature map and the weight data to obtain a convolution calculation result; a convolution calculation result is processed through a maximum pooling layer, an Upsample up-sampling layer and a Concat layer, and a pooling, up-sampling and splicing feature map is obtained; and the NMS non-maximum suppression module containing the FIFO structure performs post-processing on the pooled, up-sampled and spliced feature map. The method is suitable for the work of neural network target detection.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to the technical field of convolutional neural network accelerators, and specifically to a YOLOv3-Tiny neural network hardware accelerator implementation method based on FPGA, and a neural network re-parameterized channel pruning and quantization compression method. Background Art

[0002] With the improvement of computer computing power and storage, deep learning algorithms have ushered in unprecedented great development. Among them, convolutional neural networks (CNN) are widely used in image classification, recognition, target detection and other aspects. However, the central processing unit (CPU) is not competent for such high-computation model reasoning; the graphics processing unit (GPU) has ultra-large-scale computing performance, but its power consumption is difficult to meet the needs of embedded scenarios; the hardware design of application-specific integrated circuits (ASIC) has problems such as long R&D cycle and lack of flexibility. In contrast, field programmable gate arrays (FPGA) are flexible and can meet the characteristics of large-scale parallel computing of convolutional neural networks, and can adapt to low-power and high-efficiency scenarios. In the design of traditional FPGA-based convolutional neural network accelerators, the traditional methods for implementing CNN on FPGA are mainly model compression and hardware architecture optimization. For model compression, quantization is mainly used to quantize the single-precision floating-point numbers involved in the calculation into low-bit width integers, but this method significantly reduces the accuracy of networks such as YOLO. As for the hardware architecture, traditional accelerator designs mostly focus on optimizing performance in terms of throughput, energy efficiency, and versatility. The network is sent to the accelerator for calculation in a single layer, and the calculation and data transmission of each layer are processed separately in a time-division multiplexing manner. This will result in higher latency, but for edge target acquisition systems, low latency is the focus of optimization. Summary of the invention

[0003] In order to solve the technical problems existing in the prior art, in the existing FPGA-based convolutional neural network accelerator design, the accuracy of networks such as YOLO is significantly reduced, and there is a high delay, the technical solution provided by the present invention is:

[0004] The method for constructing a neural network target detection system based on FPGA acceleration includes:

[0005] The steps of compressing the original YOLOv3-Tiny network model to obtain a compressed network model;

[0006] The compressed network model and the preprocessed feature map are transmitted to obtain weight data and feature map data;

[0007] A step of performing convolution calculation on the preprocessed feature map and weight data to obtain a convolution calculation result;

[0008] The convolution calculation result is processed through a maximum pooling layer, an Upsample upsampling layer and a Concat layer to obtain a feature map after pooling, upsampling and splicing;

[0009] A step of post-processing the pooled, upsampled and concatenated feature map using an NMS non-maximum suppression module with a FIFO structure.

[0010] Furthermore, a preferred implementation is provided to compress the original YOLOv3-Tiny network model through re-parameterized channel pruning and quantization-aware training.

[0011] Furthermore, a preferred implementation is provided, which uses AXIDMA and MIG to perform data transmission on the compressed weight data and the preprocessed feature map.

[0012] Furthermore, a preferred implementation is provided, which performs convolution calculation on the preprocessed feature map and weight data based on a dual signed multiplication encapsulation design and a row convolution calculation pipeline architecture.

[0013] Based on the same inventive concept, the present invention also provides a device for constructing a neural network target detection system based on FPGA acceleration, comprising:

[0014] Compress the original YOLOv3-Tiny network model to obtain the module of the compressed network model;

[0015] The compressed network model and the preprocessed feature map are transmitted to obtain a module for weight data and feature map data;

[0016] A module for performing convolution calculation on the preprocessed feature map and weight data to obtain a convolution calculation result;

[0017] The convolution calculation results are processed through a maximum pooling layer, an Upsample upsampling layer and a Concat layer to obtain a module of a feature map after pooling, upsampling and splicing;

[0018] A module that uses an NMS non-maximum suppression module with a FIFO structure to post-process the feature map that has undergone pooling, upsampling and splicing.

[0019] Based on the same inventive concept, the present invention also provides a neural network target detection method based on FPGA acceleration, including:

[0020] The step of collecting image signals, feature map signals, weight data and intermediate calculation results of the target to be measured as preset data;

[0021] The step of processing the preset data according to the method to obtain the target detection result.

[0022] Based on the same inventive concept, the present invention also provides a neural network target detection device based on FPGA acceleration, comprising:

[0023] A module for collecting image signals, feature map signals, weight data and intermediate calculation results of the target to be measured as preset data;

[0024] A module for processing the preset data according to the device to obtain a target detection result.

[0025] Based on the same inventive concept, the present invention also provides a computer storage medium for storing a computer program, and when the computer program is read, the computer executes the method described.

[0026] Based on the same inventive concept, the present invention also provides a computer, including a processor and a storage medium. When the processor reads a computer program stored in the storage medium, the computer executes the method described.

[0027] Based on the same inventive concept, the present invention also provides a computer program product, which is a computer program. When the computer program is read, the method described above is implemented.

[0028] Compared with the prior art, the technical solution provided by the present invention is beneficial in that:

[0029] The present invention provides a neural network target detection method based on FPGA acceleration, and re-parameterized channel pruning (ResRep) reduces the model calculation amount by deleting unimportant channels while maintaining the accuracy of the model. This method significantly reduces the number of parameters and the amount of calculation of the model, and improves the operating efficiency of the model. Quantization-aware training further optimizes the reasoning performance of the model on the FPGA by adapting the model to quantization errors during training. Compared with traditional network compression methods (such as simple pruning or quantization), this method can better balance performance and efficiency, and is particularly suitable for resource-constrained FPGA implementations.

[0030] The FPGA-accelerated neural network target detection method provided by the present invention effectively reduces the delay of data transmission by using AXIDMA and MIG for output multiplexing and column buffer inter-layer pipeline design. This method accelerates the transmission of feature maps and weight data, allowing convolution calculations to be carried out more quickly. Compared with the traditional layer-by-layer data transmission method, this inter-layer pipeline design significantly reduces the startup delay and improves the overall efficiency of the system.

[0031] The neural network target detection method based on FPGA acceleration provided by the present invention uses DSP48E1 to perform dual signed number multiplication encapsulation, which improves the efficiency of multiplication operations and reduces the occupation of FPGA resources. The row convolution calculation pipeline architecture design further optimizes the speed of convolution calculation, making the entire convolution process more efficient. Compared with the traditional single multiplication operation method, this design can significantly improve the throughput and resource utilization of convolution calculation.

[0032] The neural network target detection method based on FPGA acceleration provided by the present invention can flexibly implement maximum value pooling of different sizes and step sizes through parameterized design of the maximum pooling layer, thereby improving the adaptability of the model. The upsampling layer designed by the nearest neighbor upsampling method simplifies the calculation while maintaining accuracy. The design of the Concat layer ensures the computational efficiency in the feature map splicing process. The optimized design of these subsidiary layer units makes the deployment of the model on the FPGA more flexible and efficient, and has higher adaptability and efficiency compared with the traditional fixed parameter design.

[0033] The FPGA-accelerated neural network target detection method provided by the present invention adopts the NMS non-maximum suppression method to improve the accuracy of target detection results by screening detection frames and removing redundant low-confidence detection frames. The FIFO structure design reduces off-chip data access and further improves the real-time performance of the system. Compared with traditional post-processing methods, this design can more effectively filter invalid detection frames, thereby improving the accuracy and reliability of detection results.

[0034] The neural network target detection method based on FPGA acceleration provided by the present invention adopts re-parameterized channel pruning and quantization-aware training compared to traditional pruning and quantization methods, which balances model performance and efficiency and is particularly suitable for FPGA implementation. The output multiplexing and column buffer layer pipeline design significantly reduce data transmission delay, while traditional methods are usually unable to achieve such efficient transmission. The dual signed multiplication encapsulation and row convolution calculation pipeline architecture improve the computing speed and resource utilization, while traditional methods are less efficient in this regard. The parameterized maximum pooling layer and flexible upsampling and splicing design improve the adaptability and efficiency of the system, while traditional methods are usually relatively fixed and not flexible enough. The design of the NMS non-maximum suppression module with FIFO structure optimizes the accuracy and real-time performance of the detection results, while traditional methods are insufficient in processing efficiency and accuracy.

[0035] The FPGA-accelerated neural network target detection method provided by the present invention solves several key technical problems of neural network target detection in FPGA practice and significantly improves the efficiency, accuracy and adaptability of the system.

[0036] The neural network target detection method based on FPGA acceleration provided by the present invention is suitable for application in the work of neural network target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is a schematic diagram of the system architecture of the neural network target detection method based on FPGA acceleration;

[0038] Figure 2 Schematic diagram of the convolution parallel multiplication and addition unit;

[0039] Figure 3 This is a schematic diagram of the overall post-processing process;

[0040] Figure 4 This is a schematic diagram of the post-processing hardware architecture;

[0041] Figure 5 This is a schematic diagram of the model compression process;

[0042] Figure 6 Schematic diagram of column-based inter-layer pipeline;

[0043] Figure 7 Schematic diagram of the row convolution calculation sequence;

[0044] Figure 8 Schematic diagram of the basic pooling unit;

[0045] Fig. 9 Schematic diagram of the hardware quantization process. DETAILED DESCRIPTION

[0046] In order to make the advantages and benefits of the technical solution provided by the present invention more clearly reflected, the technical solution provided by the present invention is now further described in detail with reference to the accompanying drawings, specifically:

[0047] Implementation method 1: This implementation method provides a method for constructing a neural network target detection system based on FPGA acceleration, including:

[0048] The steps of compressing the original YOLOv3-Tiny network model to obtain a compressed network model;

[0049] The compressed network model and the preprocessed feature map are transmitted to obtain weight data and feature map data;

[0050] A step of performing convolution calculation on the preprocessed feature map and weight data to obtain a convolution calculation result;

[0051] The convolution calculation result is processed through a maximum pooling layer, an Upsample upsampling layer and a Concat layer to obtain a feature map after pooling, upsampling and splicing;

[0052] A step of post-processing the pooled, upsampled and concatenated feature map using an NMS non-maximum suppression module with a FIFO structure.

[0053] Implementation method 2: This implementation method further limits the method for constructing a neural network target detection system based on FPGA acceleration provided in implementation method 1, and compresses the original YOLOv3-Tiny network model through re-parameterized channel pruning and quantization-aware training.

[0054] Implementation method three: This implementation method further limits the method for constructing a neural network target detection system based on FPGA acceleration provided in implementation method one, and uses AXIDMA and MIG to transmit data of compressed weight data and preprocessed feature maps.

[0055] Implementation method 4: This implementation method further limits the method for constructing a neural network target detection system based on FPGA acceleration provided in implementation method 1, and performs convolution calculation on the preprocessed feature map and weight data based on a dual signed multiplication encapsulation design and a row convolution calculation pipeline architecture.

[0056] Embodiment 5: This embodiment provides a device for constructing a neural network target detection system based on FPGA acceleration, including:

[0057] Compress the original YOLOv3-Tiny network model to obtain the module of the compressed network model;

[0058] The compressed network model and the preprocessed feature map are transmitted to obtain a module for weight data and feature map data;

[0059] A module for performing convolution calculation on the preprocessed feature map and weight data to obtain a convolution calculation result;

[0060] The convolution calculation results are processed through a maximum pooling layer, an Upsample upsampling layer and a Concat layer to obtain a module of a feature map after pooling, upsampling and splicing;

[0061] A module that uses an NMS non-maximum suppression module with a FIFO structure to post-process the feature map that has undergone pooling, upsampling and splicing.

[0062] Implementation 6: This implementation provides a neural network target detection method based on FPGA acceleration, including:

[0063] The step of collecting image signals, feature map signals, weight data and intermediate calculation results of the target to be measured as preset data;

[0064] The step of processing the preset data according to the method provided in the first embodiment to obtain the target detection result.

[0065] Specifically: This implementation includes:

[0066] Step 1: Model compression

[0067] Input: Original YOLOv3-Tiny network model

[0068] Reparameterized channel pruning: Use the ResRep method to prune some channels of CNN to reduce the amount of computation and maintain performance. Quantization-aware training: Adapt to quantization errors through training to make the model perform better during inference.

[0069] Output: The compressed YOLOv3-Tiny network model with fewer parameters but almost the same accuracy.

[0070] Step 2: Data processing and transmission

[0071] Input: Compressed network model

[0072] Output multiplexing: Feature map transmission and weight data writing through AXIDMA and MIG.

[0073] Column buffer inter-layer pipeline: Design row parallelism and column buffer inter-layer pipeline optimization to reduce startup delay.

[0074] Output: preprocessed and transmitted feature maps and weight data.

[0075] Step 3: Convolutional unit design

[0076] Input: preprocessed feature maps and weight data

[0077] Dual signed multiplication encapsulation: Use DSP48E1 to perform dual signed multiplication encapsulation to improve multiplication efficiency. Row convolution computation pipeline architecture: Design of computation pipeline architecture based on row convolution to improve computation speed.

[0078] Output: Feature map after convolution calculation.

[0079] Step 4: Subsidiary layer unit design

[0080] Input: Feature map after convolution calculation

[0081] Max pooling layer: parameterize the max pooling layer to implement left column, right column and overall maximum pooling.

[0082] Upsample upsampling layer: The upsampling layer is designed using the nearest neighbor upsampling method.

[0083] Concat layer: Design the concatenation of input feature maps to maintain pipeline calculation.

[0084] Output: Feature map after processing by the auxiliary layer.

[0085] Step 5: Post-processing unit design

[0086] Input: Feature map after processing by the auxiliary layer

[0087] NMS non-maximum suppression: decode the convolutional network output and filter the detection box.

[0088] FIFO structure design: Reduce off-chip data access and filter low-confidence detection frames.

[0089] Output: Final target detection result, including confidence and detection box location information.

[0090] in,

[0091] The collected signals are:

[0092] Image signal: usually the raw image data obtained from a camera or other imaging device.

[0093] Feature map signal: A multi-layer feature map extracted by a convolutional neural network (CNN), each layer of which contains different feature information.

[0094] Weight data: The convolution kernel weights of each layer in the convolutional neural network. These weights are optimized during the training process and saved in the model.

[0095] Intermediate calculation results: intermediate results generated during convolution calculation, pooling, upsampling, splicing, etc.

[0096] The final output results are:

[0097] Bounding Boxes: Identify the location and size of the target object in the image, usually represented by a rectangular box.

[0098] Class Labels: Classification results of objects in the detection box, such as people, cars, animals, etc.

[0099] Confidence Scores: The probability that the target object corresponding to each detection box belongs to a certain category, indicating the credibility of the detection result.

[0100] Embodiment 7: This embodiment provides a neural network target detection device based on FPGA acceleration, including:

[0101] A module for collecting image signals, feature map signals, weight data and intermediate calculation results of the target to be measured as preset data;

[0102] The device provided in the fifth embodiment processes the preset data to obtain a module for target detection results.

[0103] Embodiment 8: This embodiment provides a computer storage medium for storing a computer program. When the computer program is read, the computer executes the method provided in embodiment 1.

[0104] Embodiment 9: This embodiment provides a computer, including a processor and a storage medium. When the processor reads a computer program stored in the storage medium, the computer executes the method provided in embodiment 1.

[0105] Embodiment 10: This embodiment provides a computer program product, which is a computer program. When the computer program is read, the method provided in embodiment 1 is implemented.

[0106] Implementation Method XI: Combination Figure 1-9 This embodiment is described in detail by using specific examples to further describe the technical solution provided above. Specifically:

[0107] Model compression process:

[0108] Since the traditional pruning paradigm imposes a penalty on parameters to produce structured sparsity, this may inhibit the parameters necessary for memory. The pruning scheme is further improved by using the Re-parameterization method and the ResRep (Re-parameterization and Gradient Resetting) method, which reduces the amount of computation by pruning some channels of the CNN while maintaining the performance of the model as much as possible, and reparameterizes each target convolution layer and its subsequent BN layer into a convolution-BN-compressor structure.

[0109] The compressor is initialized to the identity matrix, where and are the re-parameterized convolution kernel and bias, and Q′ is the compressor.

[0110] This process can be expressed as

[0111]

[0112] During the training process, a new gradient reset strategy is introduced:

[0113] The gradient update process of the original part is as follows:

[0114]

[0115] During training, the standard gradient descent method is applied to the original part, while the gradient reset strategy is applied to the compressor part, and a binary m mask is introduced to adjust the gradient to achieve the decoupling of "memory" and "forgetting":

[0116]

[0117] Quantization-aware training, through back-propagation during training, the model parameters gradually adapt to the quantization error, making the quantized model perform better during inference. After training is completed, the model's weights and activation values ​​are quantized to actual low-bit width integers.

[0118] Mean Average Precision (mAP) refers to an evaluation index for multiple categories. First, find the AP of each class, and then find the average of these classes to get the mAP value, which is used to evaluate the average evaluation level of the algorithm's detection accuracy for all categories. The calculation principle of mAP is:

[0119]

[0120] Where N class is the number of detection categories of the target detection task; AP i It is the average of all the recall rates corresponding to the precision rates of each category.

[0121] The storage space compression ratio Mrr is mainly used to evaluate the compression degree of the target model storage space by the pruning algorithm. The unit is MB (Million Byte). The calculation principle is:

[0122]

[0123] Among them, M origin is the storage space of the model before pruning, M remain It is the storage space of the pruned model.

[0124] The calculation principle of the calculation amount compression ratio Orr is:

[0125]

[0126] Ops origin is the number of convolution operations of the model before pruning, Ops remain is the number of convolution operations of the pruned model.

[0127] Inter-layer pipeline design for output multiplexing and column buffering:

[0128] The system architecture of this embodiment is as follows Figure 1 As shown in the figure. The system is mainly composed of peripheral data paths and cascaded processing units (Process Engine, PE) and post-processing modules. Among them, the peripheral data path is composed of AXIDMA (Advanced Xtensible Interface Direct Memory Access) and MIG (Memory Interface Generator). AXI DMA transmits feature maps and writes weight data to DDR3 (Double Data Rate 3) mounted on the PL side with the ARM processor on the PS side of the host ZYNQ7035 through the AXI interface; the off-chip memory controller is responsible for controlling the distribution of weights to each convolution layer; MIG accesses the off-chip DRAM through the DDR3 interface for weight transmission. PE is the processing unit of each convolution layer, responsible for operations such as convolution, pooling and activation, and is the core of the CNN accelerator. PE is mainly composed of cache units such as Col_buffer and Wt_Buffer, Layer Controller control unit and computing units such as CONV, POOL and RELU. Col_buffer is responsible for caching the input of each convolutional layer; Wt_Buffer is responsible for caching the weights; Layer Controller is responsible for the scheduling of the entire PE module; CONV, POOL and Leaky are responsible for convolution, pooling and activation operations respectively.

[0129] In the initialization phase, the trained weights are preloaded from the ARM side to the DRAM via the AXI bus. In the inference phase, the input feature map is first sent to the input buffer column by column via AXI. Then, it is processed step by step by the PE in the pipeline. After the output of the convolutional network is obtained, it passes through the pipeline accelerated Detect layer and the NMS non-maximum suppression post-processing process, and the final calculation result is sent to the PC side for post-processing via AXIDMA. In particular, in order to further shorten the pipeline startup delay, the padding in the calculation of the first and last columns can be skipped. For a 3×3 convolution layer, the padding skip technology can reduce the startup delay by 1 / 3. The DSP in the PE uses the double signed multiplication encapsulation technology to double the convolution calculation speed and shorten the calculation delay. The parameterized design and the design of the adaptive addition tree enable the PE to be encapsulated into an IP, support the calculation of convolutions of different sizes, and can also be configured with different degrees of parallelism to balance the pipeline time.

[0130] Convolution unit design:

[0131] The DSP model commonly used in FPGA is DSP48E1, which is a 25×18-bit multiplier. Its two multipliers are B (maximum bit width 18 bits) and the lower 25 bits of the result after adding A and D. The output result is 48 bits (the upper 5 bits are sign extension bits, and the lower 43 bits are data bits). In the multiplication process of neural network reasoning, the weights and feature maps are quantized into 8-bit signed numbers. Since the DSP input bit width can meet the splicing input of 2 signed numbers, the DSP can be encapsulated with dual signed number multiplication to improve the use efficiency of the DSP and reduce resource consumption.

[0132] DSP encapsulation of double signed multiplication, assuming that B = {bn-1, .. . b1, b0} is an n-bit signed integer and C is an n-bit unsigned integer, then the value of B*C is:

[0133]

[0134] Since 2n-1=2n-2n-1, it can be further expressed as:

[0135]

[0136] The result of the multiplication is divided into two parts. The first part is the unsigned multiplication result of B and C when B is regarded as an unsigned number, and the second part is the correction term. If bn-1 is 1, that is, B is a negative number, the result of the unsigned multiplication of B and C minus the result of C multiplied by 2n is equal to the result of the signed multiplication of B and C, where C multiplied by 2n is achieved by shifting. When B is a positive number, bn-1 is 0, and the signed multiplication result of B and C is equal to the unsigned multiplication result of B and C.

[0137] Since the convolution process involves a large number of multiply-accumulate operations (MAC), and the FPGA on-chip storage and computing resources are limited, in order to improve the computing speed, it is necessary to reasonably design the parallelism of the multiply-accumulate calculations of the data in the convolution process. Therefore, for each layer of the convolution, only K i Group convolution kernel (due to the use of Ping-Pong Buffer, the actual memory size will be designed to be 2K i ). In each cycle, the convolution processing unit takes n inputs from the input buffer and K from the weight buffer. i ×n weights, using K i ×n÷2 DSPs, calculate K i An intermediate result. Figure 2 As shown in the figure, the calculation process of the parallel multiplication and addition unit of the convolution is shown. In the example, a represents the n inputs taken out from the input feature map cache, b and c represent the 2×n weights taken out from the weight cache, and the multiplication results are passed through the addition tree to obtain the intermediate results each time and accumulate them. After traversing all the input channels N, the final convolution output is obtained.

[0138] This implementation has designed a computational pipeline architecture based on row convolution, and the above method can be used for calculation regardless of the size of the input feature map and the convolution kernel. In addition to parallel processing of channels, the convolution kernel is also fully parallelized, and the intermediate feature map buffer is eliminated. While saving FPGA resource consumption, the computation time of each layer is equal to the time to retrieve all data from the input buffer, so that the computation time is optimized. Table 3 configures the parameters of the input and output parallelism of each convolution layer.

[0139] Table 3 Hardware configuration parameters of each convolution layer

[0140] Convolutional Layer enter Parallelism DSP Processing time (ms) CONV0 416×416×3 2×4 36 6.92 CONV1 208×208×16 4×4 72 6.92 CONV2 104×104×32 4×4 72 6.92 CONV3 52×52×64 4×4 72 6.92 CONV4 26×26×125 4×4 72 6.38 CONV5 13×13×236 2×4 36 6.18 CONV6 13×13×247 2×4 36 4.40 CONV7 13×13×166 2×2 2 3.30 CONV8 13×13×93 1×2 9 4.40 CONV9 (output layer) 13×13×111 1×2 1 3.56 CONV10 13×13×93 1×2 1 1.89 CONV11 26×26×259 2×2 72 7.25 CONV12 (output layer) 26×26×130 1×2 1 1.67 total 482

[0141] Subsidiary unit design:

[0142] In YOLOv3-Tiny, in addition to the convolution layer, there are also maximum pooling layer, Concat layer, Upsample upsampling layer, etc.

[0143] This design performs parameterized design on the maximum pooling layer. Its main parameter is the convolution kernel parallel parameter m. According to the number of m, the basic pooling unit is expanded m times and connected behind the corresponding convolution kernel. Since the convolution operation is performed by column, the pooling process is also divided into three steps by column: first obtain the maximum value of the left column, then obtain the maximum value of the right column, and finally obtain the overall maximum value as the pooling result.

[0144] For the Upsample upsampling layer, since the nearest neighbor upsampling method is used, for the upsampling layer with an upsampling kernel size of n, it is only necessary to pass the parallel convolution results through the FIFO cache and keep the corresponding output for n cycles.

[0145] For the input of the Concat layer, the column output results of the i-th layer and the i+j-th layer (j>1 and is an integer) need to be spliced ​​and then sent to the next convolution layer for calculation. Since the pipeline calculation needs to be maintained, and from the storage scheme of the input feature maps of each layer mentioned above, it can be seen that when a single column feature map is transmitted to the i+j-th layer, the output of the i-th layer has been overwritten in the feature map cache of the i+1 layer, and Concat cannot be performed. In order to avoid occupying additional bandwidth and reducing the computing speed due to data exchange with external storage, it is necessary to explore how many columns of data need to be pre-cached for the output of the i-th layer to perform correct calculations after Concat passes through multiple layers. According to the column-based pipeline structure, assuming that there are convolutional layers, pooling layers, and upsampling layers in the path crossed by the Concat layer, the number of cache columns required to pass through the convolutional layer is:

[0146] col conv =k c +1-p c +s c (n-1) (1-9)

[0147] where k c is the convolution kernel size, p c is the padding number of the convolution kernel, s c is the moving step of the convolution kernel.

[0148] The number of cache columns required after the pooling layer is:

[0149] col pool =k p +s p (n-1) (1-10)

[0150] where k p is the pooling size, s p is the moving step size of the pooling kernel.

[0151] The number of columns that need to be cached after the upsampling layer is:

[0152]

[0153] where k u is the upsampling size.

[0154] According to the above formula, it can be concluded that in order to maintain the correct column-based pipeline calculation, the number of columns that the Concat layer of YOLOv3-Tiny needs to cache for the 8th convolutional layer is 16.

[0155] Post-processing unit design:

[0156] YOLOv3-Tiny has two detection layers, which are responsible for decoding the output of the convolutional network into confidence and the position information of the detection box. The output of the two detection layers is subjected to NMS non-maximum suppression, and other detection boxes with similar positions to the ones with the maximum confidence are filtered out, and other boxes with small or no overlap are retained as another target box. The overall process is as follows: Figure 3 shown.

[0157] The proposed NMS hardware unit mainly consists of a maximum prediction box buffer and an IOU calculation unit. Figure 4 As shown in the figure. The NMS unit is connected to the position calculation unit and the confidence calculation unit through two channels, using a FIFO structure to reduce off-chip data access. A threshold unit is designed to filter out boxes with low confidence scores to reduce the workload of the NMS unit.

[0158] This implementation consists of two parts, namely, the lightweight design of the target detection network and the design of the FPGA accelerator. This implementation adopts YOLOV3-tiny as the target detection algorithm, and adopts the lightweight method of QAT quantization, reparameterized channel pruning and BN layer fusion in the VOC data set training process, which significantly compresses the model parameters while ensuring the model detection accuracy. In the FPGA implementation part, this implementation adopts an inter-layer pipeline design based on column cache, which significantly reduces the delay of network reasoning acceleration. A convolution module with configurable convolution kernel parameters is designed, which is suitable for convolution kernels of various sizes and step sizes, and improves the versatility of hardware deployment. A pooling module that supports different step sizes and a Concat module that supports inter-layer pipelining are designed, which effectively solves the storage problem of inter-layer data association based on the column buffer module. A quantization module based on PoT quantization, a post-processing module of detect and NMS, and an OV5640 camera configured on the PS end of ZYNQ to collect images in real time, and the video processed by the target detection network is displayed in real time through the HDMI display module.

[0159] Implementation Method 12: Combination Figure 1-9This embodiment is described. This embodiment embodies the advantages of the technical solution provided above through specific examples, specifically:

[0160] Example of model compression process:

[0161] This implementation optimizes the model compression effect and greatly reduces the number of parameters of YOLOV3-Tiny. The overall process of model compression is as follows: Figure 5 As shown: ① Use a sparsified method to calculate the importance of each weight in the network and sort it. ② Calculate a pruning threshold, use the re-parameterization method and the gradient reset combined ResRep (Re-parameterization and Gradient Resetting) method to re-parameterize each target convolution layer and its subsequent BN layer into a convolution-BN-compressor structure, prune some channels to reduce the amount of calculation, and maintain the performance of the model as much as possible. ③ Use QAT quantization during training to fine-tune the quantized network so that the model parameters gradually adapt to the quantization error, so that the quantized model performs better during inference. ④ After repeating multiple sparse training, channel pruning, and QAT quantization operations, a quantized compressed network is finally obtained. The performance comparison between the compressed model and the original model is shown in Table 1, and the convolution layer parameters and calculation amount are shown in Table 2.

[0162] Table 1 Performance comparison between the compressed model and the original model

[0163] Pruning method <![CDATA[ mAP (%)]]> Model size (MB) <![CDATA[Ops(x10 9 )]]> <![CDATA[ Mrr (%)]]> Orr(%) none 63.4 33.43 5.49 - - ResRep 62.8 6.47 2.58 80.65 53.01

[0164] Table 2 Parameters and computational complexity of convolutional layers after pruning

[0165]

[0166] Example of inter-layer pipeline design:

[0167] The column-based pipeline architecture can significantly reduce latency. In addition, the input feature map and intermediate results do not need to access DRAM. Based on the comparison of these multiplexing methods, combined with column-based pipeline optimization, the data flow can be organized by output multiplexing. The specific implementation method is as follows: Figure 6 The whole process is divided into four stages: ① The input of each layer is cached in the input buffer by column, first loading the input and the corresponding T m Multiply the group weights to get the output feature map T m output; ② Slide the convolution window along the height direction and repeat step 1 until all the convolutions of a column are completed; ③ Replace another T m The convolution kernel is set, the convolution window returns to the top, and the first two steps are repeated to calculate the output feature map. m④ Slide the convolution window to the right to enter the next column of convolution calculations and repeat steps 1-3.

[0168] Convolution unit design:

[0169] This implementation has designed a computational pipeline architecture based on row convolution, which processes channels in parallel and processes the convolution kernel in full parallel, and eliminates the intermediate feature map row buffer. The specific computation sequence is as follows: Figure 7 As shown in Figure 1, the whole process is divided into four stages: ① For a convolution calculation of a certain layer of size K×K, the corresponding N are taken out from the K input buffers at the same time. n The feature map FM with input parallelism n , every T time, traverse all input channels N in a single row and K columns; take out N from the weight cache n Input parallelism, M m ② Divide the kernel weights into K groups according to the number of rows and add them to the input feature map FM n The convolution parallel multiplication and addition unit is sent to the above convolution parallel multiplication and addition unit for calculation (according to the parallel degree M m The intermediate results are stored in K output registers D after two input channels N are traversed. The calculation results of the convolution kernels of the corresponding positions of each row can be obtained. ③ After KT cycles, the K output registers have corresponding calculation values, among which D1 obtains the final output feature value corresponding to a group of m output channels, and D2 obtains the K-1 convolution sum of the output feature value of the corresponding position of the second row, and so on. k The convolution sum of the output feature value of the corresponding position of the Kth row is obtained. ④ Every time the input feature map moves down one row (equivalent to the convolution kernel moving down), the weight returns to the starting position, and the same steps are continued. It only takes T cycles to get the final output feature value of the second row in D2. After that, the final output feature value of the corresponding row can be obtained once every T cycles, that is, the calculation time is not delayed relative to the time of reading data.

[0170] Example of accessory layer unit design

[0171] This embodiment designs a highly streamlined pooling unit. This design greatly improves the versatility of the pooling layer. In addition to being able to handle the case where the pooling size and the pooling step length are equal, it can also handle the case where the pooling size and the pooling step length are not equal, and achieve highly streamlined pooling. The specific principle diagram is as follows Figure 8, divided into the following four steps: ① Assume that the kernel size of the pooling is n. When performing left-column pooling, use Reg1 to store the value received for the first time. When the convolution window slides down, the second output of the left column is obtained, which is compared with the value received for the first time, and the comparison result is stored in Reg2. ② The convolution window slides down again to obtain the third output of the left column, where Reg1 compares the values ​​received for the second and third times, and Reg2 compares the maximum value of the first two outputs with the value received for the third time. ③ Similarly, after n valid inputs, Reg(n) can compare the maximum value within the pooling range of n, and the comparison step is 1. ④ For a pooling unit with a pooling size of n, a single-column pooling result with a pooling step of 1 to n can be obtained by controlling the output valid signal of Reg(n). When performing cross-column pooling, use the column buffer to cache the pooling result of the previous column and compare it with the single-column pooling result of this column to finally obtain the pipeline-level pooling output.

[0172] Example of hardware quantization process:

[0173] The hardware quantization process is as follows Fig. 9 As shown: ① The network inference result after using 8-bit quantization is only slightly lower than that of 32-bit floating-point inference. After training, the 32-bit floating-point weights are quantized to obtain 8-bit fixed-point weights. ② After convolution with 8-bit fixed-point feature data, the data is further expanded. In order to prevent data overflow, the bit width is expanded to 16 bits. ③ The output intermediate data also needs to be accumulated through multiple channels, and the output data bit width continues to expand to 32 bits. ④ Finally, the bias data is added, and according to the scaling ratio of the feature data of this layer, the 8-bit feature data required for the next layer calculation is obtained through shift quantization. ⑤ When some shifted quantized data exceeds the 8-bit bit width representation range, the truncation method can be used to limit it to the upper or lower limit of INT8. When shifting quantization, it is divided into two cases according to whether the feature data is positive or negative: when the feature data is negative, according to the ReLU activation function, the negative value is directly reset to zero. When the characteristic data is a positive number, the rounding direction is determined by judging whether the integer part after shifting is an even number and whether the decimal part after shifting is greater than 0.5 according to the Round approximation function.

[0174] The technical solution provided by the present invention is further described in detail above through several specific implementation modes in order to highlight the advantages and benefits of the technical solution provided by the present invention. However, the several specific implementation modes described above are not intended to be used as limitations on the present invention. Any reasonable modification and improvement of the present invention, combination of implementation modes and equivalent substitution within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for constructing a neural network target detection system based on FPGA acceleration, characterized in that: include: The steps of compressing the original YOLOv3-Tiny network model to obtain a compressed network model; The compressed network model and the preprocessed feature map are transmitted to obtain weight data and feature map data; A step of performing convolution calculation on the preprocessed feature map and weight data to obtain a convolution calculation result; The convolution calculation result is processed through a maximum pooling layer, an Upsample upsampling layer and a Concat layer to obtain a feature map after pooling, upsampling and splicing; A step of post-processing the pooled, upsampled and concatenated feature map using an NMS non-maximum suppression module with a FIFO structure.

2. The method for constructing a neural network target detection system based on FPGA acceleration according to claim 1, characterized in that: The original YOLOv3-Tiny network model is compressed through re-parameterized channel pruning and quantization-aware training.

3. The method for constructing a neural network target detection system based on FPGA acceleration according to claim 1, characterized in that: Use AXIDMA and MIG to transfer compressed weight data and preprocessed feature maps.

4. The method for constructing a neural network target detection system based on FPGA acceleration according to claim 1, characterized in that: Convolution calculation is performed on the preprocessed feature map and weight data based on a dual signed multiplication encapsulation design and a row convolution calculation pipeline architecture.

5. A device for constructing a neural network target detection system based on FPGA acceleration, characterized in that: include: Compress the original YOLOv3-Tiny network model to obtain the module of the compressed network model; The compressed network model and the preprocessed feature map are transmitted to obtain a module for weight data and feature map data; A module for performing convolution calculation on the preprocessed feature map and weight data to obtain a convolution calculation result; The convolution calculation results are processed through a maximum pooling layer, an Upsample upsampling layer and a Concat layer to obtain a module of a feature map after pooling, upsampling and splicing; A module that uses an NMS non-maximum suppression module with a FIFO structure to post-process the feature map that has undergone pooling, upsampling and splicing.

6. A neural network target detection method based on FPGA acceleration, characterized in that: include: The step of collecting image signals, feature map signals, weight data and intermediate calculation results of the target to be measured as preset data; The method according to claim 1 processes the preset data to obtain a target detection result.

7. A neural network target detection device based on FPGA acceleration, characterized in that: include: A module for collecting image signals, feature map signals, weight data and intermediate calculation results of the target to be measured as preset data; A module according to claim 5, which processes the preset data to obtain a target detection result.

8. A computer storage medium for storing a computer program, characterized in that: When the computer program is read, the computer executes the method of claim 1.

9. A computer, comprising a processor and a storage medium, characterized in that: When the processor reads the computer program stored in the storage medium, the computer executes the method of claim 1 .

10. A computer program product, being a computer program, characterized in that When the computer program is read, the method according to claim 1 is implemented.

Citation Information

Patent Citations

  • FPGA (Field Programmable Gate Array)-based YOLOv2-tiny neural network low-delay hardware accelerator implementation method

    CN110555516A

  • Training method, application method and training device of hybrid neural network model

    CN116362293A

  • Model compression method and device, storage medium and program product

    CN116992946A

  • Improved YOLOv8s traffic target detection method based on FPGA acceleration

    CN117237603A

  • Multi-channel image acquisition and processing system based on ZYNQ

    CN118018665A

Cited By

  • Brain-computer interface classification system and method based on FPGA acceleration

    CN121658992A

  • A brain-computer interface classification system and method based on FPGA acceleration

    CN121658992B