Neural network target detection method based on FPGA acceleration

Abstract

The application discloses a neural network target detection method based on FPGA acceleration and relates to the technical field of a convolutional neural network accelerator. To solve the technical problems that in the prior art, the accuracy of a network such as YOLO is significantly reduced, and there is high delay in the prior design of a convolutional neural network accelerator based on FPGA, the application provides a technical scheme that is a neural network target detection system construction method based on FPGA acceleration, which comprises the following steps: obtaining a compressed network model; performing data processing and transmission to obtain preprocessed feature maps and weight data; performing convolution calculation on the preprocessed feature maps and the weight data to obtain a convolution calculation result; processing the convolution calculation result through a maximum pooling layer, an Upsample up-sampling layer and a Concat layer to obtain feature maps subjected to pooling, up-sampling and splicing; and performing post-processing on the feature maps subjected to pooling, up-sampling and splicing through an NMS non-maximum suppression module with a FIFO structure. The application is suitable for application in neural network target detection.

Description

Technical Field

[0001] This relates to the technical field of convolutional neural network accelerators, specifically to an FPGA-based YOLOv3-Tiny neural network hardware accelerator implementation method, and a method for reparameterized channel pruning and quantization compression of neural networks. Background Technology

[0002] With the improvement of computing power and storage, deep learning algorithms have ushered in unprecedented development, among which Convolutional Neural Networks (CNNs) are widely used in image classification, recognition, object detection, and other fields. However, the Central Processing Unit (CPU) cannot handle such computationally intensive model inference; the Graphics Processing Unit (GPU) has ultra-large-scale computing performance, but its power consumption is difficult to meet the needs of embedded scenarios; and the hardware design of Application Specific Integrated Circuits (ASICs) suffers from problems such as long development cycles and lack of flexibility. In contrast, Field Programmable Gate Arrays (FPGAs) are flexible, can meet the characteristics of large-scale parallel computing of CNNs, and can adapt to low-power and high-energy-efficiency scenarios. In the traditional design of FPGA-based CNN accelerators, the traditional methods for implementing CNNs on FPGAs mainly involve model compression and hardware architecture optimization. For model compression, quantization is mainly used to quantize the single-precision floating-point quantization involved in the operation into low-bit-width integers, but this method has a significant decrease in accuracy for networks like YOLO. For hardware architecture, traditional accelerator designs often focus on improving performance in terms of throughput, energy efficiency, and versatility. They send the network layer by layer into the accelerator for computation and process the computation and data transmission of each layer separately in a time-division multiplexing manner. This will result in higher latency. However, for edge target acquisition systems, low latency is a key optimization priority. Summary of the Invention

[0003] To address the technical problems existing in current FPGA-based convolutional neural network accelerator designs, such as significant accuracy degradation and high latency for networks like YOLO, the present invention provides the following technical solution:

[0004] A method for constructing a neural network object detection system based on FPGA acceleration includes:

[0005] The steps to compress the original YOLOv3-Tiny network model to obtain the compressed network model;

[0006] The step of transmitting the compressed network model and the preprocessed feature map to obtain weight data and feature map data;

[0007] The step of performing convolution calculation on the preprocessed feature map and weight data to obtain the convolution calculation result;

[0008] The convolution calculation results are processed by a max pooling layer, an upsample layer, and a concat layer to obtain a feature map that has undergone pooling, upsampling, and concatenation.

[0009] The step involves post-processing the feature map that has undergone pooling, upsampling, and stitching using an NMS non-maximum suppression module with a FIFO structure.

[0010] Furthermore, a preferred implementation is provided, which compresses the original YOLOv3-Tiny network model through reparameterized channel pruning and quantized perceptual training.

[0011] Furthermore, a preferred implementation is provided, which uses AXIDMA and MIG to transmit compressed weight data and preprocessed feature maps.

[0012] Furthermore, a preferred implementation is provided, which performs convolution calculations on the preprocessed feature map and weight data based on a dual signed multiplication encapsulation design and a row convolution computation pipeline architecture.

[0013] Based on the same inventive concept, the present invention also provides an apparatus for constructing a neural network target detection system based on FPGA acceleration, comprising:

[0014] The original YOLOv3-Tiny network model is compressed to obtain the modules of the compressed network model;

[0015] The module that transmits the compressed network model and the preprocessed feature map to obtain weight data and feature map data;

[0016] A module that performs convolution calculations on the preprocessed feature maps and weight data to obtain the convolution calculation results;

[0017] The convolution calculation results are processed by a max pooling layer, an upsample layer, and a concat layer to obtain a module with a feature map that has been pooled, upsampled, and concatenated.

[0018] A module that uses an NMS non-maximum suppression module with a FIFO structure to post-process the feature map after pooling, upsampling and stitching.

[0019] Based on the same inventive concept, this invention also provides a neural network target detection method based on FPGA acceleration, comprising:

[0020] The step of collecting image signals, feature map signals, weight data, and intermediate calculation results of the target under test as preset data;

[0021] The steps of processing the preset data according to the method to obtain the target detection result.

[0022] Based on the same inventive concept, the present invention also provides an FPGA-accelerated neural network target detection device, comprising:

[0023] The module that collects image signals, feature map signals, weight data, and intermediate calculation results of the target under test as preset data;

[0024] The module that processes the preset data according to the device to obtain the target detection result.

[0025] Based on the same inventive concept, the present invention also provides a computer storage medium for storing a computer program, wherein when the computer program is read, the computer executes the method described thereon.

[0026] Based on the same inventive concept, the present invention also provides a computer, including a processor and a storage medium, wherein when the processor reads a computer program stored in the storage medium, the computer executes the method described thereon.

[0027] Based on the same inventive concept, the present invention also provides a computer program product, which, when read, implements the method described.

[0028] Compared with the prior art, the advantages of the technical solution provided by the present invention are as follows:

[0029] This invention provides an FPGA-accelerated neural network object detection method. Reparameterized channel pruning (ResRep) reduces model computation by removing unimportant channels while maintaining model accuracy. This method significantly reduces the number of model parameters and computational cost, improving model efficiency. Quantization-aware training further optimizes the model's inference performance on FPGAs by adapting the model to quantization errors during training. Compared to traditional network compression methods (such as simple pruning or quantization), this method better balances performance and efficiency, making it particularly suitable for resource-constrained FPGA implementations.

[0030] The FPGA-accelerated neural network object detection method provided in this invention effectively reduces data transmission latency by using AXIDMA and MIG for output multiplexing and column-buffered inter-layer pipeline design. This method accelerates the transmission of feature maps and weight data, enabling faster convolution calculations. Compared to traditional layer-by-layer data transmission methods, this inter-layer pipeline design significantly reduces startup latency and improves the overall system efficiency.

[0031] The FPGA-accelerated neural network object detection method provided by this invention utilizes the DSP48E1 for dual signed multiplication encapsulation, improving the efficiency of multiplication operations and reducing FPGA resource consumption. The row convolution computation pipeline architecture further optimizes the speed of convolution computation, making the entire convolution process more efficient. Compared with the traditional single multiplication operation method, this design significantly improves the throughput and resource utilization of convolution computation.

[0032] The FPGA-accelerated neural network object detection method provided in this invention improves the model's adaptability by flexibly implementing max pooling layers of different sizes and strides through parameterized design. The upsampling layer designed using the nearest neighbor upsampling method simplifies computation while maintaining accuracy. The design of the Concat layer ensures computational efficiency during feature map stitching. These optimized design features make the model's deployment on the FPGA more flexible and efficient, exhibiting higher adaptability and efficiency compared to traditional fixed-parameter designs.

[0033] This invention provides an FPGA-accelerated neural network target detection method that employs Non-Maximum Suppression (NMS) to improve the accuracy of target detection results by filtering detection boxes and removing redundant low-confidence detection boxes. The FIFO structure design reduces off-chip data access, further enhancing the system's real-time performance. Compared to traditional post-processing methods, this design more effectively filters invalid detection boxes, thereby improving the accuracy and reliability of the detection results.

[0034] The FPGA-accelerated neural network object detection method provided in this invention, compared to traditional pruning and quantization methods, employs reparameterized channel pruning and quantization-aware training, balancing model performance and efficiency, making it particularly suitable for FPGA implementation. Output multiplexing and column buffer inter-layer pipeline design significantly reduce data transmission latency, a level of efficiency typically unattainable by traditional methods. Dual signed multiplication encapsulation and row convolution computation pipeline architecture improve computational speed and resource utilization, areas where traditional methods are less efficient. Parameterized max-pooling layers and flexible upsampling and stitching designs enhance system adaptability and efficiency, areas where traditional methods are often fixed and inflexible. The design of an NMS non-maximum suppression module with a FIFO structure optimizes the accuracy and real-time performance of detection results, areas where traditional methods fall short in processing efficiency and accuracy.

[0035] The FPGA-accelerated neural network target detection method provided by this invention solves several key technical problems in the application of neural network target detection in FPGA, and significantly improves the efficiency, accuracy and adaptability of the system.

[0036] The FPGA-accelerated neural network target detection method provided by this invention is suitable for application in neural network target detection. Attached Figure Description

[0037] Figure 1 This is a schematic diagram of the system architecture for a neural network object detection method based on FPGA acceleration.

[0038] Figure 2 This is a schematic diagram of a convolution parallel multiply-accumulate unit;

[0039] Figure 3 This is a schematic diagram of the overall post-processing workflow;

[0040] Figure 4 This is a schematic diagram of the post-processing hardware architecture;

[0041] Figure 5 This is a schematic diagram of the model compression process;

[0042] Figure 6 This is a schematic diagram of a column-based interlayer pipeline.

[0043] Figure 7 This is a diagram illustrating the order of row convolution calculations;

[0044] Figure 8 This is a schematic diagram of a basic pooling unit;

[0045] Figure 9 This is a schematic diagram of the hardware quantization process. Detailed Implementation

[0046] To make the advantages and benefits of the technical solution provided by the present invention clearer, the technical solution provided by the present invention will now be described in further detail with reference to the accompanying drawings, specifically:

[0047] Implementation Method 1: This implementation method provides a method for constructing a neural network target detection system based on FPGA acceleration, including:

[0048] The steps to compress the original YOLOv3-Tiny network model to obtain the compressed network model;

[0049] The step of transmitting the compressed network model and the preprocessed feature map to obtain weight data and feature map data;

[0050] The step of performing convolution calculation on the preprocessed feature map and weight data to obtain the convolution calculation result;

[0051] The convolution calculation results are processed by a max pooling layer, an upsample layer, and a concat layer to obtain a feature map that has undergone pooling, upsampling, and concatenation.

[0052] The step involves post-processing the feature map that has undergone pooling, upsampling, and stitching using an NMS non-maximum suppression module with a FIFO structure.

[0053] Implementation Method 2: This implementation method further defines the FPGA-accelerated neural network target detection system construction method provided in Implementation Method 1. It compresses the original YOLOv3-Tiny network model through reparameterized channel pruning and quantized perceptual training.

[0054] Implementation Method 3: This implementation method further defines the FPGA-accelerated neural network target detection system construction method provided in Implementation Method 1. It uses AXIDMA and MIG to transmit compressed weight data and preprocessed feature maps.

[0055] Implementation Method 4: This implementation method further defines the FPGA-accelerated neural network target detection system construction method provided in Implementation Method 1. It performs convolution calculations on the preprocessed feature maps and weight data based on a dual signed multiplication encapsulation design and a row convolution computation pipeline architecture.

[0056] Implementation Method 5: This implementation method provides a device for constructing a neural network target detection system based on FPGA acceleration, including:

[0057] The original YOLOv3-Tiny network model is compressed to obtain the modules of the compressed network model;

[0058] The module that transmits the compressed network model and the preprocessed feature map to obtain weight data and feature map data;

[0059] A module that performs convolution calculations on the preprocessed feature maps and weight data to obtain the convolution calculation results;

[0060] The convolution calculation results are processed by a max pooling layer, an upsample layer, and a concat layer to obtain a module with a feature map that has been pooled, upsampled, and concatenated.

[0061] A module that uses an NMS non-maximum suppression module with a FIFO structure to post-process the feature map after pooling, upsampling and stitching.

[0062] Implementation Method Six: This implementation method provides a neural network target detection method based on FPGA acceleration, including:

[0063] The step of collecting image signals, feature map signals, weight data, and intermediate calculation results of the target under test as preset data;

[0064] The step of processing the preset data according to the method provided in Embodiment 1 to obtain the target detection result.

[0065] Specifically: This implementation method includes:

[0066] Step 1: Model Compression

[0067] Input: Original YOLOv3-Tiny network model

[0068] Reparameterized channel pruning: Using the ResRep method to prune some channels of the CNN, reducing computational cost while maintaining performance. Quantization-aware training: Adapting the model to quantization errors during training improves its inference performance.

[0069] Output: The compressed YOLOv3-Tiny network model, with fewer parameters but almost no change in accuracy.

[0070] Step 2: Data Processing and Transmission

[0071] Input: Compressed network model

[0072] Output multiplexing: Feature map transmission and weight data writing are performed via AXIDMA and MIG.

[0073] Inter-column buffer pipeline: The design optimizes both row-parallel and column-buffered inter-column pipelines to reduce startup latency.

[0074] Output: Preprocessed and transmitted feature maps and weight data.

[0075] Step 3: Convolutional Unit Design

[0076] Input: Preprocessed feature maps and weight data

[0077] Dual signed multiplication encapsulation: Utilizing the DSP48E1 for dual signed multiplication encapsulation improves multiplication efficiency. Row convolution computation pipeline architecture: A computation pipeline architecture based on row convolution is designed to improve computation speed.

[0078] Output: Feature map after convolution calculation.

[0079] Step 4: Subsidiary Layer Unit Design

[0080] Input: Feature map after convolution calculation

[0081] Max pooling layer: Parametrically designed max pooling layer to achieve maximum pooling of the left column, right column and overall maximum value.

[0082] Upsample layer: The nearest neighbor upsampling method is used to design the upsampling layer.

[0083] Concat layer: Designs the concatenation of input feature maps to maintain pipelined computation.

[0084] Output: Feature map after processing by the auxiliary layer.

[0085] Step 5: Post-processing unit design

[0086] Input: Feature map after processing by the secondary layer

[0087] Non-maximum suppression (NMS): Decodes the output of the convolutional network and filters the detection boxes.

[0088] FIFO structure design: reduces access to external data and filters low-confidence detection frames.

[0089] Output: Final target detection results, including confidence score and bounding box location information.

[0090] in,

[0091] The signals collected include:

[0092] Image signal: This is usually raw image data acquired from a camera or other imaging device.

[0093] Feature map signal: Multi-layer feature maps extracted by convolutional neural networks (CNN), each layer of feature map contains different feature information.

[0094] Weight data: The weights of the convolutional kernels in each layer of a convolutional neural network. These weights are optimized during training and stored in the model.

[0095] Intermediate calculation results: Intermediate results generated during convolution calculation, pooling, upsampling, and splicing.

[0096] The final output results are:

[0097] Bounding boxes: These identify the position and size of a target object in an image, and are usually represented by a rectangular box.

[0098] Class Labels: The classification results of objects within the detection box, such as people, vehicles, animals, etc.

[0099] Confidence Scores: The probability that the target object corresponding to each detection box belongs to a certain category, representing the confidence level of the detection result.

[0100] Implementation Method Seven: This implementation method provides a neural network target detection device based on FPGA acceleration, including:

[0101] The module that collects image signals, feature map signals, weight data, and intermediate calculation results of the target under test as preset data;

[0102] The module that processes the preset data according to the device provided in Embodiment 5 to obtain the target detection result.

[0103] Implementation Method 8: This implementation method provides a computer storage medium for storing a computer program. When the computer program is read, the computer executes the method provided in Implementation Method 1.

[0104] Implementation Method Nine: This implementation method provides a computer, including a processor and a storage medium. When the processor reads a computer program stored in the storage medium, the computer executes the method provided in Implementation Method One.

[0105] Implementation Method 10: This implementation method provides a computer program product. As a computer program, when the computer program is read, it implements the method provided in Implementation Method 1.

[0106] Implementation Method Eleven: Combination Figure 1-9 This embodiment describes the technical solution provided above in further detail through specific examples. Specifically:

[0107] Model compression process:

[0108] Traditional pruning paradigms penalize parameters to produce structured sparsity, which may suppress parameters necessary for memorization. To further improve the pruning scheme, a ResRep (Re-parameterization and Gradient Resetting) method, combining reparameterization and gradient resetting, is used. This reduces computation by pruning only a portion of the CNN's channels while preserving model performance as much as possible. Each target convolutional layer and its subsequent batch normalization (BN) layers are reparameterized into a convolution-BN-compactor structure.

[0109] The compressor is initialized as an identity matrix, where, and Q' is the reparameterized convolution kernel and bias, and Q′ is the compressor.

[0110] This process can be expressed by the formula as follows:

[0111]

[0112] During training, a new gradient reset strategy is introduced:

[0113] The gradient update process for the original part is as follows:

[0114]

[0115] During training, standard gradient descent is applied to the original part, while a gradient reset strategy is applied to the compressor part, introducing a binary m-mask to adjust the gradient, thereby decoupling "memory" and "forgetting".

[0116]

[0117] Quantization-aware training, through backpropagation during training, gradually adapts the model parameters to the quantization error, resulting in better inference performance of the quantized model. After training, the model's weights and activation values ​​are quantized into actual low-bit-width integers.

[0118] Mean Average Precision (mAP) is an evaluation metric for multiple classes. It is calculated by first determining the AP for each class, and then calculating the average AP across all classes. The mAP value is used to assess the average accuracy of an algorithm across all classes. The calculation principle of mAP is as follows:

[0119]

[0120] Where N class It is the number of detection categories in the object detection task; AP i It is the average of the precision corresponding to the recall for each category.

[0121] Storage space compression ratio Mrr is mainly used to evaluate the degree of compression of the target model's storage space by the pruning algorithm. The unit is MB (Million Byte), and its calculation principle is as follows:

[0122]

[0123] Where M origin M is the storage space of the model before pruning. remain It is the storage space for the pruned model.

[0124] The principle behind the computational compression ratio Orr calculation is as follows:

[0125]

[0126] Among them Ops origin Ops are the number of convolution operations in the model before pruning. remain These are the convolution operation numbers for the pruned model.

[0127] Interlayer pipeline design for output multiplexing and column buffering:

[0128] The system architecture of this embodiment is as follows: Figure 1 As shown, the system mainly consists of an external data path and cascaded processing units (Process Engines, PEs) and post-processing modules. The external data path comprises AXIDMA (Advanced Dextensible Interface Direct Memory Access) and MIG (Memory Interface Generator). AXIDMA uses the AXI interface to transfer feature maps and write weight data to the ARM processor on the PS side of the ZYNQ7035 host, writing it to the DDR3 (Double Data Rate 3) memory mounted on the PL side. The off-chip memory controller controls the distribution of weights to each convolutional layer. The MIG accesses off-chip DRAM via the DDR3 interface for weight transfer. The PEs are the processing units for each convolutional layer, responsible for convolution, pooling, and activation operations, and are the core of the CNN accelerator. Each PE mainly consists of buffer units such as Col_buffer and Wt_buffer, a Layer Controller, and computation units such as CONV, POOL, and RELU. Col_buffer is responsible for buffering the input of each convolutional layer; Wt_Buffer is responsible for buffering the weights; Layer Controller is responsible for scheduling the entire PE module; CONV, POOL, and Leaky are responsible for convolution, pooling, and activation operations, respectively.

[0129] During initialization, trained weights are preloaded from the ARM side to DRAM via the AXI bus. During inference, the input feature map is first fed column-by-column into the input buffer via AXI. Then, it is processed step-by-step by the PE in the pipeline. After the output of the convolutional network is obtained, it undergoes a pipeline-accelerated Detect layer and NMS (Non-Maximum Suppression) post-processing. The final computation result is then sent to the PC for post-processing via AXIDMA. Specifically, to further reduce pipeline startup latency, padding in the first and last column computations can be skipped. For a 3×3 convolutional layer, the padding skip technique can reduce startup latency by 1 / 3. The DSP in the PE utilizes double signed multiplication encapsulation technology to double the convolution computation speed and shorten computation latency. The parameterized design and adaptive addition tree design allow the PE to be packaged as IP, supporting the computation of convolutions of different sizes, and can also be configured with different degrees of parallelism to balance pipeline time.

[0130] Convolutional unit design:

[0131] The commonly used DSP model in FPGAs is the DSP48E1, a 25×18-bit multiplier. Its two multipliers are B (maximum 18 bits wide) and the lower 25 bits of the result of adding A and D. The output is 48 bits (the high 5 bits are the sign extension bits, and the low 43 bits are the data bits). During the multiplication operation in neural network inference, the weights and feature maps are quantized into 8-bit signed numbers. Since the DSP's input bit width can accommodate the concatenation of two signed numbers, the DSP can be packaged for dual signed number multiplication, improving its efficiency and reducing resource consumption.

[0132] In a DSP package with double signed multiplication, assuming B = {bn-1, ..., b1, b0} is an n-bit signed integer and C is an n-bit unsigned integer, then the value of B*C is:

[0133]

[0134] Since 2n-1 = 2n - 2n-1, it can be further expressed as:

[0135]

[0136] The result of the multiplication is divided into two parts. The first part is the result of the unsigned multiplication of B and C when B is considered unsigned. The second part is the correction term. If bn-1 is 1, that is, B is negative, the result of the unsigned multiplication of B and C minus the result of C multiplied by 2n is equal to the result of the signed multiplication of B and C. Here, C multiplied by 2n is achieved through bit shifting. When B is positive, bn-1 is 0, and the result of the signed multiplication of B and C is equal to the result of the unsigned multiplication of B and C.

[0137] Because convolution involves a large number of multiply-accumulate (MAC) operations, and FPGA on-chip storage and computing resources are limited, it is necessary to rationally design the parallelism of data multiplication and accumulation operations during convolution to improve computation speed. Therefore, for each layer of convolution, only K layers are cached on-chip. i The group convolution kernel (due to the use of the Ping-Pong Buffer, the actual memory size will be designed to be 2K) i In each cycle, the convolution processing unit fetches n inputs from the input buffer and K weights from the weight buffer. i ×n weights, using K i ×n÷2 DSPs, calculate K i An intermediate result. For example... Figure 2 As shown, the computation process of the parallel multiply-accumulate unit of convolution is illustrated. In the example, 'a' represents the n inputs taken from the input feature map buffer, and 'b' and 'c' represent the 2×n weights taken from the weight buffer. The result of multiplication is passed through an addition tree to obtain the intermediate result each time and then accumulated. After traversing all input channels N, the final convolution output is obtained.

[0138] This implementation design employs a row-based convolutional computational pipeline architecture, allowing computation using the aforementioned method regardless of the input feature map and kernel size. In addition to parallel processing of the channels, full parallel processing is also performed within the convolution kernel, eliminating the need for intermediate feature map buffers. This saves FPGA resources while ensuring that the computation time for each layer is equal to the time required to retrieve all data from the input buffer, thus optimizing computation time. Table 3 shows the parameters for the input-output parallelism of each convolutional layer.

[0139] Table 3 Hardware configuration parameters for each convolutional layer

[0140] CONV0 416×416×3 2×4 36 6.92 CONV1 208×208×16 4×4 72 6.92 CONV2 104×104×32 4×4 72 6.92 CONV3 52×52×64 4×4 72 6.92 CONV4 26×26×125 4×4 72 6.38 CONV5 13×13×236 2×4 36 6.18 CONV6 13×13×247 2×4 36 4.40 CONV7 13×13×166 2×2 2 3.30 CONV8 13×13×93 1×2 9 4.40 CONV9 (Output Layer) 13×13×111 1×2 1 3.56 CONV10 13×13×93 1×2 1 1.89 CONV11 26×26×259 2×2 72 7.25 CONV12 (Output Layer) 26×26×130 1×2 1 1.67 total 482

[0141] Subsidiary layer unit design:

[0142] In YOLOv3-Tiny, in addition to convolutional layers, there are also max pooling layers, concat layers, upsample layers, etc.

[0143] This design parametrically designs the max pooling layer. Its main parameter is the convolution kernel parallel parameter *m*. Based on the number of *m*, the basic pooling unit is expanded *m* times and connected to the corresponding convolution kernel. Since the convolution operation is performed column-wise, the pooling process is also divided into three steps column-wise: first, the maximum value of the left column is obtained; then, the maximum value of the right column is obtained; finally, the overall maximum value is obtained as the pooling result.

[0144] For the Upsample layer, since the nearest neighbor upsampling method is used, for an upsampling layer with an upsampling kernel size of n, it is only necessary to buffer the parallel convolution results through FIFO and keep the corresponding output for n cycles.

[0145] For the input of the Concat layer, the column outputs of the i-th layer and the (i+j)-th layer (j>1 and an integer) need to be concatenated before being sent to the next convolutional layer for computation. Since pipelined computation needs to be maintained, and as can be seen from the previously mentioned storage schemes for the input feature maps of each layer, when a single column feature map is transmitted to the (i+j)-th layer, the output of the i-th layer has already been overwritten in the feature map cache of the (i+1)-th layer, making concat impossible. To avoid consuming additional bandwidth due to data exchange with external storage, which would decrease computation speed, it is necessary to investigate how many columns of data from the i-th layer's output need to be pre-cached for correct computation after passing through multiple layers. Based on the column-based pipeline structure, assuming that the path traversed by the Concat layer includes convolutional layers, pooling layers, and upsampling layers, the number of columns required to be cached after passing through the convolutional layers is:

[0146] col conv =k c +1-p c +s c (n-1) (1-9)

[0147] Where k c p is the kernel size. c s is the padding number of the convolution kernel. c is the stride of the convolution kernel.

[0148] The number of cache columns required after the pooling layer is:

[0149] col pool =k p +s p (n-1) (1-10)

[0150] Where k p For pooling size, s p This represents the moving step size of the pooling kernel.

[0151] The number of columns that need to be cached after the upsampling layer is:

[0152]

[0153] Where k u This is the upsampling size.

[0154] Based on the above formula, it can be concluded that in order to maintain correct column-based pipeline computation, the YOLOv3-Tiny Concat layer needs to cache 16 columns for the 8th convolutional layer.

[0155] Post-processing unit design:

[0156] YOLOv3-Tiny has two detection layers responsible for decoding the output of the convolutional network into confidence scores and bounding box location information. Non-maximum suppression (NMS) is applied to the outputs of the two detection layers to filter out other bounding boxes that are close to the location with the highest confidence score, retaining those with little or no overlap as the other target boxes. The overall process is as follows: Figure 3 As shown.

[0157] The proposed NMS hardware unit mainly consists of a maximum prediction frame buffer and an IOU calculation unit, as follows: Figure 4 As shown, the NMS unit is connected to the position calculation unit and the confidence calculation unit through two channels, using a FIFO structure to reduce external data access. A threshold unit is also designed to filter out boxes with low confidence scores, thereby reducing the workload of the NMS unit.

[0158] This implementation consists of two main parts: a lightweight design for the object detection network and an FPGA accelerator design. This implementation uses YOLOv3-tiny as the object detection algorithm. During training on the VOC dataset, a lightweight method is employed, utilizing QAT quantization, reparameterized channel pruning, and BN layer fusion to significantly reduce the number of model parameters while maintaining detection accuracy. In the FPGA implementation, this implementation employs a column-buffered inter-layer pipeline design, significantly reducing latency for network inference acceleration. A convolutional module with configurable kernel parameters is designed, suitable for convolutional kernels of various sizes and strides, improving the versatility of hardware deployment. A pooling module supporting different strides and a Concat module supporting inter-layer pipelines are designed, effectively solving the storage problem of inter-layer data association based on the column-buffered module. A quantization module based on PoT quantization, post-processing modules for detect and NMS, and an OV5640 camera configured on the ZYNQ PS end for real-time image acquisition, along with a HDMI display module, display the video processed by the object detection network in real time.

[0159] Implementation Method Twelve, Combination Figure 1-9This embodiment describes the advantages of the technical solution provided above through specific examples. Specifically:

[0160] Example of model compression process:

[0161] This implementation optimizes model compression, significantly reducing the number of parameters in YOLOv3-Tiny. The overall model compression process is as follows: Figure 5 As shown: ① The importance of each weight in the network is calculated and ranked using a sparsity approach. ② A pruning threshold is calculated, and the ResRep (Re-parameterization and Gradient Resetting) method, which combines reparameterization and gradient resetting, is used to reparameterize each target convolutional layer and its subsequent BN layers into a convolution-BN-compressor structure, pruning some channels to reduce computation while maintaining model performance as much as possible. ③ QAT quantization is used during training to fine-tune the quantized network, allowing the model parameters to gradually adapt to the quantization error, resulting in better performance of the quantized model during inference. ④ After repeating the sparsity training, channel pruning, and QAT quantization operations multiple times, a quantized compressed network is finally obtained. The performance comparison between the compressed model and the original model is shown in Table 1, and the convolutional layer parameters and computational cost are shown in Table 2.

[0162] Table 1 Performance comparison between the compressed model and the original model

[0163] none 63.4 33.43 5.49 - - ResRep 62.8 6.47 2.58 80.65 53.01

[0164] Table 2. Parameters and computational cost of convolutional layers after pruning.

[0165]

[0166] Example of interlayer assembly line design:

[0167] Column-based pipeline architectures can significantly reduce latency. Furthermore, input feature maps and intermediate results do not need to access DRAM. Based on a comparison of these multiplexing methods and combined with column-based pipeline optimization, output multiplexing can be used to organize the data flow. Specific implementation details are as follows: Figure 6 As shown. The entire process is divided into four stages: ① The inputs of each layer are buffered column by column in the input buffer. First, the inputs and their corresponding Ts are loaded sequentially along the channel direction. m Multiplying the group weights yields T on the output feature map. m 1. Output; 2. Slide the convolution window along the height direction and repeat step 1 until all convolutions in one column are calculated; 3. Replace with another T m Set up the convolution kernel, return the convolution window to the top, repeat the first two steps, and calculate the T-value on the output feature map. m④ Slide the convolution window to the right to enter the next column of convolution calculation, and repeat steps 1-3.

[0168] Convolutional unit design:

[0169] This implementation design employs a computational pipeline architecture based on row convolution, performing parallel processing on both the channels and the kernel of the convolution, and eliminating the need for an intermediate feature map row buffer. The specific computation order is as follows: Figure 7 As shown, the entire process is divided into four stages: ① For a certain K×K convolution calculation, the corresponding N values ​​are simultaneously retrieved from the K input buffers. n Feature map FM with input parallelism n Every time T elapses, iterate through all the input channels N in a single row and K columns; then retrieve N from the weight buffer. n Input parallelism, M m ① A K×K kernel of weights with parallel output. ② Divide the kernel weights into K groups according to the number of rows, and group them with the input feature map FM. n The data is fed into the aforementioned convolution parallel multiply-accumulate unit for computation (according to parallelism M). m (The calculations are performed pairwise in the encapsulated DSP), obtaining intermediate results for each iteration, which are stored in K output registers D. This process continues until all input channels N have been traversed, yielding the calculation results for the convolution kernel at each corresponding position in each row. ③ After KT cycles, each of the K output registers has a corresponding calculated value. D1 obtains the final output feature values ​​corresponding to a set of m output channels, while D2 obtains the K-1 convolution sums of the output feature values ​​at the corresponding position in the second row, and so on. k ④ Obtain the convolution sum of the output feature values ​​at the corresponding position of the Kth row. ④ Each time the input feature map moves down one row (equivalent to the convolution kernel moving down), the weights return to the starting position, and the same steps are repeated. Only T cycles are needed to obtain the final output feature value of the second row in D2. After that, the final output feature value of the corresponding row can be obtained once every T cycles. That is, there is no delay in the calculation time relative to the data reading time.

[0170] Example of auxiliary layer unit design

[0171] This embodiment designs a highly fluidized pooling unit. This design greatly improves the versatility of the pooling layer, enabling it to handle both cases where the pooling size and pooling step size are equal, and cases where the pooling size and pooling step size are unequal, achieving highly fluidized pooling. A detailed schematic diagram is shown below. Figure 8The process consists of the following four steps: ① Assuming the pooling kernel size is n, during left column pooling, Reg1 stores the first received value. When the convolution window slides down to obtain the second output of the left column, it is compared with the first received value, and the comparison result is stored in Reg2. ② The convolution window slides down again to obtain the third output of the left column. Reg1 compares the second and third received values, and Reg2 compares the maximum value of the first two outputs with the third received value. ③ This process continues, and after n valid inputs, Reg(n) can compare the maximum value within the pooling range of n, with a comparison step of 1. ④ For a pooling unit with a pooling size of n, the output valid signal of Reg(n) can be controlled to obtain single-column pooling results with a pooling step of 1 to n. During cross-column pooling, a column buffer is used to cache the pooling result of the previous column and compare it with the single-column pooling result of this column, ultimately obtaining the pipelined pooling output.

[0172] Example of hardware quantization process:

[0173] Hardware quantization process as follows Figure 9 As shown: ① The inference result after using 8-bit quantization is only slightly lower than that after 32-bit floating-point inference. Quantizing the 32-bit floating-point weights after training yields 8-bit fixed-point weights. ② After convolution with the 8-bit fixed-point feature data, the data is further expanded. To prevent data overflow, the bit width is extended to 16 bits. ③ The intermediate output data needs to be accumulated through multiple channels, further expanding the output data bit width to 32 bits. ④ Finally, bias data is added. Based on the scaling ratio of the feature data in this layer, shift quantization is used to obtain the 8-bit feature data required for the next layer's calculation. ⑤ When some shift-quantized data exceeds the 8-bit bit width representation range, a truncation method can be used to limit it to the upper or lower bound of INT8. During shift quantization, there are two cases depending on whether the feature data is positive or negative: when the feature data is negative, the negative value is directly set to zero according to the ReLU activation function. When the feature data is positive, the rounding direction is determined by judging whether the integer part after shifting is even and whether the decimal part after shifting is greater than 0.5, according to the Round approximation function.

[0174] The above description of several specific embodiments further details the technical solution provided by the present invention in order to highlight the advantages and benefits of the technical solution provided by the present invention. However, the above-described specific embodiments are not intended to limit the present invention. Any reasonable modifications and improvements to the present invention, combinations of embodiments, and equivalent substitutions based on the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for constructing a neural network target detection system based on FPGA acceleration, characterized in that, include: The original YOLOv3-Tiny network model is compressed by reparameterizing the target convolutional layer and its subsequent BN layers into a convolution-BN-compressor structure. Channel pruning is performed using a combination of reparameterization and gradient reset ResRep method. The channel-pruned network model is then subjected to quantization-aware training to obtain the compressed network model. An external data path consisting of AXI DMA and MIG is constructed. AXI DMA transmits feature maps to the PS-side ARM processor of ZYNQ7035 through the AXI interface and writes weight data to the DDR3 mounted on the PL side. MIG accesses the off-chip DRAM through the DDR3 interface for weight transmission. During the inference stage, the inputs of each layer are buffered column-wise in the input buffer. Output multiplexing and column-buffered inter-layer pipelines are used to enable the cascaded processing units to process each layer step by step and provide the output feature map of the current layer directly to the column buffer of the next layer. Convolution calculations are performed on the preprocessed feature maps and weight data. The DSP48E1 is used for double signed multiplication encapsulation, and a row convolution-based computational pipeline architecture is adopted to perform parallel processing on the channels and parallel processing on the convolution kernel. The convolution calculation results are processed by max pooling layer, upsample layer and concat layer to obtain feature maps that have been pooled, upsampled and concatenated; The feature map that has been pooled, upsampled and stitched is post-processed using an NMS (Non-Maximum Suppression) hardware unit located on the FPGA side. The NMS hardware unit includes a maximum prediction box buffer, an IOU (Intersection over Union) calculation unit and a threshold unit. The NMS hardware unit is connected to the position calculation unit and the confidence calculation unit through two channels respectively, and uses a FIFO structure to buffer the post-processed data.

2. The method for constructing a neural network target detection system based on FPGA acceleration according to claim 1, characterized in that, The compression of the original YOLOv3-Tiny network model includes: calculating and ranking the importance of each weight in the network using a sparsity method, determining the pruning threshold, pruning some channels using the ResRep method, and performing QAT quantization during training; repeating the sparsity training, channel pruning, and QAT quantization to obtain the quantized compressed network.

3. The method for constructing a neural network target detection system based on FPGA acceleration according to claim 1, characterized in that, Each layer of processing units includes Col_buffer, Wt_Buffer, Layer Controller, and CONV, POOL, and RELU calculation units. Col_buffer is used to buffer the input of the current convolutional layer, Wt_Buffer is used to buffer the weights, and Layer Controller is used to schedule the processing units. During the initialization phase, the trained weights are preloaded from the ARM side to DRAM via the AXI bus. During the inference phase, the input feature map is sent column by column to the input buffer and then processed step by step by the cascaded processing units.

4. The method for constructing a neural network target detection system based on FPGA acceleration according to claim 1, characterized in that, The computational pipeline architecture based on row convolution performs fully parallel processing on the channels and within the convolution kernel, eliminating the need for intermediate feature map row buffers; the NMS non-maximum suppression hardware unit utilizes a FIFO structure to reduce external data access and uses the threshold unit to filter detection boxes with confidence levels below a threshold.

5. A device for constructing a neural network target detection system based on FPGA acceleration, characterized in that, include: The model compression module is used to reparameterize the target convolutional layer and its subsequent BN layer in the original YOLOv3-Tiny network model into a convolution-BN-compressor structure. It uses the reparameterization method and the ResRep method combined with gradient reset to perform channel pruning, and performs quantization-aware training on the channel-pruned network model. The data transmission and inter-layer pipeline module includes AXI DMA, MIG, and cascaded processing units for each layer. AXI DMA transmits feature maps to the PS-side ARM processor of ZYNQ7035 through the AXI interface and writes weight data to the DDR3 mounted on the PL side. MIG accesses the off-chip DRAM through the DDR3 interface for weight transmission. The inputs of each layer are buffered column-wise in the input buffer, and the output feature map of the current layer is directly provided to the column buffer of the next layer. The convolution computation module is used to encapsulate double signed number multiplication using DSP48E1 and to perform convolution computation on preprocessed feature maps and weight data using a row convolution-based computational pipeline architecture. The auxiliary layer processing module is used to process the convolution calculation results through the max pooling layer, the upsample layer, and the concat layer; The NMS post-processing module, located on the FPGA side, includes a maximum prediction frame buffer, an IOU calculation unit, and a threshold unit. It is connected to the position calculation unit and the confidence calculation unit through two channels, respectively, and uses a FIFO structure to buffer the post-processing data.

6. A neural network target detection method based on FPGA acceleration, characterized in that, include: Collect image signals, feature map signals, weight data, and intermediate calculation results of the target to be tested as preset data; The preset data is processed according to the method described in claim 1 to obtain the target detection result.

7. A neural network target detection device based on FPGA acceleration, characterized in that, include: The module that collects image signals, feature map signals, weight data, and intermediate calculation results of the target under test as preset data; And a module that processes the preset data to obtain the target detection result using the device according to claim 5.

8. A computer storage medium for storing computer programs, characterized in that, When the computer program is read, the computer executes the method of claim 1.

9. A computer, comprising a processor and a storage medium, characterized in that, When the processor reads the computer program stored in the storage medium, the computer executes the method of claim 1.

10. A computer program product, characterized in that, When the computer program in the computer program product is read, the method of claim 1 is implemented.

Images