FPGA-based YOLOv5 hardware acceleration method

Abstract

The invention relates to the technical field of FPGAs (Field Programmable Gate Array), in particular to an FPGA-based YOLOv5 hardware acceleration method, which comprises the following steps of: constructing an FPGA to accelerate a YOLOv5 network; performing signed integer quantization on the floating-point number of the acceleration operation; a three-input two-stage compression Compressor adder is constructed, and the addition of the acceleration operation is calculated; the multiplication of the acceleration operation is calculated; carrying out parallel partitioning on the Compressor adder, respectively calculating the calculation time delays of multiplication and addition of the input feature map, and carrying out parallel calculation of a ternary addition tree; the two IFMs are used for processing an input feature map, and the one OFW is used for processing an output feature map. The problem that when YOLOv5 is deployed on an edge device and an embedded device through an FPGA, the reasoning speed is low is solved.

Description

Technical Field

[0001] The present invention relates to the field of FPGA technology, and in particular to a YOLOv5 hardware acceleration method based on FPGA. Background Art

[0002] YOLOv5 is used for real-time target detection and has good detection speed and accuracy. More than 90% of YOLOv5's calculations are convolution calculations, and FPGA has the ability of large-scale parallel computing, which can perform multiple convolution operations at the same time, greatly improving computing efficiency. Due to YOLOv5's lightweight model structure, efficient convolution operations, and FPGA's parallel computing capabilities and customizability, YOLOv5 is very suitable for acceleration on FPGA.

[0003] Chen Chen et al. designed and implemented a deep learning target detection system based on FPGA, designed and implemented a YOLOv2 accelerator based on the Zynq7000FPGA heterogeneous platform, and adopted optimization strategies such as parameter reordering and multi-channel data transmission to design and implement an FPGA-based SIMD convolutional neural network accelerator architecture to achieve inference acceleration of the YOLOv2 network; and hardware acceleration of the YOLOv5s network model based on aerospace-grade FPGA; their FPGA accelerator design for target detection networks has the following deficiencies:

[0004] 1. Using HLS (High-level synthesis) language to write hardware circuits cannot design hardware circuits at a deeper level;

[0005] 2. Only 2-input adders are used and the optimization of the adder tree only stays at the binary tree structure, and the optimization focus is only on multiplication. Summary of the invention

[0006] In view of the shortcomings of the existing methods, the present invention solves the problem of slow inference speed when YOLOv5 is deployed on edge devices and embedded devices through FPGA; the present invention uses Verilog HDL language to implement logic circuits to accelerate the YOLOv5 network, so that the circuit can have maximum design freedom and can perform fine-grained optimization; and designs adders, addition trees and ATB units to solve the inefficiency problem of addition in convolution operations.

[0007] The technical solution adopted by the present invention is: the YOLOv5 hardware acceleration method based on FPGA comprises the following steps:

[0008] Step 1: Build FPGA to accelerate the YOLOv5 network;

[0009] As a preferred implementation of the present invention, the acceleration operation includes: CONV convolution, BN and ADD operations.

[0010] Step 2: quantize the floating point number of the acceleration operation into a signed integer;

[0011] As a preferred implementation of the present invention, quantization is to convert Float32 floating point numbers into INT8.

[0012] As a preferred implementation of the present invention, PostTrainning Static Quantization is used to perform quantization calculation.

[0013] Step 3: Construct a three-input two-stage compression adder to calculate the addition of the acceleration operation;

[0014] As a preferred embodiment of the present invention, the output formula of the addition of the acceleration operation is:

[0015]

[0016] in, represents XOR gate; · represents AND gate; + represents OR gate; M n-1 Represents the output value of the n-1th first-stage compressor; A i , B i , C i Three input values ​​of n bits.

[0017] Step 4: Calculate the multiplication of the acceleration operation;

[0018] As a preferred embodiment of the present invention, step 4 specifically includes:

[0019] Step 41, H in , W in , C in are the width, height and channel of the input feature map, T h , T w , T c Represent the width, height and channel of the feature map block, K, N b Represents the convolution size and the number of parallel calculations of input feature map block multiplications; each DSP48E2 executes a convolution operator and two feature map operators Multiplication calculation of ;

[0020] Step 42, the input feature map is divided into blocks, and each channel K×K convolution operators are calculated twice in parallel by DSP48E2; N b Parallel calculation is performed between each channel of each block of the feature map block;

[0021] Step 43: Calculate the result Send to Nb ×4 ATBs.

[0022] Step 5: Parallelize the compressor adder, calculate the multiplication and addition delays of the input feature map respectively, and perform parallel calculation of the ternary adder tree;

[0023] As a preferred embodiment of the present invention, the formula for calculating the delay of the multiplication of the input feature map is:

[0024]

[0025] Among them, f osc is the FPGA frequency; N inblock is the number of input feature map blocks; T h , T w Represents the width and height of the feature map block, N b Represents the number of parallel computations of block multiplication of the input feature map.

[0026] As a preferred embodiment of the present invention, the formula for calculating the delay of adding the input feature map is:

[0027]

[0028] Among them, T h′ , T w′ Respectively represent the width and height of the output feature map block; N a Represents the number of ATBs filled in one addition parallel calculation; N depth is the depth of the addition tree, N outblock is the number of blocks of the output feature map.

[0029] Step 6: Use 2 IFM blocks to process the input feature map, and use 1 OFW block to process the output feature map.

[0030] As a preferred embodiment of the present invention, YOLO based on FPGA v 5 Hardware acceleration system, including: a memory for storing instructions executable by a processor; a processor for executing the instructions to implement YOLO based on FPGA v 5 Hardware acceleration methods.

[0031] Beneficial effects of the present invention:

[0032] 1. Deploy the YOLOv5 network on the Xilinx XCZU15EG chip and use the Verilog HDL language to implement the logic circuit to accelerate the YOLOv5 network, so that the circuit can have the greatest design freedom and can perform fine-grained optimization;

[0033] 2. Use 3-input Compressor adders instead of the adders automatically generated by Vivado to implement addition operations in convolution calculations and tensor addition, compress the resource consumption of hardware circuits, and instantiate more adders for acceleration under the same hardware resources;

[0034] 3. The convolution calculation unit is divided into two units: multiplication calculation and addition calculation. In the addition calculation unit, a ternary tree structure is designed based on the compressor adder to reduce the accumulation calculation time of a large number of operands;

[0035] 4. Designed an input-output module with dual input cache and single output cache to reduce the delay in data transmission, and added an Addition Tree Buffer to implement the calculation of multiplication and addition units. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a flow chart of the YOLOv5 hardware acceleration method based on FPGA of the present invention;

[0037] Figure 2 It is a structural diagram of the YOLOv5 hardware accelerator based on FPGA of the present invention;

[0038] Figure 3 It is a YOLOv5 network structure diagram of the present invention;

[0039] Figure 4 It is a circuit diagram of a 3-input n-bit compressor adder of the present invention;

[0040] Figure 5 It is a schematic diagram of convolution multiplication calculation of the present invention;

[0041] Figure 6 It is a schematic diagram of convolution addition calculation of the present invention;

[0042] Figure 7 It is a schematic diagram of the input and output module of the present invention;

[0043] Figure 8 a to e represent the inference time and computing performance of the image under the YOLOv5n / s / m / l / x network respectively. DETAILED DESCRIPTION

[0044] The present invention is further described below in conjunction with the accompanying drawings and embodiments. This figure is a simplified schematic diagram, which only illustrates the basic structure of the present invention in a schematic manner, and therefore it only shows the components related to the present invention.

[0045] like Figure 1 As shown, the FPGA-based YOLOv5 hardware acceleration method includes the following steps:

[0046] Step 1: Build FPGA to accelerate the YOLOv5 network;

[0047] like Figure 3 For the YOLOv5-v6.1 version, the Focus structure is changed to a convolution operation, and the components filled with dark colors are FPGA accelerated operations, including CONV convolution, BN, and ADD operations.

[0048] The YOLOv5-v6.1 network is deployed on the FPGA chip, and the FPGA uses the Xikinx XCZU1 5EG chip.

[0049] Step 2: quantize the floating point number of the acceleration operation into a signed integer;

[0050] YOLOv5 network quantization uses the PostTrainningStatic Quantization under the Pytorch framework to replace the Fkoat32 floating-point calculations in the acceleration operations: CONV, BN, and ADD with INT8 signed integers, which improves the processing efficiency of the FPGA with a small loss of precision.

[0051] Let r be a floating point number, S be the scaling factor, g be the quantized integer, and Z be the position of the zero point after quantization. The corresponding quantization formula is:

[0052]

[0053] Where S is (r max -r min )·(q max -q min ) -1 ; Z is Round(q max r mmax ·S -1 );r max and r min The highest and lowest bits of floating point numbers; q max and q min are the highest and lowest bits of the fixed-point number; subscripts 1, 2, and 3 represent the quantized input feature map, convolution kernel, and output feature map, respectively; i represents the channel index currently processed in the convolution operation; j represents a position index in the input feature map or convolution kernel, which is used to refer to the local area of ​​the convolution calculation; k represents a position index in the output feature map after the convolution operation.

[0054] like Figure 4 ,Step 3, construct a three-input two-stage compression adder to calculate the addition of the acceleration operation;

[0055] Step three specifically includes:

[0056] A certain compressor adder AU m Divided into two levels of compression, M n and Cout n The first stage compressor FA n The first level of compression is to convert the n-bit input A i , B i , C i Input to n 3:2 compressors FA respectively n-1 Middle; Second stage compressor FA 2n M n and Cout n Compression again, because the carry value of the first level needs to be carried to the previous level, the carry of the previous level needs to be considered during the second level compression. The output formula of the adder is:

[0057]

[0058] in, represents an XOR gate; · represents an AND gate; + represents an OR gate.

[0059]

[0060] The present invention utilizes a Compressor adder to replace a traditional adder of Vivado, thereby compressing resource consumption of a hardware circuit and instantiating more adders for acceleration under the same hardware resources.

[0061] Step 4: Calculate the multiplication of the acceleration operation;

[0062] like Figure 5 , step four specifically includes:

[0063] In the quantized CONV calculation, all multiplication operations are performed in INT8 format, which means that each value is represented by an 8-bit signed integer. The Zynq Ultrascale series FPGA has a DSP48E2, which can perform 27-bit multiplication by 18-bit signed number multiplication. A single DSP48E2 can perform operations with a larger bit width, so directly using it for a single INT8 multiplication will result in a waste of resources. Mapping two INT8 multiplications to one DSP48E2 will cause the DSP48E2 to have a larger bit width in the multiplier MU. m superior.

[0064] Step 41, H in , W in , C in Represent the width, height and channel of the input feature map, T h , T w, T c Represent the width, height and channel of the feature map block, K, N b Respectively represent the convolution size and the number of parallel computations of input feature map block multiplication; N b =4, the size of K in the YOLOv5 network is 1, 3, 6; in the feature map calculation of a feature map block, each DSP48E2 can perform a convolution operator and two feature map operators Multiplication calculation of ;

[0065] Step 42: Input feature map blocks and perform K×K convolution operators twice per channel on DSP48E2 in parallel, which can process K×K×4 feature map operators; and N b Each channel of each block of the feature map block is also calculated in parallel, so N b In one multiplier cycle, N feature blocks are instantiated. b ×T c ×K×K×2 DSP48E2, processing N b ×T c ×K×K×4 feature map operators;

[0066] Step 43: Finally, calculate the result Send to N b In ×4 ATBs, in order to prevent data overflow, the output data bit width is 16 bits.

[0067]

[0068] Step 5: Parallelize the compressor adder, calculate the multiplication and addition delays of the input feature map respectively, and perform parallel calculation of the ternary adder tree;

[0069] In the addition tree, some nodes will have idle inputs. The processing method is to fill the idle inputs with 0. The input data bit width of the addition tree is 16. For each additional depth of the addition tree, the bit width increases by 2 bits, which is equivalent to a 4-fold increase in the data range. This ensures that the data of the 3-input adder does not overflow. An input channel C in =1280 feature map and 3×3 convolution, calculate an output feature map operator of 1280x3x3, then select ATB (Additiontree buffer) size of 11520; in YOLO v 5 In the network, most of the input channels and convolution sizes are less than or equal to this number. In this case, the output value can be obtained at a node in the addition tree; if the input channel and convolution size are greater than 11520, block calculation is required.

[0070] Assume that the FPGA frequency is f osc , the number of input feature map blocks is N inblock , T h , T w Represents the width and height of the feature map block, N b Represents the number of parallel calculations of block multiplication of the input feature map, then the multiplication calculation delay of the input feature map is Latency mcl The formula is:

[0071]

[0072] This latency calculation formula is used to estimate the latency of multiplication calculations in order to optimize inference performance on FPGA accelerators.

[0073] like Figure 6 , H out , W out , C out Represent the width, height and channel of the output feature map, T h′ , T w′ Represents the width and height of the output feature map block, N a Represents the number of ATBs filled in one addition parallel calculation, where N a The size is 4×N b ; According to the FPGA resources, the ternary adder tree is instantiated into 9 depths Depth1-Depth9, consuming a total of 5763 compressor adders, and can calculate a maximum of 11520 input data, which comes from the output of the multiplication calculation unit;

[0074] Assume that the FPGA frequency is f osc , the addition tree depth is N depth , the number of output feature map blocks is N outblock , the addition calculation delay of the input feature map Latency add The formula is:

[0075]

[0076]

[0077] The present invention proposes an addition calculation unit of a ternary addition tree structure constructed by a compressor adder, which reduces the depth of the tree compared to the existing binary addition tree structure, thereby greatly improving the calculation speed of the convolution.

[0078] like Figure 6 ,Step 6. Use 2 IFM blocks to process the input feature map, and use 1 OFW block to process the output feature map.

[0079] The data input and output work is implemented by the CPU scheduling the AXIDMA IP core. DMA converts the external DDR3 data into AXI-Stream streaming data through the AXI bus and inputs it into the FPGA's BRAM, or stores the data back to DDR3 through DMA. Since the read and write channels of the AXI4 master interface are independent, the read and write operations can be executed concurrently. The input feature map, convolution weights and bias are input through DMA, and the output feature map is output through DMA after the adder unit and multiplier unit are calculated. Among them, each Group implements a convolution calculation.

[0080] The convolution calculation of the present invention is to perform multiplication calculation and addition calculation separately, so Figure 7 The MU unit and AU unit in the MU are executed separately; after the MU unit completes a block calculation, when it executes the next MU block calculation, the result of the previous block will be executed by the AU, so the block calculations of MU and AU are executed in parallel in a pipelined manner; the WB unit represents the time required for the weights and biases of the input feature map and BN fusion, and the CPU unit represents the CPU processing time after the output feature map is transmitted to the DDR3 end, such as Concat and Postprocessing, which have less computational workload.

[0081] The present invention uses three memory pairs for data input and output, wherein the memory of the output feature map is one OFW; and the memory of the input feature map is two IFMs, that is, the memory of the input feature map is IFM1 and IFM2 of Group1 or IFM3 and IFM4 of Group2 and Group3; the memory of the output feature map is OFW1 of Group1 or OFW2 of Group2; in this way, when calculating a Group, the input feature map of the next Group can be transmitted in advance, wherein only the first input feature map is not transmitted in advance.

[0082] like Figure 2 , the FPGA-based YOLOv5 hardware acceleration system includes:

[0083] Figure 2The left half of the picture shows the PS side of the FPGA. The PS side obtains the image used for accelerated detection and all the CONV convolution and BN parameters of the YOLOv5 network from the SD card; the complete YOLOv5 network is run by calling the accelerator IP core on the PL side to achieve accelerated detection; the CPU on the PS side performs less computational work such as Concat, MaxPool and Post processing. For example, the pooling of the YOLOv5 network is only three times, and the pooling calculation is performed in the CPU; the PL side uses the Verilog HDL language to design the accelerator IP core, which implements a single-layer CONV convolution calculation, BN fusion and activation operations; the convolution calculation unit is divided into a multiplier MU m and Compressor Adder AU m , the AXI bus is used as the communication interface between PS and PL; the AXI bus handles the data transmission between the PS-side DDR3 and the PL-side BRAM, and the PS-side calls the PL-side by configuring the register Register; the register data is divided into three types: Command, Parameter and Address. Command is the flag that controls the execution status of the accelerator and the start and end of data reading and writing. Parameter is the input feature map width and height, number of channels, convolution kernel size, stride, padding, output feature map width and height and number of channels, precision control and other related parameters. Address is the address information of various data; on the PL side, the data stored in the BRAM is divided into 4 buffers, of which 2 buffers are IFMB1 and IFMB2 that alternately store the input feature map data and the corresponding convolution and BN fusion parameters; the other 2 buffers are an ATB that stores addition operations and an OFMB that stores output feature map data; the PS side passes the calculation data and instruction information to the PL side through the AXI bus, and the PL side passes multiple PE n The unit is divided into several blocks for calculation to achieve reasoning acceleration of the YOLOv5 network.

[0084] Experimental process:

[0085] In order to evaluate the accelerator proposed in this paper, the experiment used Xilinx XCZU15EG FPGA, which is based on the Zynq UltraScale MPSoC architecture and is equipped with a 4-core Cortex-A53 processor, a 2-core Cortex-R5 RPU and a Mali-400MP2 GPU, as well as hardware resources such as 341k LUTs, 3258 DSP units and 26.2Mbit of BRAM; in the experiment, the FPGA is connected to the external hardware platform through two DDR3 memories, each with a capacity of 4GB and 2GB; in the actual circuit, 47.33% of the FPGA's BRAM resources are used, 83.57% of the LUT resources are used, 96.20% of the DSP resources are used, 53.95% of the FF resources are used, 3.35% of the IO resources are used, and 1.73% of the BUFG resources are used; the Vivado tool estimates that the FPGA on-chip power is 13.021W, the junction temperature is 47.8℃, and the static power is 0.816W, of which the PS part accounts for 32% and the PL part accounts for 68%; the dynamic power is 12.206W, of which the PS part accounts for 61% and the PL part accounts for 39%.

[0086] Figure 8 The inference time, computing performance, and detailed information of each layer of the accelerator network are shown for a 640×640 image with an 8-bit fixed-point bit width. Figure 8 a to e represent the inference time and computing performance of the image under the YOLOv5n / s / m / l / x network respectively; Figure 8 It can be seen that the inference time of the feature layers of the first few layers and the last layer of the network is longer. This is because these feature layers have more input and output channels and larger input and output widths and heights. Therefore, when performing addition tree calculations and multiplication unit calculations, more block calculations are performed. When data is transmitted through the AXI bus for input and output, the block transmission delay is also higher. Therefore, the calculation time of these feature layers is longer.

[0087] Based on the above ideal embodiments of the present invention, the relevant staff can make various changes and modifications without departing from the technical concept of the present invention through the above description. The technical scope of the present invention is not limited to the contents of the specification, and its technical scope must be determined according to the scope of the claims.

Claims

1. The YOLOv5 hardware acceleration method based on FPGA is characterized by: The following steps are involved: Step 1: Build FPGA to accelerate the YOLOv5 network; Step 2: quantize the floating point number of the acceleration operation into a signed integer; Step 3: Construct a three-input two-stage compression adder to calculate the addition of the acceleration operation; Step 4: Calculate the multiplication of the acceleration operation; Step 5: Parallelize the compressor adder, calculate the multiplication and addition delays of the input feature map respectively, and perform parallel calculation of the ternary adder tree; Step 6: Use 2 IFM blocks to process the input feature map, and use 1 OFW block to process the output feature map.

2. The FPGA-based YOLOv5 hardware acceleration method according to claim 1, characterized in that: The output formula of the addition of the accelerated operation is: in, represents XOR gate; · represents AND gate; + represents OR gate; M n-1 Represents the output value of the n-1th first-stage compressor; A i , B i , C i Three input values ​​of n bits.

3. The YOLOv5 hardware acceleration method based on FPGA according to claim 1, characterized in that: Step 4 specifically includes: Step 41, H in , W in , C in are the width, height and channel of the input feature map, T h , T w , T c Represent the width, height and channel of the feature map block, K, N b Represents the convolution size and the number of parallel calculations of input feature map block multiplications; each DSP48E2 executes a convolution operator and two feature map operators Multiplication calculation of ; Step 42, the input feature map is divided into blocks, and each channel K×K convolution operators are calculated twice in parallel by DSP48E2; N b Parallel calculation is performed between each channel of each block of the feature map block; Step 43: Calculate the result Send to N b ×4 ATBs.

4. The FPGA-based YOLOv5 hardware acceleration method according to claim 3, characterized in that: The formula for the computational delay of the multiplication of the input feature map is: Among them, f osc is the FPGA frequency; N inblock is the number of input feature map blocks; T h , T w Represents the width and height of the feature map block, N b Represents the number of parallel computations of block multiplication of the input feature map.

5. The FPGA-based YOLOv5 hardware acceleration method according to claim 3, characterized in that: The formula for calculating the delay of adding the input feature map is: Among them, T h′ , T w′ Respectively represent the width and height of the output feature map block; N a Represents the number of ATBs filled in one addition parallel calculation; N depth is the depth of the addition tree, N outblock is the number of blocks of the output feature map.

6. The FPGA-based YOLOv5 hardware acceleration method according to claim 1, characterized in that: Accelerated operations include: CONV convolution, BN and ADD operations.

7. The FPGA-based YOLOv5 hardware acceleration method according to claim 1, characterized in that: Quantization is the process of converting Float32 floating point numbers to INT8.

8. The FPGA-based YOLOv5 hardware acceleration method according to claim 7, characterized in that: PostTrainning Static Quantization is used for quantitative calculation.

9. FPGA-based YOLOv5 hardware acceleration system, characterized by: include: a memory for storing instructions executable by a processor; A processor, configured to execute instructions to implement the FPGA-based YOLOv5 hardware acceleration method according to any one of claims 1 to 8.

Images