A target detection system based on ARM+FPGA heterogeneous platform and its hardware acceleration method

Abstract

A target detection system based on an ARM+FPGA heterogeneous platform and its hardware acceleration method. By quantizing the neural network weight data and feature map data, the original 32-bit floating-point neural network model is quantized into an 8-bit fixed-point neural network model. At the same time, a configurable neural network hardware accelerator is designed. By configuring the parameters of each module, the same hardware accelerator can be applied to calculate different convolutional layers in the neural network, thereby completing the acceleration operation of the entire neural network. In the process of convolution calculation in the convolution layer, the resource reuse of the weight and feature map cache modules is designed to reduce the frequent access to the external DDR3 memory. The control process is designed on the PS side to complete the scheduling of the entire convolution calculation process and the post-processing steps of target detection, realizing the full process design from image acquisition, target detection algorithm calculation, and detection result output display. Finally, the entire target detection system is built through software and hardware collaborative design.

Description

Technical Field

[0001] The present invention belongs to the technical field of target detection and recognition, and specifically relates to a target detection system based on an ARM+FPGA heterogeneous platform and a hardware acceleration method thereof. Background Art

[0002] Object detection technology plays an important role in various fields of social life, such as intelligent monitoring, autonomous driving, and social security. In recent years, with the rise of deep learning methods, deep learning-based object detection algorithms have become the mainstream to solve this problem. Deep learning-based object detection algorithms usually use convolutional neural networks to extract image features and complete the positioning and classification of targets. Their detection accuracy and generalization ability are greatly improved compared with traditional algorithms. However, convolutional neural networks often have a large number of parameters and calculations, as well as a variable network structure and complex calculation process. Due to the above reasons, deep learning-based object detection algorithms usually need to be deployed on high-performance computing platforms. This makes it difficult for object detection algorithms to be applied in embedded applications with limited resources, high processing speed and low power consumption. In order to meet these needs and promote the further implementation of object detection technology, FPGA (Field-Programmable Gate Array) technology has received widespread attention. Compared with other computing platforms such as CPU (Central Processing Unit / Processor) and GPU (Graphics Processing Unit), FPGA, as a flexible and programmable hardware platform, has the advantages of high parallelism, low power consumption, high computing power, and flexible design solutions, providing new possibilities for accelerating deep learning-based target detection algorithms.

[0003] However, deploying deep learning-based target detection algorithms on FPGA platforms still faces the following difficulties: (1) FPGA is not suitable for complex logic control. When calculating target detection algorithms, complex process control and data scheduling usually bring great difficulties to algorithm deployment. (2) FPGA on-chip resources are very limited. It is almost impossible to completely save the weight data of the neural network and the intermediate data generated during the calculation process on the chip. It is necessary to reasonably plan the limited resources, which also brings certain difficulties to algorithm deployment. (3) Different target detection neural network algorithms usually have different network scales and network structures. When designing FPGA target detection systems, compatibility with different network structures is also an issue that needs to be considered. Therefore, designing a target detection system based on FPGA and designing a hardware acceleration method with excellent reusability, low resource consumption and high performance to solve the above-mentioned problems and enable the network to efficiently utilize FPGA resources and computing power during calculations has important practical significance in the field of image target detection. Summary of the invention

[0004] In view of the above problems, the present invention proposes a target detection system based on an ARM+FPGA heterogeneous platform and a hardware acceleration method thereof. By quantizing the neural network weight data and feature map data, the original 32-bit floating-point neural network model is quantized into an 8-bit fixed-point neural network model, which reduces the hardware resource overhead and speeds up the reasoning speed of the neural network model. At the same time, a configurable neural network hardware accelerator is designed. By configuring the parameters of each module, the same hardware accelerator can be applied to calculate different convolutional layers in the neural network, thereby completing the acceleration operation of the entire neural network. In the process of convolution calculation in the convolution layer, by designing resource reuse of the weight cache module and the feature map cache module, frequent access to the external DDR3 memory is reduced, thereby saving bus bandwidth. And each module is optimized and designed, saving the consumption of hardware resources and the complexity of the system. The control process is designed on the PS side to complete the scheduling of the entire convolution calculation process and the post-processing steps of target detection, realizing the full process design from image acquisition, target detection algorithm calculation, and output display of detection results, and finally the entire target detection system is built through software and hardware collaborative design.

[0005] The technical solution of the present invention is as follows:

[0006] A target detection system based on ARM+FPGA heterogeneous platform.

[0007] The target detection system includes a PS terminal, a PL terminal and a peripheral part;

[0008] The PS side includes a core processor module;

[0009] The PL side is responsible for completing image acquisition, processing and display of data, as well as accelerating the processing work with large data volume and calculation amount in the target detection algorithm. The PL side includes image acquisition module, image processing module, algorithm acceleration module and video output display module.

[0010] The peripheral part is responsible for completing the image acquisition and image display functions, as well as storing the data during the operation of the target detection algorithm. The peripheral part includes a camera, an HDMI display and a DDR3 memory.

[0011] Core processor module: It is a CPU based on ARM (Advanced RISC Machines) architecture. The compiler is used to compile the control flow into machine code that the CPU can recognize. The CPU is responsible for completing the operation and scheduling tasks of the control flow, and at the same time, it issues control instructions to complete the initialization configuration of each module on the PL side, as well as the post-processing work with small computational workload in the target detection algorithm.

[0012] Image acquisition module: responsible for configuring the registers in the camera, making the registers in working state, and being able to output the correct video stream data according to the set parameters to complete the acquisition of the input image.

[0013] Image processing module: responsible for processing the input image of the image acquisition module to make it conform to the input format required by the algorithm acceleration module.

[0014] Algorithm acceleration module: responsible for accelerating the calculation of the neural network-based target detection algorithm, including instruction decoding module, data path selection module, feature map cache module, weight cache module, bias cache module, convolution window generation module, convolution calculation module, convolution result cache module, re-quantization module, activation function module, output feature map cache module and data input and output module.

[0015] 1) Instruction decoding module: used to obtain control instructions from the PS side, parse the control instructions, and send control signals to the subsequent modules to realize the calculation of the target detection algorithm. Specifically, the PS side writes data to the register of the instruction decoding module through the AXI bus. The instruction decoding module parses the parameter values ​​required by each module according to the values ​​at different positions in the register, and transmits the parsed parameter values ​​to each module. After each module obtains the parameter values, it starts to calculate the corresponding target detection algorithm.

[0016] 2) Data path selection module: responsible for controlling the data transmitted from the AXI bus and distributing it to the feature map cache module, weight cache module, and bias cache module according to different control instructions. The data path selection module contains the control logic for storing the received feature map data, weight data, and bias data into the corresponding cache module.

[0017] 3) Feature map cache module: It is responsible for caching the feature map data transmitted from the AXI bus, arranging the feature map data in sequence, and providing data stream for the convolution calculation module. The feature map cache module uses ping-pong operation to divide the cache into two spaces, namely the write space and the read space, which can meet the function of simultaneous reading and writing. The feature map cache module uses the internal address control unit to control the switching of the write space and the read space. It can meet the computing requirements of the target detection algorithm and ensure that the data of each channel can be effectively stored and read in parallel computing.

[0018] 4) Weight cache module: responsible for caching the weight data of the convolution kernel transmitted on the AXI bus. The weight cache module needs to be able to output the weight data of 8 convolution kernels and 8 channels at the same time. When outputting data, it is required that the weight data of the convolution kernels of each convolution kernel for 8 channels will be given at the same time to provide weight data for the convolution calculation module.

[0019] 5) Bias cache module: It is responsible for caching the bias data transmitted on the AXI bus and providing the bias of 8 convolution kernels at one time during calculation. The bias cache module has a bit width of 16 bits and a depth of 512. It can complete the calculation at the same time as the weight cache module, so that the internal data can be updated in the same step in the control process on the PS side.

[0020] 6) Convolution window generation module: responsible for converting the serial data output by the feature map cache module into a convolution window of appropriate size to facilitate the convolution calculation module to perform calculations. The convolution window generation module consists of two FIFOs and a read-write control unit.

[0021] 7) Convolution calculation module: responsible for receiving the feature map data, weight data and bias data output by the convolution window generation module, weight cache module and bias cache module, and performing parallel convolution calculations. The input data will enter the multiplier array to complete the element-by-element multiplication operation within the convolution window. The result of the element-by-element multiplication operation will be input into the addition tree to complete the addition operation within the convolution window. Since the calculation results of the convolution windows between different channels also need to be added during a completed convolution calculation process, after completing the addition operation within the convolution window, the data also needs to enter the second-level addition tree to complete the addition of the data between channels. The final calculated data will be input into the convolution result cache module.

[0022] 8) Convolution result cache module: responsible for temporarily storing the output results of the convolution calculation module. In the actual neural network-based target detection algorithm, the number of channels of the input feature map data is greater than 8, which means that the calculation results of the first 8 channels need to be temporarily stored during calculation. When the calculation results of subsequent channels are output, they need to be added to the output results of the previous round of 8 channels, and then the results after addition are written back to the same address. This cycle repeats until all channels are calculated.

[0023] 9) Re-quantization module: Responsible for re-quantizing the data output by the convolution calculation module and the convolution result buffer module. Since the bit width of the input data is 8 bits, the bit width of the data will accumulate after the multiplication and addition operations of the convolution calculation. When performing the convolution calculation of the next layer of the network, the required input is still 8 bits wide, so the re-quantization module is needed to scale and truncate the data bit width.

[0024] 10) Activation function module: It is used to complete the activation function operation in the neural network by scaling and truncating the output feature map of the requantization module.

[0025] 11) Output feature map cache module: used to cache data after convolution calculation, cropping, and activation function operation, so that the data input and output module can send data back to the PS end in batches.

[0026] 12) Data input and output module: It is composed of an interface module that supports the AXI-Stream protocol. It is responsible for receiving the feature map data, weight data, and bias data sent by the PS end, and sending them to the data path selection module for distribution. It is also responsible for sending the calculated data in the output feature map cache module back to the PS end.

[0027] Video output display module: The image data processed by the image processing module is stored in the DDR3 memory. The video output display module is responsible for reading the collected image data from the DDR3 memory, receiving the control instructions from the PS end, superimposing the detection result target frame output by the neural network on the original image data, and finally transmitting it to the external HDMI display through the HDMI interface to realize the output of the target detection results.

[0028] A hardware acceleration method for a target detection system based on an ARM+FPGA heterogeneous platform includes the following steps:

[0029] Step 1: Load image data and weight data;

[0030] Step 1.1: Perform fixed-point quantization on the weight data of the neural network model after training on the host computer, and convert the data type from 32-bit floating-point numbers to 8-bit fixed-point numbers.

[0031] Step 1.2: Run the control flow on the PS side, configure the register values ​​of each module on the PL side, and make the registers work in the running state.

[0032] Step 1.3: Load the weight file of the neural network after fixed-point quantization and store it in the DDR3 memory for subsequent calls.

[0033] Step 1.4: Get the initial image of the target to be detected, use the image acquisition module to complete the acquisition of the input image through the video pixel information transmitted by the camera, and use the image processing module to process the image data into an input format that meets the requirements of the algorithm acceleration module, and finally store it in the DDR3 memory.

[0034] Step 2: The algorithm acceleration module accelerates the calculation of the target detection algorithm based on the neural network;

[0035] Step 2.1: The PS side will send control instructions to the algorithm acceleration module according to the set control flow, indicating the feature map size, convolution kernel size, relevant parameters of re-quantization, and whether activation function calculation is required for this convolution calculation.

[0036] Step 2.2: After issuing a control instruction, the PS side writes data to the register of the instruction decoding module through the AXI bus. The instruction decoding module parses the parameter values ​​required by each module according to the values ​​at different positions in the register; the data path selection module transmits weight data, bias data and feature map data from the PS side to the feature map cache module, weight cache module and bias cache module of the algorithm acceleration module according to different control instructions.

[0037] Step 2.3: After receiving the control instruction to start the convolution calculation, the feature map cache module will output the feature map data of 8 channels in parallel according to the address. The feature map data will enter the convolution window generation module. After two lines of cache and three-beat storage, the convolution window data of 8 channels will be output in parallel and input into the convolution calculation module together with the weight data cached by the weight cache module and the bias data cached by the bias cache module.

[0038] Step 2.4: The convolution calculation module will perform convolution calculation on the input convolution window feature map data, bias data and weight data in real time, use a parallel multiplier array to complete the multiplication operation, use the first-level addition tree to complete the addition operation within the convolution window, use the second-level addition tree to complete the addition of data between channels, and output the calculated data to the convolution result cache module.

[0039] Step 2.5: After the convolution calculation completes the data of the first 8 channels of a batch of feature map data, the algorithm acceleration module will update the weights output by the weight cache module. At the same time, the feature map cache module will output the data of subsequent feature map channels. After the multiplication and addition operations of the convolution calculation module, it will be output to the convolution result cache module. The new input data will be added to the already written data and then rewritten to the original address. This cycle will be repeated until all channels have been calculated.

[0040] Step 2.6: After all channels have been calculated, the output feature map data will enter the re-quantization module, which will scale and truncate the data according to the incoming quantization parameters and finally output 8-bit wide data.

[0041] Step 2.7: The data after scaling and truncation operations in the requantization module will enter the activation function module, and after the activation function operation, it will enter the output feature map cache module, and will be transmitted back to the PS end through the data bus by the data input and output module. The PS end will send it back to the PL end as the output feature map data of the next convolution calculation to perform the convolution calculation of the next convolution layer until all convolution layers in the target detection algorithm are calculated.

[0042] Step 3: Display of test results;

[0043] Step 3.1: After calculating all the convolutional layers, the PS side will analyze the final output results and process them into meaningful detection result target frame data. Then, it will filter them according to the set threshold, retain the detection result target frames with high confidence, and then perform non-maximum suppression on the retained detection result target frames. Only one target frame of multiple detection result target frames that enclose the same object will be retained to obtain the final detection result.

[0044] Step 3.2: The PS side sends the detection results to the video output display module on the PL side. The video output display module reads the collected image data from the DDR3 memory, and superimposes the detection result target frame received in the control instruction on the original image data. Finally, it is transmitted to the external HDMI display through the HDMI interface to realize the output display of the target detection results.

[0045] Beneficial effects of the present invention:

[0046] 1. The present invention adopts the PS-side and PL-side software and hardware collaboration to design the target detection system, which can give full play to the advantages of the PS-side CPU being suitable for complex control operations and the PL-side FPGA being suitable for high-parallel computing and large-scale data throughput, make full use of hardware resources, and reduce the complexity of deploying the target detection neural network algorithm at the edge.

[0047] 2. The present invention has flexible configurability and excellent reusability. The feature map size, input channel, convolution step size, whether activation operation is required, and other settings input to the algorithm acceleration module can all be configured through parameters. This also allows the accelerator to adapt to different calculation methods in different algorithms, including 3×3 convolution and 1×1 convolution, etc., and can calculate different convolution layers of different target detection neural networks, with excellent reusability.

[0048] 3. The algorithm acceleration module in the present invention uses on-chip BRAM to cache the intermediate values ​​calculated by the convolution layer, reducing the number of accesses to the external memory, reducing the number of feature map transfers, and saving the interface bandwidth of the FPGA. At the same time, the multiplication and addition operations in the convolution calculation module are optimized, which can reduce FPGA resource consumption and improve resource utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is a system operation flow chart of the present invention;

[0050] Figure 2 is a system architecture diagram of the present invention;

[0051] Figure 3 This is the architecture diagram of the neural network accelerator designed by the algorithm acceleration module;

[0052] Figure 4 It is the input timing diagram of the convolution result cache module. DETAILED DESCRIPTION

[0053] In order to make the technical solution of the present invention clearer, the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0054] A target detection system based on ARM+FPGA heterogeneous platform, such as Figure 2 As shown, the target detection system includes a PS end, a PL end and a peripheral part;

[0055] The PS side includes a core processor module;

[0056] The PL side is responsible for completing image acquisition, processing and display of data, as well as accelerating the processing work with large data volume and calculation amount in the target detection algorithm. The PL side includes image acquisition module, image processing module, algorithm acceleration module and video output display module.

[0057] The peripheral part is responsible for completing the functions of image acquisition and image display, as well as storing the data during the operation of the target detection algorithm. The peripheral part includes a camera, an HDMI display and a DDR3 memory. The camera uses an OV5640 camera, and the DDR3 memory is responsible for caching the intermediate data of the algorithm (such as image data, feature map data, weight data, etc.). The data is transmitted from the OV5640 camera to the image acquisition module, and then to the image processing module, and finally controlled by the PS end to be written into the DDR3 memory.

[0058] Core processor module: It is a CPU based on ARM (Advanced RISC Machines) architecture. The compiler is used to compile the control flow into machine code recognizable by the CPU. The CPU is responsible for completing the operation and scheduling tasks of the control flow, and at the same time, it issues control instructions to complete the initialization configuration of each module on the PL side, as well as completing the post-processing tasks with low computational complexity in the target detection algorithm, such as non-maximum suppression.

[0059] Image acquisition module: responsible for configuring the registers in the camera, making the registers in working state, and being able to output the correct video stream data according to the set parameters to complete the acquisition of the input image.

[0060] Image processing module: responsible for processing the input image of the image acquisition module to make it conform to the input format required by the algorithm acceleration module.

[0061] Algorithm acceleration module: responsible for reading feature map data and weight data from DDR3 memory, accelerating the calculation of the target detection algorithm based on neural network, and transmitting the calculated data back to DDR3 memory. The final detection result will be output to the HDMI display after the video output and display module is superimposed with the original video stream. The algorithm acceleration module includes instruction decoding module, data path selection module, feature map cache module, weight cache module, bias cache module, convolution window generation module, convolution calculation module, convolution result cache module, re-quantization module, activation function module, output feature map cache module and data input and output module.

[0062] 1) Instruction decoding module: used to obtain control instructions from the PS side, parse the control instructions, and send control signals to the subsequent modules to realize the calculation of the target detection algorithm. Specifically, the PS side writes data to the register of the instruction decoding module through the AXI bus. The instruction decoding module parses the parameter values ​​required by each module according to the values ​​at different positions in the register, and transmits the parsed parameter values ​​to each module. After each module obtains the parameter values, it starts to calculate the corresponding target detection algorithm.

[0063] 2) Data path selection module: responsible for controlling the data transmitted from the AXI bus and distributing it to the feature map cache module, weight cache module, and bias cache module according to different control instructions. The data path selection module contains the control logic for storing the received feature map data, weight data, and bias data into the corresponding cache module.

[0064] 3) Feature map cache module: It is responsible for caching the feature map data transmitted from the AXI bus, arranging the feature map data in sequence, and providing data streams for the convolution calculation module. In order to improve the actual utilization of bandwidth, the feature map cache module uses ping-pong operation to divide the cache into two spaces, namely write space and read space, which can meet the function of simultaneous reading and writing. The internal address control unit in the feature map cache module controls the switching of write space and read space. It can meet the computing requirements of the target detection algorithm and ensure that the data of each channel can be effectively stored and read in parallel computing.

[0065] 4) Weight cache module: responsible for caching the weight data of the convolution kernel transmitted on the AXI bus. The weight cache module needs to be able to output the weight data of 8 convolution kernels and 8 channels at the same time. When outputting data, it is required that the weight data of the convolution kernels of each convolution kernel for 8 channels will be given at the same time to provide weight data for the convolution calculation module.

[0066] 5) Bias cache module: It is responsible for caching the bias data transmitted on the AXI bus and providing the bias of 8 convolution kernels at one time during calculation. The bias cache module has a bit width of 16 bits and a depth of 512. It can complete the calculation at the same time as the weight cache module, so that the internal data can be updated in the same step in the control process on the PS side.

[0067] 6) Convolution window generation module: responsible for converting the serial data output by the feature map cache module into a convolution window of appropriate size to facilitate the convolution calculation module to perform calculations. The convolution window generation module consists of two FIFOs and a read-write control unit.

[0068] 7) Convolution calculation module: responsible for receiving the feature map data, weight data and bias data output by the convolution window generation module, weight cache module and bias cache module, and performing parallel convolution calculations. The input data will enter the multiplier array to complete the element-by-element multiplication operation within the convolution window. The result of the element-by-element multiplication operation will be input into the addition tree to complete the addition operation within the convolution window. Since the calculation results of the convolution windows between different channels also need to be added during a completed convolution calculation process, after completing the addition operation within the convolution window, the data also needs to enter the second-level addition tree to complete the addition of the data between channels. The final calculated data will be input into the convolution result cache module.

[0069] 8) Convolution result cache module: responsible for temporarily storing the output results of the convolution calculation module. Since the parallelism of the accelerator design is to calculate the input feature map of 8 channels at a time, and in the actual neural network-based target detection algorithm, the number of channels of the input feature map data is greater than 8, it is necessary to temporarily store the calculation results of the first 8 channels during calculation. When the calculation results of subsequent channels are output, they need to be added to the output results of the 8 channels of the previous round, and the results after addition are written back to the same address, and this cycle is repeated until all channels are calculated.

[0070] 9) Re-quantization module: Responsible for re-quantizing the data output by the convolution calculation module and the convolution result buffer module. Since the bit width of the input data is 8 bits, the bit width of the data will accumulate after the multiplication and addition operations of the convolution calculation. When performing the convolution calculation of the next layer of the network, the required input is still 8 bits wide, so the re-quantization module is needed to scale and truncate the data bit width.

[0071] 10) Activation function module: It is used to complete the activation function operation in the neural network by scaling and truncating the output feature map of the requantization module.

[0072] 11) Output feature map cache module: used to cache data after convolution calculation, cropping, and activation function operation, so that the data input and output module can send data back to the PS end in batches.

[0073] 12) Data input and output module: It is composed of an interface module that supports the AXI-Stream protocol. It is responsible for receiving the feature map data, weight data, and bias data sent by the PS end, and sending them to the data path selection module for distribution. It is also responsible for sending the calculated data in the output feature map cache module back to the PS end.

[0074] Video output display module: The image data processed by the image processing module is stored in the DDR3 memory. The video output display module is responsible for reading the collected image data from the DDR3 memory, receiving the control instructions from the PS end, superimposing the detection result target frame output by the neural network on the original image data, and finally transmitting it to the external HDMI display through the HDMI interface to realize the output of the target detection results.

[0075] In order to maximize the parallel computing capability of FPGA, the accelerator in this example is designed as an 8×8×3×3 architecture, that is, the convolution calculation module simultaneously calculates 3×3 convolutions of 8 channels of 8 convolution kernels and calculates 576 multiplications per clock cycle.

[0076] Since this embodiment uses 8-channel parallel computing at the PL end to accelerate the algorithm, it means that the feature map cache module needs to store and output the feature map data of 8 channels at the same time. A dual-port BRAM is used to store the input data, with a bit width of 64 bits and a depth of 4096. One port is responsible for writing and the other port is responsible for reading.

[0077] For each channel of convolution kernel weight data, the input data is serially input, and the module needs to spend 9 clock cycles to prepare the weight of a 3×3 convolution kernel and write it to the corresponding address, and so on for a total of 8 channels. When reading data, for each convolution kernel, the convolution kernel weight data of 8 channels will be given at the same time to provide data for the convolution calculation module.

[0078] For the convolution window generation module, taking 3×3 convolution as an example, the convolution window generation module consists of two FIFOs and a read-write control unit. The two FIFOs here are cascaded, and each FIFO stores a complete row of pixels. The data at the input of the convolution window generation module is written into the first FIFO, and the data output by the first FIFO is written into the second FIFO. When a row is full, the read enable of the FIFO is pulled high, so that the output data of each FIFO and the input data of the convolution window generation module form three aligned rows of pixels.

[0079] In this example, YOLOv5-s is selected as the target detection neural network model. Taking a feature map with a convolution kernel of 3×3, a height of 320, a width of 320, and a number of channels of 32 as an example, the second layer of convolution of the backbone network is selected, the convolution step size is 2, the padding size is 1, and the padding value is 0. This patent is explained in detail through the convolution process.

[0080] A hardware acceleration method for target detection system based on ARM+FPGA heterogeneous platform, such as Figure 1 and Figure 3 As shown, the steps include:

[0081] Step 1: Load image data and weight data;

[0082] Step 1.1: Use a neural network framework such as Pytorch to build a neural network model on the host computer, prepare data and write scripts to complete the training of the neural network. Perform fixed-point quantization on the weight data of the neural network model after training on the host computer, convert the data type from 32-bit floating-point numbers to 8-bit fixed-point numbers, and use the quantized weights and scaling factors as files for the PS end to call.

[0083] Step 1.2: Run the control process on the PS side and configure the register values ​​of each module on the PL side, including writing data to the register of the OV5640 camera through the IIC protocol and configuring its output parameters to make the register work in the running state.

[0084] Step 1.3: Load the weight file of the neural network after fixed-point quantization and store it in the DDR3 memory for subsequent calls.

[0085] Step 1.4: Use an external camera and image acquisition module to obtain the initial image of the target to be detected, use the image processing module to splice the 8-bit data passed by OV5640 into 16-bit RGB565 data, and cut it into 640×640 video information required by the neural network algorithm, and then store it in the DDR3 memory through VDMA.

[0086] Step 2: The algorithm acceleration module accelerates the calculation of the target detection algorithm based on the neural network;

[0087] Step 2.1: The PS side will send control instructions to the algorithm acceleration module according to the set control flow, indicating the feature map size, convolution kernel size, re-quantization related parameters, and whether activation function calculation is required for this convolution calculation. Taking the second convolution layer of YOLOv5-s mentioned above as an example, the convolution kernel size is 3×3, the feature map height is 320, and the width is 320. Due to the addition of padding, the width and height of the actual feature map need to be increased by 2 on the original basis, the number of channels is 32, and the activation function is LeakyRelu.

[0088] Step 2.2: After the PS sends out the control instruction, it will write data to the register of the instruction decoding module through the AXI bus. The instruction decoding module parses the parameter values ​​required by each module according to the values ​​at different positions in the register; the data path selection module transmits weight data, bias data and feature map data from the PS to the feature map cache module, weight cache module and bias cache module of the algorithm acceleration module according to the different control instructions. Among them, the quantized weight is loaded into the weight cache module through the data path selection module. The bus width is 64, and the 8-bit weights of 8 channels can be transmitted at one time. There are two counters cnt_9 and cnt_8 inside the data path selection module. Whenever cnt_9 is full, it means that the data of one channel of a 3×3 convolution kernel has been filled, and cnt_8 will increase by 1. When cnt_8 is full, it means that the weights of the convolution kernels of 8 channels have been filled, and the output weight data is valid data.

[0089] Step 2.3: After receiving the control instruction to start the convolution calculation, the feature map cache module will output the feature map data of 8 channels in parallel according to the address, and the feature map data will enter the convolution window generation module. The two FIFOs in series in the convolution window generation module will cache the data. When the number of data in the first FIFO is 322, it means that the first line has been cached. At this time, the output enable signal of the first FIFO will be pulled high, and the output data will be passed to the second FIFO until the second FIFO is also full of corresponding data. At this time, the output of the two FIFOs and the original input signal can form three lines of data, and then use 9 registers to complete the three-beat storage of data. After two lines of caching and three-beat storage, the 3×3 size convolution window data of 8 channels is output in parallel, and input into the convolution calculation module together with the weight data cached by the weight cache module and the bias data cached by the bias cache module.

[0090] Step 2.4: The convolution calculation module will perform convolution calculations on the input convolution window feature map data, bias data and weight data in real time, and use a multiplier array with a parallel degree of 8×8×3×3 to simultaneously calculate 576 8-bit multiplications. Since the multiplication array uses a pipeline structure, the input data will get the output result after 4 clock cycles. Then, 8 groups of 9-input addition trees are used to complete the accumulation operation within the convolution window, and then the accumulation operation and bias accumulation between the channels of 8 groups of 9-input addition trees are used. Finally, the 8-way output is the convolution result output of the 8 convolution kernels.

[0091] Step 2.5: After the convolution calculation completes the data of the first 8 channels of a batch of feature map data, driven by the clock signal clk, the algorithm acceleration module will update the weights output by the weight cache module. Since the convolution kernel weights in the weight cache module have been arranged in the required specific order, it is only necessary to increase the value of the read address signal read_addr by 1 to output the next batch of weight data. At the same time, the feature map cache module will output the data of subsequent feature map channels, which will be output to the convolution result cache module after multiplication and addition operations of the convolution calculation module. The main data interfaces of the convolution result cache module include the read enable signal read_en, the read address signal read_addr, the write enable signal write_en, the write address signal write_addr and the current input data signal curr_data. The specific input timing of the signal is as follows: Figure 4As shown in the figure, the read_en signal and read_addr signal must be one clock cycle ahead of the curr_data signal, so that the signal of the data stored at the previous address synchronized with curr_data can be read out. The write_addr signal must lag behind the curr_data signal by one cycle, so that the accumulated signal can be written to the same address as the original one. This cycle repeats until all channels are calculated.

[0092] Step 2.6: After all channels have been calculated, the output feature map data will enter the re-quantization module. The main parameters in the re-quantization module are the scale factor, the shift factor and the zero point. The values ​​of these three factors have been given in the quantization stage. The re-quantization module will scale and truncate the data according to the incoming quantization parameters, and finally output 8-bit wide data.

[0093] Step 2.7: The data after scaling and truncation in the requantization module will enter the activation function module. In this example, the activation function is the LeakyRelu activation function. In hardware implementation, you only need to use a comparator to output the original value of numbers greater than zero and multiply the numbers less than zero by a slope α. If the value of α is selected as 0.125, you can use the shift operation instead of the multiplication operation to reduce the complexity of hardware implementation. After the activation function operation, it enters the output feature map cache module, and the data input and output module transmits it back to the PS end through the data bus. The PS end will send it back to the PL end as the output feature map data of the next convolution calculation to perform the convolution calculation of the next convolution layer until all convolution layers in the target detection algorithm are calculated.

[0094] Step 3: Display of test results;

[0095] Step 3.1: After calculating all the convolutional layers, the PS side will analyze the final output results and process them into meaningful detection result target frame data. The specific operation is to retain the confidence of the algorithm output, restore the normalized detection result target frame to x, y, w, h with practical significance, and then determine which category this detection result target frame belongs to based on the category score. Then filter according to the set threshold to keep the detection result target frame with higher confidence. Keep the detection result target frame with high confidence, and then perform non-maximum suppression on the retained detection result target frame, and keep only one target frame of multiple detection result target frames that enclose the same object to obtain the final detection result.

[0096] Step 3.2: The PS side sends the detection results to the video output display module of the PL side. The video output display module reads the collected image data from the DDR3 memory, and according to the control instructions, draws detection result target boxes of different colors for different categories of objects, and superimposes the detection result target boxes on the original image data. Finally, it is transmitted to the external HDMI display through the HDMI interface to realize the output display of the target detection results.

Claims

1. A target detection system based on ARM+FPGA heterogeneous platform, characterized in that: The target detection system includes a PS terminal, a PL terminal and a peripheral part; The PS side includes a core processor module; The PL side is responsible for completing image acquisition, processing and display of data, as well as accelerating the processing work with large data volume and large computational workload in the target detection algorithm. The PL side includes image acquisition module, image processing module, algorithm acceleration module, and video output display module. The peripheral part is responsible for completing the image acquisition and image display functions, as well as storing the data during the operation of the target detection algorithm. The peripheral part includes a camera, an HDMI display, and a DDR3 memory. Core processor module: It is an ARM-based CPU. The compiler is used to compile the control flow into machine code that the CPU can recognize. The CPU is responsible for completing the operation and scheduling tasks of the control flow, and at the same time, it issues control instructions to complete the initialization configuration of each module on the PL side, as well as the post-processing work with small computational workload in the target detection algorithm. Image acquisition module: responsible for configuring the registers in the camera, making the registers in working state, and being able to output the correct video stream data according to the set parameters to complete the acquisition of the input image; Image processing module: responsible for processing the input image of the image acquisition module to make it conform to the input format required by the algorithm acceleration module; Algorithm acceleration module: responsible for accelerating the calculation of the target detection algorithm based on the neural network, including instruction decoding module, data path selection module, feature map cache module, weight cache module, bias cache module, convolution window generation module, convolution calculation module, convolution result cache module, re-quantization module, activation function module, output feature map cache module and data input and output module; Video output display module: The image data processed by the image processing module is stored in the DDR3 memory. The video output display module is responsible for reading the collected image data from the DDR3 memory, receiving the control instructions from the PS end, and superimposing the detection result target frame output by the neural network on the original image data. Finally, it is transmitted to the external HDMI display through the HDMI interface to realize the output of the target detection result. The specific modules in the algorithm acceleration module are as follows: 1) Instruction decoding module: used to obtain the control instructions passed in from the PS end, and send control signals to the subsequent modules after parsing the control instructions to realize the calculation of the target detection algorithm; specifically, the PS end writes data to the register of the instruction decoding module through the AXI bus, and the instruction decoding module parses the parameter values ​​required by each module according to the values ​​at different positions in the register, and transmits the parsed parameter values ​​to each module. After obtaining the parameter values, each module starts to calculate the corresponding target detection algorithm; 2) Data path selection module: responsible for controlling the data transmitted from the AXI bus and distributing it to the feature map cache module, weight cache module, and bias cache module according to different control instructions; The data path selection module contains the control logic for storing the received feature map data, weight data and bias data into the corresponding cache module; the quantized weights are loaded into the weight cache module via the data path selection module. The bus width is 64, and the 8-bit weights of 8 channels can be transmitted at one time. There are two counters cnt_9 and cnt_8 inside the data path selection module. Whenever cnt_9 is full, it means that the data of one channel of a 3×3 convolution kernel has been filled, and cnt_8 will increase by 1; when cnt_8 is full, it means that the convolution kernel weights of the 8 channels have been filled, and the output weight data is valid data at this time; 3) Feature map cache module: responsible for caching the feature map data transmitted on the AXI bus, arranging the feature map data in sequence, and providing data flow for the convolution calculation module; the feature map cache module uses ping-pong operation to divide the cache into two spaces, namely write space and read space, which can meet the function of simultaneous reading and writing. The internal address control unit in the feature map cache module controls the switching of write space and read space; it can meet the calculation requirements of the target detection algorithm and ensure that the data of each channel can be effectively stored and read in parallel computing; 4) Weight cache module: responsible for caching the weight data of the convolution kernel transmitted on the AXI bus. The weight cache module needs to be able to output the weight data of 8 convolution kernels and 8 channels at the same time. When outputting data, it is required that the weight data of the convolution kernels of each convolution kernel for 8 channels will be given at the same time to provide weight data for the convolution calculation module. 5) Bias cache module: responsible for caching the bias data transmitted on the AXI bus, and giving the bias of 8 convolution kernels at one time during calculation. The bias cache module has a bit width of 16 bits and a depth of 512. It can complete the calculation at the same time as the weight cache module, so that the internal data can be updated in the same step in the control process of the PS end; 6) Convolution window generation module: responsible for converting the serial data output by the feature map cache module into a convolution window of appropriate size to facilitate the convolution calculation module to perform calculations; the convolution window generation module consists of two FIFOs and a read-write control unit; 7) Convolution calculation module: responsible for receiving the feature map data, weight data and bias data output by the convolution window generation module, weight cache module and bias cache module, and performing parallel convolution calculation; the input data will enter the multiplier array to complete the element-by-element multiplication operation in the convolution window, and the result of the element-by-element multiplication operation will be input into the addition tree to complete the addition operation in the convolution window; since the calculation results of the convolution windows between different channels also need to be added in a convolution calculation process, after completing the addition operation in the convolution window, the data also needs to enter the second-level addition tree to complete the addition of the data between channels, and the final calculated data will be input into the convolution result cache module; 8) Convolution result cache module: responsible for temporarily storing the output results of the convolution calculation module; in the actual neural network-based target detection algorithm, the number of channels of the input feature map data is greater than 8, which means that the calculation results of the first 8 channels need to be temporarily stored during calculation. When the calculation results of subsequent channels are output, they need to be added to the output results of the previous round of 8 channels, and then the results after addition are written back to the same address, and this cycle is repeated until all channels are calculated; 9) Re-quantization module: responsible for re-quantizing the data output by the convolution calculation module and the convolution result buffer module; since the bit width of the input data is 8 bits, and after the multiplication and addition operations of the convolution calculation, the bit width of the data will accumulate, and when performing the convolution calculation of the next layer of the network, the required input is still 8 bits wide, so the re-quantization module is needed to scale and truncate the data bit width; the parameters in the re-quantization module include the scaling factor scale, the shift factor shift and the zero point zero_point; 10) Activation function module: used to complete the LeakyRelu activation function operation in the neural network by scaling and truncating the output feature map of the re-quantization module; in hardware implementation, it is only necessary to use a comparator to output the original value of the number greater than zero and multiply the number less than zero by a slope α. The value of α is selected as 0.125, so that the shift operation can be used instead of the multiplication operation; 11) Output feature map cache module: used to cache data after convolution calculation, cropping, and activation function operation, so that the data input and output module can send the data back to the PS end in batches; 12) Data input and output module: It is composed of an interface module that supports the AXI-Stream protocol. It is responsible for receiving the feature map data, weight data, and bias data sent by the PS end, and sending them to the data path selection module for distribution. It is also responsible for sending the calculated data in the output feature map cache module back to the PS end.

2. The hardware acceleration method of the target detection system based on the ARM+FPGA heterogeneous platform according to claim 1 is characterized in that: The steps include: Step 1: Load image data and weight data; Step 1.1: Perform fixed-point quantization on the weight data of the neural network model after training on the host computer, and convert the data type from 32-bit floating-point numbers to 8-bit fixed-point numbers; Step 1.2: Run the control flow on the PS side, configure the register values ​​of each module on the PL side, and make the registers work in the running state; Step 1.3: Load the weight file of the neural network after fixed-point quantization and store it in the DDR3 memory for subsequent calls; Step 1.4: Obtain the initial image of the target to be detected, and complete the acquisition of the input image through the image acquisition module using the video pixel information transmitted by the camera. The image processing module processes the image data into an input format that meets the requirements of the algorithm acceleration module, and finally stores it in the DDR3 memory; Step 2: The algorithm acceleration module accelerates the calculation of the target detection algorithm based on the neural network; Step 2.1: The PS side will send control instructions to the algorithm acceleration module according to the set control flow, indicating the feature map size, convolution kernel size, relevant parameters of re-quantization, and whether activation function calculation is required for this convolution calculation; Step 2.2: After the PS sends out the control instruction, it will write data to the register of the instruction decoding module through the AXI bus. The instruction decoding module parses the parameter values ​​required by each module according to the values ​​at different positions in the register; the data path selection module transmits weight data, bias data and feature map data from the PS to the feature map cache module, weight cache module and bias cache module of the algorithm acceleration module according to different control instructions; the quantized weights are loaded into the weight cache module via the data path selection module. The bus width is 64, and the 8-bit weights of 8 channels can be transmitted at one time. There are two counters cnt_9 and cnt_8 inside the data path selection module. Whenever cnt_9 is full, it means that the data of one channel of a 3×3 convolution kernel has been filled, and cnt_8 will increase by 1; when cnt_8 is full, it means that the convolution kernel weights of 8 channels have been filled, and the output weight data is valid data at this time; Step 2.3: After receiving the control instruction to start the convolution calculation, the feature map cache module will output the feature map data of 8 channels in parallel according to the address. The feature map data will enter the convolution window generation module. After two lines of cache and three beats of storage, the convolution window data of 8 channels will be output in parallel and input into the convolution calculation module together with the weight data cached by the weight cache module and the bias data cached by the bias cache module; Step 2.4: The convolution calculation module will perform convolution calculation on the input convolution window feature map data, bias data and weight data in real time, use the parallel multiplier array to complete the multiplication operation, use the first-level addition tree to complete the addition operation within the convolution window, use the second-level addition tree to complete the addition of data between channels, and output the calculated data to the convolution result cache module; Step 2.5: After the convolution calculation completes the data of the first 8 channels of a batch of feature map data, the algorithm acceleration module will update the weights output by the weight cache module. At the same time, the feature map cache module will output the data of the subsequent feature map channels. After the multiplication and addition operations of the convolution calculation module, it will be output to the convolution result cache module. The new input data will be added to the already written data and then rewritten to the original address. This cycle repeats until all channels are calculated; Step 2.6: The output feature map data after all channels have been calculated will enter the re-quantization module, which will scale and truncate the data according to the incoming quantization parameters, and finally output 8-bit wide data; the parameters in the re-quantization module include scale factor, shift factor and zero point; Step 2.7: The data after scaling and truncation of the re-quantization module will enter the activation function module. The activation function is the LeakyRelu activation function. When implemented in hardware, the comparator is used to output the original value of the number greater than zero and multiply the number less than zero by a slope α. The value of α is selected as 0.125, and the shift operation can be used instead of the multiplication operation; after the activation function operation, it enters the output feature map cache module, and the data input and output module transmits it back to the PS end through the data bus. The PS end will send it back to the PL end as the output feature map data of the next convolution calculation to perform the convolution calculation of the next convolution layer until all convolution layers in the target detection algorithm are calculated; Step 3: Display of test results; Step 3.1: After calculating all the convolutional layers, the PS side will analyze the final output results and process them into meaningful detection result target frame data, and then filter them according to the set threshold, retain the detection result target frame with high confidence, and then perform non-maximum suppression operation on the retained detection result target frame, and retain only one target frame of multiple detection result targets that frame the same object, and obtain the final detection result; Among them, the process of processing into meaningful detection result target frame data is: retain the confidence output by the algorithm acceleration module, restore the normalized detection result target frame to x, y, w, h with practical significance, and then determine which category this detection result target frame belongs to according to the category score; Step 3.2: The PS side sends the detection results to the video output display module on the PL side. The video output display module reads the collected image data from the DDR3 memory, and superimposes the detection result target frame received in the control instruction on the original image data. Finally, it is transmitted to the external HDMI display through the HDMI interface to realize the output display of the target detection results.

Images