Convolutional neural network image accelerator system based on ZYNQ and implementation method

Abstract

The invention provides a convolutional neural network image accelerator system based on ZYNQ and an implementation method. The method comprises the steps of storing trained and optimized convolutional neural network parameters into an SD card, loading the parameters through a PS end of ZYNQ, constructing a control subsystem of the whole system at the PS end, constructing an acceleration subsystem at a PL end, and arranging storage modules at the PS end and the PL end respectively. The camera acquires image data and stores the image data into the storage module, the control module controls data flow and an acceleration instruction, receives feedback and caches the data into the acceleration module, and the acceleration module completes calculation acceleration according to the instruction and outputs an image detection result to the external screen through the HDMI. The forward reasoning speed of the convolutional neural network is increased, the instruction set construction speed is increased, power consumption is low, and expandability is achieved.

Description

Technical Field

[0001] The present application relates to the field of target detection technology, and in particular to a ZYNQ-based convolutional neural network image accelerator system and an implementation method thereof. Background Art

[0002] With the advent of the intelligent era, artificial intelligence is rapidly affecting all walks of life, and convolutional neural networks (CNNs) play an important role in the field of artificial intelligence. In terms of target detection, convolutional neural networks have recognition capabilities close to or even better than human vision, and have huge development potential and application value. However, the forward reasoning calculation of convolutional neural networks is a computationally intensive task. With the continuous development of algorithms, while the detection accuracy increases, the structure of the network becomes more complex, and the amount of model calculation increases exponentially, making it difficult to efficiently apply it in border inspection calculations. The storage capacity and computing resources of most terminal devices are relatively limited, and cannot meet the resource requirements required for the deployment of convolutional neural networks. In order to improve this pain point, it is necessary to design a dedicated convolutional neural network accelerator to reduce the hardware requirements for model deployment.

[0003] The mainstream hardware acceleration platforms at present are GPU, CPU, ASIC and FPGA. GPU has high computing parallelism and significant acceleration effect, but its running power consumption is too high and it is not suitable for edge devices. CPU has low acceleration efficiency due to its hardware architecture and serial computing limitations. As a special chip design, ASIC has the advantages of high efficiency and low power consumption, but its long development cycle and high tape-out cost make it not a good choice. FPGA has the advantages of low power consumption, reconfigurability and parallel computing, which is very suitable for the inference stage of convolutional neural networks. At the same time, convolutional neural networks are in a state of rapid iterative development, and FPGA can flexibly change the hardware structure and customize hardware solutions in a targeted manner, so the convolutional neural network acceleration solution based on FPGA has always been widely concerned.

[0004] However, traditional FPGAs have slow instruction set construction and high timing complexity when processing complex task mobilization and data allocation, resulting in low development efficiency, low precision, and poor versatility. For example, the patent application number is 202410049260.1, and its name is a ZYNQ-based convolutional neural accelerator implementation device and method. Although this solution reduces the difficulty of algorithm design, it will also lose certain performance, which is contrary to the design with high performance and low power consumption as the goal. In summary, it is necessary to design a suitable method for building a convolutional neural network image accelerator. Summary of the invention

[0005] The present invention provides a ZYNQ-based convolutional neural network image accelerator system and implementation method, which are used to improve the problems of slow construction speed of traditional FPGA instruction sets, low architecture development efficiency, and poor versatility of acceleration solutions.

[0006] The technical solution to achieve the purpose of the present invention is:

[0007] On the one hand, the present invention proposes a method for implementing a convolutional neural network image accelerator based on ZYNQ, the method comprising:

[0008] Step 1: Select a convolutional neural network model, obtain the target image to be detected to create a data set, and store the data set on a server with a GPU;

[0009] Step 2: Train the selected convolutional network model on a server with a GPU according to the acquired data set to obtain the weights and bias parameters of the trained floating-point convolutional neural network;

[0010] Step 3: Quantize the convolutional neural network model and convert the floating-point weights and bias parameters trained on the GPU into fixed-point numbers.

[0011] Step 4: Select the FPGA core board and design the architecture of the convolutional neural network accelerator based on the selected hardware board;

[0012] Step 5: Load the convolutional neural network model into the hardware;

[0013] Step 6: After reading the data from the SD card into the memory, program the driver modules of the camera and the display on the programmable logic array side;

[0014] Step 7: After the image data starts to be transmitted, turn on the accelerator to perform image preprocessing, forward reasoning of the convolutional neural network model, and output image post-processing.

[0015] Furthermore, the FPGA core board is a ZYNQ chip, including an FPGA chip at the logic end and an ARM chip at the processor system end; wherein:

[0016] The FPGA chip end is responsible for designing the input module, acceleration module, on-chip storage module and output module corresponding to the image preprocessing unit, convolutional neural network calculation unit, cache unit and image post-processing unit. The input module is connected to the off-chip storage module. The data of the off-chip storage module is transmitted to the on-chip storage module cache according to the command of the network structure control module, and then enters the acceleration module, and is finally output through the output module.

[0017] The ARM chip side is responsible for building a forward reasoning framework for the convolutional neural network and writing data processing functions, designing a network structure control module and a DDR controller. The ARM side builds a forward reasoning framework through the network structure control module, and calls the storage module of the FPGA core board through the DDR controller to send data to the acceleration module.

[0018] Furthermore, step 5 specifically includes: loading the convolutional neural network model into the hardware, using an external SD card to store the quantized weight parameters, bias parameters and activation function data, reading the file stored in the SD card into the DDR through the xilffs library on the ARM chip side, temporarily storing the data in the off-chip storage module, waiting for the network structure control module to issue an instruction to call, and accelerating in the acceleration module on the FPGA chip side.

[0019] Furthermore, the step 6 specifically includes: after reading the data from the SD card to the memory, writing the driver module of the camera and the display on the programmable logic array end, driving the camera through the FPGA to obtain the image and transferring the image data to the off-chip storage module, and finally displaying the picture on the display screen of the FPGA chip.

[0020] Furthermore, image preprocessing specifically includes:

[0021] The image data input by the camera is converted from 8-bit data to 24-bit image data in RGB888 format; according to the convolutional neural network model, the converted image data is resized and scaled to a size recognizable by the acceleration module; the resized image data is quantized to complete the image preprocessing operation.

[0022] Furthermore, the forward reasoning of the convolutional neural network model specifically includes: after reading the parameter file inside the SD card, generating the corresponding control command, then loading the quantized preprocessed data, performing forward reasoning of the network, through the GP and HP interfaces inside ZYNQ, according to the AXI bus protocol, the network structure control module sends parameters to the acceleration module, and loads the image feature data, bias data and weight data into the on-chip storage module for caching, waiting for the acceleration module to calculate these data, after one layer of structure calculation is completed, the result is stored in DDR as the input data of the next layer, and according to the control module instructions, the next round of calculation is performed, and it is repeated continuously, waiting for the convolutional neural network model acceleration calculation to be completed, and the final output data is obtained.

[0023] Furthermore, the output image post-processing specifically includes:

[0024] According to the output data of the convolutional neural network model, the target frame and confidence score are decoded, and the detection frame of each frame image is obtained according to the target frame confidence score;

[0025] Convert the image's detection box position into actual image coordinates and send them to the FPGA;

[0026] Based on the received coordinate data, FPGA constructs an OSD interface on the display screen and draws a detection frame corresponding to the target; and outputs the result to the display screen through the HDMI interface.

[0027] On the other hand, the present invention provides a ZYNQ-based convolutional neural network accelerator system, comprising: an image input module, a control module, a storage module, a computing acceleration module and an output module;

[0028] The input module consists of a camera driver module and an image preprocessing module; the camera driver module is used to acquire images and transfer image data to the off-chip storage module, and the image preprocessing module is used for image preprocessing;

[0029] The control module consists of a data flow control issuing module, a parameter configuration module and an interrupt receiving module. The data flow control issuing module is used to issue calculation instructions to the acceleration module. The parameter configuration module is used to receive the training parameters stored in the SD card and load them into the off-chip memory. The interrupt receiving module is used to receive the calculation process feedback of the acceleration module.

[0030] The storage module is composed of an on-chip storage module and an off-chip storage module. The camera driver module acquires images and transfers image data to the off-chip storage module, and the data of the off-chip storage module is transferred to the on-chip storage module.

[0031] The acceleration module consists of a convolution calculation module, an upsampling calculation module, and a pooling calculation module. The convolution calculation module is used to accelerate the convolution operation in the convolutional neural network and complete the extraction of feature data. The upsampling calculation module accelerates the upsampling calculation in the network layer, amplifies the extracted feature image, and maintains the quality of the feature image. The pooling calculation module is used to accelerate the pooling operation in the network layer, reduce the size of the parameter matrix in the layer, improve the calculation speed, and prevent the network from overfitting.

[0032] The output module consists of a target frame drawing module and a display screen driving module. The target frame drawing module is used to read the predicted frame diagram coordinates calculated by the convolutional neural network model and realize the drawing of the target frame diagram on the PL side. The display screen driving module is used to display the output results of the output module.

[0033] Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention selects and trains a neural network model, quantizes the trained neural network model, selects an FPGA development board, designs a global architecture suitable for the neural network model based on the ZYNQ chip, places the convolution calculation module, upsampling module and pooling module, which account for a large proportion of the calculation in the network, on the programmable logic end, and realizes calculation acceleration by building multi-channel parallel, convolution loop expansion, ping-pong buffer, pipeline architecture, data bit width cutting and the like. The convolutional neural network data flow scheduling task and network structure deployment are placed on the PS end, and the inter-layer data control instructions are written with the help of software language to realize the forward reasoning of the convolutional neural network. While the recognition accuracy meets the requirements, the reasoning speed is improved to complete the acceleration task of the convolutional neural network. The method has fast detection speed, low power consumption and high precision, and can be used in high-speed and low-power edge computing devices. This system solution can adjust each submodule according to actual needs, and can also adjust the control module instructions according to different networks. It is suitable for accelerating other networks on other ZYNQ platforms and has certain versatility. It improves the problems of slow construction of traditional FPGA instruction sets and high timing complexity, which leads to low development efficiency, low precision and poor versatility. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 The present invention is a flowchart of a method for implementing a convolutional neural network image accelerator based on ZYNQ in one embodiment of the present invention.

[0035] Figure 2 FIG. 4 is a schematic diagram of symmetric quantization in an embodiment of the present invention.

[0036] Figure 3 FIG. 4 is a schematic diagram of asymmetric quantization in an embodiment of the present invention.

[0037] Figure 4 FIG. 4 is a schematic diagram of a circular convolution expansion in one embodiment of the present invention.

[0038] Figure 5 It is a schematic diagram of pipeline calculation in one embodiment of the present invention.

[0039] Figure 6 Flow chart of a convolution calculation scheme in one embodiment of the present invention.

[0040] Figure 7 Schematic diagram of the architecture of a ZYNQ-based convolutional neural network image accelerator system in one embodiment of the present invention. DETAILED DESCRIPTION

[0041] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0042] See also Figure 1, Figure 1 The flowchart of the method for implementing the image accelerator of the present invention is a method for implementing the convolutional neural network image accelerator based on ZYNQ, comprising the following steps:

[0043] Step 1: Select a convolutional neural network model, obtain the target image to be detected to create a data set, and store the data set on a server with a GPU;

[0044] Step 2: Train the selected convolutional neural network model on a server with a GPU according to the acquired data set to obtain the weights and bias parameters of the trained convolutional neural network;

[0045] Step 3: quantize the model and convert the floating-point weights and bias parameters trained on the GPU into fixed-point numbers. Because the trained network cannot be directly deployed on the FPGA, it needs to be quantized first. In the present invention, the weight parameters are quantized into the INT8 data type. Quantization can reduce the model size and quantize the 32-bit floating-point number into an 8-bit fixed-point number. Similarly, the FPGA has a faster computing speed for integers, and quantization can achieve faster computing speed.

[0046] See also Figure 2 and Figure 3 , is a schematic diagram of asymmetric quantization and symmetric quantization in the present invention, wherein Figure 2 For asymmetric quantization, Figure 3 Symmetric quantization.

[0047] According to whether the quantization zero point Z is 0, linear quantization can be divided into symmetric quantization and asymmetric quantization. The advantage of symmetric quantization is that the calculation is simple. It uses the same quantization step to map positive and negative numbers, thereby reducing the calculation complexity, but the distribution of the quantized data may not be uniform enough. In contrast, asymmetric quantization has a better mapping range. The disadvantage is that the quantization zero point needs to be calculated during the quantization process, which brings additional hardware overhead. Considering that symmetric quantization shows a certain compatibility when used in conjunction with power-of-2 quantization, the invention chooses to use a symmetric quantization strategy.

[0048] Power-of-2 quantization is a special quantization strategy that requires that during the quantization process, the scaling factor S is always within the domain of n-th power with 2 as the base, that is, S can be expressed as: S = 2 n Where n is an integer. Since the FPGA platform has optimizations related to bit operations when it is designed, when the scaling factor S is equal to 2 to the power of n, the scaling operations originally designed for multiplication and division can be simplified to shift operations. Combined with the symmetric quantization strategy with the quantization zero point Z being 0, the quantization and dequantization operations in the model inference process can be easily completed by shifting, thereby effectively improving the data processing speed on the FPGA side.

[0049] The quantification principle is

[0050] r=S(qZ)

[0051] Where r represents the original floating-point parameter value; S is the conversion factor, which is a floating-point number; q is the integer result after quantization, which is an int8 fixed-point number. Z is a zero point, a fixed-point number, which represents the value of the floating-point number 0 mapped to the fixed-point number.

[0052] in and So the quantized value can be

[0053]

[0054] Step 4: Select the FPGA core board and design the architecture of the convolutional neural network accelerator based on the selected hardware board;

[0055] Among them, the FPGA core board is selected as the ZYNQ chip, which is a heterogeneous chip, including the FPGA chip at the logic end (PL end) and the ARM chip at the processor system end (PS end).

[0056] Estimate the amount of resources required to deploy the convolutional neural network model, focusing on the number of DSP and BRAM resources provided by the hardware platform.

[0057] The FPGA side is responsible for designing the input module system, acceleration module, on-chip storage module, and output module corresponding to the image preprocessing unit, convolutional neural network computing unit, cache unit, and image post-processing unit. The input module is connected to the off-chip storage module. The data is transmitted to the on-chip storage module cache according to the command of the network structure control module, and then enters the acceleration module, and is finally output through the output module.

[0058] The ARM side is responsible for building the forward reasoning framework of the convolutional neural network and writing data processing functions. Design the network structure control module, DDR controller, ARM builds the forward reasoning framework through the network structure control module, and calls the storage module through the DDR controller to send data to the acceleration module.

[0059] The specific chip selected is the ZYNQ7035 chip from Xilinx, which has 275k logic units and 900 DSP computing resources.

[0060] Step 5: Load the model into the hardware, use an external SD card to store the quantized weight parameters, bias parameters, and activation function data, and read the files stored in the SD card into DDR through the xilffs library on the ARM side. The data is temporarily stored in the off-chip storage module, waiting for the control module to issue instructions for calling, and then accelerated in the FPGA acceleration module.

[0061] Step 6: After reading the data from the SD card to the memory, write the driver module of the camera and the display on the PL side, drive the camera through the FPGA to obtain the image and transfer the image data to the off-chip storage module, and finally display the picture on the display driven by the FPGA. For the input feature map, it can be a picture in the SD card or the pixel data of the camera.

[0062] Step 7: After the image data starts to be transmitted, turn on the accelerator and perform image preprocessing first.

[0063] Image preprocessing is included in the input module, including the following:

[0064] Convert the image data input by the camera from 8-bit data to 24-bit image data in RGB888 format;

[0065] Specifically, the connection between the camera of the present invention and the system relies on the DVP interface, which transmits 8-bit data one by one in units of each clock cycle. The original data format output by the camera is RGB565, that is, each pixel is composed of 16-bit data (including 5-bit red component, 6-bit green component and 5-bit blue component). Therefore, it takes two clock cycles to receive a complete pixel data. Subsequently, we need to convert these RGB565 format data to meet the RGB888 standard. This conversion process usually involves extending the high-order data to the low-order: for the red (R) channel, we copy the original high-order 3-bit data to the newly added low-order 3 bits; for the green (G) channel, the high-order 2-bit data is expanded to the low-order 6 bits; the processing method of the blue (B) channel is consistent with the red channel.

[0066] According to the requirements of the convolutional neural network model, the converted image data is resized to a size that can be recognized by the acceleration module;

[0067] The specific scaling method is bilinear interpolation.

[0068] The resized image data is quantized to complete the image preprocessing operation.

[0069] Step 8: Perform forward reasoning of the model. After reading the parameter file inside the SD card, generate the corresponding control command, then load the quantized preprocessed data, and perform forward reasoning of the network. Through the GP and HP interfaces inside ZYNQ, according to the AXI bus protocol, the network structure control module sends parameters to the acceleration module, loads the image feature data, bias data and weight data into the on-chip storage module for caching, and waits for the acceleration module to calculate these data. After one layer of structure calculation is completed, the result is stored in DDR as the input data for the next layer. According to the control module instructions, perform the next round of calculation, repeat continuously, wait for the network model acceleration calculation to be completed, and obtain the final output data.

[0070] Specifically, the weight data is sent by the PS end. The quantized parameters have been loaded into the DDR (off-chip storage module) through the SD card, including weights, biases, and quantization parameters. Then, through DMA, the weights required for a calculation are sent to the on-chip storage module of the storage system as a weight cache area, waiting for the call of the acceleration module.

[0071] After sending the weight parameters, the bias data is sent and also transferred to the bias buffer area through DMA.

[0072] The data cache areas required for acceleration of the on-chip storage module are respectively set up to cache input data, weight data, bias data and control instructions. The input data, weight data and bias data are all transmitted quickly through DMA through the MIO port of zynq. The DMA is set to transmit 64 bits at a time, and the control instructions are transmitted through AXI_LITE. The commands are transmitted to the command register group, and a total of 3 32-bit command data are set. This register stores the parameter configuration commands of the convolutional neural network model and can be freely constructed.

[0073] The control instructions sent by the control module include DMA read valid signal, DMA write valid signal, convolution type signal, including 1*1 convolution and 3*3 convolution, pooling signal, data batch signal, whether to use activation function signal, whether to use padding signal, send data batch signal, image feature map row and column signal, etc.

[0074] The data cache module is specially designed to realize multi-channel parallel computing. If the weight data and bias data exceed the number allocated to each cache area by the on-chip storage module, the data needs to be sent in batches according to the batch signal of the control instruction. After each transmission is completed, the control instruction re-updates the parameters.

[0075] After sending the weight data and bias data, the image feature data is sent. Entering the acceleration module for calculation, the convolution accelerator calculates the image feature data with the weight data and bias data. At the same time, 8-channel data input is set, and 8-channel convolution calculations can be performed simultaneously. Each calculation consists of a group of multipliers and adders. The multiplication and addition calculations inside the FPGA can be completed by the DSP. Since the data bit width of the DSP is fixed, the convolution data is 8 bits, and the multiplication associative law can be used to make one DSP do two multiplications.

[0076] The control module allocates computing tasks, and each FPGA core represents a thread. Therefore, the thread parallelism can be used to implement pipeline design, without having to wait for a frame of image to be read before processing, thereby realizing loop unrolling convolution ( Figure 4 ) and pipeline computing ( Figure 5 ), calculate the pixels at different positions in the same feature map and the same convolution window simultaneously within one clock cycle, which is equivalent to parallelizing different elements of a convolution window. Input the original image frame by frame, transfer the data to the input buffer on the acceleration module, and start the convolution calculation process ( Figure 6 ), the convolution module not only occupies the key resources of the overall calculation, but also is the core module of the accelerator. When the instruction is sent to the convolution module, the data channel is first padded with zeros according to whether the image data is layer0, and then it will determine whether the convolution kernel of the convolution layer is a 3*3 convolution kernel. After the determination is completed, the data is passed to the matrix generation module, and then the sliding window of the convolution is generated, and the data in this window is passed to the DSP for calculation, and then the processed data is passed to the pooling module; the pooling module determines whether pooling operation is needed, and writes the data to the output data buffer after completing the maximum value pooling.

[0077] After each cache storage and calculation step on the PL side is completed, an interrupt signal will be sent to the PS side. After the control module receives the interrupt signal, it knows that the current operation has been completed and then executes the next operation to prevent data conflicts.

[0078] After the acceleration module inputs the calculated data into the output buffer, the accelerator's AXI_stream host will initiate a handshake signal, and the DMA receiving channel is the slave. After the AXI handshake is successful, the output data will be transferred back to the DDR. The storage space address of the DDR is controlled by the PS. After the transmission is completed, a DMA interrupt signal is sent, and a round of calculation is completed. This process can be repeated until the convolutional neural network structure calculation is completed. The calculation method of different structural layers is the same, and the parameters will change, which needs to be controlled by control instructions.

[0079] Step 9: Output image post-processing: According to the final output data, decode to obtain the target frame and confidence score, and obtain the detection frame of each frame image according to the target frame confidence score;

[0080] Convert the image's detection box position into actual image coordinates and send them to the FPGA;

[0081] Based on the received coordinate data, FPGA constructs an OSD interface on the display screen, draws the detection frame corresponding to the target, and outputs the result to the display screen through the HDMI interface.

[0082] The present invention also provides a ZYNQ-based convolutional neural network accelerator system, which consists of an image input module, a control module, a storage module, a computing acceleration module, and an output module. The system is based on a ZYNQ heterogeneous chip, and the system structure diagram is shown in FIG. Figure 7 Shown

[0083] The input module consists of a camera driver module and an image preprocessing module;

[0084] The control module is composed of a data flow control issuing module, a parameter configuration module and an interrupt receiving module;

[0085] The storage module is composed of an on-chip storage module and an off-chip storage module;

[0086] The acceleration module consists of a convolution calculation module, an upsampling calculation module, and a pooling calculation module;

[0087] The output module consists of a target frame drawing module and a display screen driver module;

[0088] Among them, the input module, acceleration module, output module, and on-chip storage module are designed by the FPGA side, and the configuration of the control module and off-chip storage module are designed by the ARM side;

[0089] The system design uses the vivado platform and the vitis platform. The input module is connected to an external camera to obtain image data.

[0090] The video stream system composed of the input module, storage module and output module can be divided into two paths, one of which is the original video stream, which obtains and displays the original resolution picture without passing through the acceleration module and the control module;

[0091] The other is an image accelerator that loads a convolutional neural network. The input module preprocesses the image on this path, completes the image size change, and is connected to the control module. The off-chip storage module is mobilized to load and configure parameters. Through data flow control, the parameters and feature data are stored in the on-chip storage module, waiting for the acceleration module to run. The acceleration module is connected to the storage system. This video does not output an image, but after the convolutional neural network acceleration operation is completed, the image coordinate data of the target detection frame is obtained, and the image coordinates of the target detection frame are transmitted to the output module. The output module draws the detection frame in the output screen of the original video stream based on the coordinate information, so that the two are superimposed and transmitted to the display screen for display.

[0092] The present invention improves the forward reasoning speed of the convolutional neural network, improves the instruction set construction speed, has low power consumption and is scalable.

[0093] The above specific process is used to illustrate the present invention, which is only used to help understand the acceleration process of the present invention, and is not intended to limit the present invention. Any local modification or replacement within the technical scope disclosed by the present invention by anyone familiar with the technology should be included in the scope of the present invention.

[0094] The above-described embodiments of the present application do not constitute a limitation on the protection scope of the present application.

Claims

1. A method for implementing a convolutional neural network image accelerator based on ZYNQ, the method comprising: Step 1: Select a convolutional neural network model, obtain the target image to be detected to create a data set, and store the data set on a server with a GPU; Step 2: Train the selected convolutional network model on a server with a GPU according to the acquired data set to obtain the weights and bias parameters of the trained floating-point convolutional neural network; Step 3: Quantize the convolutional neural network model and convert the floating-point weights and bias parameters trained on the GPU into fixed-point numbers. Step 4: Select the FPGA core board and design the architecture of the convolutional neural network accelerator based on the selected hardware board; Step 5: Load the convolutional neural network model into the hardware; Step 6: After reading the data from the SD card into the memory, program the driver modules of the camera and the display on the programmable logic array side; Step 7: After the image data starts to be transmitted, turn on the accelerator to perform image preprocessing, forward reasoning of the convolutional neural network model, and output image post-processing.

2. The method for implementing a ZYNQ-based convolutional neural network image accelerator according to claim 1, characterized in that: The FPGA core board is a ZYNQ chip, including an FPGA chip at the logic end and an ARM chip at the processor system end; wherein: The FPGA chip end is responsible for designing the input module, acceleration module, on-chip storage module and output module corresponding to the image preprocessing unit, convolutional neural network calculation unit, cache unit and image post-processing unit. The input module is connected to the off-chip storage module. The data of the off-chip storage module is transmitted to the on-chip storage module cache according to the command of the network structure control module, and then enters the acceleration module, and is finally output through the output module. The ARM chip side is responsible for building a forward reasoning framework for the convolutional neural network and writing data processing functions, designing a network structure control module and a DDR controller. The ARM side builds a forward reasoning framework through the network structure control module, and calls the storage module of the FPGA core board through the DDR controller to send data to the acceleration module.

3. The method for implementing a ZYNQ-based convolutional neural network image accelerator according to claim 2, characterized in that: The step 5 specifically includes: loading the convolutional neural network model into the hardware, using an external SD card to store the quantized weight parameters, bias parameters and activation function data, reading the file stored in the SD card into the DDR through the xilffs library on the ARM chip side, temporarily storing the data in the off-chip storage module, waiting for the network structure control module to issue an instruction to call, and accelerating in the acceleration module on the FPGA chip side.

4. The method for implementing a ZYNQ-based convolutional neural network image accelerator according to claim 3, characterized in that: The step 6 specifically includes: after reading the data from the SD card to the memory, programming the camera and display driver modules on the programmable logic array end, driving the camera through FPGA to acquire images and transferring the image data to the off-chip storage module, and finally displaying the picture on the display screen of the FPGA chip.

5. The method for implementing a ZYNQ-based convolutional neural network image accelerator according to claim 1, characterized in that: Image preprocessing specifically includes: The image data input by the camera is converted from 8-bit data to 24-bit image data in RGB888 format; according to the convolutional neural network model, the converted image data is resized and scaled to a size recognizable by the acceleration module; the resized image data is quantized to complete the image preprocessing operation.

6. The method for implementing a ZYNQ-based convolutional neural network image accelerator according to claim 1, characterized in that: The forward reasoning of the convolutional neural network model specifically includes: after reading the parameter file inside the SD card, generating the corresponding control command, then loading the quantized preprocessed data, and performing forward reasoning of the network. Through the GP and HP interfaces inside ZYNQ, according to the AXI bus protocol, the network structure control module sends parameters to the acceleration module, and loads the image feature data, bias data and weight data into the on-chip storage module for caching, waiting for the acceleration module to calculate these data. After one layer of structure calculation is completed, the result is stored in DDR as the input data of the next layer. According to the control module instructions, the next round of calculation is performed, and it is repeated continuously, waiting for the acceleration calculation of the convolutional neural network model to be completed, and the final output data is obtained.

7. The method for implementing a ZYNQ-based convolutional neural network image accelerator according to claim 1, characterized in that: The output image post-processing specifically includes: According to the output data of the convolutional neural network model, the target frame and confidence score are decoded, and the detection frame of each frame image is obtained according to the target frame confidence score; Convert the image's detection box position into actual image coordinates and send them to the FPGA; Based on the received coordinate data, FPGA constructs an OSD interface on the display screen and draws a detection frame corresponding to the target; and outputs the result to the display screen through the HDMI interface.

8. A ZYNQ-based convolutional neural network accelerator system for implementing any of the methods of claims 1 to 7, characterized in that: include: It includes an image input module, a control module, a storage module, a computing acceleration module and an output module; The input module consists of a camera driver module and an image preprocessing module; the camera driver module is used to acquire images and transfer image data to the off-chip storage module, and the image preprocessing module is used for image preprocessing; The control module consists of a data flow control issuing module, a parameter configuration module and an interrupt receiving module. The data flow control issuing module is used to issue calculation instructions to the acceleration module. The parameter configuration module is used to receive the training parameters stored in the SD card and load them into the off-chip memory. The interrupt receiving module is used to receive the calculation process feedback of the acceleration module. The storage module is composed of an on-chip storage module and an off-chip storage module. The camera driver module acquires images and transfers image data to the off-chip storage module, and the data of the off-chip storage module is transferred to the on-chip storage module. The acceleration module consists of a convolution calculation module, an upsampling calculation module, and a pooling calculation module. The convolution calculation module is used to accelerate the convolution operation in the convolutional neural network and complete the extraction of feature data. The upsampling calculation module is used to accelerate the upsampling calculation in the network layer and amplify the extracted feature image. The pooling calculation module is used to accelerate the pooling operation in the network layer. The output module consists of a target frame drawing module and a display screen driving module. The target frame drawing module is used to read the predicted frame diagram coordinates calculated by the convolutional neural network model and realize the drawing of the target frame diagram on the PL side. The display screen driving module is used to display the output results of the output module.

Images