FPGA (Field Programmable Gate Array) hardware acceleration system and method for YOLOv5 jujube trauma target detection, storage medium and electronic equipment

Abstract

The invention discloses an FPGA (Field Programmable Gate Array) hardware acceleration system and method for YOLOv5 jujube trauma target detection, a storage medium and electronic equipment. The FPGA hardware acceleration system comprises an ARM (Advanced RISC Machines) end, an FPGA end, an AXI bus and an external storage equipment SD (Secure Digital) card, the ARM end is responsible for reading data in the DDR, initializing each module, and scheduling forward reasoning, Yolo Head detection and non-maximum suppression of the whole YOLOv5 algorithm network to obtain a detection frame; the FPGA end mainly comprises a 3 * 3 convolution IP with the step length of 2 and 1, a 1 * 1 convolution IP with the step length of 1, an up-sampling IP, a 5 * 5 maximum pooling IP with the step length of 2 and a matrix addition IP; the ARM end and the FPGA end are connected through an AXI bus. According to the method, the architecture of time division multiplexing of a single calculation unit is adopted, and meanwhile, the strategy of maximum block transmission calculation is adopted, so that the occupation of the FPGA resource quantity is greatly reduced, and the reliability and the real-time performance are relatively high.

Description

Technical Field

[0001] The present invention relates to the field of target detection technology, and in particular to an FPGA hardware acceleration system, method, storage medium and electronic device for YOLOv5 jujube wound target detection. Background Art

[0002] At present, in recent years, with the continuous advancement of science and technology and the vigorous development of the field of artificial intelligence, image intelligent recognition and detection technology has made significant progress in various fields, including jujube trauma target detection technology. With the development of modern agriculture and the improvement of farmers' income level, jujube, as an important agricultural product, has gradually expanded its planting scale. However, jujube is threatened by various diseases and trauma, which makes target detection of jujube trauma a key link in agricultural production. Traditional target detection algorithms have many problems, such as low detection efficiency, poor accuracy, and high resource consumption, especially in the detection of jujube trauma targets in complex scenes. However, in recent years, with the continuous development of machine learning and GPU parallel technology, new target detection algorithms have emerged in an endless stream, and these algorithms have shown good potential in jujube trauma detection. Among them, the YOLO (You Only Look Once) series is one of the most widely used target detection algorithms. Through multiple version iterations and algorithm optimization, the YOLO algorithm has achieved significant advantages in performance. However, the accurate detection of jujube trauma still faces certain challenges. This may be due to factors such as the diversity of jujube trauma morphology, changes in lighting conditions, and possible occlusions in the image.

[0003] At present, compared with traditional image recognition methods, the deep learning-based jujube wound recognition method can automatically extract target features and has a higher recognition accuracy, so more and more scholars are studying this method. However, deep learning algorithms usually have the characteristics of many network layers, large number of parameters, and complex models. At present, most deep learning acceleration tools rely on high-performance CPU or GPU devices, but these devices have high power consumption, large size, and obvious heat generation, making them difficult to use in complex jujube garden environments. Compared with CPUs and GPUs, FPGAs have the advantages of high parallelism, reconfigurability, and low power consumption, and have become the mainstream hardware for deep learning to deploy neural networks.

[0004] Beijing University of Technology has disclosed a general hardware accelerator system platform that can be quickly deployed for the YOLO algorithm in its invention patent technology application "A general hardware accelerator system platform that can be quickly deployed for the YOLO algorithm" patent application number: 202210056834.9, publication number: CN 114662681A for the hardware implementation of the YOLO algorithm. A general hardware accelerator system platform including an ARM subsystem, an FPGA subsystem, and an off-chip memory is designed, which can quickly deploy different versions of the YOLO algorithm and realize modular design and efficient computing. The ARM subsystem is responsible for logic control and small-scale data processing, the FPGA subsystem performs high-density computing of the YOLO algorithm through parallel computing and resource optimization, and the off-chip memory handles large-scale data storage. However, the system also has some shortcomings, such as the system architecture and modular design increase the complexity and development and maintenance costs, the ability to adapt to other neural network algorithms is limited, high-density computing requires high FPGA resources, and data transmission and storage efficiency may become bottlenecks when processing large-scale data. Summary of the invention

[0005] The purpose of the present invention is to provide an FPGA hardware acceleration system, method, storage medium and electronic device for YOLOv5 jujube wound target detection, which can utilize the high parallelism of FPGA to accelerate the improved YOLOv5 neural network to achieve real-time detection of jujube wound targets.

[0006] The technical solution adopted by the present invention is:

[0007] An FPGA hardware acceleration system for YOLOv5 jujube wound target detection includes an ARM side, an FPGA side, an AXI bus, and an external storage device SD card; the ARM side is responsible for reading data in DDR, initializing each module, scheduling the forward reasoning of the entire YOLOv5 algorithm network, Yolo Head detection, and non-maximum suppression to obtain a detection frame; the FPGA side mainly includes 3×3 convolution IPs with step sizes of 2 and 1, 1×1 convolution IPs with a step size of 1, upsampling IPs, 5×5 maximum pooling IPs with a step size of 2, and matrix addition IPs; the ARM side and the FPGA side are connected via an AXI bus;

[0008] Among them, ARM is responsible for the scheduling and operation of the entire neural network accelerator. The weights and the images to be detected after layer fusion and quantization are stored in the SD card, and all the weights and detection images are read into the off-chip memory DDR: first, the input image is loaded into the on-chip block memory BRAM through the AXI bus, and then ARM will call the convolution unit, pooling unit and other computing units in the YOLO accelerator in sequence according to the network structure of YOLOv5. At the same time, the corresponding weights are loaded through the AXI bus and the intermediate results are stored in DDR again, waiting for the calculation of the next network layer; when all network layers are calculated, the output feature map of the network is stored in DDR, and ARM performs detection box decoding and non-maximum suppression NMS processing to obtain the detection result.

[0009] The steps include:

[0010] S1: Build a jujube injury target detection model based on improved YOLOv5. Specifically: In order to reduce the complexity of the model, the first layer of convolution of the jujube injury target detection model based on improved YOLOv5 is set to a 3×3 convolution that can be reused more times, and the activation function in the jujube injury target detection model based on improved YOLOv5 is set to LeakyRelu;

[0011] S2: Train the jujube wound target detection model based on improved YOLOv5 to obtain the weight data of the best model;

[0012] S3: The convolution layer and the BN layer are layer-fused, and then the weight data of the best improved YOLOv5 jujube wound target detection model is squared fixed-point quantized and exported as a binary bin file for initialization of the neural network accelerator;

[0013] S4: Use Vivado HLS tool to design and improve YOLOv5's date-like wound target detection network accelerator IP. Specifically, a double buffer mechanism is adopted to create two buffer layers in the on-chip BRAM to enable loading and computing operations to be performed simultaneously, thereby covering the loading time and improving the throughput of the accelerator;

[0014] S5: Use the Vivado tool to connect the input and output feature map interface and the read weight bias interface of the accelerator top-level IP to the ZYNQ PS end through the AXI bus, and connect the configuration information of each network layer IP in the accelerator to the ZYNQ PS end through the AXI-lite bus. Use the Vitis tool to schedule the orderly operation of the macro network, so that each network layer IP can correctly read the input feature map and weight bias from the DDR, and correctly save the calculation results to the DDR.

[0015] The step S2 specifically includes the following steps: constructing a jujube wound detection data set, and using the constructed data set to train the jujube wound target detection model of the improved YOLOv5, specifically:

[0016] By collecting samples through the designed image data acquisition device and making labels through manual and machine joint annotation, a data set of jujubes with multiple varieties, cross-origin, unified standards and complete indicators was constructed;

[0017] According to the actual sorting needs and the observation and analysis of the sampling data, the jujubes are initially divided into normal, crannied, crushed, and copied. The reflections and shadows in the data set are preprocessed using the illumination correction algorithm to improve the image quality.

[0018] For sample categories with a small proportion, CycleGAN and Auto Augment image enhancement strategies are used to expand samples;

[0019] After the data set is processed, the constructed jujube wound detection data set is used to train the improved YOLOv5 jujube wound target detection model to obtain the optimal weight data.

[0020] The step S3 specifically includes the following steps: Specifically: the principle of the convolutional layer and the BN layer before fusion is as follows:

[0021] y conv =wx+b, where w is the weight, x is the input, b is the bias, and y is the conv is the convolution output;

[0022] where y BN is the output of the BN layer, y conv is the convolution output, is the mean value of x, σ 2 is the variance of x, γ and β are learnable parameters, and ε is the minimum value that prevents the denominator from being zero;

[0023] The output of the convolution and BN layers after layer fusion is as follows:

[0024] Among them: Y is the output after convolution BN fusion, w is the convolution weight, b is the convolution bias, is the mean value of x, σ 2 is the variance of x, γ and β are learnable parameters, and ε is the minimum value that prevents the denominator from being zero;

[0025] The weight of the convolutional BN layer after layer fusion is recorded as: Y = w′x + b′ Among them: Y is the output after convolution BN fusion, w' is the fusion weight, b' is the fusion bias, w is the convolution weight, b is the convolution bias, is the mean value of x, σ 2 is the variance of x, γ and β are learnable parameters, and ε is the minimum value to prevent the denominator from being zero. After layer fusion, one convolution operation can represent one convolution plus one BN operation, which reduces the number of cache times of intermediate results.

[0026] By square quantizing the fused weights, the multiplication operation can be converted into a shift operation, which can be implemented using the lookup table (LUT) resources on the FPGA, reducing the consumption of the DSP multiplier, which is beneficial to improving resource utilization and accelerating the inference speed.

[0027] The principle of square quantization is as follows: for a weight ω with a bit width of m, the quantization value scaling factor α is multiplied by the quantization level,

[0028]

[0029] Map the original weight value to the mth power of 2, and the quantization function is defined as follows:

[0030]

[0031] is the mapping function from the original weight to the quantized weight. The function h(·) maps the range of the weight value from [-1, +1] to [0, 1]. The square of the quantized weight value 2 b When multiplied by the activation function value a, it is equivalent to shifting a b times. The formula is as follows:

[0032]

[0033] The improved YOLOv5 convolutional neural network accelerator in step S4 mainly includes 3×3 convolution IPs with step sizes of 2 and 1, 1×1 convolution IPs with a step size of 1, upsampling IPs, 5×5 maximum pooling IPs with a step size of 2, and matrix addition IPs.

[0034] In the step D, a maximum block transmission calculation strategy is designed for the 3×3 and 1×1 convolution IPs with the largest computational complexity: each time an array of the size of the complete output feature map is applied to Bram, TR×TC×TM data is calculated each time, and the coordinates of the corresponding input feature map and convolution weights are inverted according to the parameters of the convolution kernel and the shape of the input feature map to perform convolution operations and traverse the entire output feature map in sequence. Specifically:

[0035] Assuming that the width and height of the input image are fm_size, the number of zero padding bits of the convolution kernel is P, the size is K, and the step size is S, the shape of the output feature map is:

[0036]

[0037] According to the shape of the output feature map, the size of the input feature map is inferred as:

[0038] fm_size=Ofm*S-(2*P+SK)

[0039] When the block shape of the output feature map read once is TR, the block shape Trin of the corresponding input feature map is:

[0040] TRin=(Tr-1)*S-2*P+K

[0041] By moving the computing blocks on the output feature map in sequence to traverse the entire output map, the final output feature map is divided into blocks and passed into DDR; TR and TC are set as large as possible according to the shape distribution of the feature map in the network structure, and TM is made as large as possible according to the board resource limitations, so that the accelerator can achieve the highest performance.

[0042] The configuration information of each network layer in step S5 includes the input feature map size, the number of input feature map channels, the output feature map size, the number of output feature map channels, the kernel size, the step size, and the activation enable, specifically:

[0043] (1) When the kernel size is 3, call the 3×3 convolution IP to read the corresponding weights and biases for calculation, and at the same time judge the two cases of convolution kernel step size of 1 and 2 to complete different calculation strategies. Finally, according to the activation enable judgment, the calculation result is LeakyRelu activated and the calculation result is cached in DDR; (2) When the kernel size is 1, call the 1×1 convolution IP to read the corresponding weights and biases for calculation. Finally, according to the activation enable judgment, the calculation result is LeakyRelu activated and the calculation result is cached in DDR;

[0044] (3) When the kernel size is 5, the 5×5 maximum pooling IP is called to perform a pooling operation with a step size of 2 on the input feature map, and the calculation results are cached in DDR;

[0045] For upsampling layers and matrix addition layers, the kernel sizes of these two layers can be agreed in advance, and then the input and output feature map sizes and the number of channels can be compared with the input and output feature map sizes and the number of channels of the relevant layers in the YOLOv5 network structure to call the correct calculation IP for calculation;

[0046] Finally, after the above three calculation results are saved in DDR, ARM uses the anchor frame data obtained through training to decode the calculation results into detection frames. After decoding, NMS processing is performed to filter out redundant detection frames to obtain the final target position, target category and category confidence.

[0047] A computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the device where the computer-readable storage medium is located executes any one of the FPGA hardware acceleration methods for YOLOv5 jujube wound target detection.

[0048] An electronic device comprises: a memory and a processor, wherein the memory stores a program that can be run on the processor, and when the processor executes the program, any of the FPGA hardware acceleration methods for YOLOv5 jujube wound target detection is implemented.

[0049] The present invention adopts a single computing unit time-sharing multiplexing architecture and a maximum block transmission calculation strategy, which greatly reduces the occupation of FPGA resources and is backward compatible with low-cost FPGA1 with limited resources. It realizes the acceleration of the jujube wound target detection model based on YOLOv5 on FPGA, has the advantages of high parallelism, low power consumption and low cost, ensures its faster detection speed and higher detection accuracy, and has high reliability and real-time performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0051] Figure 1 The system structure diagram of the present invention is

[0052] Figure 2 This is a structural diagram of the improved YOLOv5 described in the present invention.

[0053] Figure 3 Computer graphics for the double buffering mechanism of the present invention.

[0054] Figure 4 This is a diagram showing the effect of the maximum block transmission strategy described in the present invention.

[0055] Figure 5 This is a diagram showing the detection results of a date image according to an example of the present invention.

[0056] Figure 6 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0057] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] like Figure 1 , 2 As shown in Figure 3, an FPGA hardware acceleration system for YOLOv5 jujube wound target detection is shown.

[0059] It includes ARM side, FPGA side, AXI bus and external storage device SD card; ARM side is responsible for reading data in DDR, initializing each module, scheduling forward reasoning of the entire YOLOv5 algorithm network, Yolo Head detection and non-maximum suppression to obtain detection frame; FPGA side mainly includes 3×3 convolution IP with step size of 2 and 1, 1×1 convolution IP with step size of 1, upsampling IP, 5×5 maximum pooling IP with step size of 2 and matrix addition IP; ARM side and FPGA side are connected through AXI bus;

[0060] Among them, ARM is responsible for the scheduling and operation of the entire neural network accelerator. The weights and the images to be detected after layer fusion and quantization are stored in the SD card, and all the weights and detection images are read into the off-chip memory DDR: first, the input image is loaded into the on-chip block memory BRAM through the AXI bus, and then ARM will call the convolution unit, pooling unit and other computing units in the YOLO accelerator in sequence according to the network structure of YOLOv5. At the same time, the corresponding weights are loaded through the AXI bus and the intermediate results are stored in DDR again, waiting for the calculation of the next network layer; when all network layers are calculated, the output feature map of the network is stored in DDR, and ARM performs detection box decoding and non-maximum suppression NMS processing to obtain the detection result.

[0061] This system adopts a single computing unit design, implements a single implementation of the network layer involved in the YOLOv5 algorithm, and uses time-sharing multiplexing. It introduces off-chip DDR memory to cache weight data and intermediate results, and performs fixed-point square quantization on weight data to further reduce storage resource consumption. At the same time, a block reading and block computing strategy is designed for the computing IP of each network layer, which can more flexibly balance the throughput and resource consumption of the accelerator.

[0062] FPGA hardware acceleration method for YOLOv5 jujube wound target detection, such as Figure 6 As shown, the following steps are included:

[0063] S1: Build a jujube injury target detection model based on improved YOLOv5. Specifically: In order to reduce the complexity of the model, the first layer of convolution of the jujube injury target detection model based on improved YOLOv5 is set to a 3×3 convolution that can be reused more times, and the activation function in the jujube injury target detection model based on improved YOLOv5 is set to LeakyRelu;

[0064] like Figure 2 , set the first layer of convolution of the model to 3×3 convolution that can be reused more times to improve the utilization of FPGA hardware resources, and set the activation function in the model to LeakyRelu to improve the efficiency of the accelerator. The LeakyRelu formula is as follows:

[0065]

[0066] Where β is a scaling factor when x is negative, and the default value is 0.01.

[0067] S2: Train the jujube injury target detection model based on the improved YOLOv5 to obtain the weight data of the best model; in actual use, first construct a jujube injury detection dataset, and use the constructed dataset to train YOLOv5. Specifically: collect samples through the designed image data acquisition device, use manual and machine joint annotation to make labels, and construct a jujube dataset with multiple varieties, cross-origin, unified standards, and perfect indicators. According to the actual sorting needs and the observation and analysis of the sampling data, the jujube is initially divided into normal, crannied, crushed, and scratched categories. The reflections and shadows in the dataset are preprocessed using the illumination correction algorithm to improve the image quality. For the sample categories with a small proportion, CycleGAN (Cycle Generative Adversarial Networks) and Auto Augment image enhancement strategies are used to expand the samples. After the dataset is processed, the constructed jujube injury detection dataset is used to train the jujube injury target detection model of the improved YOLOv5 to obtain the weight data of the best model.

[0068] S3: The convolution layer and the BN layer are layer-fused, and then the weight data of the best improved YOLOv5 jujube injury target detection model is squared and quantized and exported as a binary bin file for initialization of the neural network accelerator; the steps of S3 specifically include the following steps: Specifically: The principles of the convolution layer and the BN layer before fusion are as follows:

[0069] y conv =wx+b, where w is the weight, x is the input, b is the bias, and y is theconv is the convolution output;

[0070] where y BN is the output of the BN layer, y conv is the convolution output, is the mean value of x, σ 2 is the variance of x, γ and β are learnable parameters, and ε is the minimum value that prevents the denominator from being zero;

[0071] The output of the convolution and BN layers after layer fusion is as follows:

[0072] Among them: Y is the output after convolution BN fusion, w is the convolution weight, b is the convolution bias, is the mean value of x, σ 2 is the variance of x, γ and β are learnable parameters, and ε is the minimum value that prevents the denominator from being zero;

[0073] The weight of the convolutional BN layer after layer fusion is recorded as: Y = w′x + b′ Among them: Y is the output after convolution BN fusion, w' is the fusion weight, b' is the fusion bias, w is the convolution weight, b is the convolution bias, is the mean value of x, σ 2 is the variance of x, γ and β are learnable parameters, and ε is the minimum value to prevent the denominator from being zero. After layer fusion, one convolution operation can represent one convolution plus one BN operation, which reduces the number of cache times of intermediate results.

[0074] By square quantizing the fused weights, the multiplication operation can be converted into a shift operation, which can be implemented using the lookup table (LUT) resources on the FPGA, reducing the consumption of the DSP multiplier, which is beneficial to improving resource utilization and accelerating the inference speed.

[0075] The principle of square quantization is as follows: for a weight ω with a bit width of m, the quantization value scaling factor α is multiplied by the quantization level,

[0076]

[0077] Map the original weight value to the mth power of 2, and the quantization function is defined as follows:

[0078]

[0079] is the mapping function from the original weight to the quantized weight. The function h(·) maps the range of the weight value from [-1, +1] to [0, 1]. The square of the quantized weight value 2 b When multiplied by the activation function value a, it is equivalent to shifting a b times. The formula is as follows:

[0080]

[0081] S4: Use Vivado HLS tool to design and improve YOLOv5's jujube trauma target detection network accelerator IP. Specifically, a double buffer mechanism is adopted to create two buffer layers in the on-chip BRAM to enable loading and computing operations to be performed simultaneously, thereby covering the loading time and improving the throughput of the accelerator. In actual use, the improved YOLOv5 jujube trauma target detection network accelerator mainly includes 3×3 convolution IPs with strides of 2 and 1, 1×1 convolution IP with stride of 1, upsampling IP, 5×5 maximum pooling IP with stride of 2, and matrix addition IP.

[0082] In the neural network accelerator IP design, this application adopts a double buffer mechanism, creating two buffer layers in the on-chip BRAM to allow loading and computing operations to be performed simultaneously, thereby covering the loading time and improving the throughput of the accelerator. Ping-pong buffering is a key technology in program optimization, which is used to improve memory access efficiency and parallel processing capabilities. It has two working principles. First, the double buffer mechanism is as follows: Figure 3 , the system has two buffers. When one buffer is used for data calculation (such as convolution operation), the other buffer loads the next batch of data from the memory. Secondly, continuous data flow, this method can overlap the delay of on-chip processing and the delay of reading data off-chip (DRAM) and writing data back, minimize the idle time of the processor, ensure the continuity of data flow, and improve the overall system efficiency.

[0083] Considering the limited FPGA resources of low-cost edge devices, a maximum block transmission calculation strategy is designed for the 3×3 convolution IP with the largest computational workload. Figure 4 , each time an array of the size of the complete output feature map is requested in Bram, TR×TC×TM data is calculated each time, and the coordinates of the corresponding input feature map and convolution weights are inverted according to the parameters of the convolution kernel and the shape of the input feature map to perform convolution operations and traverse the entire output feature map in turn. Specifically:

[0084] Assuming that the width and height of the input image are fm_size, the padding of the convolution kernel is P, the size is K, and the step size is S, the shape of the output feature map is:

[0085]

[0086] According to the shape of the output feature map, the size of the output feature map is inferred as:

[0087] fm_size=Ofm*S-(2*P+SK)

[0088] When the block shape of the output feature map read once is TR, the block shape Trin of the corresponding input feature map is:

[0089] TRin=(Tr-1)*S-2*P+K

[0090] By moving the computation blocks on the output feature map sequentially to traverse the entire output map, the final output feature map is divided into blocks and transferred to DDR. TR and TC are set as large as possible according to the shape distribution of the feature map in the network structure, and TM is set as large as possible according to the board resource limitations to achieve the highest performance of the accelerator.

[0091] S5: Use the Vivado tool to connect the input and output feature map interface and the read weight bias interface of the accelerator top-level IP to the ZYNQ PS end through the AXI bus, and connect the configuration information of each network layer IP in the accelerator to the ZYNQ PS end through the AXI-lite bus. Use the Vitis tool to schedule the orderly operation of the macro network, so that each network layer IP can correctly read the input feature map and weight bias from the DDR, and correctly save the calculation results to the DDR.

[0092] The configuration information of each network layer in step S5 includes the input feature map size, the number of input feature map channels, the output feature map size, the number of output feature map channels, the kernel size, the step size, and the activation enable, specifically:

[0093] (1) When the kernel size is 3, the 3×3 convolution IP is called to read the corresponding weights and biases for calculation. At the same time, the two cases of convolution kernel step size of 1 and 2 are judged to complete different calculation strategies. Finally, Leakyrelu activation is performed on the calculation result according to the activation enable judgment time, and the calculation result is cached in DDR.

[0094] (2) When the kernel size is 1, the 1×1 convolution IP is called to read the corresponding weights and biases for calculation. Finally, Leakyrelu activation is performed on the calculation results according to the activation enable judgment, and the calculation results are cached in DDR.

[0095] (3) When the kernel size is 5, the 5×5 maximum pooling IP is called to perform a pooling operation with a step size of 2 on the input feature map, and the calculation results are cached in DDR.

[0096] For upsampling layers and matrix addition layers, the kernel sizes of these two layers can be agreed in advance, and then the input and output feature map sizes and the number of channels can be compared with the input and output feature map sizes and the number of channels of the relevant layers in the YOLOv5 network structure to call the correct calculation IP for calculation.

[0097] Finally, after the calculation results of the three detection heads are saved in DDR, ARM uses the anchor frame data obtained through training to decode the calculation results. After decoding, NMS processing is performed to filter out redundant detection frames to obtain the final target position, target category and category confidence. The visual detection results are shown in the figure. Figure 5 .

[0098] Finally, it is also possible to evaluate whether the detection accuracy and detection speed of the accelerator meet the actual needs, specifically including step S6: comparing the detection results of an image by the FPGA accelerator and the Pytorch-based network model, judging the detection accuracy and correctness of the accelerator, and evaluating whether it can meet the requirements of real-time detection based on the time consumed by the accelerator to detect an image.

[0099] This design has completed the deployment and implementation of the improved YOLOv5n jujube wound detection model on the PYNQ-Z2 platform, and detected jujube images with a size of 640*640. The detection results are as follows Figure 5 As shown in the figure, the detection accuracy is similar to that of the YOLOv5 model running on the GPU, and the latency is only 0.2s.

[0100] A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the device where the computer-readable storage medium is located executes the FPGA hardware acceleration method for YOLOv5 jujube wound target detection as described above. The computer program includes computer program code, which may be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory and other memory.

[0101] An electronic device comprises: a memory and a processor, wherein the memory stores a program that can be run on the processor, and when the processor executes the program, the FPGA hardware acceleration method for YOLOv5 jujube wound target detection as described above is implemented.

[0102] If the module / unit integrated in the electronic device described in this application is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware devices through a computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by the processor, the steps of each of the above-mentioned method embodiments can be implemented.

[0103] Furthermore, the computer-readable storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the blockchain node, etc.

[0104] Computer-readable instructions are stored in the computer-readable storage medium, and the computer-readable instructions are executed by a processor in an electronic device to implement the FPGA hardware acceleration method for YOLOv5 jujube wound target detection described in any of the above embodiments.

[0105] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

[0106] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0107] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the patent application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent application shall be subject to the attached claims.

[0108] It should be noted that the terms "including" and "having" and any variations thereof in the specification and claims of the present application are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or inherent to these processes, methods, products or apparatuses.

[0109] Note that the above are only preferred embodiments of the present invention and the principles of the application technology. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention is described in more detail through the above embodiments, the present invention is not limited to the specific embodiments described herein, and may include more other effective embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. An FPGA hardware acceleration system for YOLOv5 jujube wound target detection, characterized by: It includes ARM side, FPGA side, AXI bus and external storage device SD card; ARM side is responsible for reading data in DDR, initializing each module, scheduling forward reasoning of the entire YOLOv5 algorithm network, Yolo Head detection and non-maximum suppression to obtain detection frame; FPGA side mainly includes 3×3 convolution IP with step size of 2 and 1, 1×1 convolution IP with step size of 1, upsampling IP, 5×5 maximum pooling IP with step size of 2 and matrix addition IP; ARM side and FPGA side are connected through AXI bus; Among them, ARM is responsible for the scheduling and operation of the entire neural network accelerator. The weights and the images to be detected after layer fusion and quantization are stored in the SD card, and all the weights and detection images are read into the off-chip memory DDR: first, the input image is loaded into the on-chip block memory BRAM through the AXI bus, and then ARM will call the convolution unit, pooling unit and other computing units in the YOLO accelerator in sequence according to the network structure of YOLOv5. At the same time, the corresponding weights are loaded through the AXI bus and the intermediate results are stored in DDR again, waiting for the calculation of the next network layer; when all network layers are calculated, the output feature map of the network is stored in DDR, and ARM performs detection box decoding and non-maximum suppression NMS processing to obtain the detection result.

2. According to claim 1, a FPGA hardware acceleration method for YOLOv5 jujube wound target detection is characterized in that: The steps include: S1: Build a jujube injury target detection model based on improved YOLOv5. Specifically: In order to reduce the complexity of the model, the first layer of convolution of the jujube injury target detection model based on improved YOLOv5 is set to a 3×3 convolution that can be reused more times, and the activation function in the jujube injury target detection model based on improved YOLOv5 is set to LeakyRelu; S2: Train the jujube wound target detection model based on improved YOLOv5 to obtain the weight data of the best model; S3: The convolution layer and the BN layer are layer-fused, and then the weight data of the best improved YOLOv5 jujube wound target detection model is squared fixed-point quantized and exported as a binary bin file for initialization of the neural network accelerator; S4: Use Vivado HLS tool to design and improve YOLOv5's date-like wound target detection network accelerator IP. Specifically, a double buffer mechanism is adopted to create two buffer layers in the on-chip BRAM to enable loading and computing operations to be performed simultaneously, thereby covering the loading time and improving the throughput of the accelerator; S5: Use the Vivado tool to connect the input and output feature map interface and the weight bias reading interface of the accelerator's top-level IP to the ZYNQ PS end through the AXI bus. At the same time, connect the configuration information of each network layer IP in the accelerator to the ZYNQ PS end through the AXI-lite bus. Use the Vitis tool to schedule the orderly operation of the macro network, so that each network layer IP can correctly read the input feature map and weight bias from the DDR, and correctly save the calculation results to the DDR.

3. The FPGA hardware acceleration method for YOLOv5 jujube wound target detection according to claim 1 is characterized in that: The step S2 specifically includes the following steps: constructing a jujube wound detection data set, and using the constructed data set to train the jujube wound target detection model of the improved YOLOv5, specifically: By collecting samples through the designed image data acquisition device and making labels through manual and machine joint annotation, a data set of jujubes with multiple varieties, cross-origin, unified standards and complete indicators was constructed; According to the actual sorting needs and the observation and analysis of the sampling data, the jujubes are initially divided into normal, crannied, crushed, and copied. The reflections and shadows in the data set are preprocessed using the illumination correction algorithm to improve the image quality. For sample categories with a small proportion, CycleGAN and Auto Augment image enhancement strategies are used to expand samples; After the data set is processed, the constructed jujube wound detection data set is used to train the improved YOLOv5 jujube wound target detection model to obtain the optimal weight data.

4. The FPGA hardware acceleration method for YOLOv5 jujube wound target detection according to claim 2 is characterized in that: The step S3 specifically includes the following steps: Specifically: The principle of the convolutional layer and the BN layer before fusion is as follows: conv =wx+b, where w is the weight, x is the input, b is the bias, y is conv is the convolution output; where y BN is the output of the BN layer, y conv is the convolution output, is the mean value of x, σ 2 is the variance of x, γ and β are learnable parameters, and ε is the minimum value that prevents the denominator from being zero; The output of the convolution and BN layers after layer fusion is as follows: Among them: Y is the output after convolution BN fusion, w is the convolution weight, b is the convolution bias, is the mean value of x, σ 2 is the variance of x, γ and β are learnable parameters, and ε is the minimum value that prevents the denominator from being zero; The weight of the convolutional BN layer after layer fusion is recorded as: Y = w′x + b′ Among them: Y is the output after convolution BN fusion, w' is the fusion weight, b' is the fusion bias, w is the convolution weight, b is the convolution bias, is the mean value of x, σ 2 is the variance of x, γ and β are learnable parameters, and ε is the minimum value to prevent the denominator from being zero. After layer fusion, one convolution operation can represent one convolution plus one BN operation, which reduces the number of cache times of intermediate results. By square quantizing the fused weights, the multiplication operation can be converted into a shift operation, which can be implemented using the lookup table (LUT) resources on the FPGA, reducing the consumption of the DSP multiplier, which is beneficial to improving resource utilization and accelerating the inference speed.

5. The FPGA hardware acceleration method for YOLOv5 jujube wound target detection according to claim 3 is characterized in that: The principle of square quantization is as follows: for a weight ω with a bit width of m, the quantization value scaling factor α is multiplied by the quantization level, Map the original weight value to the mth power of 2, and the quantization function is defined as follows: is the mapping function from the original weight to the quantized weight. The function h(·) maps the range of the weight value from [-1, +1] to [0, 1]. The square of the quantized weight value 2 b When multiplied by the activation function value a, it is equivalent to shifting a b times. The formula is as follows:

6. The FPGA hardware acceleration system for YOLOv5 jujube wound target detection according to claim 1, characterized in that: The improved YOLOv5 convolutional neural network accelerator in step S4 mainly includes 3×3 convolution IPs with step sizes of 2 and 1, 1×1 convolution IPs with a step size of 1, upsampling IPs, 5×5 maximum pooling IPs with a step size of 2, and matrix addition IPs.

7. The FPGA hardware acceleration method for YOLOv5 jujube wound target detection according to claim 3 is characterized in that: In the step D, a maximum block transmission calculation strategy is designed for the 3×3 and 1×1 convolution IPs with the largest computational complexity: each time an array of the size of the complete output feature map is applied to Bram, TR×TC×TM data is calculated each time, and the coordinates of the corresponding input feature map and convolution weights are inverted according to the parameters of the convolution kernel and the shape of the input feature map to perform convolution operations and traverse the entire output feature map in sequence. Specifically: Assuming that the width and height of the input image are fm_size, the number of zero padding of the convolution kernel is P, the size is K, and the step size is S, the shape of the output feature map is: According to the shape of the output feature map, the size of the input feature map is inferred as: fm_size=Ofm*S-(2*P+SK) When the block shape of the output feature map read once is TR, the block shape Trin of the corresponding input feature map is: TRin=(Tr-1)*S-2*P+K By moving the computing blocks on the output feature map in sequence to traverse the entire output map, the final output feature map is divided into blocks and passed into DDR; TR and TC are set as large as possible according to the shape distribution of the feature map in the network structure, and TM is made as large as possible according to the board resource limitations, so that the accelerator can achieve the highest performance.

8. The FPGA hardware acceleration method for YOLOv5 jujube wound target detection according to claim 3 is characterized in that: The configuration information of each network layer in step S5 includes the input feature map size, the number of input feature map channels, the output feature map size, the number of output feature map channels, the kernel size, the step size, and the activation enable, specifically: (1) When the kernel size is 3, call the 3×3 convolution IP to read the corresponding weights and biases for calculation. At the same time, judge the two cases where the convolution kernel step size is 1 and 2 to complete different calculation strategies. Finally, according to the activation enable judgment, LeakyRelu is activated for the calculation result, and the calculation result is cached in DDR; (2) When the kernel size is 1, call the 1×1 convolution IP to read the corresponding weights and biases for calculation. Finally, perform LeakyRelu activation on the calculation results according to the activation enable judgment, and cache the calculation results in DDR. (3) When the kernel size is 5, the 5×5 maximum pooling IP is called to perform a pooling operation with a step size of 2 on the input feature map, and the calculation results are cached in DDR; For upsampling layers and matrix addition layers, the kernel sizes of these two layers can be agreed in advance, and then the input and output feature map sizes and the number of channels can be compared with the input and output feature map sizes and the number of channels of the relevant layers in the YOLOv5 network structure to call the correct calculation IP for calculation; Finally, after the above three calculation results are saved in DDR, ARM uses the anchor frame data obtained through training to decode the calculation results into detection frames. After decoding, NMS processing is performed to filter out redundant detection frames to obtain the final target position, target category and category confidence.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, the device where the computer-readable storage medium is located executes the FPGA hardware acceleration method for YOLOv5 jujube wound target detection as described in any one of claims 1 to 7.

10. An electronic device, characterized in that: include: A memory and a processor, wherein the memory stores a program that can be run on the processor, and when the processor executes the program, the FPGA hardware acceleration method for YOLOv5 jujube wound target detection as described in any one of claims 1-7 is implemented.

Images