FPGA-based target detection method and system

Abstract

The application provides a target detection method and system based on FPGA, comprising: designing a target detection model according to FPGA hardware resources, wherein the target detection model comprises a hardware accelerator based on GPU and a YOLOv5 model; performing clock domain optimization configuration on an IP core of the GPU; adapting and training the YOLOv5 model based on the GPU after optimization configuration; and performing fast target detection on input images by using the target detection model based on single instruction multiple data stream technology. In this way, the synchronization and coordination between various components of the FPGA are ensured, the inference speed and the overall performance of the system are significantly improved while maintaining the algorithm accuracy, the data throughput of the target detection model is significantly improved, the target detection scheme has the advantages of high flexibility and low cost due to the characteristics of FPGA design, and therefore high-precision, high-flexibility and low-cost target detection on the basis of FPGA is realized, and the problem of poor flexibility and high cost of the hardware acceleration scheme in the existing target detection task is solved.

Description

Technical Field

[0001] This invention relates to the field of target detection technology, and in particular to a target detection method and system based on FPGA. Background Technology

[0002] With the development of computer technology, more and more tasks rely on computers to achieve efficient and accurate task completion. For example, in the field of deep learning acceleration, especially in object detection tasks, the YOLOv5 model has received widespread attention due to its efficient algorithm design and excellent performance.

[0003] However, in existing object detection tasks, hardware acceleration solutions primarily rely on dedicated deep learning accelerators, such as graphics processing units (GPUs) or application-specific integrated circuits (ASICs). While GPUs and ASICs offer faster processing speeds, they also have limitations in terms of algorithm accuracy, flexibility, and cost-effectiveness. For example, while GPU-based solutions can achieve good training results and inference speeds, their structural characteristics and operating mechanisms often prevent them from providing sufficient flexibility and scalability while maintaining high accuracy. Furthermore, the high power consumption and cost of GPUs are uneconomical for many application scenarios. Conversely, while ASICs offer higher power consumption and can be customized to meet specific business needs, their development cycle is long and costly, and once designed, they are difficult to adapt to new algorithm requirements.

[0004] In addition, commonly used hardware acceleration solutions typically require complex software stacks and high hardware costs, which limits their widespread application in edge computing and resource-constrained environments. Summary of the Invention

[0005] The purpose of this invention is to provide an FPGA-based target detection method and system, so as to at least solve the problems of poor flexibility and high cost of existing hardware acceleration solutions in target detection tasks.

[0006] To address the aforementioned technical problems, this invention provides an FPGA-based target detection method, comprising:

[0007] Based on FPGA hardware resources, a target detection model is designed, which includes a hardware accelerator based on a deep learning processor, a YOLOv5 model, and a CPU.

[0008] Optimize the clock domain configuration of the IP cores of the deep learning processor;

[0009] Based on the optimized configuration of the deep learning processor, the YOLOv5 model is adapted and trained.

[0010] Based on single instruction multiple data stream (SMILE) technology, this method utilizes a target detection model to perform fast target detection on the input image.

[0011] Optionally, in the FPGA-based target detection method, the method for designing the target detection model based on FPGA hardware resources includes:

[0012] Design a hardware accelerator based on a deep learning processor, taking advantage of FPGA hardware resources.

[0013] A runtime environment is built using Petalinux, which includes drivers for hardware accelerators, library functions, and necessary middleware.

[0014] Connect the YOLOv5 model and CPU to the hardware accelerator for communication.

[0015] Optionally, in the FPGA-based target detection method, the method for designing a hardware accelerator based on a deep learning processor includes:

[0016] Zynq UltraScale+ MPSoC 3.4 was selected as the processing system for the hardware accelerator;

[0017] Define the resource consumption of the deep learning processor, where the parallel number of input feature channels and the parallel number of output feature channels are both 16, the parallel number of pixels is 8, and each MAC operation consists of one multiplication and one addition.

[0018] Configure the Ultra-RAM of the deep learning processor to 65 and the ALU Parallel number to 4;

[0019] The softmax function, which is used for model normalization calculation in deep learning processors, is transferred to the CPU for calculation.

[0020] The AXI Interconnection IP is used to connect the IP core of the deep learning processor to the processing system and to allocate register addresses.

[0021] Optionally, in the FPGA-based target detection method, when using AXI Interconnection IP to connect the IP core of the deep learning processor to the processing system, the IP core of the deep learning processor is equipped with three main interfaces, one for instruction extraction and the other two for data access. The main interface for instruction extraction is a low-bandwidth interface, and the main interface for data access is a high-bandwidth interface.

[0022] Optionally, in the FPGA-based target detection method, the method for optimizing the clock domain configuration of the deep learning processor's IP core includes:

[0023] The register clock frequency of the IP core of the deep learning processor is configured to be 100MHz;

[0024] The data controller clock frequency of the IP core configured for the deep learning processor is 295MHz;

[0025] The computing engine clock frequency of the IP core configured for the deep learning processor is 590MHz, and the computing engine clock is aligned with the data controller clock.

[0026] Set up two parallel data processing paths in the deep learning processor.

[0027] Optionally, in the FPGA-based target detection method, the method for adapting and training the YOLOv5 model based on the optimized deep learning processor includes:

[0028] For the feature extraction part of the YOLOv5 model, the convolutional layer using the SiLU activation function is replaced with the convolutional layer using the LeakyReLU activation function to obtain the CBL layer, where the parameter of the LeakyReLU activation function is set to 0.1015625 when it is less than 0.

[0029] The original convolutions in the CBL layer of the YOLOv5 model are replaced with 1×1 convolutions and depthwise separable convolutions.

[0030] Change the splicing layer in layer C3 of the YOLOv5 model to a summation layer;

[0031] The configured YOLOv5 model was trained and evaluated using the COCO dataset;

[0032] If the evaluation results meet the preset requirements, training will stop, and a trained YOLOv5 model will be obtained.

[0033] Optionally, in the FPGA-based target detection method, the method for training and evaluating the configured YOLOv5 model using the COCO dataset includes:

[0034] Set a minimum of 200 training cycles;

[0035] Data augmentation techniques were used to process the COCO dataset;

[0036] In each training cycle, the YOLOv5 model is trained using the data-augmented COCO dataset. The YOLOv5 model learns data features through forward propagation and optimizes network weights through backpropagation.

[0037] The performance of the trained YOLOv5 model was evaluated using the mAP metric.

[0038] Optionally, in the FPGA-based target detection method, the method for evaluating the performance of the trained YOLOv5 model using the mAP metric includes:

[0039] The accuracy of the YOLOv5 model is measured by the mAP value at a confidence threshold of 0.5, where a higher mAP value indicates better accuracy of the YOLOv5 model.

[0040] The robustness of the YOLOv5 model is measured by the average mAP value within a confidence threshold range of 0.5 to 0.95, where a higher average mAP value indicates better robustness of the YOLOv5 model.

[0041] Optionally, in the FPGA-based target detection method, the method for fast target detection of the input image using a target detection model based on single instruction multiple data stream technology includes:

[0042] Construct a pipeline computation graph, which includes several tensors and several operators. The tensors are used to define the dimensions and shape of the data, and the operators are used to define computational operations.

[0043] Preprocess the input image;

[0044] Inference is performed on the preprocessed image to perform object detection.

[0045] Optionally, in the FPGA-based target detection method, the method of performing inference on the preprocessed image to perform target detection includes:

[0046] Construct a read queue and a display queue;

[0047] The preprocessed image is placed into the read queue, and the deep learning processor is used to perform forward reasoning on the image to obtain tensor data in NHWC format of the prediction result.

[0048] Convert tensor data from NHWC format to NCHW format;

[0049] The tensor data in NCHW format is normalized to obtain the position information of the prediction box relative to the original input image.

[0050] Based on the set confidence threshold, the predicted boxes are filtered, and non-maximum suppression is performed on the remaining predicted boxes after filtering to obtain the final predicted boxes.

[0051] The final prediction box is used for inference to obtain the prediction result;

[0052] The inferred image is sent to the display queue so that the image with the prediction result is displayed frame by frame.

[0053] Optionally, in the FPGA-based target detection method, the method for normalizing the NCHW format tensor data to obtain the position information of the prediction box relative to the original input image includes:

[0054] The softmax function is used to normalize the tensor data in NCHW format to obtain coordinate information;

[0055] The coordinate information is transformed to obtain the initial position information of the prediction box. The transformation formula is as follows:

[0056]

[0057]

[0058]

[0059]

[0060] in, , These are the x and y coordinates of the center point of the prediction box, respectively. To predict the actual width of the bounding box, To predict the actual height of the bounding box, For cell width, The height of the cell. This is the horizontal offset of the center point of the prediction box relative to the top-left corner of its cell. The vertical offset of the center point of the prediction box relative to the top-left corner of its cell. The width of the secondary prediction box outside the prediction box. The height of the secondary prediction box outside the prediction box;

[0061] The initial position information of the prediction box is mapped back to the size of the original input image to obtain the position information of the prediction box relative to the original input image.

[0062] To address the aforementioned technical problems, the present invention also provides an FPGA-based target detection system, comprising a memory located on the FPGA, a processor, and an executable program stored in the memory and capable of being run by the processor; when the processor runs the executable program, it executes the FPGA-based target detection method as described in any of the preceding claims.

[0063] The present invention provides an FPGA-based target detection method and system, comprising: designing a target detection model based on FPGA hardware resources, wherein the target detection model includes a hardware accelerator based on a deep learning processor, a YOLOv5 model, and a CPU; optimizing the clock domain configuration of the IP core of the deep learning processor; adapting and training the YOLOv5 model based on the optimized deep learning processor; and using the target detection model to perform fast target detection on the input image based on single instruction multiple data stream (SID) technology. Optimizing the clock domain configuration of the IP core of the deep learning processor ensures synchronization and coordination between the IP core of the deep learning processor and other components of the FPGA; adapting and training the YOLOv5 model significantly improves inference speed and overall system performance while maintaining algorithm accuracy; and significantly improves the data throughput of the target detection model through SID technology. Since the target detection model is based on an FPGA design, it benefits from the flexibility and low cost of FPGA design, giving the target detection scheme high flexibility and low cost advantages. This achieves high-precision and high-flexibility target detection on an FPGA basis, solving the problem of poor flexibility and high cost of existing hardware acceleration schemes in target detection tasks. Attached Figure Description

[0064] Figure 1 This is a flowchart of the FPGA-based target detection method provided in this embodiment;

[0065] Figure 2 This is a schematic diagram illustrating the process of building a runtime environment using Petalinux in this embodiment;

[0066] Figure 3 A schematic diagram illustrating the clock domain optimization configuration of the IP core of the deep learning processor provided in this embodiment;

[0067] Figure 4 This is an optimized schematic diagram of the feature extraction part in the YOLOv5 model provided in this embodiment;

[0068] Figure 5 This is a schematic diagram of the lightweight CBL provided in this embodiment;

[0069] Figure 6 This is a schematic diagram of the lightweight C3 layer provided in this embodiment;

[0070] Figure 7 This is a schematic diagram of the optimized YOLOv5 model structure provided in this embodiment;

[0071] Figure 8 The mAP value of the optimized YOLOv5 model provided in this embodiment;

[0072] Figure 9 This is a comparative diagram showing the execution methods of target detection provided in this embodiment;

[0073] Figure 10 This is a schematic diagram of the computation graph tensor provided in this embodiment;

[0074] Figure 11 This is a schematic diagram of the relative positions of the prediction boxes provided in this embodiment;

[0075] Figure 12 This is a schematic diagram of the video stream prediction process provided in this embodiment. Detailed Implementation

[0076] The FPGA-based target detection method and system proposed in this invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that the drawings are all in a very simplified form and use non-precise scales, used only to facilitate and clarify the illustration of the embodiments of this invention. Furthermore, the structures shown in the drawings are often part of the actual structure. In particular, different figures may emphasize different aspects and sometimes use different scales.

[0077] It should be noted that the terms "first," "second," etc., used in the specification, claims, and drawings of this invention are used to distinguish similar objects in order to describe embodiments of the invention, and are not used to describe a specific order or sequence. It should be understood that such uses of terminology are interchangeable where appropriate. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0078] With the development of semiconductor technology, FPGA (Field Programmable Gate Array) has emerged. FPGA is a programmable integrated circuit pre-designed and implemented on a silicon chip. It can be configured into a specified circuit structure according to the designer's needs, freeing customers from relying on ASIC chips designed and manufactured by chip manufacturers. This makes integrated circuit design and software development more flexible and reduces development costs.

[0079] Based on this, this embodiment provides a target detection method based on FPGA, such as... Figure 1 As shown, it includes:

[0080] S1. Based on FPGA hardware resources, design a target detection model, wherein the target detection model includes a hardware accelerator based on a deep learning processor, a YOLOv5 model, and a CPU.

[0081] S2 optimizes the clock domain configuration of the deep learning processor's IP core;

[0082] S3, based on an optimized deep learning processor, adapts and trains the YOLOv5 model;

[0083] S4, based on single instruction multiple data stream technology, uses a target detection model to perform fast target detection on the input image.

[0084] The FPGA-based target detection method provided in this embodiment optimizes the clock domain configuration of the deep learning processor's IP core, ensuring synchronization and coordination between the deep learning processor's IP core and other FPGA components. By adapting and training the YOLOv5 model, it significantly improves inference speed and overall system performance while maintaining algorithm accuracy. Single instruction multiple data stream (SID) technology significantly enhances the data throughput of the target detection model. Since the target detection model is based on an FPGA design, it benefits from the flexibility and low cost of FPGA design, resulting in a highly flexible and cost-effective target detection solution. This achieves high-precision and highly flexible target detection on an FPGA platform, solving the problem of poor flexibility and high cost of existing hardware acceleration solutions for target detection tasks.

[0085] Specifically, in this embodiment, step S1, the method for designing a target detection model based on FPGA hardware resources, includes:

[0086] S11, based on FPGA hardware resources, designs a hardware accelerator based on a deep learning processor (GPU).

[0087] In this embodiment, the method for designing a hardware accelerator based on a deep learning processor includes: selecting Zynq UltraScale+ MPSoC 3.4 as the processing system side of the hardware accelerator; defining the resource consumption of the deep learning processor, wherein the parallel number of input feature channels and the parallel number of output feature channels are both 16, the parallel number of pixels is 8, and each MAC operation consists of one multiplication and one addition; configuring the Ultra-RAM of the deep learning processor to 65 and the ALU Parallel number to 4; transferring the softmax function for model normalization calculation in the deep learning processor to the CPU for calculation; using AXIInterconnection IP to communicate between the IP core of the deep learning processor and the processing system side, and allocating register addresses.

[0088] In one specific embodiment, the Vivado 2022.1 design suite can be used for hardware design and implementation. For example, the Xilinx DPUCZDX8G 1.4.1 version DPU IP core is used for efficient block design, and corresponding hardware resources and clock domains are configured. During the design process, the Zynq UltraScale+ MPSoC 3.4 is selected as the processing system (PS) of the hardware accelerator to provide powerful computing capabilities and flexible system interfaces. Regarding DPU configuration, the B4096 configuration option is selected, which defines the DPU's resource consumption. The formula for calculating resource consumption is: DPU resource consumption = number of parallel input feature channels × number of parallel output feature channels × number of parallel pixels. In this design, the number of parallel pixels is configured to 8, and the number of parallel input and output feature channels is 16 each. Each MAC operation consists of one multiplication and one addition. Therefore, the DPU can perform 8 × 16 × 16 × 2 = 4096 operations (ops) in one clock cycle.

[0089] Ideally, to optimize resource utilization, each DPU is configured with 65 Ultra-RAM units and 4 ALU parallel units. Furthermore, considering computational complexity, the softmax function for lightweight model normalization is moved to the CPU for computation; therefore, the SFM core configuration of the DPU is omitted to save hardware resources and reduce power consumption.

[0090] In terms of interface configuration and connectivity, the DPUCZDX8G IP core is connected to the processing system (PS) via the AXI Interconnection IP, and a register address is assigned to it. The DPUCZDX8G core is equipped with three main interfaces, one for instruction fetching and two for data access. The main interface for instruction fetching is a low-bandwidth interface, while the main interface for data access is a high-bandwidth interface, to accommodate different data flow requirements.

[0091] The specific connection configuration is as follows:

[0092] - The slave port S_AXI of the DPUCZDX8G is connected to the M_AXI_HPM0_LPD of the PS;

[0093] - DPUO_M_AXI_DATAO is connected to PS's S_AXI_HPO_FPD;

[0094] - DPUO_M_AXI_DATA1 is connected to PS's S_AXI_HP1_FPD;

[0095] - DPUO_M_AXI_INSTR is connected to PS's S_AXI_LPD;

[0096] After completing the DPUCZDX8G connection, the next step is to allocate register addresses for the AXI slave interface. The DPUCZDX8G requires a minimum of 16 MB of space, and its slave interface can be allocated to any starting address accessible by the host CPU. A reference configuration is shown in Table 1 below, which details the interface connections and resource allocation:

[0097]

[0098] Table 1. Interface Configuration Table

[0099] S12, using Petalinux to build a runtime environment, which includes hardware accelerator drivers, library functions, and necessary middleware.

[0100] In one specific embodiment, the Petalinux operating system build tools and the Vitis AI development kit can be used to customize and optimize the hardware accelerator's drivers, library functions, and necessary middleware to support efficient model deployment and inference processes.

[0101] In this embodiment, as Figure 2 As shown, the Petalinux project is first started based on an XSA (Xilinx Support Archive) file generated by Vivado. The XSA file contains comprehensive information about the FPGA hardware design, such as logic structure, clock configuration, and interface definitions. It defines the hardware resources and constraints of the FPGA, providing a foundation for subsequent software development and hardware integration.

[0102] Next, the Petalinux project configures the Linux kernel, with a particular emphasis on including the Xilinx Deep Learning Processing Unit (DPU) driver. This allows the operating system to recognize and manage DPU resources, providing the necessary support for deep learning object detection models; the integration of the DPU driver ensures that the operating system can communicate effectively with the DPU, enabling efficient data processing.

[0103] Ideally, to further enhance the system's functionality, the Vitis AI Computing Library can also be integrated. This is a software library developed by Xilinx specifically for deep learning applications, containing algorithms and models optimized for deep learning, as well as practical tools for data processing and model deployment, making it possible to execute AI models on FPGAs.

[0104] Next, a rootfs (root filesystem) is built, containing a filesystem image of the operating system and application files required by the system. By packaging the rootfs with the system files, a Petalinux image file (petalinux-sdimage.wic.gz) is generated. This file can then be copied to an SD card for loading and execution when the FPGA system boots. This ensures that the system can correctly load all necessary components and configurations at startup.

[0105] Furthermore, regarding the dynamic loading of hardware accelerators, the Device Tree File (DTS), the bitstream file (.bit) of the hardware design, and the dynamic loading configuration file can be packaged together. These files collectively describe the hardware topology, configuration, and loading parameters, ensuring that the hardware design can be correctly identified and loaded by the system. Specifically, this loading process can be executed using the xmutil command-line tool provided by the Petalinux system. This tool is responsible for loading the hardware design into the programmable logic section of the FPGA, thereby enabling the dynamic deployment of the hardware accelerator. The use of the xmutil tool simplifies the loading and management process of hardware accelerators, improving the system's flexibility and scalability.

[0106] S13 connects the YOLOv5 model and CPU to the hardware accelerator for communication.

[0107] The implementation method of connecting the YOLOv5 model and CPU to the hardware accelerator via FPGA is well known to those skilled in the art, and will not be described in detail here. At this point, the prototype of the target detection model has been designed. Next, the target detection model needs to be optimized and configured.

[0108] Furthermore, in this embodiment, as Figure 3 As shown, step S2, the method for optimizing the clock domain configuration of the deep learning processor (GPU) IP cores, includes:

[0109] S21, configure the register clock frequency of the deep learning processor IP core to 100MHz. The register clock (s_axi_aclk) is dedicated to the register configuration of the DPUCZDX8G IP core. Configuring it to 100MHz ensures high-speed and reliable register access, and this independent clock helps improve the sharing efficiency with the M-AXI clock domain, thereby optimizing the overall system performance.

[0110] S22 configures the data controller clock frequency of the deep learning processor IP core to 295MHz. The data controller clock (m_axi_dpu_aclk) is connected to the AXI_MM master clock to ensure the synchronization and consistency of data transmission. Configuring it to 295MHz allows the data controller to efficiently schedule data streams between the processing system (PS) and the DPU.

[0111] S23 configures the deep learning processor's IP core's computing engine clock frequency to 590MHz, and the computing engine clock is aligned with the data controller clock. Since the computing engine's DSP slice runs at a higher frequency, the computing clock (dpu_2x_clk) is configured to be twice the data controller clock, and this clock is aligned with the data controller clock to ensure synchronization between clock domains during high-speed computing and guarantee system stability.

[0112] S24, two parallel data processing paths are set up in the deep learning processor. In this embodiment, to further improve the parallelism and efficiency of data processing, data is read simultaneously from the processing system (PS) through two paths, DPU0_M_AXI_DATA0 and DPU0_M_AXI_DATA1. This allows for the simultaneous processing of more data, thereby significantly improving data throughput and overall inference speed.

[0113] It should be noted that the order of the above steps can be adjusted or performed simultaneously as needed. This application does not limit the execution order of the above steps.

[0114] Based on actual testing, the final resource consumption of the hardware accelerator is shown in Table 2 below:

[0115]

[0116] Table 2. Final Resource Consumption

[0117] The final design power consumption is shown in Table 3 below:

[0118]

[0119] Table 3. Final Design Power Consumption

[0120] As can be seen from Tables 2 and 3, the utilization rate of the look-up table (LUT) in the hardware accelerator provided in this embodiment reaches approximately 50%. This efficiency is achieved thanks to a carefully designed resource allocation strategy aimed at balancing performance and power consumption. Similarly, the resource consumption of flip-flops (FFs) and digital signal processors (DSPs) is also controlled at approximately 50%, providing sufficient margin for the system to support future functional expansion and the integration of acceleration cores.

[0121] In terms of power consumption, the total on-chip power is controlled to within 7.156 watts (W), of which dynamic power accounts for the vast majority, approximately 6.817 watts (W), or about 95% of the total power consumption. In contrast, static power consumption is relatively low, at only 0.339 watts (W), accounting for about 5% of the total power consumption, reflecting the low energy consumption characteristics of the system in the inactive state.

[0122] Furthermore, in this embodiment, step S3, the method for adapting and training the YOLOv5 model based on the optimized deep learning processor configuration, includes:

[0123] S31, For the feature extraction part of the YOLOv5 model, the convolutional layer using the SiLU activation function is replaced with a convolutional layer using the LeakyReLU activation function to obtain the CBL layer, where the parameter of the LeakyReLU activation function is set to 0.1015625 when it is less than 0.

[0124] Specifically, such as Figure 4 As shown, the original YOLOv5 model's feature extraction part (Conv) includes convolution, batch normalization, and the SiLU activation function. In this embodiment, convolutional layers with the LeakyReLU activation function are used to replace the SiLU activation function convolutional layers. This allows the replaced feature extraction part (CBL) to enhance the model's feature extraction capability using convolutional layers, improve the model's training stability using batch normalization layers, and increase the model's non-linear expression using the LeakyReLU activation function.

[0125] In this embodiment, the LeakyReLU activation function is:

[0126]

[0127] Where x is the input value of the feature map.

[0128] S32 replaces the original convolutions in the CBL layer of the YOLOv5 model with 1×1 convolutions and depthwise separable convolutions.

[0129] To lightweight CBL while maintaining feature extraction efficiency and reducing model parameters and computational cost, this embodiment replaces the original convolutions in the CBL layer of the YOLOv5 model with 1×1 convolutions and depthwise separable convolutions. Specifically, as follows... Figure 5 As shown, the lightweight CBL includes a 1×1 convolutional layer, a batch normalization layer, a ReLU activation function, a depthwise separable convolution, a batch normalization layer, a ReLU activation function, a 1×1 convolutional layer, and a batch normalization layer.

[0130] S33, change the splicing layer in the C3 layer of the YOLOv5 model to a summation layer.

[0131] Typically, in the computationally intensive and repetitive C3 layer, the Concat layer concatenates two or more tensors along a specified dimension. This operation is often used to connect the outputs of different layers to merge feature information and enhance the network's expressive power. However, this method increases computational cost due to the increased tensor dimension. Therefore, this embodiment modifies the C3 layer in the non-feature extraction part, i.e., the prediction stage C3 layer, by changing the original Concat operation to a summation, further reducing the number of parameters and computational cost. Specifically, as follows... Figure 6 As shown, in the lightweight C3 layer, a summation operation is performed after the bottleneck layer, and the summation result is input into the convolutional layer. The original convolutional layer is replaced by 1×1 convolution and depthwise separable convolution for lightweight processing.

[0132] This concludes the optimization of the YOLOv5 model. The optimized YOLOv5 model structure is as follows: Figure 7 As shown, by optimizing the YOLOv5 model, not only is the deployment efficiency of the model improved in resource-constrained environments, but the accuracy of the object detection task is also maintained. This design significantly reduces hardware resource requirements while preserving model performance, making the model more suitable for edge computing and embedded systems.

[0133] S34. Use the COCO dataset to train and evaluate the configured YOLOv5 model.

[0134] The COCO dataset is a widely recognized benchmark dataset for large-scale object detection, segmentation, and keypoint detection, containing over 200,000 images covering 80 different object categories.

[0135] In this embodiment, the method for training and evaluating the configured YOLOv5 model using the COCO dataset includes: First, to fully train the model and achieve convergence, at least 200 training epochs are set; then, data augmentation techniques are used to process the COCO dataset, such as random cropping, horizontal flipping, and color adjustment, to improve the model's generalization ability; next, in each training epoch, the YOLOv5 model is trained using the data-augmented COCO dataset, wherein the YOLOv5 model learns data features through forward propagation and optimizes network weights through backpropagation; finally, after the model training is completed, the performance of the trained YOLOv5 model is evaluated using the mAP metric.

[0136] mAP is a commonly used evaluation metric in object detection, measuring the average accuracy of a model at different confidence thresholds. In this embodiment, the mAP value at a confidence threshold of 0.5 is used to measure the accuracy of the YOLOv5 model, where a higher mAP value indicates better accuracy. Furthermore, the average mAP value across confidence thresholds ranging from 0.5 to 0.95 is used to measure the robustness of the YOLOv5 model, where a higher average mAP value indicates better robustness.

[0137] Through practical verification, such as Figure 8 As shown, the YOLOv5 model provided in this embodiment has an mAP value of 0.6 at a confidence threshold of 0.5, indicating that the model has high accuracy in identifying targets; and the average mAP value also shows its robustness at different confidence levels within a confidence range of 0.5 to 0.95.

[0138] S35. If the evaluation result meets the preset requirements, stop training and obtain the trained YOLOv5 model.

[0139] This embodiment trains and evaluates the YOLOv5 model, ensuring that the lightweight YOLO network structure can still deliver high-quality object detection performance while maintaining model size and computational efficiency. These results provide a solid foundation for the deployment of the model in practical applications, especially in edge computing scenarios requiring real-time object detection.

[0140] Furthermore, in this embodiment, step S4, based on single-instruction multiple-data stream technology, uses a target detection model to perform fast target detection on the input image, which is divided into three stages: pre-frame processing, frame prediction, and post-frame processing, to ensure efficient and accurate target detection. Specifically, it includes:

[0141] S41, construct a pipeline computation graph, which includes several tensors and several operators. The tensors are used to define the dimensions and shape of the data and are the basic units for data flow in the model. The operators are used to define computational operations, including but not limited to convolution, pooling, activation functions, etc., which are key computational units for realizing the model's functions.

[0142] like Figure 9 As shown, target detection is usually performed in a serial manner. However, in this embodiment, in addition to parallel execution, a multi-threaded and pipelined approach is used to greatly improve the inference speed and throughput of the final result during the inference process of the video stream.

[0143] Specifically, in this embodiment, a detailed computation graph can be constructed using the compiled xmodel file. This computation graph is the core of model execution and accurately describes the computation process of the object detection model. Figure 10 A specific implementation of computation graph tensors is presented.

[0144] S42, preprocess the input image. This step is called frame preprocessing.

[0145] Specifically, to ensure the DPU can effectively execute the model, the input image format needs to be precisely set and preprocessed. In this embodiment, the input image size is set to 640x640x3 pixels to ensure that the image meets the requirements of the YOLO model input and to balance computational resources and detection accuracy. This reduces unnecessary computational burden while maintaining high detection accuracy, thereby improving overall system performance. Furthermore, preprocessing includes normalization, color space conversion, and resizing to adapt to the model's input requirements. These preprocessing steps ensure consistency between the input data and the data used during model training, thus guaranteeing the model's generalization ability in practical applications.

[0146] In practical applications, the CPU can be used to sequentially read each frame of the video stream and perform the necessary preprocessing steps.

[0147] S43, perform inference on the preprocessed image to perform object detection. This step is frame prediction.

[0148] Specifically, in this embodiment, the process includes: First, constructing a read queue and a display queue. The read queue is used to temporarily store image frames read sequentially from the video stream, while the display queue is used to store image frames after object detection processing for subsequent display or analysis. The design of these two queues ensures the sequential and real-time nature of video stream processing, avoiding data loss or corruption.

[0149] Then, the preprocessed image is placed into a read queue, and the deep learning processor performs forward inference on the image to obtain tensor data in NHWC format for the prediction results. In this embodiment, after the DPU performs model prediction, it generates three feature map data blocks of different sizes: 80×80×255, 40×40×255, and 20×20×255. These feature maps represent detection results at different scales, which are crucial for recognizing target objects of various sizes. The spatial resolution of each feature map is determined by n. i ×n i Decision (where n) i (It can be 80, 40, or 20), which reflects that the input image is segmented into n. i ×n i Each grid cell is responsible for predicting the target object whose center point is located within that cell. The 255-dimensional information of each grid cell represents the prediction information of each grid cell. The 255-dimensional information of each grid cell includes three prediction boxes of different sizes. Each prediction box has 85-dimensional information, including the coordinates of each bounding box, the object confidence score, and C class probability values, where C represents the total number of target classes. For the COCO dataset used in this invention, C is 80.

[0150] Next, the tensor data is converted from NHWC format to NCHW format (a general format for neural networks). This conversion is to adapt to subsequent deep learning frameworks and processing procedures.

[0151] Next, the NCHW format tensor data is normalized to obtain the positional information of the predicted bounding boxes relative to the original input image. In this embodiment, the softmax function is used to normalize the NCHW format tensor data (85-dimensional tensors for each predicted bounding box), normalizing its output values ​​to the range of 0 to 1 to obtain coordinate information. This step is crucial for improving the accuracy and reliability of the model's predictions; for example... Figure 11 As shown, the coordinate information is transformed to obtain the initial position information of the prediction box. The transformation formula is as follows:

[0152]

[0153]

[0154]

[0155]

[0156] in, , These are the x and y coordinates of the center point of the prediction box, respectively. To predict the actual width of the bounding box, To predict the actual height of the bounding box, For cell width, The height of the cell. This is the horizontal offset of the center point of the prediction box relative to the top-left corner of its cell. The vertical offset of the center point of the prediction box relative to the top-left corner of its cell. The width of the secondary prediction box outside the prediction box. The height of the sub-predicted bounding boxes is calculated. Since the output feature maps correspond to downsampling of the original image by 8x, 16x, and 32x respectively, the initial position information of the predicted bounding boxes needs to be mapped back to the dimensions of the original input image to obtain the position information of the predicted bounding boxes relative to the original input image, ensuring the accuracy of the predicted bounding boxes. This mapping process involves precise calculation of the downsampling ratio and coordinate transformation.

[0157] During training, the final output will converge to near 0. Values ​​near 0 will approach 0.5 after passing through the sigmoid function. For the normalized result, we want the value to be between 0 and 1; therefore, we scale it by a factor of 2 to improve the accuracy of the model's predictions. The predicted value of the center point coordinates of the predicted bounding box is multiplied by 2 and subtracted by 0.5, resulting in a range of (-0.5, 1.5). This is to enable predictions across half a grid point, thus improving the recall of predicted bounding boxes around that grid point.

[0158] Next, based on the set confidence threshold, the predicted bounding boxes are filtered, and non-maximum suppression (NMS) is performed on the remaining predicted bounding boxes to obtain the final predicted bounding boxes. Filtering removes predicted bounding boxes that are unlikely to contain the target, thereby reducing the possibility of false positives and improving detection accuracy; NMS eliminates overlapping predicted bounding boxes, ensuring that each target is predicted only once, thereby improving the clarity and accuracy of the detection results.

[0159] Then, the final prediction frame is used for reasoning to obtain the prediction result.

[0160] At this point, the frame prediction process ends, followed by the frame post-processing stage: the inferred image is sent to the display queue so that the image with the prediction result is displayed frame by frame. In this embodiment, the CPU reads data from the queue and displays the prediction result frame by frame on the display device. This step ensures that the user can observe the target detection result in real time, providing an intuitive basis for real-time monitoring and decision-making.

[0161] The video stream prediction process provided in this embodiment is as follows: Figure 12As shown, the CPU processes the video stream in frames and preprocesses the frame images, then sends them to the read queue (read frame queue). The DPU executes the computation graph in the order of the images in the read queue to predict the images, and the CPU calculates the prediction box. Finally, the prediction results are output to the screen in sequence through the display queue to visualize the prediction results.

[0162] The FPGA-based target detection method provided in this embodiment not only improves the output quality of the target detection model, but also ensures the real-time performance and accuracy of the processing results, thereby providing strong technical support for various application scenarios that require real-time target detection, such as video surveillance and autonomous driving.

[0163] This embodiment also provides an FPGA-based target detection system, including a memory located on the FPGA, a processor, and an executable program stored in the memory and capable of being run by the processor; when the processor runs the executable program, it executes the FPGA-based target detection method as described above.

[0164] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to mutually. In addition, different parts between embodiments can also be combined with each other, and this invention does not limit this.

[0165] This embodiment provides an FPGA-based target detection method and system, comprising: designing a target detection model based on FPGA hardware resources, wherein the target detection model includes a hardware accelerator based on a deep learning processor, a YOLOv5 model, and a CPU; optimizing the clock domain configuration of the deep learning processor's IP core; adapting and training the YOLOv5 model based on the optimized deep learning processor; and using the target detection model to perform fast target detection on the input image based on single instruction multiple data stream (SID) technology. Optimizing the clock domain configuration of the deep learning processor's IP core ensures synchronization and coordination between the deep learning processor's IP core and other FPGA components; adapting and training the YOLOv5 model significantly improves inference speed and overall system performance while maintaining algorithm accuracy; and significantly improves the data throughput of the target detection model through SID technology. Since the target detection model is based on an FPGA design, it benefits from the flexibility and low cost of FPGA design, giving this target detection scheme the advantages of high flexibility and low cost. This achieves high-precision, high-flexibility target detection on an FPGA basis, solving the problem of poor flexibility and high cost of existing hardware acceleration schemes in target detection tasks.

[0166] The above description is merely a description of preferred embodiments of the present invention and is not intended to limit the scope of the present invention in any way. Any changes or modifications made by those skilled in the art based on the above disclosure shall fall within the protection scope of the claims.

Claims

1. A target detection method based on FPGA, characterized in that, include: Based on FPGA hardware resources, a target detection model is designed, which includes a hardware accelerator based on a deep learning processor, a YOLOv5 model, and a CPU. Optimize the clock domain configuration of the IP cores of the deep learning processor; Based on the optimized configuration of the deep learning processor, the YOLOv5 model is adapted and trained, which includes: for the feature extraction part of the YOLOv5 model, the convolutional layer using the SiLU activation function is replaced with the convolutional layer using the LeakyReLU activation function to obtain the CBL layer, wherein the parameter of the LeakyReLU activation function is set to 0.1015625 when it is less than 0. The original convolutions in the CBL layer of the YOLOv5 model are replaced with 1×1 convolutions and depthwise separable convolutions. Change the splicing layer in the C3 layer of the prediction phase in the YOLOv5 model to a summation layer; The configured YOLOv5 model was trained and evaluated using the COCO dataset; If the evaluation results meet the preset requirements, training will stop and a trained YOLOv5 model will be obtained. Based on the Single Instruction Multiple Data Stream (SIMDS) technology, a target detection model is used to perform fast target detection on the input image. The method includes: constructing a pipelined computation graph, which includes several tensors and several operators. The tensors are used to define the dimensions and shape of the data, and the operators are used to define computational operations. Preprocess the input image; Inference is performed on the preprocessed image for object detection, specifically including: constructing a read queue and a display queue; The preprocessed image is placed into the read queue, and the deep learning processor is used to perform forward reasoning on the image to obtain tensor data in NHWC format of the prediction result. Convert tensor data from NHWC format to NCHW format; The tensor data in NCHW format is normalized to obtain the position information of the prediction box relative to the original input image. Based on the set confidence threshold, the predicted boxes are filtered, and non-maximum suppression is performed on the remaining predicted boxes after filtering to obtain the final predicted boxes. The final prediction box is used for inference to obtain the prediction result; The inferred image is sent to the display queue so that the image with the prediction result is displayed frame by frame.

2. The FPGA-based target detection method according to claim 1, characterized in that, The method for designing a target detection model based on FPGA hardware resources includes: Design a hardware accelerator based on a deep learning processor, taking advantage of FPGA hardware resources. A runtime environment is built using Petalinux, which includes drivers, library functions, and middleware for hardware accelerators. Connect the YOLOv5 model and CPU to the hardware accelerator for communication.

3. The FPGA-based target detection method according to claim 2, characterized in that, The method for designing a hardware accelerator based on a deep learning processor includes: Zynq UltraScale+ MPSoC 3.4 was selected as the processing system for the hardware accelerator; Define the resource consumption of the deep learning processor, where the parallel number of input feature channels and the parallel number of output feature channels are both 16, the parallel number of pixels is 8, and each MAC operation consists of one multiplication and one addition. Configure the Ultra-RAM of the deep learning processor to 65 and the ALU Parallel number to 4; The softmax function, which is used for model normalization calculation in deep learning processors, is transferred to the CPU for calculation. The AXI Interconnection IP is used to connect the IP core of the deep learning processor to the processing system and to allocate register addresses.

4. The FPGA-based target detection method according to claim 1, characterized in that, The method for optimizing the clock domain configuration of the IP cores of the deep learning processor includes: The register clock frequency of the IP core of the deep learning processor is configured to be 100MHz; The data controller clock frequency of the IP core configured for the deep learning processor is 295MHz; The computing engine clock frequency of the IP core configured for the deep learning processor is 590MHz, and the computing engine clock is aligned with the data controller clock. Set up two parallel data processing paths in the deep learning processor.

5. The FPGA-based target detection method according to claim 1, characterized in that, The method for training and evaluating the configured YOLOv5 model using the COCO dataset includes: Set a minimum of 200 training cycles; Data augmentation techniques were used to process the COCO dataset; In each training cycle, the YOLOv5 model is trained using the data-augmented COCO dataset. The YOLOv5 model learns data features through forward propagation and optimizes network weights through backpropagation. The performance of the trained YOLOv5 model was evaluated using the mAP metric.

6. The FPGA-based target detection method according to claim 1, characterized in that, The method for normalizing NCHW format tensor data to obtain the position information of the prediction box relative to the original input image includes: The softmax function is used to normalize the tensor data in NCHW format to obtain coordinate information; The coordinate information is transformed to obtain the initial position information of the prediction box. The transformation formula is as follows: in, , These are the x and y coordinates of the center point of the prediction box, respectively. To predict the actual width of the bounding box, To predict the actual height of the bounding box, For cell width, The height of the cell. This is the horizontal offset of the center point of the prediction box relative to the top-left corner of its cell. This is the vertical offset of the center point of the prediction box relative to the top-left corner of its cell. The width of the secondary prediction box outside the prediction box. The height of the secondary prediction box outside the prediction box; The initial position information of the prediction box is mapped back to the size of the original input image to obtain the position information of the prediction box relative to the original input image.

7. A target detection system based on FPGA, characterized in that, It includes a memory located on an FPGA, a processor, and an executable program stored in the memory and capable of being run by the processor; when the processor runs the executable program, it performs the FPGA-based target detection method as described in any one of claims 1 to 6.

Images