A target recognition implementation method based on MPSOC

Abstract

The application provides an MPSOC-based target recognition implementation method, and belongs to the technical field of artificial intelligence, and specifically comprises the following steps: calling a DPUCZDX8G IP core on an FPGA end based on a multi-core system-level chip platform, designing a multi-core deep learning processor parallel data calculation module, and performing parallel acceleration calculation on a convolutional neural network target recognition algorithm model; based on a target recognition algorithm yolox, a target recognition algorithm network model compatible with multi-core deep learning processor instruction operation is compiled, and a compiled network model instruction file yolox.xmodel is deployed into an MPSOC platform FPGA end DPU module for execution. Through the processing scheme, the real-time performance and reliability of the target recognition system are improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a target recognition method based on MPSOC. Background Technology

[0002] Currently, target recognition technologies based on optical images are mainly divided into two types: traditional template matching methods and the now mainstream neural network-based target recognition algorithms. Traditional template matching algorithms have poor robustness, heavily rely on template sample features, and have low target recognition accuracy. Therefore, the mainstream research direction now is to achieve target recognition based on neural network processing.

[0003] Neural network-based target recognition methods, trained on large amounts of sample data, can achieve generalized target detection with high accuracy. However, the application of AI target recognition technology in the aviation airborne field is not yet mature, mainly because existing target recognition algorithms are difficult to deploy on airborne equipment. High-performance GPU computing platforms that typically support AI algorithms suffer from drawbacks such as high power consumption, low reliability, and limited hardware scalability, making them unsuitable for the harsh aviation airborne environment. Summary of the Invention

[0004] In view of this, this application provides a target recognition implementation method based on MPSOC, which solves the problems in the prior art and improves the real-time performance and reliability of the target recognition system.

[0005] The target recognition method based on MPSOC provided in this application adopts the following technical solution:

[0006] A target recognition implementation method based on MPSOC includes:

[0007] Step 1: Based on the FPGA platform of a multi-core system-on-a-chip, the DPUCZDX8G IP core is called to design a parallel data computing module for a multi-core deep learning processor to accelerate the parallel computing of the convolutional neural network target recognition algorithm model.

[0008] Step two: Based on the YOLOx target recognition algorithm, compile a network model for the target recognition algorithm that is compatible with multi-core deep learning processor instructions.

[0009] Step 3: Deploy the compiled network model instruction file yolox.xmodel to the FPGA-side DPU module of the MPSOC platform for execution.

[0010] Optionally, step 1 includes:

[0011] A multi-core DPU data computing module is designed on the FPGA side, adding two DPUCZDX8G IP cores and ZYNQ UltraScale+MPSOC IP. The DPUCZDX8G IP cores and ZYNQ UltraScale+MPSOC IP cores are connected. Each DPUCZDX8G IP core includes three AXI bus master interfaces and one AXI bus slave interface. Among them, the three AXI bus master interfaces include two data interfaces and one instruction interface, which are used to acquire data and instructions respectively.

[0012] Connect the AXI bus data master interfaces of the two DPUCZDX8G IP cores to the high-speed AXI bus interfaces S_AXI_HP0_FPD, S_AXI_HP1_FPD, S_AXI_HP2_FPD and S_AXI_HP3_FPD of the ZYNQ UltraScale+MPSOC IP core respectively.

[0013] The AXI bus instruction master interfaces of the two DPUCZDX8G IP cores are connected to the S_AXI_LPD interface of the ZYNQ UltraScale+MPSOC IP core through the AXI bus interconnect.

[0014] Then, all DPUCZDX8G IP cores are connected to the M_AXI_HPM0_LPD interface of the ZYNQUltraScale+MPSOC IP core via the AXI bus from the interface. The CPU accesses the deep learning processor register space through the M_AXI_HPM0_LPD interface.

[0015] The DPUCZDX8G IP core and the ZYNQ UltraScale+MPSOC core are designed to interact via the AXI bus to form the deep learning processor computing module of the target recognition system.

[0016] Optionally, a VDMA core can be added to the FPGA and mounted on the AXI bus. The VDMA core is also connected to the digital component serial interface acquisition module and the ZYNQ UltraScale+MPSOC core. The acquired SDI video stream is buffered in the FPGA's double rate synchronous dynamic random access memory. When the CPU reads image data, the VDMA core accelerates the reading of data from the FPGA's double rate synchronous dynamic random access memory and pushes it onto the AXI bus for transmission to the CPU.

[0017] Optionally, step 2 includes: improving the target recognition algorithm YOLOx to make it compatible with the execution of deep learning processor instructions, forming the required new YOLOx network model. The specific steps are as follows:

[0018] Remove the focus structure at the input of the YOLOx network, and use a 3*3*3 convolution kernel to perform convolution operation on the three channels of the input image. Set the convolution stride to 2, and the convolution operation has the same size as the feature map obtained by the focus structure.

[0019] By changing the Silu activation function in the YOLOx network structure to the ReLU activation function and adjusting the data training method, the change of activation function has a limited impact on the overall network performance.

[0020] Remove the sigmod function from the YOLOx network output detector header and place its implementation in the post-processing stage of the network output results, i.e., it is not included in the entire target recognition network.

[0021] Optionally, the new YOLOx network can be built using the PyTorch deep learning framework and trained on a dataset to generate a network model weight file yolox.pth. The generated model weights are 32-bit floating-point weights. The PyTorch quantization tool in the Xilinx Vitis AI toolchain can be used to quantize the model weights into 8-bit integers.

[0022] Optionally, during the quantization process, the quantization weights are calibrated for forward network inference using a calibration dataset.

[0023] Optionally, step 3 specifically includes:

[0024] The quantization model is compiled using the Vitis AI compiler vai_c_xir to generate a yolox network model computation graph. The operations in the computation graph are then mapped to the deep learning processor instruction sequence to generate the deep learning processor instruction file yolox.xmodel.

[0025] The instruction file yolox.xmodel is deployed to the deep learning processor module on the FPGA side of the MPSOC platform for execution.

[0026] Optionally, step 3 further includes: designing a multi-threaded task scheduling mechanism to deploy the algorithm. The multi-threaded task scheduling mechanism runs on the CPU. First, six thread tasks are created in the main program. The first thread is used for image data reception and preprocessing. The first thread calls the direct memory module on the CPU to receive the digital component serial image data on the AXI bus and caches it in the CPU-side double-rate synchronous dynamic random access memory for image preprocessing. The second thread to the first thread, a total of four threads, are used to call the DPU engine to run the algorithm network model. Before calling the deep learning processor engine to run, yolox.xmo is loaded once in the main program. The DEL file is loaded, and the instruction file is serialized to form a computation sequence graph. Then, based on the computation sequence graph, four deep learning processor engines are created and called in four threads. The deep learning processor engine in each thread transmits image data and network model parameters to the deep learning processor module on the FPGA side via the AXI bus, and then performs forward inference calculation of the network model. After the calculation is completed, the result information is sent back to the CPU side via the AXI bus. The sixth thread is used to receive the recognition result information sent back from the FPGA side, and post-process the output result, superimposing the target recognition information onto the original image and displaying it in real time through the DP interface.

[0027] In summary, this application includes the following beneficial technical effects:

[0028] To address the complex airborne environment, this application presents a target recognition method and system based on the MPSOC platform, integrating data acquisition, transmission, target recognition network acceleration computation, and local display of recognition results. Based on the system's CPU+FPGA architecture, a combination of bilateral DMA synchronization and DPU acceleration computation is employed, significantly improving the system's data transmission and algorithm inference computation speed. The system features high versatility, high real-time performance, high stability, and low power consumption. The convolutional neural network target recognition algorithm implemented in this system boasts high recognition rate, diverse target detection types, and strong stability, solving the problems of inaccurate and inefficient manual target recognition in traditional airborne vision systems, thereby improving the accuracy and real-time performance of target recognition in airborne environments. Attached Figure Description

[0029] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is a general block diagram of the target recognition system of this application.

[0031] Figure 2 This is a block diagram of the multi-core DPU data computing module design in this application.

[0032] Figure 3 This is a schematic diagram of the working principle of the target recognition system of this application. Detailed Implementation

[0033] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0034] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. This application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be based on different...

[0035] The same viewpoints and applications may be modified or altered in various ways without departing from the spirit of this application. It should be noted that, unless otherwise specified, the following embodiments and features described herein can be combined with each other. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0036] It should be noted that the following description covers various aspects of embodiments within the scope of the appended claims. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this application, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the apparatus and / or practice. Additionally, aspects other than those set forth herein can be used...

[0037] Other structures and / or functionalities besides one or more of the aspects may be used to implement this device and / or practice this method. 5 It should also be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this application. The drawings only show components related to this application and are not drawn according to the actual number, shape, and size of the components in the actual implementation. In the actual implementation, the type, quantity, and proportion of each component may be arbitrarily changed, and the component layout may also be more complex.

[0038] Furthermore, specific details are provided in the following description to facilitate a thorough understanding of the examples. However, those skilled in the art will understand that the described aspects can be practiced without these specific details.

[0039] This application provides a target recognition method based on MPSOC.

[0040] A target recognition implementation method based on MPSOC includes:

[0041] like Figure 1-3 As shown, step one: Based on the multi-processor system on chip (MPSOC) platform FPGA side, call DPUCZDX8G IP core to design a multi-core deep learning processor (DPU) parallel data computing module to perform parallel acceleration computing on the convolutional neural network target recognition algorithm model;

[0042] Step two: Based on the YOLOx target recognition algorithm, compile a network model for the target recognition algorithm that is compatible with multi-core deep learning processor instructions.

[0043] Step 3: Deploy the compiled network model instruction file yolox.xmodel to the FPGA-side DPU module of the MPSOC platform for execution.

[0044] Step 1 includes:

[0045] like Figure 2 As shown, a multi-core DPU data computation module is designed on the FPGA side, adding two DPUCZDX8G IP cores and a ZYNQ UltraScale+MPSOC IP. The two DPUCZDX8G IP cores are designated as DPU_0 and DPU_1. The DPUCZDX8G IP cores are connected to the ZYNQ UltraScale+MPSOC IP core. Each DPUCZDX8G IP core includes three AXI bus master interfaces and one AXI bus slave interface.

[0046] The three AXI (Advanced eXtensible Interface) bus master interfaces include two data interfaces and one instruction interface, used to acquire data and instructions respectively. Because the data interface requires high bandwidth, the AXI bus data master interfaces of the two DPUCZDX8G IP cores are connected to the high-speed AXI bus interfaces S_AXI_HP0_FPD, S_AXI_HP1_FPD, S_AXI_HP2_FPD5, and S_AXI_HP3_FPD of the ZYNQ UltraScale+MPSOC IP core, respectively. The AXI bus instruction master interfaces of the two DPUCZDX8G IP cores are connected to the S_AXI_LPD interface of the ZYNQ UltraScale+MPSOC IP core via the AXI bus interconnect. All the AXI bus slave interfaces of the DPUCZDX8G IP cores are then connected to the M_AXI_HPM0_LPD interface of the ZYNQ UltraScale+MPSOC IP core via the AXI interconnect. The CPU accesses the deep learning processor register space through the M_AXI_HPM0_LPD interface. The IP core and the 0ZYNQ UltraScale+MPSOC core are designed to interact via the AXI bus to form the deep learning processor computing module of the target recognition system.

[0047] A VDMA (Video Direct Memory Access) core is added to the FPGA and mounted on the AXI bus. The VDMA core is also connected to the serial digital interface (SDI) acquisition module and the ZYNQUltraScale+MPSOC core, buffering the acquired SDI video stream to the FPGA's Double Data Rate (DDR) synchronous dynamic random access memory.

[0048] When the CPU reads image data, it accelerates the reading of data from the FPGA's double-rate synchronous dynamic random access memory (DDR) through the VDMA core and pushes the data onto the AXI bus for transmission to the CPU.

[0049] After the logic circuit design is completed, it is synthesized and compiled in Vivado to generate the final bit stream file, which is then burned to the FPGA for execution.

[0050] Step 2 includes: improving the object recognition algorithm YOLOx to make it compatible with deep learning processor instructions, thus forming the required new YOLOx network model. The specific steps are as follows:

[0051] 1. Remove the focus structure at the input of the YOLOx network and use a 3*3*3 convolution kernel to perform convolution operation on the three channels of the input image. Set the convolution stride to 2. The size of the feature map obtained by the convolution operation is the same as that obtained by the focus structure.

[0052] 2. By changing the Silu activation function in the YOLOx network structure to the ReLU activation function and adjusting the data training method, the change of activation function has a limited impact on the overall network performance.

[0053] 3. Remove the sigmod function from the YOLOx network output detector head and place its implementation in the post-processing stage of the network output results, i.e., it is not included in the entire target recognition network.

[0054] A new YOLOx network was built using the PyTorch deep learning framework and trained on a dataset, generating a network model weight file named yolox.pth. The trained model weights were 32-bit floating-point weights. The PyTorch quantization tool in the Xilinx VitisAI toolchain was used to quantize the model weights to 8-bit integers. During the quantization process, the quantized weights were calibrated using a calibration dataset for forward network inference, thereby improving the effectiveness of the quantized weights.

[0055] Step 3 specifically includes:

[0056] The quantized model weights need to be compiled for a specific DPU architecture before they can be loaded into the DPU for execution. The Vitis AI compiler vai_c_xir is used to compile the quantized model, generating a YOLOx network model computation graph. The operations in the computation graph are then mapped to the deep learning processor instruction sequence to generate the deep learning processor instruction file yolox.xmodel. The instruction file yolox.xmodel is then deployed to the deep learning processor module on the FPGA side of the MPSOC platform for execution.

[0057] A multi-threaded task scheduling mechanism is designed to deploy the algorithm. This mechanism runs on the CPU. First, six threads are created in the main program. The first thread is used for image data reception and preprocessing, and it calls direct memory on the CPU. The Access (DMA) module receives digital component serial SDI image data from the AXI bus and caches it in the CPU-side Double Data Rate (DDR) synchronous dynamic random access memory for image preprocessing. Four threads (from the second to the first) call the DPU engine to run the algorithm network model. Before calling the deep learning processor engine, the yolox.xmodel file is loaded once in the main program. After loading, the instruction file is serialized to form a computation sequence diagram. Then, four deep learning processor engines are created based on the computation sequence diagram and called in the four threads respectively. Each thread's deep learning processor engine transmits image data and network model parameters to the FPGA-side deep learning processor module via the AXI bus, then performs forward inference calculations of the network model. After the calculation is completed, the result information is returned to the CPU via the AXI bus. The sixth thread receives the recognition result information returned from the FPGA and performs post-processing on the output results, overlaying the target recognition information onto the original image and displaying it in real time via the DP interface.

[0058] In one embodiment, a multi-threaded task scheduling mechanism is designed on the CPU side. The CPU main program loads the yolox.xmodel file once, and after loading, it performs serialization operations on the instruction file to form a computation sequence graph. An image receiving and preprocessing sub-thread is created, which calls the DMA module to receive SDI image data from the AXI bus, scales the acquired image resolution to 640*640, and then performs 8-bit quantization processing on the scaled image data. The pixel values ​​are processed in parallel using the spatial matrix multiplication method.

[0059] The quantization calculation involves creating a coefficient matrix of the same size as the image and initializing all values ​​in the matrix to a fixed coefficient value of scale. Then, a matrix dot product is performed with the image matrix to obtain the quantized image pixel values. These quantized pixel values ​​are 8-bit integers that can be directly input into the DPU engine. Four sub-threads are created to call the DPU engine. Each sub-thread synchronously reads the calculation sequence diagram from the main program by calling the Vitis AI library (VART) and creates a DPU execution engine. Each thread calls the DPU engine to transmit the quantized image data and network model parameters to the FPGA-side DPU0 calculation module via the AXI bus. The DPU then performs forward inference calculations on the network model and outputs the algorithm processing results.

[0060] Finally, a sub-thread for post-processing and displaying the recognition information is created to receive the recognition results from the FPGA, analyze and filter the information, overlay the correct target recognition information onto the original image, and then display it in real time through the CPU's DP (DisplayPort) interface.

[0061] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A target recognition method based on MPSOC, characterized in that, include: Step 1: Based on the FPGA platform of a multi-core system-on-a-chip, the DPUCZDX8G IP core is called to design a parallel data computing module for a multi-core deep learning processor to accelerate the parallel computing of the convolutional neural network target recognition algorithm model. Step two: Based on the YOLOx target recognition algorithm, compile a network model for the target recognition algorithm that is compatible with multi-core deep learning processor instructions. Step 3: Deploy the compiled network model instruction file yolox.xmodel to the FPGA-side DPU module of the MPSOC platform for execution; Step one includes: A multi-core DPU data computing module is designed on the FPGA side, adding two DPUCZDX8G IP cores and ZYNQ UltraScale+MPSOC IP. The DPUCZDX8G IP cores and ZYNQ UltraScale+MPSOC IP cores are connected. Each DPUCZDX8G IP core includes three AXI bus master interfaces and one AXI bus slave interface. Among them, the three AXI bus master interfaces include two data interfaces and one instruction interface, which are used to acquire data and instructions respectively. Connect the AXI bus data master interfaces of the two DPUCZDX8G IP cores to the high-speed AXI bus interfaces S_AXI_HP0_FPD, S_AXI_HP1_FPD, S_AXI_HP2_FPD and S_AXI_HP3_FPD of the ZYNQ UltraScale+MPSOC IP core respectively. The AXI bus instruction master interfaces of the two DPUCZDX8G IP cores are connected to the S_AXI_LPD interface of the ZYNQ UltraScale+MPSOC IP core through the AXI bus interconnect. Then, all DPUCZDX8G IP cores are connected to the M_AXI_HPM0_LPD interface of the ZYNQUltraScale+MPSOC IP core via the AXI bus from the interface. The CPU accesses the deep learning processor register space through the M_AXI_HPM0_LPD interface. The DPUCZDX8G IP core and the ZYNQ UltraScale+MPSOC core are designed to interact via the AXI bus to form the deep learning processor computing module of the target recognition system.

2. The target recognition method based on MPSOC according to claim 1, characterized in that, A VDMA core is added to the FPGA and mounted on the AXI bus. The VDMA core is also connected to the digital component serial interface acquisition module and the ZYNQ UltraScale+MPSOC core. The acquired SDI video stream is buffered in the FPGA's double rate synchronous dynamic random access memory. When the CPU reads image data, the VDMA core accelerates the reading of data from the FPGA's double rate synchronous dynamic random access memory and pushes it onto the AXI bus for transmission to the CPU.

3. The target recognition method based on MPSOC according to claim 1, characterized in that, Step two includes: improving the object recognition algorithm YOLOx to make it compatible with deep learning processor instructions, thus forming the required new YOLOx network model. The specific steps are as follows: Remove the focus structure at the input of the YOLOx network and use 3 3 A 3-kernel convolution operation is performed on the three channels of the input image, and the convolution stride is set to 2. The size of the feature map obtained by the convolution operation is the same as that of the focus structure. By changing the Silu activation function in the YOLOx network structure to the ReLU activation function and adjusting the data training method, the change of activation function has a limited impact on the overall network performance. Remove the sigmod function from the YOLOx network output detector header and place its implementation in the post-processing stage of the network output results, i.e., it is not included in the entire target recognition network.

4. The target recognition method based on MPSOC according to claim 3, characterized in that, The new YOLOx network was built using the PyTorch deep learning framework and trained on a dataset to generate the network model weight file yolox.pth. The generated model weights were 32-bit floating-point weights. The PyTorch quantization tool in the Xilinx Vitis AI toolchain was used to quantize the model weights into 8-bit integers.

5. The target recognition method based on MPSOC according to claim 4, characterized in that, During the quantization process, the quantization weights are calibrated by forward network inference using a calibration dataset.

6. The target recognition method based on MPSOC according to claim 4, characterized in that, Step 3 specifically includes: The quantization model is compiled using the Vitis AI compiler vai_c_xir to generate a yolox network model computation graph. The operations in the computation graph are then mapped to the deep learning processor instruction sequence to generate the deep learning processor instruction file yolox.xmodel. The instruction file yolox.xmodel is deployed to the deep learning processor module on the FPGA side of the MPSOC platform for execution.

7. The target recognition method based on MPSOC according to claim 6, characterized in that, Step 3 further includes: designing a multi-threaded task scheduling mechanism to deploy the algorithm. The multi-threaded task scheduling mechanism runs on the CPU. First, six thread tasks are created in the main program. The first thread is used for image data reception and preprocessing. The first thread calls the direct memory module on the CPU to receive the digital component serial image data on the AXI bus and caches it in the CPU's double-rate synchronous dynamic random access memory for image preprocessing. The second to first threads, a total of four threads, are used to call the DPU engine to run the algorithm network model. Before calling the deep learning processor engine, the yolox.xmodel file is loaded once in the main program. After loading, the instruction file is serialized to form a computation sequence diagram. Then, four deep learning processor engines are created according to the computation sequence diagram and called in the four threads respectively. The deep learning processor engine in each thread will transmit the image data and network model parameters to the FPGA-side deep learning processor module through the AXI bus, and then perform forward inference calculation of the network model. After the calculation is completed, the result information is returned to the CPU through the AXI bus. The sixth thread is used to receive the recognition result information returned by the FPGA and perform post-processing on the output result, superimposing the target recognition information onto the original image and displaying it in real time through the DP interface.

Images