Distributed target detection system based on heterogeneous computing platform

Abstract

The invention discloses a distributed target detection system based on a heterogeneous computing platform, and relates to the technical field of embedded artificial intelligence and computer vision, the distributed target detection system comprises a master node and at least one slave node, the master node and the slave node are connected through a PCIe high-speed interface; the master node comprises: a video processing module for receiving a video stream and segmenting a video frame into a plurality of subframes; the first PCIe transmission module is used for transmitting a part of subframes to the slave node through a PCIe interface; the local reasoning module is used for executing target detection on the sub-frame reserved by the main node; the fusion module is used for fusing the reasoning result returned by the slave node with a local reasoning result; according to the distributed target detection system based on the heterogeneous computing platform, through PCIe DMA high-speed transmission and lightweight metadata return strategies, transmission delay is greatly reduced, bandwidth resources are effectively saved, and meanwhile the stability risk caused by large-capacity data transmission is avoided.

Description

Technical Field

[0001] This invention relates to the field of embedded artificial intelligence and computer vision technology, specifically a distributed target detection system based on a heterogeneous computing platform. Background Technology

[0002] With the increasing demand for real-time target recognition in fields such as video surveillance and autonomous driving, high-definition video real-time inference places higher demands on the computing power of embedded hardware. Embedded development boards integrating high-performance NPUs, such as the Rockchip RK3588, have become the mainstream choice for edge computing. Their accompanying RKNN-Toolkit2 framework can efficiently schedule NPU computing power and support the deployment and optimization of models such as YOLO.

[0003] Currently, a multi-edge device inference cluster scheme is disclosed in patent number CN119271393A, in which the master device allocates video image data to the slave device for inference and uses conventional network protocol for transmission, aiming to optimize resource allocation; while patent number CN116886940A improves processing efficiency through asynchronous multi-channel video segmented download, batch preprocessing and parallel inference.

[0004] The existing technologies provided by the aforementioned patents have the following drawbacks: most solutions use traditional network protocols to transmit complete video frames, resulting in high transmission latency and large bandwidth consumption; when using the image backhaul and stitching mode, the difference in preprocessing parameters of different nodes often leads to screen segmentation problems; the lack of a precise frame synchronization strategy can easily cause the inference results to be misaligned with the original video frames; the task scheduling is not optimized for specific hardware architectures, resulting in unbalanced computing unit load; and the storage of a large amount of intermediate data leads to excessive memory consumption, which can easily cause system instability. Summary of the Invention

[0005] The purpose of this invention is to provide a distributed target detection system based on a heterogeneous computing platform to address the shortcomings of the prior art.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a distributed target detection system based on a heterogeneous computing platform, comprising: a master node and at least one slave node, wherein the master node and the slave node are connected via a PCIe high-speed interface; the master node comprises: a video processing module for receiving video streams and dividing video frames into multiple subframes; a first PCIe transmission module for transmitting some subframes to the slave node via the PCIe interface; a local inference module for performing target detection on the subframes retained by the master node; and a fusion module for fusing the inference results returned by the slave node with the local inference results; the slave node comprises: a second PCIe transmission module for receiving subframe data transmitted by the master node; a slave node inference module for performing target detection on the received subframes; and a result encapsulation module for encapsulating the inference results into lightweight metadata and returning it to the master node via the PCIe interface; and also includes a frame synchronization mechanism to ensure that the inference results of the master node and the slave node correspond to the same original video frame.

[0007] Preferably, the video processing module divides the video frame into a left half-frame and a right half-frame along a horizontal center line, wherein the left half-frame is processed by the master node and the right half-frame is transmitted to the slave node for processing.

[0008] Preferably, the lightweight metadata includes target bounding box coordinates, confidence level, category identifier, and frame number, and the data volume is less than 10% of the original image data.

[0009] Preferably, the first PCIe transmission module and the second PCIe transmission module use direct memory access to implement data transmission without the need for central processing unit intervention.

[0010] Preferably, the frame synchronization mechanism includes: a global frame sequence number allocation module, which assigns a unique identifier to each video frame by the master node; a frame sequence number tracing module, which is used to ensure the consistency of frame sequence numbers during transmission, inference, and return; and a timeout retransmission module, which is used to trigger the slave node to retransmit when the master node does not receive the return data of the corresponding frame sequence number within a preset time.

[0011] Preferably, the fusion module uses a non-maximum suppression algorithm to deduplicatize and fuse the detection boxes from the master node and the slave node.

[0012] Preferably, it also includes a memory management module that uses a fixed-size buffer reuse mechanism to avoid memory fragmentation caused by dynamic memory allocation.

[0013] Preferably, the heterogeneous computing platform is a development board equipped with an RK3588 chip, and the local inference module and the slave node inference module perform task balancing for the three-core architecture of the RK3588 neural network processing unit.

[0014] Preferably, it also includes a performance monitoring module for real-time collection and output of neural network processing unit utilization, transmission latency, inference frame rate and fusion time metrics.

[0015] Preferably, in video streaming processing at 720p resolution and 60Hz refresh rate, the overall frame rate is not less than 120 frames per second and the transmission delay is not more than 3 milliseconds.

[0016] In the above technical solution, the distributed target detection system based on a heterogeneous computing platform provided by this invention brings significant benefits in terms of transmission efficiency, image fusion quality, system synchronization accuracy, computing resource utilization, system stability, deployment and maintenance, and technical practicality. Through PCIe DMA high-speed transmission and a lightweight metadata backhaul strategy, transmission latency is significantly reduced and bandwidth resources are effectively saved, while avoiding stability risks caused by large-capacity data transmission. In image processing, unified rendering and intelligent fusion technology are adopted to completely eliminate image segmentation artifacts that may occur during multi-node processing, significantly improving the accuracy and visual consistency of target recognition. By constructing a full-process frame sequence number synchronization mechanism and timeout retransmission guarantee, the system achieves precise timing alignment, effectively preventing frame misalignment problems. For the multi-core architecture of the heterogeneous computing platform, the system has undergone load balancing and parallel collaboration optimization, fully utilizing hardware computing power and improving overall throughput and resource utilization. In memory management, buffer reuse and lightweight storage strategies are adopted to effectively avoid memory fragmentation and overflow, enhancing the long-term reliability of the system. Furthermore, the system is implemented on a general-purpose development board, offering advantages such as low deployment cost, good compatibility, and ease of expansion and maintenance. It also includes built-in performance monitoring capabilities to support system optimization. This solution not only meets the high requirements of real-time processing of high-definition video but also provides a complete, efficient, and stable technical solution for distributed AI applications in embedded edge computing scenarios, demonstrating significant technological innovation value and practical application potential. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0018] Figure 1 This invention provides a connection diagram of various components of a distributed target detection system based on a heterogeneous computing platform, as shown in an embodiment of the invention. Figure 2 This is a flowchart of a distributed target detection system based on a heterogeneous computing platform, provided as an embodiment of the present invention. Detailed Implementation

[0019] To enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.

[0020] Please see Figure 1-2 This invention provides a distributed target detection system based on a heterogeneous computing platform, comprising: a master node and at least one slave node, the master node and slave node being connected via a PCIe high-speed interface; the master node includes: a video processing module for receiving video streams and dividing video frames into multiple subframes; a first PCIe transmission module for transmitting some subframes to the slave node via the PCIe interface; a local inference module for performing target detection on the subframes retained by the master node; and a fusion module for fusing the inference results returned by the slave node with the local inference results; the slave node includes: a second PCIe transmission module for receiving subframe data transmitted by the master node; a slave node inference module for performing target detection on the received subframes; and a result encapsulation module for encapsulating the inference results into lightweight metadata and returning it to the master node via the PCIe interface; and also includes a frame synchronization mechanism to ensure that the inference results of the master node and the slave node correspond to the same original video frame.

[0021] The video processing module divides the video frame into a left half-frame and a right half-frame along the horizontal center line. The left half-frame is processed by the master node, while the right half-frame is transmitted to the slave node for processing.

[0022] The lightweight metadata includes target bounding box coordinates, confidence level, category identifier, and frame number, and its data volume is less than 10% of the original image data.

[0023] The first PCIe transmission module and the second PCIe transmission module use direct memory access to achieve data transmission without the need for central processing unit intervention.

[0024] The frame synchronization mechanism includes: a global frame sequence number allocation module, which assigns a unique identifier to each video frame by the master node; a frame sequence number tracing module, which ensures the consistency of frame sequence numbers during transmission, inference, and return; and a timeout retransmission module, which triggers the slave node to retransmit when the master node does not receive the return data of the corresponding frame sequence number within a preset time.

[0025] The fusion module uses a non-maximum suppression algorithm to deduplicatize and fuse detection boxes from the master node and slave node.

[0026] It also includes a memory management module that uses a fixed-size buffer reuse mechanism to avoid memory fragmentation caused by dynamic memory allocation.

[0027] The heterogeneous computing platform is a development board equipped with an RK3588 chip. The local inference module and the slave node inference module perform task balancing for the three-core architecture of the RK3588 neural network processing unit.

[0028] It also includes a performance monitoring module, which is used to collect and output neural network processing unit utilization, transmission latency, inference frame rate and fusion time in real time.

[0029] In video streaming processing at 720p resolution and 60Hz refresh rate, the overall frame rate is no less than 120 frames per second, and the transmission delay is no more than 3 milliseconds.

[0030] Based on the above, the system workflow includes the following steps: S1: The master node receives the video stream and extracts frames; S2: The video frame is divided into a left half-frame and a right half-frame according to the horizontal center line; S3: The right half-frame is transmitted to the slave node via PCIe DMA; S4: The master node and slave node perform half-frame inference in parallel; S5: The slave node encapsulates the inference result into metadata and sends it back; S6: The master node matches the inference results of the left and right half-frames according to the frame sequence number; S7: The detection results are fused using a non-maximum suppression algorithm; S8: The complete frame detection result is output and displayed.

[0031] In a preferred embodiment of this invention, the system hardware employs two NanoPC-T6 development boards equipped with RK3588 chips, connected via a PCIe 3.0 transmission line. The master and slave nodes are each equipped with independent power supplies and cooling systems to ensure stable operation under high loads.

[0032] Specifically, the software environment configuration is as follows: Operating system: Ubuntu 22.04 LTS (arm64 architecture); Inference framework: RKNN-Toolkit2 V1.6; Image processing library: OpenCV 4.5.5; PCIe communication library: a dedicated transmission module developed based on Rockchip's official driver.

[0033] Based on the above, the master node is board A, and the slave node is board B. Specifically, in this embodiment of the invention, the video processing module (main_rc.cc) on board A includes calling the OpenCV VideoCapture class to read a 7-channel video stream (path is local file . / mp4 or camera / dev / video0); the frame retrieval time is controlled within 3.65ms, and the FPS reaches 273.9, ensuring the stability of the video stream reading. Secondly, the original 720×1280 resolution frame is split into a left half-frame (columns 0-639) and a right half-frame (columns 640-1279) according to column coordinate 640; the left half-frame is stored locally for inference, and the right half-frame is stored in the PCIe transmission buffer, awaiting DMA transfer. Further, the rknn_init interface is called to initialize the left half-frame inference model, and the left half-frame data is input into the model; the inference time is 12.63ms, the FPS reaches 79.2, and the NPU three-core utilization is evenly distributed at 58% / 58% / 58%.

[0034] In this process, based on the first and second PCIe transmission modules, board A calls the PCIe_send interface to write the right half-frame data and a unique frame sequence number (frame_id, an auto-incrementing integer) into the PCIe buffer; board B directly reads the buffer data through the PCIe_recv interface, with a transmission time of 2.25ms and no CPU intervention. Furthermore, after inference is completed, board B only encapsulates the target bounding box coordinates (x1, y1, x2, y2), confidence score (float), class ID (int), and frame_id as 128 bytes of metadata; this is then transmitted back to board A via PCIeDMA, reducing the data volume by 90% compared to a complete image, with a transmission time of <1ms.

[0035] The inference module (slave node inference module) (main_ep.cc) of board B obtains the right half frame data and frame_id through the PCIe communication module; it calls the RKNN inference interface to process the right half frame, and the inference time is the same as that of board A (12.63ms). The NPU three-core utilization rate is 57% / 57% / 57%; the inference results are packaged according to the agreed format and PCIe return is triggered.

[0036] In the fusion module, after receiving the returned data, the A board matches the frame_id with the frame_id of the local left half-frame inference result, and only performs fusion on the results with the same ID; the NMS threshold is set to 0.5 to remove duplicate detection boxes in the left and right half-frames, and the fusion takes 18.18ms; the fused complete frame is displayed through the OpenCV imshow interface, and the FPS reaches 140.8.

[0037] The foregoing has only described certain exemplary embodiments of the present invention by way of illustration. Undoubtedly, those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the foregoing drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

Claims

1. A distributed target detection system based on a heterogeneous computing platform, characterized in that, include: A master node and at least one slave node, wherein the master node and the slave node are connected via a PCIe high-speed interface; the master node includes: The video processing module is used to receive video streams and divide video frames into multiple subframes. The first PCIe transmission module is used to transmit a portion of the subframes to the slave node via the PCIe interface; The local inference module is used to perform target detection on the subframes retained by the master node; The fusion module is used to fuse the inference results returned from the node with the local inference results; The slave nodes include: The second PCIe transmission module is used to receive subframe data transmitted by the master node. The inference module from the node is used to perform target detection on the received subframes; The result encapsulation module is used to encapsulate the inference results into lightweight metadata and return them to the master node through the PCIe interface. It also includes a frame synchronization mechanism to ensure that the inference results of the master node and the slave node correspond to the same original video frame.

2. The distributed target detection system based on a heterogeneous computing platform according to claim 1, characterized in that, The video processing module divides the video frame into a left half-frame and a right half-frame along the horizontal center line. The left half-frame is processed by the master node, and the right half-frame is transmitted to the slave node for processing.

3. The distributed target detection system based on a heterogeneous computing platform according to claim 1, characterized in that, The lightweight metadata includes target bounding box coordinates, confidence level, category identifier, and frame number, and the data volume is less than 10% of the original image data.

4. The distributed target detection system based on a heterogeneous computing platform according to claim 1, characterized in that, The first PCIe transmission module and the second PCIe transmission module use direct memory access to implement data transmission without the need for central processing unit intervention.

5. A distributed target detection system based on a heterogeneous computing platform according to claim 1, characterized in that, The frame synchronization mechanism includes: The global frame sequence number allocation module assigns a unique identifier to each video frame by the master node. The frame sequence number tracing module is used to ensure the consistency of frame sequence numbers during transmission, inference, and return. The timeout retransmission module is used to trigger the slave node to retransmit when the master node does not receive the return data of the corresponding frame sequence number within a preset time.

6. The distributed target detection system based on a heterogeneous computing platform according to claim 1, characterized in that, The fusion module uses a non-maximum suppression algorithm to deduplicate and fuse detection boxes from the master node and slave node.

7. A distributed target detection system based on a heterogeneous computing platform according to claim 1, characterized in that, It also includes a memory management module that uses a fixed-size buffer reuse mechanism to avoid memory fragmentation caused by dynamic memory allocation.

8. A distributed target detection system based on a heterogeneous computing platform according to claim 1, characterized in that, The heterogeneous computing platform is a development board equipped with an RK3588 chip. The local inference module and the slave node inference module perform task balancing for the three-core architecture of the RK3588 neural network processing unit.

9. A distributed target detection system based on a heterogeneous computing platform according to claim 1, characterized in that, It also includes a performance monitoring module, which is used to collect and output neural network processing unit utilization, transmission latency, inference frame rate and fusion time in real time.

10. A distributed target detection system based on a heterogeneous computing platform according to claim 1, characterized in that, In video streaming processing at 720p resolution and 60Hz refresh rate, the overall frame rate is no less than 120 frames per second, and the transmission delay is no more than 3 milliseconds.

Images