Ultra-high resolution image double-gpu parallel pipeline real-time target detection method and system

Abstract

The application discloses a kind of ultra-high resolution image double GPU parallel pipeline real-time target detection method and system, real-time acquisition high-resolution original gray or color image, with multiple detector detection pool to receive original large image and be segmented into two half images in pipeline mode, two inference threads of detector respectively send two half images into two GPUs with TensorRT accelerated YOLOv8 inference engine and implement inference, after inference is completed, result is transmitted back to host memory asynchronously;Main thread waits the synchronization signal of two inference threads, when two parts of results are ready, two groups of detection frames are merged into a result set;Global non-maximum suppression operation is performed on the merged result, and the merged detection result is sent to the target tracking algorithm ByteTrack for interframe correlation, timestamp is marked, and the result is output in order.The application utilizes the hardware advantage of double graphics cards, carries out "macroscopic two division+parallel calculation", and realizes real-time under the premise of guaranteeing 100% field of view coverage through hardware stacking and software scheduling.

Description

Technical Field

[0001] This invention belongs to the field of target detection technology, and in particular relates to a real-time target detection method and system for ultra-high resolution images using a dual-GPU parallel pipeline. Background Technology

[0002] With the development of aerial remote sensing and wide-area surveillance technologies, image resolution has reached the 100-megapixel level (12000×8000). These images are characterized by wide field of view, small targets, and large data volume. In fields such as military reconnaissance and urban security, systems are required to output detection results in real time at video streaming speed (typically 25fps, i.e., 40ms interval between frames). However, due to the huge image size, image processing requires more storage space, wider transmission bandwidth, and longer processing latency. If ultra-high-resolution images are directly input into a graphics card and deep learning models (such as the YOLO series) are used for target detection, it will lead to memory overflow or excessively long inference time. Even a single high-end graphics card cannot complete the calculation within 40ms, thus making it impossible to complete normal real-time target detection applications.

[0003] Existing technologies typically employ the following schemes to process ultra-high resolution images: Patent CN202210834864 uses a two-stage "coarse-fine" strategy. First, the image is downsampled for density estimation to find "Regions of Interest (ROIs)." Then, these regions are cropped from the high-resolution image for fine-grained detection. The inference time of this method is dynamically changing. If the scene has dense targets and many sub-images need to be cropped, the inference time will surge instantly, causing severe frame rate fluctuations and failing to meet the stringent 25Hz real-time synchronization requirements of industrial applications. Furthermore, the processing flow of this method is complex (two stages). For dense targets or complex backgrounds, it is easy to miss targets in the first stage, leading to irrecoverable target loss in the second stage. Patent CN202210999507 describes cropping ultra-high resolution images into a large number of tiny sub-images (such as 416x416) and processing them using multiple concurrent neural networks. This method cuts large images into hundreds or thousands of small fragments, which brings huge I / O overhead and post-processing (NMS) pressure, making it difficult to meet the real-time requirement of 25fps. In addition, the small images fragment the semantic information of large targets, which can easily lead to missed detections. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to enable multiple GPUs to process different parts of the current frame simultaneously and to schedule different frames simultaneously through parallel pipeline technology, so as to meet the real-time capture requirements of high-speed moving targets.

[0005] To address the aforementioned technical problems, this invention proposes a real-time target detection method for ultra-high resolution images using a dual-GPU parallel pipeline. This method is implemented through an image acquisition module, a segmentation scheduling module, a dual-GPU parallel inference engine, and a result fusion module, and includes the following steps:

[0006] Step 1: Detector pool initialization and hardware binding

[0007] When the system starts, a detector object pool (Detector Array) is initialized, containing N detector instances. The main thread receives the raw grayscale or color image from the camera and puts it into the input queue of the currently idle detector. When each detector is instantiated, two independent inference threads are automatically created, each responsible for processing the upper and lower halves of the image.

[0008] Step 2: Spatial Domain Segmentation and Overlapping Slices

[0009] The detector's internal worker thread (frameworker) retrieves images from the step 1 queue. To leverage dual-GPU acceleration, the original large image is split into two halves, upper and lower.

[0010] Overlapping area design: The size of the overlapping area in the height direction of the upper and lower halves of the image is designed to be greater than half of the maximum possible size of the target to be detected in the height direction, so as to ensure that objects located on the dividing line can appear completely in at least one slice, and avoid the target being "cut in half" and thus missing detection.

[0011] Step 3: Dual-thread binding dual-GPU parallel inference

[0012] The detector starts two independent inference threads to work in parallel, uploading the slice data they are responsible for to the corresponding GPU memory. The two GPUs run the YOLOv8 inference engine based on TensorRT acceleration at the same time. After the inference is completed, the results are asynchronously sent back to the host memory.

[0013] Step 4: Time-domain pipeline scheduling

[0014] The main program simulates the camera frame rate or directly accesses camera callbacks. During frame distribution, the main program maintains a detector index. When the k-th frame arrives, it is sent to the detector corresponding to that index in the detector object pool. Image sending is non-blocking; the main thread returns quickly after sending the image to prepare for receiving the next frame, without waiting for the previous frame to finish processing. The background continuously monitors the result queue and collects the results.

[0015] Step 5: Mapping the coordinates of the detection results

[0016] After the two inference threads complete inference and return results, coordinate restoration is performed: the coordinates of the detection boxes in the upper half of the image remain unchanged, and the Y coordinate of the detection boxes in the lower half of the image needs to be added with the segmentation offset Offset Y = 3840 pixels, that is, (x, y + 3840).

[0017] Step 6: Global Fusion and NMS

[0018] The main thread waits for the synchronization signals of the two inference threads. When the results of both parts are ready, the two sets of detection boxes are merged into a result set. Global non-maximum suppression (Global NMS) operation is performed on the merged results. NMS retains the one with the highest confidence and removes redundancy.

[0019] Step 7: Object Tracking and Result Output

[0020] The merged detection results are sent into the object tracking algorithm ByteTrack for inter-frame association, timestamp is marked, and the results are output in sequence.

[0021] A super high-resolution image dual-GPU parallel pipeline system for real-time object detection, comprising:

[0022] An image acquisition module for collecting real-time raw grayscale or color images of the detected object;

[0023] A segmentation scheduling module for taking out the images inside the detector and splitting the original large image into two half-images, which can be split horizontally up and down, or vertically left and right, or for wider images, a "field" shape split can be used, or an asymmetric split can be used;

[0024] A dual-channel GPU parallel inference engine, where two GPUs simultaneously run the YOLOv8 inference engine accelerated by TensorRT, and after inference is completed, the results are asynchronously sent back to the host memory;

[0025] A result fusion module, where the main thread waits for the synchronization signals of the two inference threads. When the results of both parts are ready, the two sets of detection boxes are merged into a result set. Global non-maximum suppression (Global NMS) operation is performed on the merged results.

[0026] Compared with the prior art, the beneficial effects of the present invention are:

[0027] (1) The present invention only splits the large image into 2 parts, and the dual cards pass in parallel at one time, greatly reducing the CPU overhead of pre-processing and post-processing of slicing, and the logic is simpler and more stable.

[0028] (2) The present invention only needs to process the overlapping fusion of one horizontal split line, reducing the probability of missed detection of cross-domain targets, and the NMS calculation amount is smaller.

[0029] (iii) The use of intra-frame parallelism significantly reduces the inference latency of a single frame and improves throughput.

[0030] (iv) Utilizing the advantages of dual graphics card hardware, “macroscopic binary search + parallel computing” is performed. Under the premise of ensuring 100% field of view coverage, single frame latency is controlled within 80ms and throughput reaches 25fps through hardware stacking and software scheduling, which meets the real-time capture requirements of high-speed moving targets. Attached Figure Description

[0031] Figure 1 This is a diagram of the real-time target detection method and system architecture for ultra-high resolution images using a dual-GPU parallel pipeline, as described in this invention.

[0032] Figure 2 This is a diagram of the internal structure of the detector of the present invention;

[0033] Figure 3 This is the thread pipeline timing diagram of the present invention. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0035] like Figure 1 As shown, a specific example is used to introduce the specific implementation steps and modules of the present invention. This embodiment is based on C++ and CUDA architecture, and the hardware environment is an industrial control computer equipped with dual NVIDIA RTX 4090 graphics cards.

[0036] A real-time object detection method for ultra-high resolution images using a dual-GPU parallel pipeline includes the following steps:

[0037] Step 1: Detector pool initialization and hardware binding

[0038] When the dual-GPU parallel pipelined real-time object detection system starts, it initializes a detector object pool (DetectorArray) containing N detector instances (N=6 in this embodiment). The main thread receives the raw grayscale or color image (12000*8000 grayscale image in this example) from the camera and places it into the input queue of the currently idle detector. Upon instantiation of each detector, two independent inference threads (yolo_worker1, yolo_worker2) are automatically created: yolo_worker1 is bound to GPU0 and is responsible for processing the upper half of the image, while yolo_worker2 is bound to GPU1 and is responsible for processing the lower half of the image. This static binding avoids the overhead of CUDA context switching.

[0039] Step 2: As Figure 2As shown, the spatial domain segmentation and overlapping slice strategy

[0040] The working thread (frame_worker) inside the detector retrieves the image from the queue in step 1. To utilize dual-GPU acceleration, the original large image is divided into two half-images: the截取 area of the upper half-image slices is Rect(0, 0, 12000, 4160), and the截取 area of the lower half-image slices is Rect(0, 3840, 12000, 4160); the original large image can also be vertically divided left and right, or for wider images, a "field" - shaped division can be used, or an asymmetric division (such as the stronger card processing 70% of the area and the weaker card processing 30% of the area), and only the division line coordinates need to be adjusted;

[0041] Overlapping area design: The total height of the upper and lower half-images is 8320 pixels, the height of the original image is 8000 pixels, and the overlapping area is 320 pixels. The size of the overlapping area (320 pixels) is designed to be greater than half of the maximum possible size of the target to be detected in the height direction, ensuring that the target located on the division line can completely appear in at least one slice, avoiding missed detections caused by the target being "halved".

[0042] Step 3: Dual-thread binding for dual-GPU parallel inference

[0043] The detector starts two independent inference threads to work in parallel, uploads the slice data it is responsible for to the corresponding GPU video memory, and both GPUs run the YOLOv8 inference engine accelerated by TensorRT simultaneously. After the inference is completed, the results are asynchronously sent back to the host memory. At this time, the processing time for a single half-image is about 70ms. Due to dual-GPU parallelism, the physical calculation time for the entire image is still controlled at about 70ms (limited by the slower card).

[0044] Step 4: As Figure 3 shown, time-domain pipeline scheduling

[0045] The main program simulates the camera frame rate or directly accesses the camera callback. When distributing frames, the main program maintains a detector index. When the k-th frame image arrives, it is sent to the detector corresponding to this index in the detector object pool. Sending the image is non-blocking. After the main thread sends the image, it does not need to wait for whether the previous frame has been processed and quickly returns to prepare to receive the next frame, and the background continuously monitors the result queue for result recovery.

[0046] Although the single-frame processing delay is about 70ms - 80ms, this delay includes the transmission and post-processing time. However, due to the system adopting a detector array (Detector Array) rotation mechanism:

[0047] T = 0ms: Frame 1 enters detector 1 (GPU0 / 1 starts working);

[0048] T=40ms: Frame 2 enters detector 2 (GPU0 / 1 is under high load, but detector 2 can queue or take advantage of the gap);

[0049] T=80ms: Frame 1 completes output.

[0050] This process, which involves overlapping multiple frames on the timeline, is then implemented.

[0051] With this pipeline design, the system can continuously output results at a rate of 25fps (40ms interval), meeting real-time performance requirements.

[0052] Step 5: Mapping the coordinates of the detection results

[0053] After the two inference threads complete their inference and return the results, coordinate restoration is performed: the coordinates of the detection boxes in the upper half of the image remain unchanged, while the Y coordinate of the detection boxes in the lower half of the image needs to be increased by the segmentation offset Offset Y = 3840 pixels, i.e. (x, y +3840).

[0054] Step 6: Global Fusion and NMS

[0055] The main thread waits for synchronization signals from the two inference threads. Once both sets of results are ready, the two sets of detection boxes are merged into a single result set. Global Nonmaximum Suppression (Global NMS) is then performed on the merged result, focusing on handling duplicate boxes in overlapping regions. Since the same target may be detected in both the upper and lower halves of the image and their coordinates may overlap after reconstruction, NMS will retain the one with the highest confidence and remove redundancy.

[0056] Step 7: Target Tracking and Result Output

[0057] The merged detection results are fed into the target tracking algorithm ByteTrack (or DeepSORT, BoT-SORT) for inter-frame correlation, timestamp marking, and the results are output in sequence.

[0058] A dual-GPU parallel pipeline system for ultra-high resolution image processing in real-time object detection includes:

[0059] The image acquisition module is used to acquire raw grayscale or color images of the target being detected in real time; it is not limited to RGB images, but can also be used to acquire other types of images such as infrared images;

[0060] The segmentation scheduling module is used to extract the image inside the detector and divide the original large image into two halves. It can be divided horizontally, vertically, or in a grid pattern for wider images, or asymmetrically divided (e.g., the strong card processes 70% of the area and the weak card processes 30% of the area). Only the coordinates of the segmentation line need to be adjusted.

[0061] The dual-GPU parallel inference engine runs the YOLOv8 inference engine based on TensorRT simultaneously on two GPUs. After inference is completed, the results are asynchronously sent back to the host memory.

[0062] In the result fusion module, the main thread waits for the synchronization signals of the two inference threads. When both sets of results are ready, the two sets of detection boxes are merged into one result set. Global nonmaximum suppression (Global NMS) is then performed on the merged result.

Claims

1. A real-time target detection method for ultra-high resolution images using a dual-GPU parallel pipeline, characterized in that, Includes the following steps: Step 1: Detector pool initialization and hardware binding When the dual-GPU parallel pipeline real-time target detection system starts, it initializes a detector object pool. The main thread receives the raw grayscale or color image from the camera and puts it into the input queue of the currently idle detector. Step 2: Spatial Domain Segmentation and Overlapping Slices The working thread inside the detector retrieves the image from the queue in step 1 and divides the original image into two upper and lower halves; Overlapping area design: The size of the overlapping area between the upper and lower halves of the image in the height direction is designed to be greater than half of the maximum size of the target to be detected in the height direction; Step 3: Dual-thread binding dual-GPU parallel inference The detector starts two independent inference threads to work in parallel, uploading the slice data they are responsible for to the corresponding GPU memory. The two GPUs run the YOLOv8 inference engine based on TensorRT acceleration at the same time. After the inference is completed, the results are asynchronously sent back to the host memory. Step 4: Time-domain pipeline scheduling The main program simulates the camera frame rate or directly accesses the camera callback. During frame distribution, the main program maintains a detector index. When the k-th frame image arrives, it is sent to the detector corresponding to the index in the detector object pool. The image sending is non-blocking. After the main thread sends the image, it does not need to wait for the previous frame to be processed. It quickly returns to prepare to receive the next frame. The background continuously monitors the result queue and collects the results. Step 5: Mapping the coordinates of the detection results After the two inference threads have completed their inference and returned the results, the coordinates are restored. Step 6: Global Fusion and NMS The main thread waits for the synchronization signal of the two inference threads. When both sets of results are ready, the two sets of detection boxes are merged into one result set, and a global nonmaximum suppression operation is performed on the merged result. Step 7: Target Tracking and Result Output The merged detection results are fed into the target tracking algorithm ByteTrack for inter-frame correlation, timestamp marking, and the results are output in sequence.

2. The real-time target detection method for ultra-high resolution images using a dual-GPU parallel pipeline according to claim 1, characterized in that: Step 1 specifically includes: A detector object pool contains N detectors, where N=6. The main thread receives 12000*8000 grayscale images from the camera. When each detector is instantiated, two independent inference threads are automatically created: Inference Thread 1 and Inference Thread 2. Inference Thread 1 is bound to GPU0 and is responsible for processing the upper half of the image, while Inference Thread 2 is bound to GPU1 and is responsible for processing the lower half of the image. This static binding avoids the overhead of CUDA context switching.

3. The real-time target detection method for ultra-high resolution images using a dual-GPU parallel pipeline according to claim 1, characterized in that: Step 2 specifically includes: The cropped area of ​​the upper half of the original large image is Rect(0, 0, 12000, 4160), and the cropped area of ​​the lower half of the image is Rect(0, 3840, 12000, 4160). The total height of the upper and lower half-images is 8320 pixels, the height of the original image is 8000 pixels, and the overlapping area is 320 pixels. The size of the overlapping area is designed to be greater than half of the maximum possible size of the target to be detected, ensuring that the target located on the dividing line can appear completely in at least one slice and avoiding missing detections caused by the target being "halved". The original image segmentation includes left and right vertical segmentation, "grid" segmentation, and asymmetric segmentation.

4. The ultra-high resolution image dual-GPU parallel pipeline real-time target detection method according to claim 1, wherein: The processing time of a single half-image is 70 ms. Due to dual-card parallelism, the physical computing time of the whole image is 70 ms.

5. The ultra-high resolution image dual-GPU parallel pipeline real-time target detection method according to claim 1, wherein: Although the single-frame processing delay is 70 ms - 80 ms, this delay includes transmission and post-processing time. However, due to the detector array rotation mechanism adopted by the system: T = 0 ms: Frame 1 enters Detector 1, that is, GPU0 / 1 starts working; T = 40 ms: Frame 2 enters Detector 2, that is, GPU0 / 1 is under high load, but Detector 2 can queue or utilize the gaps; T = 80 ms: Frame 1 completes output; And so on, realizing the overlapping processing of multiple frames on the time axis; Through this pipeline design, the system can continuously output results at a rate of 25 fps, that is, at an interval of 40 ms, meeting the real-time index.

6. The ultra-high resolution image dual-GPU parallel pipeline real-time target detection method according to claim 1, wherein: The specific content of step 5 includes: The coordinates of the detection boxes in the upper half-image remain unchanged, and the Y coordinate of the detection boxes in the lower half-image needs to be added with the segmentation offset Offset Y = 3840 pixels, that is, (x, y + 3840).

7. The ultra-high resolution image dual-GPU parallel pipeline real-time target detection method according to claim 1, wherein: The specific content of step 6 includes: Since the same target is detected in both the upper and lower half-images and the restored coordinates are highly coincident in height, NMS will retain the one with the highest confidence and remove the redundancy.

8. A dual-GPU parallel pipeline system for ultra-high resolution image processing in real-time object detection, characterized in that, It includes: An image acquisition module for collecting real-time original grayscale or color images of the target to be detected; It is not limited to RGB images and can also be used for the collection of infrared images and other types of images; A segmentation scheduling module for taking out the images inside the detector and dividing the original large image into two half-images, including upper and lower horizontal segmentation, left and right vertical segmentation, "grid" segmentation, or asymmetric segmentation; A dual-path GPU parallel inference engine, where two GPUs simultaneously run the YOLOv8 inference engine accelerated by TensorRT. After the inference is completed, the results are asynchronously transmitted back to the host memory; A result fusion module, where the main thread waits for the synchronization signals of the two inference threads. When the results of both parts are ready, the two groups of detection boxes are merged into a result set; Perform global non-maximum suppression operation on the merged results.

Images