A large-format image three-stage detection-based three-stage pipeline parallel inference method and system

Abstract

This invention discloses a three-stage pipelined parallel inference method and system for large-format images based on two-stage detection. The method first divides the large-format image into several strips, which are then streamed to the first-stage buffer pool via CUDA by the image distribution thread. The feature extraction processing unit thread synchronously extracts the strips and segments them into patches. The first-stage inference engine completes the initial screening stage of the two-stage detection model, obtaining feature maps and heatmaps. Based on the heatmap, sparse feature sub-blocks of foreground candidate regions are extracted and written into the second-stage annular feature buffer. The second-stage inference consumption unit thread accumulates the feature sub-blocks to a fixed batch and completes the target detection stage of the two-stage detection model in the second-stage inference engine, obtaining the full-image detection result. The image distribution thread, the feature extraction processing unit thread, and the second-stage inference consumption unit thread of this invention execute in parallel, forming a three-stage parallel pipeline, significantly improving GPU utilization and reducing the initial detection latency.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and high-performance heterogeneous computing technology, and in particular to a three-stage pipelined parallel inference method and system for large-format images based on two-stage detection. Background Technology

[0002] Large-format images are widely used in industrial inspection, print quality monitoring, remote sensing, and large-format imaging. These images are generated from various image data sources, and a single frame can reach over 600 million pixels, far exceeding the input size that a general-purpose graphics processing unit (GPU) can accept for a single neural network inference (typically...). or Since the image is divided into pixels, it must be processed in blocks. It should be noted that different image data sources arrive in different forms: some provide the complete image at once, further increasing the engineering difficulty of real-time detection of large-format images.

[0003] Current processing solutions for ultra-large pixel scale images fall into three categories, but none of them can meet the requirements for real-time and near-real-time detection, as detailed below.

[0004] The first type is a fully serial pipeline that first reads the entire image and then performs serial inference in blocks. The SAHI slice-assisted inference framework is a typical example and is currently the most widely used naive processing scheme. The core process of this pipeline is as follows: First, the complete large-format image is read from the storage medium or acquisition device into the CPU's host memory. After the entire image is stored in the host memory, it is divided into multiple blocks according to a preset slice size and overlap ratio. Then, each block is preprocessed sequentially (normalization, format conversion, etc.), and the preprocessed blocks are transferred one by one to the GPU's video memory for inference. Finally, the detection results of each block are merged and subjected to non-maximum suppression to obtain the full image result. This fully serial architecture suffers from several core drawbacks: high image loading latency, large memory peaks, and severe data throughput bottlenecks. Firstly, it requires waiting for the entire image to be fully loaded before any subsequent processing can begin. The latency of the first detection result equals the sum of the entire image loading time and the total inference time, which is insufficient for real-time or near-real-time requirements when the image size reaches hundreds of millions of pixels. Secondly, the entire image must reside in the host memory, resulting in very high memory peaks for images with billions of pixels. Thirdly, image data undergoes multiple transfers and reads / writes between storage / acquisition devices, host memory, slice buffers, and GPU memory, with the same pixel being repeatedly accessed, making data transfer itself a throughput bottleneck. Fourthly, the data transmission, preprocessing, and inference stages are strictly sequential, leaving the GPU idle for extended periods while the CPU slices and transmits data, resulting in low GPU utilization.

[0005] The second approach is a stage-sequential pipeline that supports batch inference. Building upon a fully sequential pipeline, multiple tiles obtained from segmenting the same large-format image are packaged into a batch and fed into the inference engine all at once. This reduces the startup overhead of the inference engine and improves the utilization of the GPU per inference cycle. However, this optimization only makes localized improvements to the inference stage; at the system level, it remains a stage-sequential structure of "read → preprocessing → batch inference → postprocessing." The inference engine idles until the batch assembly is complete, and the transmission and preprocessing of the next batch cannot begin until the previous batch's inference is finished. Furthermore, this batch inference method requires holding all tile data covering the entire image at once, causing memory usage to increase linearly with image size, further exacerbating memory pressure on large-format images. Moreover, this method still does not solve the fundamental problem of waiting for the entire image to be read before processing can begin.

[0006] The third approach is a naive multi-stream parallel or multi-threaded inference pipeline. To address the GPU idle problem in serial solutions, this approach attempts to achieve partial overlap between transmission and computation by having multiple CUDA streams on the GPU execute inference for multiple tiles in parallel, or by using multi-threaded parallel execution of slicing and preprocessing on the CPU. However, this approach still has significant drawbacks: First, frequent stream synchronization interface blocking while waiting for transmission completion causes repeated synchronization between the CPU and GPU, resulting in significant scheduling overhead. Second, when the production speed exceeds the consumption speed, memory slots are easily exhausted, or conversely, the pipeline may starve. Third, the preprocessing of each tile (slicing, grayscale to color expansion, normalization, data layout format conversion) is often implemented as independent multi-pass kernel functions or multiple host / device round trips, resulting in considerable cumulative kernel startup overhead and memory read / write operations, which cannot be ignored in large-format images. Fourth, the detection results generated by each tile are generally collected one by one on the CPU before global non-maximum suppression is performed. This post-processing stage, due to cross-tile result merging and repeated copying from device to host, also becomes a bottleneck at the tail end of the pipeline.

[0007] Furthermore, traditional solutions typically employ a single end-to-end object detection model (such as the YOLO series models) to detect objects in images, applying the exact same detection calculations to each tile obtained from the image segmentation in large-format scenes. However, the object distribution in large-format images is extremely sparse, with most areas consisting of background without objects. Applying the same calculation method to foreground and background regions results in limited GPU computing power being consumed by a large number of targetless background areas, causing severe redundant computation.

[0008] In summary, existing large-format image detection pipelines suffer from problems such as redundant background calculations due to indiscriminate inference across the entire domain, resulting in extremely low computing power utilization. Furthermore, their architecture, which relies on loading the entire image before processing, causes significant delays in the first result, excessive peak memory usage on the host machine, and ultimately, data synchronization between the CPU and GPU creates a system throughput bottleneck. Summary of the Invention

[0009] Purpose of the invention: The purpose of this invention is to provide a three-stage pipelined parallel inference method and system for large-format images based on two-stage detection. This method can skip sparse background regions to avoid redundant calculations and can maximize GPU utilization by using streaming processing to fully parallelize data loading and inference at each stage.

[0010] Technical solution: The present invention provides a three-stage pipelined parallel inference method for large-format images based on two-stage detection, comprising the following steps:

[0011] Step 1: Divide the large-format image to be detected into several strips evenly along the vertical direction;

[0012] Step 2: The image distribution thread asynchronously transmits the pixel data of each stripe from the host memory to the idle GPU memory slot in the first-level buffer pool via a dedicated CUDA stream. The transmission is recorded by a CUDA event when it is completed.

[0013] Step 3: After each strip is transmitted, the feature extraction processing unit thread retrieves the strip from the first-level buffer pool and divides the strip into several patches;

[0014] Step 4: The feature extraction processing unit thread sends the image patch into the first-level inference engine. The first-level inference engine is used to perform the initial screening stage of the two-stage detection model to obtain feature maps and heat maps. The feature extraction processing unit thread extracts sparse feature sub-blocks of the foreground candidate region based on the heat map and writes the feature sub-blocks into the second-level circular feature buffer.

[0015] Step 5: The second-level inference consumption unit thread accumulates feature sub-blocks from the second-level circular feature buffer. When the accumulated number reaches the predetermined batch size, the second-level inference engine processes a batch of the feature sub-blocks. The second-level inference engine then performs the target detection stage of the two-stage detection model to obtain the full-image detection result.

[0016] The image distribution thread, the feature extraction and processing unit thread, and the second-level inference and consumption unit thread are executed in parallel.

[0017] Furthermore, in step 2, a CUDA event is recorded after the transmission of each media strip is completed; in step 3, the feature extraction processing unit thread waits on the CUDA stream for the host-to-device transmission completion event of the strip via the stream wait event interface, and after waiting for the CUDA event, the strip is retrieved from the first-level buffer pool.

[0018] Furthermore, in step 1, the starting ordinate of each strip in the large-format image (also known as the full image) is recorded as coordinate traceability metadata; in step 4, the coordinate traceability table of each feature sub-image is recorded; in step 5, the relative coordinates of the detection boxes output in the target detection stage are restored to the full image coordinates to obtain the full image detection result.

[0019] The coordinate tracing table includes the large-format image number from which the feature sub-image originates (to prevent confusion when reasoning through multiple complete images in a single step), the global number of the plot it belongs to, the horizontal and vertical offsets of the top-left corner of the plot in the large-format image, and the starting horizontal and vertical coordinates in the feature map.

[0020] Furthermore, the coordinate mapping kernel function performs a three-level cascaded transformation on the relative coordinates output by the second-level inference engine:

[0021] The first layer transforms the feature coordinates of the feature sub-block into pixel coordinates of the image block based on the downsampling factor of the feature map and the starting coordinates of the feature sub-block in the feature map; the second layer transforms the pixel coordinates of the image block into pixel coordinates of the whole image based on the offset of the image block in the coordinate tracing table; the third layer generates the corner coordinates of the detection box based on the center coordinates and width and height of the detection box.

[0022] Furthermore, the first-level buffer pool consists of several pre-allocated GPU memory slots. The image distribution thread calls a slot-fetching operation before transmission and blocks and waits when there are no free slots in the first-level buffer pool. After the feature extraction processing unit thread completes processing, it calls a return operation to return the GPU memory slots and wake up the image distribution thread.

[0023] Furthermore, the second-level circular feature buffer adopts a design that separates physical and logical indices, wherein both the logical write pointer and the logical read pointer are monotonically increasing and never roll back; the physical index is obtained by taking the modulo of the corresponding logical index with the capacity of the second-level circular feature buffer;

[0024] When the feature sub-block to be written crosses the end of the circular buffer, the feature sub-block is automatically split into two segments, which are then written to the end segment and the beginning segment of the buffer through two device-to-device asynchronous copies.

[0025] Furthermore, the second-level inference consumption unit thread continuously pulls feature sub-block batches from the batch queue to advance the consumable boundary. When the difference between the consumable boundary and the logical read pointer reaches the predetermined batch size, a second-level inference is executed once. When the batch queue has been closed and there are still remaining feature sub-blocks that are less than the predetermined batch size, the remaining feature sub-blocks are padded with zeros and then the last second-level inference is executed.

[0026] Furthermore, the feature extraction processing unit thread calls the fusion block kernel function to divide the strip into several blocks in the GPU memory slot according to a preset step size. In one call, the fusion block kernel function also completes the format conversion from high-width channel arrangement to channel high-width arrangement, and / or normalization from unsigned integer to single-precision floating-point number, and / or grayscale to three-channel broadcast expansion.

[0027] Furthermore, the feature extraction processing unit thread asynchronously and serially executes four GPU kernel functions on the CUDA stream to extract feature sub-blocks:

[0028] The first kernel function applies a Sigmoid transform to the heatmap and compares it with a threshold before performing neighborhood local maxima suppression to obtain an active pixel mask. The second kernel function divides the feature map into a preset grid and generates candidate feature sub-blocks for the grid containing active pixels. The third kernel function performs a prefix sum on the candidate feature sub-block counts for each patch and calculates the offset of each patch in the global candidate feature sub-block array. The fourth kernel function extracts fixed-size feature sub-blocks from the feature map based on the output results of the first three kernel functions.

[0029] Based on the same inventive concept, the large-format image three-stage pipelined parallel inference system based on two-stage detection described in this invention includes:

[0030] The image distribution unit is used to divide the large-format image to be detected into several strips in a vertical direction. The image distribution thread transmits the pixel data of each strip asynchronously from the host memory to the idle GPU memory slot in the first-level buffer pool through a dedicated CUDA stream. The transmission is recorded by CUDA event when it is completed.

[0031] The feature extraction processing unit is used to: after each strip transmission is completed, retrieve the strip from the first-level buffer pool and divide the strip into several image patches; send the image patches into the first-level inference engine, which is used to perform the initial screening stage of the two-stage detection model to obtain feature maps and heat maps; and extract sparse feature sub-blocks of the foreground candidate region based on the heat map and write the feature sub-blocks into the second-level circular feature buffer.

[0032] The second-level inference consumption unit is used to accumulate feature sub-blocks from the second-level circular feature buffer by the second-level inference consumption unit thread. When the accumulated number reaches the predetermined batch size, the second-level inference engine will process a batch of the feature sub-blocks. The second-level inference engine will execute the target detection stage of the two-stage detection model to obtain the full image detection result.

[0033] The image distribution thread, the feature extraction and processing unit thread, and the second-level inference and consumption unit thread are executed in parallel.

[0034] Based on the same inventive concept, the large-format image target detection method based on two-stage detection described in this invention includes the following steps:

[0035] The image patch is input into the target detection model, which employs a two-stage detection process, including an initial screening stage and a target detection stage. The target detection model is deployed according to the three-stage pipelined parallel inference method for large-format images based on two-stage detection.

[0036] The object detection model performs initial screening of the map tiles using a first-level inference engine to obtain feature maps and heatmaps; then, it performs object detection on the feature sub-maps extracted from the heatmaps using a second-level inference engine to obtain detection boxes.

[0037] The electronic device of the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is loaded onto the processor, it implements the large-format image three-stage pipeline parallel inference method based on two-stage detection, or the large-format image target detection method based on two-stage detection.

[0038] The computer-readable storage medium of the present invention stores a computer program, which, when executed by a processor, implements the large-format image three-stage pipeline parallel inference method based on two-stage detection, or the large-format image target detection method based on two-stage detection.

[0039] The computer program product of the present invention includes a computer program that, when executed by a processor, implements the large-format image three-stage pipeline parallel inference method based on two-stage detection, or the large-format image target detection method based on two-stage detection.

[0040] Beneficial effects: Compared with the prior art, the advantages of the present invention are as follows:

[0041] (1) The three-stage pipelined parallel inference method of the present invention fully overlaps the three stages of data reading, first-stage inference and second-stage inference in time, eliminating the idle time of the device in serial processing, and significantly improving the GPU utilization and overall throughput; and the present invention adopts streaming processing, which does not require loading the complete image into the processor memory at once, but loads it strip by strip and infers it in real time, so that data loading and inference calculation overlap. In the case where the complete image is ready, it avoids the peak of the video memory and waiting delay of loading the whole image at once. In the case where the image data arrives in batches, it can be received and processed at the same time, further reducing the delay of the first detection result.

[0042] (2) The present invention adopts cross-stream synchronization based on CUDA events, which avoids CPU polling and frequent stream synchronization blocking, and reduces the scheduling overhead between the host and the device.

[0043] (3) The fusion block kernel function of the present invention completes slicing, channel expansion, normalization and format conversion in a single kernel call, reducing kernel startup and memory round trip; the heat map driven sparse feature sub-block extraction only performs second-level inference on the activated foreground candidate region, skipping the background region which accounts for the vast majority, greatly reducing the computational load and redundancy of the second level.

[0044] (4) The circular buffer that separates the physical index and the logical index of the present invention, combined with the cross-boundary two-segment copy and back pressure mechanism, realizes efficient and safe transfer of feature sub-blocks between threads; the differentiated batch size strategy takes into account both the large throughput of the first level and the batch utilization of the second level.

[0045] (5) The two-stage detection model segmentation avoids repeatedly extracting backbone features from different patches of the same original image; page-locked mapping memory enables zero-copy access to grayscale data, further reducing transmission overhead. Attached Figure Description

[0046] Figure 1 This is a flowchart of the three-stage pipelined parallel inference method for large-format images according to an embodiment of the present invention.

[0047] Figure 2 This is a timing diagram for the parallel execution of a three-stage pipeline according to an embodiment of the present invention.

[0048] Figure 3 This is a schematic diagram of large-format image strip division and block segmentation according to an embodiment of the present invention.

[0049] Figure 4 This is a schematic diagram of the first-level buffer pool and back pressure control mechanism in an embodiment of the present invention.

[0050] Figure 5 This is a flowchart of the feature extraction processing unit thread processing according to an embodiment of the present invention.

[0051] Figure 6 The heatmap processing of the four kernel function data flow graphs in this embodiment of the invention is described.

[0052] Figure 7 This is a schematic diagram of the physical / logical index separation and cross-boundary two-segment copying of the second-level circular feature buffer in an embodiment of the present invention.

[0053] Figure 8 This is a flowchart of the batch cumulative inference process for the second-level inference consumption unit in an embodiment of the present invention.

[0054] Figure 9This is a schematic diagram of the three-layer coordinate mapping in an embodiment of the present invention.

[0055] Figure 10 This is a flowchart illustrating the cascading shutdown of bounded blocking queues according to an embodiment of the present invention.

[0056] Figure 11 This is a diagram illustrating the two-stage detection model splitting and quantization method according to an embodiment of the present invention. Detailed Implementation

[0057] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0058] The parallel inference method of this invention is used to implement a two-stage detection scheme including an initial screening stage. The first stage performs a coarse screening of the image tiles to locate sparse foreground candidate regions, and the second stage performs a fine screening only on the candidate regions, thereby skipping the background regions that account for the majority and eliminating redundant calculations. Representative schemes of the two-stage detection scheme include ESOD and OAN.

[0059] Typically, an object detector consists of three parts: a backbone network (STEM), a feature aggregation module (Neck), and a detection head (Head). The ESOD scheme inserts an object discovery module as a preliminary screening stage, after the early stage of the backbone network (STEM) and before feature aggregation (Neck). This module predicts a class-independent objectness mask from the high-resolution feature map output by the STEM (downsampled 8 times relative to the input patches) to locate foreground candidate regions. Subsequently, in the object detection stage, an adaptive feature slicer performs feature-level patch segmentation on the same STEM output feature map as the object discovery module, discarding feature patches without objects and only sending feature patches containing objects to the Neck for subsequent feature aggregation. Finally, the SparseHead performs object detection at sparse activation locations.

[0060] The OAN approach does not segment the intermediate feature layers of the detection model. Instead, it attaches a lightweight fully convolutional initial screening network to the feature maps at the end of the backbone network (for example, the C5 stage in ResNet50, which is downsampled by 32 times relative to the input). This initial screening network evenly divides the input map patches into several grids (e.g., ...). The input map is divided into The OAN constructs a grid of pixels, predicts an objectness score for each pixel, and takes the maximum score as the confidence level for whether the input pixel contains an object. During inference, OAN uses a gating operation to only feed the features of an input pixel into the detector's subsequent modules (neck and head) if the confidence level exceeds a threshold; otherwise, the entire input pixel is discarded and does not enter the subsequent detection process. Therefore, the segmentation points of OAN are located at the ends of the backbone network.

[0061] To leverage parallelism, this invention splits the detection model into two independent inference engines and constructs a three-stage pipeline, enabling data preprocessing (strip partitioning), the first-stage inference, and the second-stage inference to execute simultaneously, thereby maximizing GPU utilization. The method employs streaming processing, eliminating the need to load the entire image into processor memory all at once. Instead, it loads the image strip by strip and performs inference in real-time, allowing data loading and inference computation to overlap in time. This approach is applicable to both input formats where the entire image arrives at once and where image data arrives in batches. Furthermore, this invention provides complementary model segmentation and quantization deployment methods, as well as access methods for various image data sources. It should be noted that this invention does not involve improvements to the neural network model itself, but focuses on the engineering pipeline architecture for efficiently deploying the detection model on large-format images, along with its segmentation and quantization preparation methods.

[0062] like Figure 1 As shown, the large-format image three-stage pipelined parallel inference method in this embodiment adopts a three-layer producer-consumer parallel model, consisting of an image distribution thread, a feature extraction processing unit thread pool (one by default), and a second-stage inference consumption unit thread. These three threads respectively handle data reading, the first-stage inference, and the second-stage inference. After the large-format image is divided into strips, the image distribution thread transmits each strip to the first-stage buffer pool. The feature extraction processing unit thread consumes strips from the first-stage buffer pool, performs slicing, first-stage inference, and heatmap processing, and writes the results to the second-stage circular feature buffer. The second-stage inference consumption unit thread consumes feature sub-blocks from the second-stage circular feature buffer, performs second-stage inference, coordinate decoding, and non-maximum suppression, and finally outputs a list of full-image detection results. It should be noted that this device operates in a streaming mode, and can proceed with inference strip by strip without waiting for the entire image to be loaded.

[0063] like Figure 2 As shown, the three layers of threads run concurrently. At a certain moment, the image distribution thread is uploading the (k+2)th strip, the feature extraction and processing unit thread is inferring the (k+1)th strip, and the second-level inference and consumption unit thread is processing the feature sub-blocks generated by the kth strip. This forms a staggered, fully pipelined parallelism, allowing the three stages to fully overlap in time.

[0064] The following section will introduce each thread separately.

[0065] (1) Image distribution thread

[0066] See Figure 3The large-format image is uniformly divided vertically into strips with a height equal to the slice size (640 pixels in this example), and the width of each strip is equal to the width of the entire image. The number of strips is equal to the total height of the entire image divided by the slice size and rounded up. Each strip carries metadata such as the strip number, the starting ordinate of the strip in the entire image, the actual pixel height of the strip, and the allocated GPU buffer slot number; the starting ordinate is used for subsequent coordinate reconstruction.

[0067] See Figure 4 The first-level buffer pool consists of a set of pre-allocated GPU memory slots. The size of a single slot is calculated using the following formula:

[0068] Single slot size = Full image width × Slice size × Number of channels × Number of bytes per pixel;

[0069] The first-level buffer pool has a capacity of several slots (typically two). Before transmission, the image distribution thread calls a slot acquisition operation; if no slots are available, it blocks and waits. After the feature extraction processing unit thread finishes processing, it calls a return operation, returning the memory slots and waking up the image distribution thread. Thus, when all slots are occupied, the image distribution thread is blocked, achieving backpressure control upstream, preventing both GPU memory overflow and upstream image data source overflow.

[0070] In terms of host-to-device transmission optimization, a dedicated CUDA stream is used to handle host-to-device transmission, and page-locked mapped memory is used to avoid intermediate buffer copying: page-locked host memory is pre-allocated in a mappable manner and its corresponding device pointer is obtained. During transmission, the striped data is first written to the page-locked memory on the host side, and then asynchronously copied to the target video memory slot via the transport stream. The transmission is completed by recording the CUDA event.

[0071] (2) Feature extraction processing unit thread

[0072] See Figure 5 The processing flow of the feature extraction processing unit thread is as follows: retrieve the stripe from the corresponding queue of the first-level buffer pool; wait for the host-to-device transmission completion event of the stripe on the working CUDA stream via the stream waiting event interface, thus eliminating the need for CPU polling; call the fusion block kernel function to complete slicing and format conversion; call the first-level inference engine to perform inference; perform heatmap processing to extract sparse feature sub-blocks; write the feature sub-blocks to the second-level circular feature buffer via cross-boundary two-stage writing; finally return the memory slot and process the next stripe in a loop.

[0073] Specifically, the fusion block kernel function directly divides the stripe into multiple square tiles in the GPU memory. The horizontal tile step size is smaller than the tile width (640 pixels in this embodiment), and the vertical step size is the tile size (there is no vertical overlap within the stripe, and the continuity between stripes is guaranteed by the cutting). This kernel function completes the following in a single call: optional grayscale to three-channel broadcast expansion, normalization from eight-bit unsigned integer to single-precision floating-point, and format conversion from height-width channel layout to channel height-width layout; sampling positions beyond the image boundary are zero-padded to avoid out-of-bounds access.

[0074] Specifically, in the first-level inference stage, multiple parallel feature extraction processing unit threads can be configured to retrieve stripes from the first-level buffer pool, depending on the hardware. These threads share the same first-level inference engine object to save GPU memory. Each feature extraction processing unit thread holds its own independent execution context. Since the execution context exclusively occupies the workspace, concurrent calls can lead to workspace pollution; therefore, a global mutex lock is used to serialize calls to the first-level inference engine. The first-level inference engine outputs two results: one is a P3 feature map (in this embodiment, the number of channels is 64, and the spatial size is...). The other path is the heatmap logarithmic probability (single channel, spatial size and characteristics). Figure 1 (To).

[0075] Specifically, the heatmap processing method involves asynchronously and serially executing four GPU kernel functions on the working CUDA stream. The first kernel function applies a sigmoid transformation to the logarithmic probability of the heatmap to obtain a probability, which is then compared with a threshold (0.3 in this embodiment) before... Local maximum suppression is performed within the neighborhood, and the activated pixel mask is output. The second kernel function divides the feature map according to a preset grid (in this embodiment...). The first kernel divides the feature map into four parts. For each grid containing active pixels, a candidate feature sub-block is generated. The coordinate range of each feature sub-block in the feature map and the feature sub-block count for each block are output. The second kernel performs a prefix sum on the feature sub-block counts of each block and calculates the offset of each block in the global feature sub-block array. The third kernel extracts a fixed-size (in this embodiment, 1000 pixels) feature from the feature map based on the above results. The feature sub-blocks of the coordinate traceability table are generated.

[0076] The coordinate tracing table records metadata for each feature sub-block, including: the full map number from which it originated, the global map number from which it originated, the horizontal and vertical offsets of the top-left corner of the sub-block within the full map, and the starting x and y coordinates of the feature sub-block within the feature map. This coordinate tracing table provides the basis for the subsequent three-layer coordinate mapping.

[0077] (3) Second-level inference consumption unit thread

[0078] See Figure 7 The second-level circular feature buffer is used by the feature extraction processing unit thread to transfer feature sub-block data to the second-level inference consumption unit thread. It employs a design that separates physical and logical indexes. The logical index includes a logical write pointer updated by the feature extraction processing unit thread and a logical read pointer updated by the second-level inference consumption unit thread; both are monotonically increasing and never rollback. The physical index is obtained by taking the corresponding logical index modulo the buffer capacity and is used to map it to the actual video memory location. The circular buffer stores the feature sub-block buffer and coordinate tracing table in video memory; in this embodiment, the capacity is 1024 feature sub-blocks.

[0079] Furthermore, regarding cross-boundary writes, when a feature sub-block to be written by the feature extraction processing unit thread crosses the end of the second-level circular feature buffer, it is automatically split into two segments: the first segment is written from the current physical location to the end of the buffer, and the second segment is written from the beginning of the buffer to the remaining part, completed through two asynchronous device-to-device copies respectively; when there is no cross-boundary writing, it is completed with a single asynchronous copy. The second-level inference consumption unit thread works similarly when reading, linearizing potentially discontinuous data in the circular buffer to a temporary buffer for use by the second-level inference engine.

[0080] Furthermore, regarding back pressure, when the space of the second-level circular feature buffer is insufficient, the feature extraction processing unit thread is blocked and waits for the second-level inference consumption unit thread to consume; after the second-level inference consumption unit thread consumes, it updates the logical read pointer and notifies the waiting thread, thereby achieving flow control between the two threads.

[0081] See Figure 8 The second-level inference consumption unit thread employs a batch accumulation strategy. It maintains a consumable boundary representing the largest known consumable logical position. The thread continuously pulls feature sub-block batches from the batch queue to advance the consumable boundary; when the difference between the consumable boundary and the logical read pointer reaches the fixed batch size, a second-level inference is executed, and the logical read pointer is advanced accordingly; when the batch queue is closed and there are still remaining feature sub-blocks smaller than the fixed batch size, the remaining feature sub-blocks are padded with zeros before a final second-level inference is executed. The purpose of fixed-batch accumulation is that, in sparse scenarios, the actual number of feature sub-blocks per strip is much smaller than the grid limit. Accumulating across multiple stripes to a fixed batch before inference improves the GPU utilization of the second-level inference engine and avoids excessive small-batch inference.

[0082] Before executing the second-level inference, the second-level inference consumer unit thread first copies the potentially discontinuous feature sub-block data and their coordinate traceability table from the circular buffer to a contiguous temporary buffer before sending them into the second-level inference engine.

[0083] See Figure 9The second-level inference engine outputs relative coordinates to the feature map of the feature sub-block. These coordinates are then transformed into full-image coordinates through a three-layer cascaded transformation using a coordinate mapping kernel function. The first layer transforms the feature coordinates of the feature sub-block into pixel coordinates based on the downsampling factor of the feature map relative to the original image and the starting coordinates of the feature sub-block in the feature map. The second layer transforms the pixel coordinates of the pixel into full-image pixel coordinates based on the horizontal and vertical offsets of the pixel in the coordinate source table. The third layer generates the coordinates of the two corner points of the final detection box based on the center coordinates of the full image and the target's width and height. Confidence is then filtered, and the product of the target-specific confidence and the maximum class confidence is used as the final confidence. When this confidence is not less than a threshold and the detection box area is not less than a preset minimum area, it is written to the global detection result array via atomic operations.

[0084] After the detection results from all feature extraction processing unit threads and all second-level inference batches are written to the global GPU memory detection result array, a global GPU non-maximum suppression is performed when the second-level inference consumption unit thread completes its last inference: each thread processes one detection box and compares it with the same type of box with higher confidence using cross-union. If the suppression threshold is exceeded, the current box is marked as suppressed. This unifies the processing of all detection results across units and batches.

[0085] The thread synchronization and cascading shutdown methods in the large-format image three-stage pipeline parallel inference method of this embodiment include the following:

[0086] See Figure 10 The first-level buffer pool and its corresponding queues, as well as the batch queues, are implemented using the same bounded blocking queues. The closing operation sets the closing flag and wakes up all waiting threads; the retrieval operation returns failure when the queue is empty and closed as an exit signal; the insertion operation directly returns failure to discard data when the queue is closed. The graceful exit of this device follows a cascading shutdown process: the image distribution thread closes the corresponding queue of the first-level buffer pool after transmitting all stripes; the feature extraction processing unit thread exits its loop and synchronizes its workflow after its retrieval operation returns failure; the last exiting feature extraction processing unit thread closes the batch queue; the second-level inference consumption unit thread refreshes the remaining feature sub-blocks and exits after its retrieval operation returns failure; the main thread merges all threads and collects the results.

[0087] The two-stage detection model segmentation and quantization deployment method in the large-format image three-stage pipeline parallel inference method of this embodiment includes the following:

[0088] See Figure 11This embodiment uses YOLO series models (such as YOLOv5 and YOLOv8) that employ the ESOD scheme for introduction. The 8x downsampled feature map of its backbone network corresponds to the P3 feature layer in the FPN. This embodiment divides the model at the P3 feature layer of the backbone network into a first-level inference engine and a second-level inference engine: the first-level inference engine performs initial screening, inputting image patches and outputting the log-probability of the P3 feature map and heatmap. The heatmap, derived from the segmentation branch, is used to locate sparse foreground candidate regions. The second-level inference engine performs fine-tuning, inputting fixed-size feature sub-blocks and outputting detection boxes. Only feature sub-blocks located by the heatmap are extracted and fed into the second-level inference engine, thus skipping background regions and avoiding redundant computation. This segmentation also avoids repeatedly extracting backbone features from different patches of the same original image and allows the second-level inference engine to perform independent inference in fixed batches, facilitating parallel processing with the first-level inference engine in the pipeline. The reasons for choosing the P3 feature layer as the segmentation point are: P3 has the highest resolution, thus preserving more details; the heatmap of the segmentation branch is based on the P3 output, which is consistent with the segmentation point; the feature sub-block size is fixed, allowing the second-level inference engine to use fixed batch independent inference; the number of feature channels after P3 is moderate, and the feature sub-block size is appropriate.

[0089] Specifically, regarding the batch size strategy, the first-level inference engine uses a larger maximum batch size (24 in this embodiment, corresponding to the number of patches generated by a strip at a given horizontal step size plus a margin); the second-level inference engine uses a fixed batch size (48 in this embodiment), accumulating feature sub-blocks across multiple strips into a fixed batch before inference, in order to improve its GPU utilization.

[0090] Specifically, the inference engine construction process is as follows: the trained weights are exported as intermediate representations (including split branches during export), and then the inference framework interface or command-line tool is used to build the inference engine, naming it according to the maximum batch size and precision. During construction, the minimum, optimal, and maximum configurations of the dynamic batch dimension are set; half-precision (approximately twice the speed of single precision) or integer quantization (requiring data calibration) is enabled depending on platform support; and an appropriate workspace size is set.

[0091] Specifically, for integer quantization calibration, representative patches need to be extracted from real large-format images as the calibration set: each representative large-format image is divided into strips, and each strip is further divided into patches according to horizontal step size, collecting inputs that conform to the true distribution; the calibration set should cover different lighting and target density conditions, include normal frames and background frames with sparse targets, and have a sufficient number. The calibration of the second-level inference engine targets feature sub-blocks, collecting actual feature sub-blocks extracted from the output of the first-level inference engine through heatmap processing from the normal inference process as the calibration set.

[0092] Specifically, regarding engine caching, cache files are named according to the model base name, maximum batch size, and precision based on the cache path. Cache validity is determined by the modification time of the engine cache file and the intermediate representation file: if the engine cache file does not exist or its modification time is earlier than that of the intermediate representation file, it is rebuilt and written to the cache; otherwise, the cache is loaded directly. Regarding runtime binding, this device automatically identifies feature map output and heatmap output by scanning the binding name and dimension of the engine (e.g., determining a feature map based on the number of channels greater than a threshold, and a heatmap based on the number of channels equal to one), and automatically obtains the number of feature channels. For the second-level inference engine, the number of detection boxes is automatically derived from its output dimension.

[0093] The image data source access and streaming processing method in the large-format image three-stage pipeline parallel inference method of this embodiment includes the following:

[0094] This embodiment does not limit the specific source of the large-format image, but defines a unified strip data structure compatible with multiple image data sources. This structure includes the strip's starting ordinate in the entire image, strip width, actual strip height, pixel data pointers, and slot numbers assigned by the image data source and to be returned after processing. With this unified structure, different sources such as file reading, whole-image memory buffering, and batch-arriving data streams can all be connected to the pipeline via the same path.

[0095] Based on the data arrival pattern, image data sources can be divided into two scenarios, both processed in a streaming manner. The first scenario involves the entire image arriving at once: In this case, the image distribution thread does not wait to load the entire image into memory, but loads it strip by strip and sends it to the pipeline immediately. This avoids the memory spike caused by loading the entire image at once and allows downstream inference to begin without waiting for the entire data to be ready. The second scenario involves image data arriving in batches: In this case, whenever a strip of data arrives, the image distribution thread immediately sends it to the pipeline, achieving on-demand processing and further reducing the latency of the first detection result. Regardless of the scenario, backpressure control of the first-level buffer pool prevents upstream data source overflow.

[0096] Specifically, in grayscale-to-color conversion, when a large-format image is a single-channel grayscale image while the neural network requires three-channel color input, the conversion can be completed in two places: firstly, on the host side, page-locked memory is expanded to copy each grayscale pixel to the three channels; secondly, in the grayscale mode of the fusion block kernel function, the single-channel grayscale pixels are directly broadcast and expanded to three channels and normalized on the GPU side. This embodiment uses page-locked mapped memory to receive the grayscale stripes output from the image data source, so that the pointers on the host side and the device side are mapped to the same physical memory, thereby achieving zero-copy access, and the expansion is completed in grayscale mode in the fusion block kernel function, avoiding intermediate copying within the host.

[0097] The large-format image three-stage pipelined parallel inference system based on two-stage detection described in this invention includes:

[0098] The image distribution unit is used to divide the large-format image to be detected into several strips in a vertical direction. The image distribution thread transmits the pixel data of each strip asynchronously from the host memory to the idle GPU memory slot in the first-level buffer pool through a dedicated CUDA stream. The transmission is recorded by CUDA event when it is completed.

[0099] The feature extraction processing unit is used to: after each strip transmission is completed, retrieve the strip from the first-level buffer pool and divide the strip into several image patches; send the image patches into the first-level inference engine, which is used to perform the initial screening stage of the two-stage detection model to obtain feature maps and heat maps; and extract sparse feature sub-blocks of the foreground candidate region based on the heat map and write the feature sub-blocks into the second-level circular feature buffer.

[0100] The second-level inference consumption unit is used to accumulate feature sub-blocks from the second-level circular feature buffer by the second-level inference consumption unit thread. When the accumulated number reaches the predetermined batch size, the second-level inference engine will process a batch of the feature sub-blocks. The second-level inference engine will execute the target detection stage of the two-stage detection model to obtain the full image detection result.

[0101] The image distribution thread, the feature extraction and processing unit thread, and the second-level inference and consumption unit thread are executed in parallel.

[0102] The large-format image target detection method based on two-stage detection described in this invention includes the following steps:

[0103] The image patch is input into the target detection model, which employs a two-stage detection process, including an initial screening stage and a target detection stage. The target detection model is deployed according to the three-stage pipelined parallel inference method for large-format images based on two-stage detection.

[0104] The object detection model performs initial screening of the map blocks using a first-level inference engine to obtain feature maps and heatmaps; then, it performs object detection on the feature sub-maps extracted from the heatmaps using a second-level inference engine to obtain detection boxes.

[0105] The electronic device of the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is loaded onto the processor, it implements the large-format image three-stage pipeline parallel inference method based on two-stage detection, or the large-format image target detection method based on two-stage detection.

[0106] The computer-readable storage medium of the present invention stores a computer program, which, when executed by a processor, implements the large-format image three-stage pipeline parallel inference method based on two-stage detection, or the large-format image target detection method based on two-stage detection.

[0107] The computer program product of the present invention includes a computer program that, when executed by a processor, implements the large-format image three-stage pipeline parallel inference method based on two-stage detection, or the large-format image target detection method based on two-stage detection.

[0108] The computer-readable storage medium may include RAM, ROM, EEPROM, CD-ROM or other optical disc storage devices, magnetic disk storage devices or other magnetic storage devices, flash memory, or any other media that can be used to store program code in the form of instructions or data structures and is accessible by a computer.

[0109] The processor is used to execute a computer program stored in memory to implement the various steps in the methods described in the above embodiments.

Claims

1. A three-stage pipelined parallel inference method for large-format images based on two-stage detection, characterized in that, Includes the following steps: Step 1: Divide the large-format image to be detected into several strips evenly along the vertical direction; Step 2: The image distribution thread asynchronously transmits the pixel data of each stripe from the host memory to the idle GPU memory slot in the first-level buffer pool via a dedicated CUDA stream. The transmission is recorded by a CUDA event when it is completed. Step 3: After each strip is transmitted, the feature extraction processing unit thread retrieves the strip from the first-level buffer pool and divides the strip into several patches; Step 4: The feature extraction processing unit thread sends the image patch into the first-level inference engine. The first-level inference engine is used to perform the initial screening stage of the two-stage detection model to obtain feature maps and heat maps. The feature extraction processing unit thread extracts sparse feature sub-blocks of the foreground candidate region based on the heat map and writes the feature sub-blocks into the second-level circular feature buffer. Step 5: The second-level inference consumption unit thread accumulates feature sub-blocks from the second-level circular feature buffer. When the accumulated number reaches the predetermined batch size, the second-level inference engine processes a batch of the feature sub-blocks. The second-level inference engine then performs the target detection stage of the two-stage detection model to obtain the full-image detection result. The image distribution thread, the feature extraction and processing unit thread, and the second-level inference and consumption unit thread are executed in parallel.

2. The large-format image three-stage pipelined parallel inference method based on two-stage detection according to claim 1, characterized in that, In step 2, after the transmission of each media strip is completed, a CUDA event is recorded; in step 3, the feature extraction processing unit thread waits on the CUDA stream for the host-to-device transmission completion event of the strip via the stream wait event interface, and after waiting for the CUDA event, the strip is retrieved from the first-level buffer pool.

3. The large-format image three-stage pipelined parallel inference method based on two-stage detection according to claim 1, characterized in that, In step 1, the starting ordinate of each strip in the large-format image is recorded as coordinate traceability metadata; in step 4, the coordinate traceability table of each feature sub-image is recorded; in step 5, the relative coordinates of the detection boxes output in the target detection stage are restored to the full-image coordinates to obtain the full-image detection result. The coordinate tracing table includes the large-format image number from which the feature sub-image originates, the global number of the image block it belongs to, the horizontal and vertical offsets of the upper left corner of the image block in the large-format image, and the starting horizontal and vertical coordinates in the feature image.

4. The large-format image three-stage pipelined parallel inference method based on two-stage detection according to claim 3, characterized in that, The coordinate mapping kernel function performs a three-level cascaded transformation on the relative coordinates output by the second-level inference engine: The first layer transforms the feature coordinates of the feature sub-blocks into pixel coordinates of the image blocks based on the downsampling factor of the feature map and the starting coordinates of the feature sub-blocks in the feature map; the second layer transforms the pixel coordinates of the image blocks into pixel coordinates of the entire image based on the offset of the image blocks in the coordinate tracing table; the third layer generates the corner coordinates of the detection box based on the center coordinates and width and height of the detection box.

5. The large-format image three-stage pipeline parallel inference method based on two-stage detection according to claim 1, characterized in that, The first-level buffer pool consists of several pre-allocated GPU memory slots. The image distribution thread calls a slot-fetching operation before transmission and blocks and waits when there are no free slots in the first-level buffer pool. After the feature extraction processing unit thread completes processing, it calls a return operation to return the GPU memory slots and wake up the image distribution thread.

6. The large-format image three-stage pipelined parallel inference method based on two-stage detection according to claim 1, characterized in that, The second-level circular feature buffer adopts a design that separates physical and logical indices. The logical write pointer and logical read pointer are both monotonically increasing and never roll back. The physical index is obtained by taking the modulo of the corresponding logical index with the capacity of the second-level circular feature buffer. When the feature sub-block to be written crosses the end of the circular buffer, the feature sub-block is automatically split into two segments, which are then written to the end segment and the beginning segment of the buffer through two device-to-device asynchronous copies.

7. The large-format image three-stage pipelined parallel inference method based on two-stage detection according to claim 6, characterized in that, The second-level inference consumption unit thread continuously pulls feature sub-block batches from the batch queue to advance the consumable boundary. When the difference between the consumable boundary and the logical read pointer reaches the predetermined batch size, a second-level inference is executed once. When the batch queue is closed and there are still remaining feature sub-blocks that are less than the predetermined batch size, the remaining feature sub-blocks are padded with zeros and then the last second-level inference is executed.

8. The large-format image three-stage pipelined parallel inference method based on two-stage detection according to claim 1, characterized in that, The feature extraction processing unit thread calls the fusion block kernel function to divide the strip into several blocks in the GPU memory slot according to a preset step size. In one call, the fusion block kernel function also completes the format conversion from high-width channel arrangement to channel high-width arrangement, and / or normalization from unsigned integer to single-precision floating-point number, and / or grayscale to three-channel broadcast expansion.

9. The large-format image three-stage pipeline parallel inference method based on two-stage detection according to claim 1, characterized in that, The feature extraction processing unit thread asynchronously and serially executes four GPU kernel functions on the CUDA stream to extract feature sub-blocks: The first kernel function applies a Sigmoid transformation to the heatmap and compares it with a threshold before performing neighborhood local maxima suppression to obtain the active pixel mask. The second kernel function divides the feature map into a preset grid and generates candidate feature sub-blocks for the grid containing active pixels; the third kernel function performs a prefix sum on the candidate feature sub-block counts of each patch and calculates the offset of each patch in the global candidate feature sub-block array; the fourth kernel function extracts fixed-size feature sub-blocks from the feature map based on the output results of the first three kernel functions.

10. A three-stage pipelined parallel inference system for large-format images based on two-stage detection, characterized in that, include: The image distribution unit is used to divide the large-format image to be detected into several strips in a vertical direction. The image distribution thread transmits the pixel data of each strip asynchronously from the host memory to the idle GPU memory slot in the first-level buffer pool through a dedicated CUDA stream. The transmission is recorded by CUDA event when it is completed. The feature extraction processing unit is used to retrieve the strip from the first-level buffer pool after each strip transmission is completed, and to divide the strip into several patches. The feature extraction processing unit thread sends the map patch into the first-level inference engine, which is used to perform the initial screening stage of the two-stage detection model to obtain feature map and heat map. Additionally, the feature extraction processing unit thread extracts sparse feature sub-blocks of the foreground candidate region based on the heatmap, and writes the feature sub-blocks into the second-level circular feature buffer. The second-level inference consumption unit is used to accumulate feature sub-blocks from the second-level circular feature buffer by the second-level inference consumption unit thread. When the accumulated number reaches the predetermined batch size, the second-level inference engine will process a batch of the feature sub-blocks. The second-level inference engine will execute the target detection stage of the two-stage detection model to obtain the full image detection result. The image distribution thread, the feature extraction and processing unit thread, and the second-level inference and consumption unit thread are executed in parallel.

11. A large-format image target detection method based on two-stage detection, characterized in that, Includes the following steps: The image patch is input into the target detection model, which employs a two-stage detection process, including an initial screening stage and a target detection stage; the target detection model is deployed according to the method described in any one of claims 1-9; The object detection model performs initial screening of the map blocks using a first-level inference engine to obtain feature maps and heatmaps; then, it performs object detection on the feature sub-maps extracted from the heatmaps using a second-level inference engine to obtain detection boxes.

12. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the computer program is loaded into the processor, it implements the three-stage pipelined parallel inference method for large-format images based on two-stage detection as described in any one of claims 1-9, or the target detection method for large-format images based on two-stage detection as described in claim 11.

13. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the three-stage pipelined parallel inference method for large-format images based on two-stage detection as described in any one of claims 1-9, or the target detection method for large-format images based on two-stage detection as described in claim 11.

14. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the three-stage pipelined parallel inference method for large-format images based on two-stage detection as described in any one of claims 1-9, or the target detection method for large-format images based on two-stage detection as described in claim 11.

Images