A sparse infrared small target detection acceleration method based on local window

Abstract

This invention belongs to the field of infrared small target detection technology and discloses an accelerated method for sparse infrared small target detection based on local windows. Addressing the problem of excessive inference overhead caused by using high-resolution features for detection in infrared small target tasks, this invention first decouples the training and inference stages through network structure adaptation, ensuring normal iterative optimization during training while reducing redundant overhead during inference using local windows. Second, it constructs different inference paths based on fixed-size local window computation and full-size computation, deploying them as small and large inference engines respectively, solving the problem of difficulty in deploying algorithms with local window designs on the edge. Finally, it completes collaborative detection between the large and small inference engines based on asynchronous concurrent pipelines, achieving overall inference acceleration by performing fast detection on most common samples and error correction processing on a few complex samples.

Description

Technical Field

[0001] This invention belongs to the field of infrared small target detection technology, specifically relating to an accelerated method for sparse infrared small target detection based on local windows. Background Technology

[0002] Infrared small target detection technology is widely used in military and civilian fields, serving as a crucial component in precision guidance, early warning detection, and civilian security applications. With the development of computer vision, deep learning methods have been widely applied in this field due to their superior detection performance. However, the inherent characteristics of infrared small targets present challenges to detection tasks: affected by long-distance imaging, the target size is extremely small, lacking shape and texture features; simultaneously, the target signal is extremely faint, easily submerged in low signal-to-noise ratio background clutter. Therefore, the mainstream approach in this field is to perform detection using high-resolution low-level features, but this leads to a significant increase in detection overhead, failing to meet the real-time requirements of infrared small target detection applications.

[0003] On the other hand, infrared small target detection has significant task characteristics: the targets to be detected are small in size and their spatial distribution is often sparse. To leverage this characteristic and reduce redundant overhead, one approach is to use sparse convolution algorithms. However, this requires the introduction of a limited library of sparse convolution operators, placing specific dependencies and requirements on network structure, hardware platform, and software ecosystem. Another approach is to use a design similar to local windows to filter out invalid background computations, which can also reduce most of the redundant overhead. Currently, there are two main types of technical solutions based on this: The first type deploys the complete detection algorithm in a single inference engine. Its advantages are simple implementation and ease of unified scheduling within the inference system; its disadvantage is the generation of a dynamic computation graph containing dynamically shaped intermediate tensors or control flow involving conditional judgments, making it difficult for existing deep learning compilers to perform effective compilation optimizations. The second type decomposes the detection algorithm and deploys it across multiple inference engines for cascaded inference. Its advantage is that it avoids the aforementioned problems; its disadvantage is that it may involve repetitive forward inference computations or the cross-engine transfer of large-scale intermediate tensors, thereby introducing additional computational or memory access overhead. It is evident that the additional overhead introduced by existing technical solutions in actual deployment often offsets the acceleration benefits gained through local window design.

[0004] Therefore, how to provide a local window-based method for accelerating the detection of sparse infrared small targets, while preserving the detection gain brought by high-resolution features to the greatest extent, and solving the problem of poor acceleration performance of algorithms with local window design when deployed on edge platforms, is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] The purpose of this invention is to solve the technical problems of excessive inference overhead caused by the use of high-resolution features in infrared small target detection, and the poor inference acceleration effect caused by the introduction of additional overhead or the obstruction of compilation optimization when the existing local window algorithm is deployed on the edge.

[0006] The technical solution adopted in this invention is as follows:

[0007] This invention provides an accelerated method for sparse infrared small target detection based on local windows, comprising the following steps:

[0008] S1: Select a basic object detection algorithm and preprocess the original image according to the requirements of the algorithm, dividing it into a training set and a test set;

[0009] S2: Adapt and adjust the network structure of the basic target detection algorithm, including introducing a coarse-grained prediction branch, and complete model training based on the adjusted network structure.

[0010] S3: During the inference phase, using the trained model weights and the output of the coarse-grained prediction branch, a local window is constructed to build different inference paths based on full-size calculation and local window calculation, and the corresponding hyperparameters are determined.

[0011] S4: Extract the intermediate representations of the models for the different inference paths, and export them as the large inference engine responsible for error correction and the engine responsible for fast computation, respectively;

[0012] S5: Acquire the image to be detected, and implement collaborative detection between the large inference engine and the small inference engine based on the asynchronous concurrent pipeline, and output the final detection result.

[0013] In the above scheme, step S1 includes:

[0014] Step 1.1: A convolutional neural network object detection algorithm based on dense prediction; or an end-to-end object detection algorithm based on the Transformer architecture as the basic object detection algorithm;

[0015] Step 1.2: Preprocess the original image according to the requirements of the basic target detection algorithm, and divide it into the training set and the test set.

[0016] In the above scheme, step S2, which involves adapting and adjusting the network structure of the basic target detection algorithm, specifically includes:

[0017] If the basic target detection algorithm includes a feature fusion network with a feature pyramid structure, then in the bottom-up fusion path, the cross-scale fusion of the bottom high-resolution feature map and its upper layer feature map in the network structure is discarded, and only the intra-channel fusion of the bottom high-resolution feature map is retained.

[0018] The coarse-grained prediction branch is trained based on a one-to-one label matching strategy;

[0019] If the basic target detection algorithm is a convolutional neural network based on dense prediction, then an additional small target-specific prediction branch is introduced on the underlying high-resolution feature map;

[0020] If the basic object detection algorithm is an end-to-end object detection algorithm based on the Transformer architecture, then the prediction head in its query selection stage is reused as the coarse-grained prediction branch, and the candidate range of its query vector is limited to the underlying high-resolution feature map.

[0021] In the above scheme, step S3, the determination of hyperparameters and the construction of the local window specifically includes:

[0022] The hyperparameters for the inference phase are determined, including: fixed window size L, number of selected features n, and quality score threshold. Maximum tolerance score difference 1. Increase the spacing a;

[0023] Using the quality fraction threshold Difference from maximum tolerance score The prediction results output by the coarse-grained prediction branch are filtered to determine the effective prediction targets corresponding to the number of selected features, and the minimum bounding rectangle covering all effective prediction targets is calculated.

[0024] The minimum bounding rectangle is expanded by the expansion spacing a to obtain the estimated region; the position coordinates of a fixed-size window of size L are generated based on the position distribution of the estimated region.

[0025] In the above scheme, the horizontal coordinate index of the lower left corner endpoint of the fixed-size window Calculate using the following formula:

[0026]

[0027] in This is the horizontal coordinate index of the bottom left corner of the local window. The horizontal starting coordinate index of the estimated region, For the estimated lateral termination coordinate index of the region, It is calculated from the following characteristic function:

[0028]

[0029] in The width of the underlying high-resolution feature map, the Represents the characteristic function.

[0030] In the above scheme, step S3, the construction process of different inference paths based on full-size calculation and local window calculation specifically includes:

[0031] Local window computation inference path:

[0032] If the basic target detection algorithm includes a feature fusion network with a feature pyramid structure, then feature fusion calculation is performed only on the deep feature maps in the feature fusion network structure. After obtaining the local window, feature fusion calculation is then performed on the low-level high-resolution feature maps in the feature fusion network structure within the local window.

[0033] If the basic object detection algorithm is an end-to-end object detection algorithm based on the Transformer architecture, then the matrix multiplication operation related to the feature map size in the decoder stage is restricted to the local window, and the candidate range of the query vector in the query selection stage is also restricted to the local window.

[0034] When performing multi-scale deformable attention calculation in the decoder stage, the local window is used to reduce the input size of the feature vector to be sampled, and the sampling points are transformed by combining the size and position of the local window to complete the sampling.

[0035] The full-size computational inference path:

[0036] Inference computation is performed based on the full-size feature map, and all computational steps related to the coarse-grained prediction branch are discarded.

[0037] In the above scheme, in step S4, the large inference engine and the small inference engine are respectively:

[0038] The large inference engine is derived from the intermediate representation of the model that calculates the inference path in full size. It takes the image to be detected as input and outputs the category and coordinates of the detection box.

[0039] The small inference engine is derived from the intermediate representation of the model that calculates the inference path using the local window. It takes the image to be detected as input and outputs the detection box category and coordinates, as well as validity information.

[0040] The validity information is used to indicate whether the current local window can completely cover the estimated region containing all the targets to be detected, and is used to guide the collaborative reasoning between the large inference engine and the small inference engine in step S5.

[0041] In the above scheme, the asynchronous concurrent pipeline and the collaborative detection of the large inference engine and the small inference engine in step S5 specifically include:

[0042] Build a thread pool for multiple small inference engines to execute concurrently, the thread where the large inference engine resides, a task queue, an output queue, and a shared buffer;

[0043] The thread containing the small inference engine works concurrently, retrieves tasks from the task queue, unconditionally submits them to the small inference engine for inference, and writes the output inference results into the shared buffer, wherein the inference results include target detection results and validity information;

[0044] The thread containing the large inference engine retrieves the inference result from the shared buffer and determines whether the inference is valid based on the validity information: if valid, the target detection result is directly sent to the output queue; if invalid, the thread submits the task to the large inference engine again to perform inference and sends the corrected inference result to the output queue.

[0045] The beneficial effects of this invention are:

[0046] 1. To address the prerequisite of preserving the detection gain brought by high-resolution features to the greatest extent, this invention adapts and adjusts the network structure. On the one hand, it ensures the quality of key feature vectors for infrared small target detection; on the other hand, it decouples the detection algorithm from the training and inference phases, enabling it to reduce redundant overhead by utilizing local windows during the inference phase while maintaining tensor size alignment during the training phase, thus ensuring iterative optimization of the model.

[0047] 2. To address the issue that conventional solutions, which split detection algorithms and deploy them across multiple inference engines for cascaded inference, often introduce significant additional computational or memory overhead, this invention constructs different inference paths during the inference phase. These paths are deployed as large and small inference engines, respectively, and their collaborative detection is achieved through an asynchronous concurrent pipeline. Under this design, most common image samples are detected quickly using the small inference engine, while a few complex image samples are handled by the large inference engine for error correction and compensation. This approach accelerates the overall inference process while ensuring detection accuracy.

[0048] 3. To address the issue in conventional solutions where the entire detection algorithm is deployed in a single inference engine, the dynamic computation graph involved can hinder deep learning compiler optimization. This invention employs a fixed-size local window design to avoid additional communication overhead caused by dynamic intermediate tensors. Furthermore, by constructing independent inference paths, it circumvents the low optimization efficiency of deep learning compilers for algorithms with control flow. Moreover, the local window construction process generates the required local windows with low computational cost. Through these designs, the computational efficiency of the small inference engine responsible for fast detection is ensured.

[0049] 4. Synergistic effect:

[0050] This invention achieves the following novel technical effects by deeply coupling "network adaptation" (feature A) and "multi-engine asynchronous inference" (feature B) through "decoupling different inference paths" (feature C):

[0051] If only feature A is available (network adaptation: introducing coarse-grained branches, etc.): While this allows the algorithm to maintain detection accuracy and construct local windows using coarse-grained predictions, thereby reducing redundant computation, deploying the algorithm to a single inference engine still faces challenges such as compiler optimization failures due to the "dynamically shaped intermediate tensors" of the dynamic computation graph or "control flow involving conditional judgments," as well as frequent CPU-GPU communication overhead. (The theoretical speedup benefits cannot be obtained).

[0052] If only feature B (multi-engine asynchronous inference) is available: If features A and C are not combined, when deploying the algorithm to multiple inference engines, although the deployment difficulties caused by the dynamic computation graph can be avoided, there will still be additional computational overhead caused by "repeated forward inference", or additional memory access overhead caused by "transferring large-scale tensors across engines" (reintroducing other additional overhead);

[0053] The combined effect is as follows: The "network adaptation" in Feature A and the "decoupling of different inference paths" in Feature C work together to enable a single inference engine in Feature B to reduce redundant overhead based on a fixed-size local window. This eliminates the problem of low single-engine inference efficiency caused by "dynamically shaped intermediate tensors" or "control flow involving conditional judgments." Simultaneously, different inference engines in Feature B only need to transfer a small amount of detection results to achieve collaborative inference, avoiding the additional overhead caused by "repeated forward inference" or "transferring large-scale tensors across engines" when multiple engines work asynchronously. Furthermore, it can perform error correction inference on samples not covered by the local window, ensuring the correctness of the inference results. In addition, the combination of Features A and C ensures that the model maintains baseline performance during the training phase while decoupling the inference and training phases, minimizing the accuracy loss introduced by this decoupling.

[0054] The two work together to completely break down the final barrier between "accelerating algorithm prototypes" and "accelerating edge deployments" while preserving the accuracy gains brought by high-resolution features. Attached Figure Description

[0055] Figure 1 This is a flowchart of a sparse infrared small target detection acceleration method based on local windows according to the present invention.

[0056] Figure 2 This is a schematic diagram illustrating the collaborative detection between the large inference engine and the small inference engine in an embodiment of the present invention.

[0057] Figure 3 This is a schematic diagram of the asynchronous concurrent pipeline described in an embodiment of the present invention. Detailed Implementation

[0058] The method of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0059] Example 1

[0060] This embodiment describes the specific application of the method of the present invention in an end-to-end target detection algorithm based on the Transformer architecture. As a specific implementation, this embodiment uses the RT-DETR detection algorithm, a representative Transformer-based end-to-end real-time target detection method, which can be used as an example implementation of the method of the present invention. This embodiment is based on the NUAA-SIRST infrared small target public dataset, which covers real infrared target images against complex backgrounds.

[0061] like Figure 1 The diagram shows a flowchart of a sparse infrared small target detection acceleration method based on a local window according to an embodiment of the present invention. The method specifically includes the following steps:

[0062] S1. Select a basic target detection algorithm and preprocess the original image according to the requirements of the algorithm, dividing it into a training set and a test set;

[0063] In this embodiment, an end-to-end object detection algorithm based on the Transformer architecture is selected as the basic object detection algorithm, specifically RT-DETR. The complete network structure of this algorithm mainly includes: a backbone network, an encoder, a decoder, and a prediction head.

[0064] In the infrared small target detection task, high-resolution features are required for detection. The RT-DETR algorithm in this embodiment uses a configuration that meets this requirement. Specifically, the backbone network is configured to output four layers of multi-scale feature maps, including one low-level high-resolution feature map with a size of 160×160 and rich in detail, and three higher-level feature maps with sizes of 20×20, 40×40, and 80×80 respectively and rich in semantic information. The encoder includes a feature fusion network with a feature pyramid structure. The feature fusion network adopts a structure similar to a path aggregation network and includes top-down and bottom-up feature fusion processes to perform cross-scale feature fusion between the four layers of feature maps. The decoder includes a query selection stage and a multi-scale deformable attention calculation stage, wherein the multi-scale deformable attention calculation stage is extended to perform sampling on all four layers of feature maps. The prediction head is responsible for outputting the final detection box and category.

[0065] Furthermore, for dataset partitioning and preprocessing, the original infrared target image data in the dataset to be processed is first divided into training set and test set in a 7:3 ratio, and the annotation information is uniformly converted into the standard COCO format; then preprocessing is performed, the original infrared images are uniformly scaled to 640×640 size, the image data is normalized, and data augmentation operations such as random cropping, random horizontal flipping, and random optical distortion are performed.

[0066] S2. Adapt and adjust the network structure of the basic target detection algorithm, including introducing a coarse-grained prediction branch, and complete model training based on the adjusted network structure.

[0067] In this embodiment, for the feature fusion network, top-down feature fusion between all feature maps is retained.

[0068] Furthermore, bottom-up cross-scale feature fusion between higher-level feature maps is preserved. Specifically, lower-level feature maps are downsampled through a 3×3 convolutional layer with a stride of 2, then concatenated with adjacent upper-level feature maps via channel concatenation, and finally input into a cross-stage local network for computation.

[0069] Furthermore, for the low-level high-resolution feature map, its cross-scale feature fusion with the upper-level feature map is discarded. Specifically, the low-level high-resolution feature map is directly output to subsequent stages after being fused intra-channel by a 1×1 channel-wise convolution.

[0070] The adjustment of the network topology facilitates the construction of inference paths based on local window computation in subsequent steps. Through this adjustment, the most crucial low-level high-resolution features for small object detection are still fully fused, thus incurring only a small actual accuracy cost.

[0071] In this embodiment, the candidate range of query vectors in the query selection stage is limited to the underlying high-resolution feature vectors. Specifically, the query selection stage includes a quality score prediction head and a reference point prediction head. First, the serialized underlying high-resolution feature vectors are input into the quality score prediction head. Then, the top 300 feature vectors are selected as query vectors based on their output quality scores in descending order. Finally, the query vectors are input into the reference point prediction head to calculate the reference point coordinates required for subsequent attention calculations.

[0072] The quality score prediction head is configured as a 256×1 linear mapping layer to output the quality score of the features. The reference point prediction head is configured as a multilayer perceptron network, including two 256-dimensional hidden layers with ReLU activation functions, and a linear output layer with an output dimension of 4 to generate reference point coordinates.

[0073] Through the above implementation method, the query vector input to subsequent stages contains richer underlying detailed information.

[0074] In this embodiment, a coarse-grained prediction branch for generating local windows is introduced. This coarse-grained prediction branch includes a classification head and a regression head. Here, the quality score prediction head and reference point prediction head from the aforementioned query selection stage are reused to construct the coarse-grained prediction branch, and the loss corresponding to this branch is added to the total loss function. The loss function of the coarse-grained prediction branch is composed of a weighted average of Focal Loss, L1 Loss, and IoU Loss, with preferred weight coefficients of 1.0, 5.0, and 2.0, and is trained based on a one-to-one label matching strategy.

[0075] The above implementation avoids the additional computational overhead caused by introducing a dedicated prediction branch and avoids the constraint of non-maximum suppression post-processing on the subsequent rapid construction process of local windows.

[0076] S3. In the inference phase, using the trained model weights, a local window is constructed using the output of the coarse-grained prediction branch. Different inference paths based on full-size computation and local window computation are constructed. The inference path based on full-size computation is configured as the large inference engine, and the inference path based on local window computation is configured as the small inference engine.

[0077] In this embodiment, the inference phase constructs one inference path based on full-size computation and one inference path based on local window computation. It should be noted that the inference path based on local window computation supports configuration for multiple different size levels; this embodiment uses a single size level local window as an example. The two inference paths share the model weights trained in step S2. Furthermore, for the inference path based on local window computation, additional hyperparameters are needed to construct the local window. In this embodiment, the preferred values ​​for each parameter are as follows: fixed window size L is 60, the number of features selected n is 20, and the quality score threshold... The maximum tolerable score difference is -1.8. The value is 1.3, and the expansion spacing a is 0.15.

[0078] In this embodiment, for the inference path based on local window computation, when the model forward inferences to the feature fusion network in the encoder, firstly, a complete top-down feature fusion is performed only on the higher-level feature maps output by the backbone network, and a linear mapping is performed on the 80×80 feature map output by this stage to align the channel dimension features. Subsequently, a complete bottom-up feature fusion is performed on the aforementioned higher-level feature maps, and the output 20×20 and 40×40 feature maps are fused intra-channel separately through channel-wise convolution. Finally, the fused higher-level feature maps are combined with the lower-level high-resolution features that have not undergone cross-scale feature fusion processing. Figure 1 And output it to subsequent stages.

[0079] Furthermore, when the model forward inferences to the query selection stage of the decoder, the four-layer feature map enhanced by the encoder is serialized into a one-dimensional feature sequence and fed into the coarse-grained prediction branch. Based on the output of this branch, a local window of fixed size L×L is constructed, and then the corresponding feature map is truncated based on this local window, restricting the compensated feature fusion calculation to within the local window.

[0080] The acquisition of the local window is specifically implemented by first constructing an estimated region based on coarse-grained prediction, and then placing a local window of a preset fixed size according to the position of the estimated region.

[0081] The estimated region is constructed using the following steps:

[0082] A1. Input the feature vector sequence corresponding to the higher-level feature map into the coarse-grained prediction branch, sort the quality scores output by the classification head in descending order, and extract the first n corresponding feature vectors. Then input the extracted feature vectors into the regression head to obtain the coordinates of each coarse prediction detection box.

[0083] A2. For all coarsely predicted bounding boxes, remove those with a quality score below the quality score threshold. The detection boxes are defined, and placeholder processing is performed on the rejected detection boxes;

[0084] A3. Examine a descending sequence of quality scores. If the difference between adjacent quality scores is greater than the maximum tolerance score difference... If so, all coarse prediction detection boxes after that position will be removed, and placeholder processing will be performed;

[0085] A4. Execute the initial screening logic based on absolute threshold: remove all coarse prediction detection boxes with quality scores lower than the preset quality score threshold, and perform placeholder processing on the corresponding prediction boxes that are removed.

[0086] A5. Calculate the minimum bounding rectangle that covers all remaining valid target coarse prediction detection boxes, and expand it around the perimeter by an expansion spacing a.

[0087] A6. Perform boundary truncation on the expanded rectangle to ensure that it does not exceed the original feature map boundary, and round the coordinates of the truncated rectangle to integer multiples of 2 to obtain the final estimated region.

[0088] After determining the estimated region, place a fixed-size local window with a preset size of L×L according to the following formula:

[0089]

[0090] in This is the horizontal coordinate index of the bottom left corner of the local window. The horizontal starting coordinate index of the estimated region, The horizontal termination coordinate index of the estimated region, where L is the fixed window size. It is calculated from the following characteristic function:

[0091]

[0092] in The width of the underlying high-resolution feature map is set to 160. The vertical coordinate index of the lower left corner endpoint of the local window is calculated using the same logic described above, and will not be repeated here. This completes the construction of the local window.

[0093] The extraction of the local feature map is specifically implemented as follows: First, based on the size and position of the local window, the one-dimensional index corresponding to the target region is pre-calculated. Then, from the 160×160-size low-level high-resolution feature map output by the encoder and its upper-level 80×80-size feature map, the aggregation operator is used to extract the feature vector sequences of the corresponding L×L and L / 2×L / 2-size local window feature maps, respectively. Finally, these are rearranged into a two-dimensional feature map. Since shape tensors with data dependencies are not deployment-friendly, slicing operations are avoided here to obtain the corresponding local feature maps directly.

[0094] For the compensation feature fusion calculation, the specific implementation is as follows: the L / 2×L / 2 size feature map truncated by the local window is upsampled, and then channel-wise concatenated with the L×L size feature map, and then fed into the cross-stage local network for cross-scale fusion; finally, the complete 80×80 size higher layer feature map and the L×L size lower layer high-resolution feature map after feature fusion are respectively processed by channel-wise convolution to perform intra-channel fusion.

[0095] Through the above implementation method, a compact fixed-size local window is obtained with low computational cost, and a large number of redundant convolution operations are filtered out using this local window, thereby significantly reducing the computational overhead of the inference stage.

[0096] In this embodiment, when the model forward inferences to the query selection stage of the decoder, the candidate range of the query vector and related calculations are restricted to the local window. The specific implementation is similar to the corresponding operation in step S2, but the computationally intensive matrix multiplication operations in the query selection stage are all restricted to the local window, including operations such as quality score prediction and reference point coordinate calculation.

[0097] Furthermore, during the cross-attention stage of forward inference to the decoder, the input size of the sampled value feature vector is reduced based on the local window, and the projection mapping matrix multiplication operation of the value vector is restricted to the reduced value feature vector. The projection mapping is configured as a linear mapping layer with a size of 256×256. Subsequently, during multi-scale deformable attention calculation, the grid sampling space coordinates involved in the attention are transformed according to the local window parameters to ensure alignment between the training and inference phases. For corner-aligned grid sampling configurations, the transformation of the lateral sampling coordinates is performed according to the following formula:

[0098]

[0099] in These are the normalized sampled coordinates after coordinate transformation. These are the normalized sampled coordinates obtained from the linear layer input by the query vector; the meanings of the other symbols are the same as before. The transformation method for the vertical sampled coordinates is logically similar and will not be repeated here.

[0100] The above implementation method limits computationally intensive large-scale matrix multiplication operations in the model, further reduces the computational overhead in the inference stage, and reduces a large amount of redundant memory access overhead in the multi-scale deformable attention computation process.

[0101] In this embodiment, for the inference path based on full-size computation, it performs a complete forward inference computation based on full-size feature maps of 20×20, 40×40, 80×80 and 160×160, and removes all computation steps related to the coarse-grained prediction branch.

[0102] S4. Extract the intermediate representations of the models for the different inference paths, and export them as the large inference engine responsible for error correction and the engine responsible for fast computation, respectively.

[0103] In this embodiment, the target deployment platform is an NVIDIA RTX 4090 GPU, and the TensorRT inference framework is selected for edge deployment.

[0104] In this embodiment, either quantization-aware training or post-training quantization can be selected to perform model quantization. If QAT quantization is used, INT8 data precision is applied to convolution operators and matrix multiplication operators in the model, while FP16 data precision is applied to multi-scale deformable attention modules and multi-head attention modules. After quantization, the precision loss is detected layer by layer using the precision analysis tool Polygraphy; if the precision loss of a certain layer is large, the quantization configuration of that layer is adjusted, and the above process is repeated until the overall precision loss is within an acceptable range.

[0105] In this embodiment, the ONNX format static computation graphs based on full-size computation inference paths and local window computation inference paths are obtained using the tracing method, and then converted into TensorRT inference engines with a batch size of 1.

[0106] The TensorRT inference engine corresponding to the static computation graph based on the full-size computational inference path is a large inference engine, with an input of a 640×640×3 image to be detected and an output of 300×5 bounding box categories and coordinates. The TensorRT inference engine corresponding to the static computation graph based on a local window is a small inference engine, with the same input specifications as the large inference engine, and an output of 300×5 bounding box categories and coordinates, as well as a Boolean flag. The Boolean flag corresponds to the validity information of the inference result; if the preset fixed-size local window can completely cover the estimated area during the forward inference process, the flag is set to 1; otherwise, it is set to 0.

[0107] S5. Acquire the image to be detected, and based on the asynchronous concurrent pipeline, implement collaborative detection between the large inference engine and the small inference engine, and output the final detection result:

[0108] In this embodiment, the asynchronous concurrent pipeline includes: a thread pool for multiple small inference engines to execute concurrently, a thread containing the large inference engine, a task queue, an output queue, and a shared buffer.

[0109] In this implementation, a thread pool allocates one thread to each small inference engine (4 threads in this example). Multiple `IExecutionContext` contexts are created based on the same small engine, and each thread is bound to a CUDA stream and an `IExecutionContext` context. The large inference engine is handled by its own dedicated thread. The task queue is a thread-safe queue storing task indices and corresponding physical addresses of image data. The output queue is also a thread-safe queue storing task sequence numbers and detection results. The shared buffer is a lock-free circular buffer implemented using paged memory allocated by `cudaMallocHost`, ensuring thread safety through atomic operations. Internally, it stores task indices, detection results, and flags.

[0110] In this example, for the collaborative detection of the large inference engine and the small inference engine, such as Figure 2 As shown, the specific implementation is as follows: the thread containing the small inference engine performs multi-threaded inference, concurrently retrieving tasks from the task queue. It binds the input address of the corresponding IExecutionContext context to the physical address of the image data retrieved from the queue to achieve zero-copy. Then, the inference task is enqueued into the CUDA stream bound to this thread for execution. The output of the small inference engine is written to the shared buffer along with the task index. The thread containing the large inference engine polls and retrieves inference results from the shared buffer. If the flag bit is 1, the other part of the detection results is directly sent to the output queue; if it is 0, the corresponding task is resubmitted to the large inference engine for error correction inference, and the output is then sent to the output queue after completion.

[0111] In this embodiment, there are also related peripheral supporting components, such as Figure 3 As shown, this includes: an image data memory pool for storing image data, which will be retained until the small inference engine completes effective inference or the large inference engine completes error correction inference; a main thread responsible for image data handling, memory pool-related operations, and summarizing detection results based on task indexes; and one or more preprocessing threads and a corresponding memory pool, which serve as an independent pipeline for performing image scaling and data normalization.

[0112] Example 2

[0113] This embodiment describes the specific application of the method of the present invention in a dense prediction-based convolutional neural network object detection algorithm. As a specific implementation, this embodiment uses the YOLOv5 detection algorithm for illustration. This algorithm is a representative dense prediction-based convolutional neural network object detection algorithm and can be used as an example implementation of the method of the present invention.

[0114] This embodiment is based on the IRSTD-1k small target public dataset, which covers real infrared target images in various scenarios.

[0115] like Figure 1 The diagram shows a flowchart of a sparse infrared small target detection acceleration method based on a local window according to an embodiment of the present invention. The method specifically includes the following steps:

[0116] S1. Select a basic target detection algorithm and preprocess the original image according to the requirements of the algorithm, dividing it into a training set and a test set;

[0117] In this embodiment, a convolutional neural network-based object detection algorithm based on dense prediction is selected as the basic object detection algorithm, specifically YOLOv5. The complete network structure of this algorithm mainly includes: a backbone network, a neck network, and a prediction head.

[0118] In the infrared small target detection task, high-resolution features are required for detection. The YOLOv5 algorithm in this embodiment uses a configuration that meets this requirement. Specifically, the backbone network is configured to output four layers of multi-scale feature maps: a P2 feature map of size 160×160 pixels, a P3 feature map of size 80×80 pixels, a P4 feature map of size 40×40 pixels, and a P5 feature map of size 20×20 pixels. The neck network includes a feature fusion network with a feature pyramid structure. The feature fusion network adopts a path aggregation network structure and includes top-down and bottom-up feature fusion processes to perform cross-scale feature fusion between the four feature maps. The prediction head is responsible for outputting the final detection box and category.

[0119] Furthermore, for dataset partitioning and preprocessing, the original infrared target image data in the dataset to be processed is first divided into training and testing sets in a 7:3 ratio, and the annotation information is uniformly converted into the standard COCO format. Then, preprocessing is performed. During training, Mosaic data augmentation is used, and the minimum black border is adaptively added using the Letterbox algorithm to uniformly scale the images to 640×640 size. At the same time, pixel values ​​are normalized, and various combined data augmentation operations such as random geometric transformation and color space perturbation are performed.

[0120] S2. Adapt and adjust the network structure of the basic target detection algorithm, including introducing a coarse-grained prediction branch, and complete model training based on the adjusted network structure.

[0121] In this embodiment, for the feature fusion network, the top-down feature fusion between the P5 and P2 feature maps is retained.

[0122] Furthermore, bottom-up cross-scale feature fusion between higher-level feature maps is preserved. Specifically, the lower-level feature map is passed through a 3×3 convolutional layer with a stride of 2, where the channel dimensions are adjusted while downsampling. It is then concatenated with the adjacent upper-level feature map and input into the C3 module for computation.

[0123] Furthermore, for the low-level high-resolution feature map, its cross-scale feature fusion with the upper-level feature map is discarded. Specifically, the low-level high-resolution feature map is directly output to subsequent stages after top-down feature fusion.

[0124] In this embodiment, a coarse-grained prediction branch for generating local windows is introduced, comprising a classification head and a regression head. The classification head is configured as a convolutional neural network, including a depthwise separable feature layer consisting of one 3×3 deep convolutional layer and one 1×1 pointwise convolutional layer, both layers having SiLU activation and batch normalization, and a final output layer with a 3×3 kernel and a feature dimension of 1, used to predict feature quality scores. The regression head is configured as a convolutional neural network, including a feature extraction layer with a feature dimension of 256, a 3×3 kernel, and SiLU activation and batch normalization, and a linear channel mapping layer with a 1×1 kernel and a feature dimension of 4, used to generate coarsely predicted bounding box coordinates. The loss corresponding to this branch is added to the total loss function. The loss function of the coarse-grained prediction branch is composed of a weighted average of BCE Loss and CIoU Loss. The weights are dynamically adjusted according to the progressive loss balancing strategy of YOLOv26, and training is performed based on the YOLOv26 dual-head collaborative label matching strategy. This strategy introduces one-to-many label matching during training, but only retains one-to-one branch heads during inference, thus it is also an optional implementation method of this approach.

[0125] Through the above implementation method, a coarse-grained prediction branch that does not require non-maximum suppression post-processing is introduced into the convolutional neural network target detection algorithm based on dense prediction.

[0126] In this embodiment, a small target-specific prediction branch is introduced into the prediction head. The small target-specific prediction branch performs prediction based on the P2 feature map, while the configuration of the remaining network layers and loss function reuses the configuration of the YOLOv26 algorithm described above.

[0127] S3. In the inference phase, the trained model weights are used to construct different inference paths based on full-size computation and local window computation, and the corresponding hyperparameters are determined.

[0128] In this embodiment, the inference phase constructs one inference path based on full-size computation and one inference path based on local window computation. These two inference paths share the model weights trained in step S2. Furthermore, for the inference path based on local window computation, additional hyperparameters are introduced to construct the local window. In this embodiment, the preferred values ​​for each parameter are as follows: fixed window size L is 66, the number of features selected n is 20, and the quality score threshold... The maximum tolerance score difference is -2.3. The value is 1.5, and the expansion spacing a is 0.13.

[0129] In this embodiment, for the inference path based on local window computation, when the model forward inferences to the feature fusion network of the neck network, firstly, a complete top-down feature fusion is performed only on the P5 to P3 feature maps output by the backbone network, and a separate linear mapping is performed on the P3 feature map output by this stage to align the channel dimension features. Subsequently, a complete bottom-up feature fusion is performed on the P3 to P5 feature maps. Finally, the fused P3 to P5 feature maps are combined with the P2 feature maps that have not undergone cross-scale feature fusion processing. Figure 1 And output it to subsequent stages.

[0130] Furthermore, the P3 feature map output from the neck network is fed into the coarse-grained prediction branch. Based on the output of this branch, a local window of fixed size L×L is constructed. Then, the corresponding feature map is extracted based on this local window, and the compensation feature fusion calculation is restricted to the local window.

[0131] The specific implementation method for obtaining the local window is the same as in Embodiment 1, and will not be repeated here.

[0132] The specific implementation method for extracting the local feature map is the same as in Embodiment 1, and will not be repeated here.

[0133] For the feature fusion calculation of compensation, the specific implementation is as follows: nearest neighbor upsampling is performed on the L / 2×L / 2 size feature map truncated by the local window, and then channel concatenation is performed with the L×L size feature map, which is then sent to the C3 module for calculation. Subsequently, using this as input, the forward inference of the small target-specific branch in the prediction head is advanced to the current stage, and the calculation is terminated, discarding the prediction results of other branches.

[0134] In this embodiment, for the inference path based on full-size computation, inference is performed using the full-size P2 to P5 feature maps, and computations related to coarse-grained prediction branches are removed, while only forward computations of small target-specific branches are retained in the prediction head.

[0135] S4. Extract the intermediate representations of the models for the different inference paths, and after mixed precision quantization, export them as a large inference engine and a small inference engine responsible for error correction and fast computation, respectively.

[0136] The specific implementation method is similar to that described in Example 1, so it will not be repeated here.

[0137] S5. Acquire the image to be detected, and based on the asynchronous concurrent pipeline, implement collaborative detection between the large inference engine and the small inference engine, and output the final detection result:

[0138] The specific implementation method is similar to that described in Example 1, so it will not be repeated here.

[0139] In summary, addressing the challenges of infrared small target detection characterized by small target size and sparse spatial distribution, this invention decouples the training and inference phases by adapting the network structure of a given detection algorithm. This allows the model to maintain tensor alignment for iterative optimization during training, while simultaneously reducing redundant overhead during inference using local windows. By constructing different inference paths based on fixed-size local window computation and full-size computation, the invention overcomes the problem of existing deployment techniques for algorithms with local window designs, which easily introduce additional inference overhead and hinder optimization by deep learning compilers. Asynchronous concurrent pipelines enable the collaborative operation of large and small inference engines to perform fast detection on most common samples and error correction on a few complex samples, thereby accelerating inference overall. This invention significantly reduces redundant computation and memory access overhead during inference while preserving the accuracy gains from high-resolution features, achieving algorithmic acceleration for a given detection algorithm and making it suitable for real-time infrared small target detection scenarios with similar task characteristics.

[0140] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of this invention.

Claims

1. A method for accelerating the detection of sparse infrared small targets based on local windows, characterized in that, Includes the following steps: Step S1: Select a basic object detection algorithm and preprocess the original image according to the requirements of the algorithm, dividing it into a training set and a test set; Step S2: Adapt and adjust the network structure of the basic target detection algorithm, including introducing a coarse-grained prediction branch, and complete model training based on the adjusted network structure; Step S3: In the inference phase, using the trained model weights, construct a local window using the output of the coarse-grained prediction branch, construct different inference paths based on full-size calculation and local window calculation, and determine the corresponding hyperparameters; Step S4: Extract the intermediate representations of the models for the different inference paths, and after mixed precision quantization, export them as the large inference engine responsible for error correction and the small inference engine responsible for fast computation, respectively. Step S5: Acquire the image to be detected, and implement collaborative detection between the large inference engine and the small inference engine based on the asynchronous concurrent pipeline, and output the final detection result.

2. The accelerated detection method for sparse infrared small targets based on local windows according to claim 1, characterized in that, Step S1 includes: Step 1.1: A convolutional neural network object detection algorithm based on dense prediction; or an end-to-end object detection algorithm based on the Transformer architecture as the basic object detection algorithm; Step 1.2: Preprocess the original image according to the requirements of the basic target detection algorithm, and divide it into the training set and the test set.

3. The accelerated detection method for sparse infrared small targets based on local windows according to claim 1, characterized in that, The adaptation and adjustment of the network structure of the basic target detection algorithm in step S2 specifically includes: The basic target detection algorithm includes a feature fusion network with a feature pyramid structure. In the bottom-up fusion path, the cross-scale fusion of the bottom high-resolution feature map and the feature map of the previous layer in the network structure is discarded, and only the intra-channel fusion of the bottom high-resolution feature map is retained. The coarse-grained prediction branch is trained based on a one-to-one label matching strategy.

4. The accelerated detection method for sparse infrared small targets based on local windows according to claim 1, characterized in that, The adaptation and adjustment of the network structure of the basic target detection algorithm in step S2 specifically includes: The basic target detection algorithm is a convolutional neural network based on dense prediction, and an additional small target-specific prediction branch is introduced on the underlying high-resolution feature map. If the basic object detection algorithm is an end-to-end object detection algorithm based on the Transformer architecture, then the prediction head in its query selection stage is reused as the coarse-grained prediction branch, and the candidate range of its query vector is limited to the underlying high-resolution feature map. The coarse-grained prediction branch is trained based on a one-to-one label matching strategy.

5. The accelerated detection method for sparse infrared small targets based on local windows according to claim 1, characterized in that, In step S3, the determination of hyperparameters and the construction of the local window specifically include: The hyperparameters for the inference phase are determined, including: fixed window size L, number of selected features n, and quality score threshold. Maximum tolerance score difference 1. Increase the spacing a; Using the quality fraction threshold Difference from maximum tolerance score The prediction results output by the coarse-grained prediction branch are filtered to determine the effective prediction targets corresponding to the number of selected features, and the minimum bounding rectangle covering all effective prediction targets is calculated. The minimum bounding rectangle is expanded by the expansion spacing a to obtain the estimated region; the position coordinates of a fixed-size window of size L are generated based on the position distribution of the estimated region.

6. The accelerated detection method for sparse infrared small targets based on local windows according to claim 5, characterized in that, The horizontal coordinate index of the lower left corner of the fixed-size window Calculate using the following formula: in This is the horizontal coordinate index of the bottom left corner of the local window. The horizontal starting coordinate index of the estimated region, For the estimated lateral termination coordinate index of the region, It is calculated from the following characteristic function: in The width of the underlying high-resolution feature map, the Represents the characteristic function.

7. The accelerated detection method for sparse infrared small targets based on local windows according to claim 1, characterized in that, In step S3, the construction process of different inference paths based on full-size calculation and local window calculation specifically includes: Local window computation inference path: The basic target detection algorithm includes a feature fusion network with a feature pyramid structure. Feature fusion calculation is performed only on the deep feature maps in the feature fusion network structure. After obtaining the local window, feature fusion calculation is performed on the low-level high-resolution feature maps in the feature fusion network structure within the local window. The full-size computational inference path: Inference computation is performed based on the full-size feature map, and all computational steps related to the coarse-grained prediction branch are discarded.

8. The accelerated detection method for sparse infrared small targets based on local windows according to claim 1, characterized in that, In step S3, the construction process of different inference paths based on full-size calculation and local window calculation specifically includes: Local window computation inference path: The basic target detection algorithm is an end-to-end target detection algorithm based on the Transformer architecture. Therefore, the matrix multiplication operation related to the feature map size in the decoder stage is restricted to the local window, and the candidate range of the query vector in the query selection stage is also restricted to the local window. When performing multi-scale deformable attention calculation in the decoder stage, the local window is used to reduce the input size of the feature vector to be sampled, and the sampling points are transformed by combining the size and position of the local window to complete the sampling. The full-size computational inference path: Inference computation is performed based on the full-size feature map, and all computational steps related to the coarse-grained prediction branch are discarded.

9. The accelerated detection method for sparse infrared small targets based on local windows according to claim 1, characterized in that, In step S4, the large inference engine and the small inference engine are respectively: The large inference engine is derived from the intermediate representation of the model that calculates the inference path in full size. It takes the image to be detected as input and outputs the category and coordinates of the detection box. The small inference engine is derived from the intermediate representation of the model that calculates the inference path using the local window. It takes the image to be detected as input and outputs the detection box category and coordinates, as well as validity information. The validity information is used to indicate whether the current local window can completely cover the estimated region containing all the targets to be detected, and is used to guide the collaborative reasoning between the large inference engine and the small inference engine in step S5.

10. The accelerated detection method for sparse infrared small targets based on local windows according to claim 1, characterized in that, The asynchronous concurrent pipeline and the collaborative detection of the large inference engine and the small inference engine in step S5 specifically include: Build a thread pool for multiple small inference engines to execute concurrently, the thread where the large inference engine resides, a task queue, an output queue, and a shared buffer; The thread containing the small inference engine works concurrently, retrieves tasks from the task queue, unconditionally submits them to the small inference engine for inference, and writes the output inference results into the shared buffer, wherein the inference results include target detection results and validity information; The thread containing the large inference engine retrieves the inference result from the shared buffer and determines whether the inference is valid based on the validity information: if valid, the target detection result is directly sent to the output queue; if invalid, the thread submits the task to the large inference engine again to perform inference and sends the corrected inference result to the output queue.

Images