An Infrared Small Target Detection Method Based on Multi-Scale Self-Attention Motion Background Modeling

Abstract

The present invention discloses a method for detecting small infrared targets based on multi-scale self-attention motion background modeling. Multiple consecutive frames are selected from infrared sequence images as model input, image features are extracted using a backbone network, and the network's multi-scale global features are fully exploited through a cross-scale self-attention encoding structure to model the global background of the time-series image. The shallowest features containing the finest granularity are grouped and weighted to form a grouped query, and a cross-scale iterative decoding method is used to form the final query, thereby retaining more target details and more effectively suppressing the motion background. A confidence map is generated through a detection head, and a threshold segmentation method is used to obtain the final detection result. The multi-scale self-attention encoding network can achieve the fusion of the global background features of the sequence image and the local features of the target. The multi-scale self-attention decoding network can enhance the multi-frame detail information of the query vector during the decoding process, improving the ability to model the motion background and express target features.

Description

Technical Field

[0001] The present invention belongs to the technical field of target detection and recognition, and relates to a method for detecting small infrared moving targets, and in particular to a method for detecting small infrared targets based on multi-scale self-attention moving background modeling. Background Art

[0002] In recent years, infrared imaging systems have been widely used in fields such as disaster prevention and control, weather forecasting, and target detection, with target detection being a core research area. In many applications, the detection platform is often far away from the target, resulting in a lack of target structure and texture in infrared images. Furthermore, the imaging scene is complex, and the target is submerged in background clutter, resulting in weak and small characteristics. This poses significant challenges to target detection. Detection methods based on single-frame images only exploit the local saliency of the target relative to the background, making them prone to missed detections and false positives. Moving targets in infrared scenes, such as ships, drones, and vehicles, are the primary targets for detection. Therefore, leveraging target motion characteristics in target detection algorithms is crucial, offering superior performance and application results compared to single-frame image-based infrared small target detection. Numerous studies have been conducted on infrared small moving target detection algorithms, including traditional methods based on prior models and data-driven deep learning approaches. In actual application scenarios, as the target approaches or moves away from the detection platform, the target's image plane motion speed, imaging scale and energy are always changing. Therefore, traditional methods based on prior assumptions cannot adapt well to task requirements. Deep learning methods can mine the high-level semantic feature differences between target and background information from sequential image data, thereby achieving accurate target positioning.

[0003] Currently, target detection in infrared image sequences still faces several challenges that need to be addressed: 1) The complex random motion of the detection platform causes irregular background changes, making it difficult to extract temporally differentiated features between the target and background. 2) When preprocessing the input images and performing image registration using the SIFT method, the scene structure details in infrared images are not as rich as those in visible light images, resulting in significant errors in traditional registration methods. Summary of the Invention

[0004] Aiming at the problem that it is difficult to extract target energy and motion information when the energy of weak targets is close to the spatial clutter fluctuations in infrared sequence image target detection tasks, the present invention provides an infrared small target detection method based on multi-scale self-attention motion background modeling.

[0005] The purpose of the present invention is achieved through the following technical solutions:

[0006] A method for detecting small infrared targets based on multi-scale self-attention motion background modeling includes the following steps:

[0007] Step 1: For the input multi-frame infrared sequence image, it is divided into patches according to the specified scale and embedding mapping is performed. Then, Video SwinTransformer is used as the backbone to extract spatiotemporal features and output four levels of features;

[0008] Step 2: Construct a multi-scale self-attention encoding network consisting of three parts: an intar-scale encoder, a cross-scale encoder, and a feature pyramid network (FPN) to extract global multi-scale features of the input sequence image. The intar-scale encoder calculates the global attention weight for a single scale and generates global features. The cross-scale encoder uses the features of adjacent scales to calculate the attention affinity from coarse to fine to measure the feature correlation between scales. After the top-level features of the FPN are modulated by the spatiotemporal attention weights, they are transferred from top to bottom to features at all levels and finally output to the multi-scale self-attention decoding network.

[0009] Step 3: Construct a multi-scale self-attention decoding network consisting of three parts: query filtering, query iteration, and feature solution. Query filtering uses input features to weight randomly initialized queries to increase the adaptability of target queries to moving targets and backgrounds. Query iteration uses a top-down approach to gradually decode to adapt to target morphological changes. Feature solution uses target queries and input features to calculate features for infrared small target detection.

[0010] Step 4: Use the detection head composed of four layers of 3D convolution to receive the features used to perform infrared small target detection and output the final target detection results.

[0011] Compared with the prior art, the present invention has the following advantages:

[0012] (1) A multi-scale self-attention encoding network is proposed for extracting global multi-scale features from input sequence images. The network first calculates the global attention weight for single-scale features and generates global features. Then, it uses the features of adjacent scales to calculate the attention affinity from coarse to fine scales to measure the feature correlation between scales. Finally, it transfers the rich semantic information from top to bottom to each level of features, thus achieving the fusion of global background features and local features of the target in the sequence image.

[0013] (2) A multi-scale self-attention decoding network suitable for infrared small target detection is proposed. The network first uses input features to weight randomly initialized queries to increase the adaptability of target queries to moving targets and backgrounds. Then, it gradually decodes in a top-down manner to adapt to the morphological changes of the target. Finally, it uses the target query and the underlying input features to calculate the features for infrared small target detection. The network can enhance the multi-frame detail information of the query vector during the decoding process, thereby improving the ability to model moving backgrounds and express target features. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 Schematic diagram of the overall architecture of the infrared small target detection method based on multi-scale self-attention motion background modeling;

[0015] Figure 2 This is a schematic diagram of the query iteration module;

[0016] Figure 3 The detection results of various algorithms in different scenarios of the NUDT-MIRSDT and TSIRMT datasets are shown. The real targets are highlighted and enlarged in the lower left corner. The red circles indicate accurately detected targets, the blue circles indicate missed detections, and the yellow circles indicate false alarms. DETAILED DESCRIPTION

[0017] The technical solution of the present invention is further described below with reference to the accompanying drawings, but is not limited thereto. Any modification or equivalent replacement of the technical solution of the present invention that does not depart from the spirit and scope of the technical solution of the present invention should be included in the scope of protection of the present invention.

[0018] The present invention provides an infrared small target detection method based on multi-scale self-attention motion background modeling. It selects multiple consecutive frames from infrared sequence images as model input, uses a backbone network to extract image features, and fully mines the network's multi-scale global features through a cross-scale self-attention encoding structure to model the global background of the time series image; the shallowest features containing the finest granularity information are grouped and weighted to form a group query, and a cross-scale iterative decoding method is used to form a final query, thereby retaining more target details and more fully suppressing the motion background; a confidence map is generated by the detection head, and a threshold segmentation method is used to obtain the final detection result. Figure 1 As shown, the specific steps include:

[0019] Step 1: For the input multi-frame infrared sequence image, divide it into patches according to the specified scale and perform embedding mapping. Then use Video SwinTransformer as the backbone to extract spatiotemporal features and output four levels of features. The specific steps are as follows:

[0020] First, Video Swin Transformer is used as the backbone to extract the spatiotemporal features of the input sequence images. T×H×W×3 As the network input, T represents the number of frames of the input infrared sequence image, H and W represent the height and width of the image respectively, and then it is divided into T / 2×H / 4×W / 4 patches according to the scale of 2×4×4×3. Each patch contains 96-dimensional features, which are then mapped to dimension C after embedding. The network uses 3D non-overlapping windows and translation windows to divide patches to reduce the complexity of Transformer calculation and extract the global features of the image sequence. Finally, 4 layers of features F1, F2, F3 and F4 are output, with dimensions T×H / 8×W / 8×2C, T×H / 16×W / 16×4C, T×H / 32×W / 32×4C and T×H / 32×W / 32×4C respectively. The output features at all levels will serve as the input of the multi-scale self-attention network.

[0021] Step 2: Construct a multi-scale self-attention encoding network consisting of three parts: an intar-scale encoder, a cross-scale encoder, and a feature pyramid network (FPN) to extract global multi-scale features of the input sequence image. The intar-scale encoder calculates the global attention weight for a single scale and generates global features. The cross-scale encoder uses the features of adjacent scales to calculate the attention affinity from coarse to fine to measure the feature correlation between scales. After the top-level features of the FPN are modulated by the spatiotemporal attention weight, they are transferred from top to bottom to features at all levels and finally output to the multi-scale self-attention decoding network. The specific steps are as follows:

[0022] Step 2-0: Define input features and After the attention Z h The calculated result is:

[0023]

[0024] Where C is the number of channels, and for The corresponding learning weight matrix, h represents the hth attention head. After the multi-head attention H calculation, we get:

[0025]

[0026] Where E is the number of attention heads, W ois the weight matrix of multi-head output, and cat is the concatenation operation.

[0027] Step 2-1: In-scale encoder

[0028] Multi-layer features extracted by the backbone network The n-th layer feature F in n , calculate the corresponding position encoding result H n and W n represents the height and width of scale n, N is the number of scales, and then F is calculated according to equation (2) n The self-attention of is:

[0029] I s (F n ,p n )=H(F n +p n ,F n +p n ,F n ) (3)

[0030] to I s (F n ,p n ) performs Feed-Forward and normalization to obtain the output features of scale n After performing the same operation on each scale, the final output features of the encoder within the scale can be obtained

[0031] Step 2-2: Inter-scale encoder

[0032] Features of the encoder output within the scale Calculate the attention of adjacent scales and calculate deep features containing more semantic information from coarse to fine Compared with shallow features containing rich detail information The attention weights between them are calculated and the shallow features are modulated to achieve cross-scale global modeling of the input sequence image. The attention calculation between scales follows the principle of equation (1). According to equation (4), and The cross-scale attention is:

[0033]

[0034] right Perform Feed-Forward and normalization to obtain the output features of scale n After all scale features are calculated, the output features of the inter-scale encoder are obtained Perform a rearrangement operation on it to make the output features consistent with the original input features Keep the same number of frames, height and width to get the features

[0035] Step 2-3: FPN

[0036] FPN is used to gradually transfer rich semantic information to shallow features, thereby enhancing the global information in shallow features, and the final output features It will be used as the input of the multi-scale self-attention decoding network to achieve the detection of small infrared targets.

[0037] Step 3: Construct a multi-scale self-attention decoding network consisting of three parts: query filtering, query iteration, and feature solution. Query filtering uses input features to weight randomly initialized queries to increase the adaptability of target queries to moving targets and backgrounds. Query iteration uses a top-down approach to gradually decode to adapt to target morphological changes. Feature solution uses target queries and input features to calculate features for infrared small target detection. The specific steps are as follows:

[0038] Step 3-1: Query Filtering

[0039] The general Transformer decoder uses randomly initialized queries as input and calculates cross attention with the features output by the Transformer encoder to generate the final query, but the randomly initialized original query With multi-frame features It is irrelevant and easily leads to feature degradation due to target and background motion. Therefore, the present invention introduces group query based on the original query. That is, use The global feature of the original query is used to calculate the grouping weight of the query to obtain the grouped query, which is then used as the initial query input into the decoder. The specific calculation method is as follows:

[0040] Construct randomly initialized raw queries according to the traditional Transformer encoder query design idea Group query T is the number of frames of input features, R is the number of original queries per frame, C is the number of channels, G is the number of grouped queries per frame, R=tG, t is set to 4. For the i-th frame feature, group query It is calculated as follows:

[0041]

[0042] In the formula represents the weight of the j-th group convolution, for The jth group of original queries in .

[0043] because The shallowest feature in Contains the richest fine-grained information and the largest span of target and background inter-frame motion displacement, so it uses Calculate the group weights to eliminate the feature degradation phenomenon to the greatest extent, so the total weight of the i-th frame is The calculation is as follows:

[0044]

[0045] In the formula yes The i-th frame feature in G p Represents global pooling, M is a mapping function consisting of two layers of linear transformation layers and activation functions, mapping the features to G×t dimensions.

[0046] Simple use Training will increase the difficulty of training, so the present invention will and As the input of the multi-scale self-attention decoding network, the query dimensions of the two inputs are different. In order to share weights during the training phase so that they constrain each other, the original query Take the average of each group to get

[0047] Step 3-2: Query Iteration

[0048] The decoding structure designed by the present invention follows the principle of top-down and iteratively generates the final target query, which can well adapt to the morphological changes of the target and achieve accurate positioning. The input of each step includes the output of the previous decoding layer and The decoding structure includes self-attention and cross-attention modules as the conventional Transformer decoder. According to formula (2), for the k-th iteration of the n-th layer, the input query The self-attention calculation result is:

[0049]

[0050] Where, Represents a query The position encoding is obtained through learning. Combined with the features Calculate cross attention:

[0051]

[0052] Where p N-n+1 Representation characteristics Positional encoding, A learnable scale positional encoding of 1×C dimensions Copy T×H N-n+1 ×W N-n+1 After N (N=4) layers of decoding and K (K=3) rounds of iteration, the target query is finally output.

[0053] For the original query and grouped queries After query iteration, the final original target query is generated and grouped target queries The two computation processes share weights during the training phase, and the original query path is removed during the inference phase to reduce the computational overhead of the network.

[0054] Step 3-3: Feature Solving

[0055] Due to the bottom-level characteristics Contains richer fine-grained information, as well as larger target morphological changes and inter-frame displacements, so and and Calculate the attention map separately and This allows the model to dynamically adjust attention based on the input features, thereby achieving more accurate pixel-level positioning of the target. The calculation process is as follows:

[0056]

[0057]

[0058] Where E is the number of attention heads, and for and The corresponding learning weights, yes The corresponding learning weight, h represents the hth attention head. and Then separate them with Splicing to obtain the final features for infrared small target detection as well as

[0059] Step 4: Use the detection head composed of four layers of 3D convolution to receive the features used to perform infrared small target detection and output the final target detection results. The overall training process of the network is constrained by the original loss and grouping loss. The specific steps are as follows:

[0060] Obtaining features for infrared small target detection as well as After that, the confidence map is formed by the detection head and the threshold segmentation method is used to form the final detection result. The detection head consists of four layers of 3D convolution. The size of the first layer 3D convolution kernel is 1×3×3, the size of the second and third layers of convolution kernel is 3×3×3, and the size of the last layer of convolution kernel is 1×1×1. and The original loss can be obtained and packet loss

[0061]

[0062]

[0063] Where D h is the detection head, G m is the target mask true value, FL is focal loss, DICE is diceloss, then detection loss for:

[0064]

[0065] Where λ is the balance weight between the original loss and the group loss, and its value is 0.1. Therefore, the final loss of the entire network is for:

[0066]

[0067] Step 5: Experiments were conducted using the publicly available TSIRMT and NUDT-MIRSDT datasets. The datasets were split into training and test sets in a ratio of 7:3, and data augmentation strategies were applied, including random cropping, normalization, and scaling. To maintain temporal continuity in multi-frame object detection, the same augmentation operation was applied to the multi-frame image segments input to the network. In all experiments, the network took 5 frames as input, the weights were initialized by Kaiming, and the AdamW optimizer was used for 30 epochs, with a learning rate ranging from 1×10 -4 Start with a polynomial strategy with a decay exponent of 0.9 and decay to 1×10 -7 . 5 queries are used per frame image and the batch size is set to 1. The network is implemented using PyTorch and runs on an Nvidia GeForce 4080 GPU.

[0068] Figure 3The detection results of various algorithms in different scenarios of TSIRMT dataset and NUDT-MIRSDT dataset are given. Figure 3 (b, d, f) It can be seen that traditional methods have difficulty in evolving the target shape based on fixed assumptions or manual features and distinguishing it from the background, which makes it difficult to detect extended targets of a certain size; Figure 3 (b,c,e,f) It can be seen that the single-frame deep learning method cannot fully utilize the temporal information and has difficulty distinguishing the target from the background with similar spatial features; although other multi-frame deep learning methods use spatiotemporal information to reduce some background clutter, they lack the extraction of local motion features of the target, resulting in their poor performance when the background moves ( Figure 3 The present invention adopts a multi-level coding structure to integrate multi-level global background semantic information, and uses group query from the shallowest feature map to achieve fine-grained background modeling, effectively suppressing mobile background interference (such as Figure 3 (a, b, e, f)), the best detection results were obtained.

Claims

1. A method for infrared small target detection based on multi-scale self-attention motion background modeling, characterized by The method comprises the following steps: Step 1: For the input multi-frame infrared sequence image, it is divided into patches according to the specified scale and embedding mapping is performed. Then, Video SwinTransformer is used as the backbone to extract spatiotemporal features and output four levels of features; Step 2: Construct a multi-scale self-attention encoding network consisting of three parts: an intra-scale encoder, an inter-scale encoder, and an FPN to extract global multi-scale features of the input sequence image. The intra-scale encoder calculates the global attention weight for a single scale and generates global features. The inter-scale encoder uses the features of adjacent scales to calculate the attention affinity from coarse to fine to measure the feature correlation between scales. After the top-level features of the FPN are modulated by the spatiotemporal attention weight, they are transferred from top to bottom to the features at all levels and finally output to the multi-scale self-attention decoding network. The specific steps are as follows: Step 2-0: Define input features and After the attention Z h The calculated result is: Where C is the number of channels, and for The corresponding learning weight matrix, h represents the h-th attention head, after the multi-head attention H calculation, is obtained: Where E is the number of attention heads, W o is the weight matrix of multi-head output, and cat is the concatenation operation; Step 2-1: In-scale encoder Multi-layer features extracted by the backbone network The n-th layer feature F in n , calculate the corresponding position encoding result H n and W n represents the height and width of scale n, N is the number of scales, and then F is calculated according to equation (2) n The self-attention of is: I s (F n ,p n )=H(F n +p n ,F n +p n ,F n ) (3) to I s (F n ,p n ) performs Feed-Forward and normalization to obtain the output features of scale n After performing the same operation on each scale, the final output features of the encoder within the scale are obtained Step 2-2: Inter-scale encoder Features of the encoder output within the scale Calculate the attention of adjacent scales and calculate deep features containing more semantic information from coarse to fine Compared with shallow features containing rich detail information The attention weights between them are calculated and the shallow features are modulated to achieve cross-scale global modeling of the input sequence image. The attention calculation between scales follows the principle of equation (1). According to equation (4), and The cross-scale attention is: right Perform Feed-Forward and normalization to obtain the output features of scale n After all scale features are calculated, the output features of the inter-scale encoder are obtained Perform a rearrangement operation on it to make the output features consistent with the original input features Keep the same number of frames, height and width to get the features Step 2-3: FPN FPN is used to gradually transfer rich semantic information to shallow features, thereby enhancing the global information in shallow features, and the final output features It will be used as the input of the multi-scale self-attention decoding network to detect small infrared targets; Step 3: Construct a multi-scale self-attention decoding network consisting of three parts: query filtering, query iteration, and feature solution. Query filtering uses input features to weight randomly initialized queries to increase the adaptability of target queries to moving targets and backgrounds. Query iteration uses a top-down approach to gradually decode to adapt to target morphological changes. Feature solution uses target queries and input features to calculate features for infrared small target detection. Step 4: Use the detection head composed of four layers of 3D convolution to receive the features used to perform infrared small target detection and output the final target detection results.

2. The infrared small target detection method based on multi-scale self-attention motion background modeling according to claim 1 is characterized in that The specific steps of step 1 are as follows: First, Video Swin Transformer is used as the backbone to extract the spatiotemporal features of the input sequence images. As the network input, T represents the number of frames of the input infrared sequence image, H and W represent the height and width of the image respectively, and then it is divided into T / 2×H / 4×W / 4 patches according to the scale of 2×4×4×3. Each patch contains 96-dimensional features, which are then mapped to dimension C after embedding. The network uses 3D non-overlapping windows and translation windows to divide patches to reduce the complexity of Transformer calculation and extract the global features of the image sequence. Finally, 4 layers of features F1, F2, F3 and F4 are output, with dimensions T×H / 8×W / 8×2C, T×H / 16×W / 16×4C, T×H / 32×W / 32×4C and T×H / 32×W / 32×4C respectively. The output features at all levels will serve as the input of the multi-scale self-attention network.

3. The infrared small target detection method based on multi-scale self-attention motion background modeling according to claim 1 is characterized in that The specific steps of step 3 are as follows: Step 3-1: Query Filtering Construct randomly initialized raw queries according to the traditional Transformer encoder query design idea Group query R is the number of original queries per frame, G is the number of grouped queries per frame, and for the i-th frame feature, grouped queries It is calculated as follows: In the formula represents the weight of the j-th group convolution, for The jth group of original queries in ; use Calculate the grouping weights to eliminate feature degradation to the greatest extent possible. The total weight of the i-th frame is The calculation is as follows: In the formula yes The i-th frame feature in G p Represents global pooling, M is a mapping function consisting of two layers of linear transformation layers and activation functions, mapping the features to G×t dimensions; Will and Together as the input of the multi-scale self-attention decoding network, the original query Take the average of each group to get Step 3-2: Query Iteration According to formula (2), for the k-th iteration of the n-th layer, the input query The self-attention calculation result is: Where, Represents a query Position encoding, combined with features Calculate cross attention: Where p N-n+1 Representation characteristics Position encoding, A learnable scale positional encoding of 1×C dimensions Copy T×H N-n+1 ×W N-n+1 After N layers of decoding and K rounds of iteration, the target query is finally output. For the original query and grouped queries After query iteration, the final original target query is generated and grouped target queries Step 3-3: Feature Solving The lowest level features and and Calculate the attention map separately and The calculation process is as follows: Where E is the number of attention heads, and for and The corresponding learning weights, yes The corresponding learning weight, h represents the hth attention head, is obtained and Then separate them with Splicing to obtain the final features for infrared small target detection as well as

Images