Infrared small target detection method based on multi-scale self-attention motion background modeling

Abstract

The invention discloses an infrared small target detection method based on multi-scale self-attention motion background modeling, and the method comprises the steps: selecting a plurality of continuous frames from an infrared sequence image as model input, and extracting image features through a backbone network; fully mining the multi-scale global features of the network through a cross-scale self-attention coding structure to model a global background of a time sequence image; grouping and weighting the shallowest layer features containing the finest granularity information to form a grouping query, and forming a final query by adopting a cross-scale iterative decoding mode, so as to reserve more target details and more sufficiently suppress a motion background; a confidence map is generated through the detection head, and a final detection result is obtained through a threshold segmentation method. The multi-scale self-attention coding network can realize fusion of sequence image background global features and target local features, and the multi-scale self-attention decoding network can enhance multi-frame detail information of query vectors in the decoding process and improve the capabilities of motion background modeling and target feature expression.

Description

Technical Field

[0001] The present invention belongs to the technical field of target detection and recognition, and relates to a method for detecting small infrared moving targets, and in particular to a method for detecting small infrared targets based on multi-scale self-attention moving background modeling. Background Art

[0002] In recent years, infrared imaging systems have been widely used in disaster prevention, weather forecasting, target detection and other fields, among which target detection is the core research direction. In many application scenarios, the detection platform is often far away from the target, resulting in the lack of target structure and texture in the infrared image. In addition, the imaging scene is complex, and the target is submerged in the background clutter, showing weak and small characteristics, which brings great difficulty to the target detection task. The detection method based on a single frame image only uses the local saliency of the target relative to the background, which is prone to missed detection and false alarms. Moving targets in infrared scenes, such as ships, drones, and vehicles, are the main detection objects. Therefore, it is of great significance to use the target motion characteristics in the target detection algorithm, which has better performance and application effect than the infrared small target detection task based on a single frame image. At present, a large number of scholars have conducted in-depth research on infrared moving small target detection algorithms, including traditional methods based on prior models and data-driven deep learning methods. In actual application scenarios, as the target approaches or moves away from the detection platform, the target's image plane motion speed, imaging scale and energy are always changing. Therefore, traditional methods based on prior assumptions cannot adapt well to task requirements. Deep learning methods can mine high-level semantic feature differences between target and background information from sequence image data, thereby achieving accurate positioning of the target.

[0003] At present, there are still some legacy issues to be solved in infrared sequence image target detection: 1) The complex random motion of the detection platform causes irregular background changes, making it difficult to mine the temporal differentiation characteristics of the target and background. 2) When preprocessing the input image and using the SIFT method for image registration, the scene structure details in the infrared image are not as rich as those in the visible light image, and the traditional registration method has large errors. Summary of the invention

[0004] Aiming at the problem that it is difficult to extract target energy and motion information when weak target energy and spatial clutter fluctuations are close in infrared sequence image target detection tasks, the present invention provides an infrared small target detection method based on multi-scale self-attention motion background modeling.

[0005] The objective of the present invention is achieved through the following technical solutions:

[0006] A method for infrared small target detection based on multi-scale self-attention motion background modeling includes the following steps:

[0007] Step 1: For the input multi-frame infrared sequence image, it is divided into patches according to the specified scale and embedding mapping is performed. Then Video SwinTransformer is used as the backbone to extract spatiotemporal features and output four levels of features;

[0008] Step 2: Construct a multi-scale self-attention encoding network consisting of an intar-scale encoder, a cross-scale encoder, and a feature pyramid network (FPN) to extract global multi-scale features of the input sequence image. The intar-scale encoder calculates the global attention weight for a single scale and generates global features. The cross-scale encoder uses the features of adjacent scales to calculate the attention affinity from coarse to fine to measure the feature correlation between scales. After the top-level features of the FPN are modulated by the spatiotemporal attention weights, they are transmitted from top to bottom to features at all levels and finally output to the multi-scale self-attention decoding network.

[0009] Step 3: Construct a multi-scale self-attention decoding network consisting of query filtering, query iteration and feature solution, where query filtering uses input features to weight randomly initialized queries to increase the adaptability of target queries to moving targets and backgrounds, query iteration gradually decodes in a top-down manner to adapt to the morphological changes of the target, and feature solution uses target queries and input features to calculate features for infrared small target detection;

[0010] Step 4: Use the detection head composed of four layers of 3D convolution to receive the features used to perform infrared small target detection and output the final target detection results.

[0011] Compared with the prior art, the present invention has the following advantages:

[0012] (1) A multi-scale self-attention encoding network suitable for extracting global multi-scale features of input sequence images is proposed. The network first calculates the global attention weight for single-scale features and generates global features. Then, it uses the adjacent scale features to calculate the attention affinity from coarse to fine to measure the feature correlation between scales. Finally, it transfers the rich semantic information from top to bottom to each level of features, thus realizing the fusion of global features of the sequence image background and local features of the target.

[0013] (2) A multi-scale self-attention decoding network suitable for infrared small target detection is proposed. The network first uses the input features to weight the randomly initialized query to increase the adaptability of the target query to the moving target and background, then gradually decodes in a top-down manner to adapt to the morphological changes of the target, and finally uses the target query and the underlying input features to calculate the features for infrared small target detection. The network can enhance the multi-frame detail information of the query vector during the decoding process, thereby improving the ability of moving background modeling and target feature expression. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 The overall architecture diagram of the infrared small target detection method based on multi-scale self-attention motion background modeling;

[0015] Figure 2 This is a schematic diagram of the query iteration module;

[0016] Figure 3 The detection results of various algorithms in different scenarios of the NUDT-MIRSDT and TSIRMT datasets. The real target is highlighted and enlarged in the lower left corner. The red circle indicates the accurately detected target, the blue circle indicates the missed detection, and the yellow circle indicates the false alarm. DETAILED DESCRIPTION

[0017] The technical solution of the present invention is further described below in conjunction with the accompanying drawings, but is not limited thereto. Any modification or equivalent replacement of the technical solution of the present invention without departing from the spirit and scope of the technical solution of the present invention should be included in the protection scope of the present invention.

[0018] The present invention provides an infrared small target detection method based on multi-scale self-attention motion background modeling. Multiple consecutive frames are selected from infrared sequence images as model input, image features are extracted using a backbone network, and the multi-scale global features of the network are fully mined through a cross-scale self-attention encoding structure to model the global background of the time series image; the shallowest features containing the finest granular information are grouped and weighted to form a group query, and a cross-scale iterative decoding method is used to form a final query, thereby retaining more target details and more fully suppressing the motion background; a confidence map is generated through a detection head, and a threshold segmentation method is used to obtain the final detection result. Figure 1 As shown, the specific steps include:

[0019] Step 1: For the input multi-frame infrared sequence image, it is divided into patches according to the specified scale and embedding mapping is performed. Then Video SwinTransformer is used as the backbone to extract spatiotemporal features and output four levels of features. The specific steps are as follows:

[0020] First, Video Swin Transformer is used as the backbone to extract the temporal and spatial features of the input sequence images. T×H×W×3 As the network input, T represents the number of frames of the input infrared sequence image, H and W represent the height and width of the image respectively. Then it is divided into T / 2×H / 4×W / 4 patches according to the scale of 2×4×4×3. Each patch contains 96-dimensional features, which are then mapped to dimension C after embedding. The network uses 3D non-overlapping windows and translation windows to divide patches to reduce the complexity of Transformer calculation and extract the global features of the image sequence. Finally, 4 layers of features F1, F2, F3 and F4 are output, with dimensions T×H / 8×W / 8×2C, T×H / 16×W / 16×4C, T×H / 32×W / 32×4C and T×H / 32×W / 32×4C respectively. The output features at all levels will be used as the input of the multi-scale self-attention network.

[0021] Step 2: Construct a multi-scale self-attention encoding network consisting of an intar-scale encoder, a cross-scale encoder, and a feature pyramid network (FPN) to extract global multi-scale features of the input sequence image. The intar-scale encoder calculates the global attention weight for a single scale and generates global features. The cross-scale encoder uses the features of adjacent scales to calculate the attention affinity from coarse to fine to measure the feature correlation between scales. After the top-level features of the FPN are modulated by the spatiotemporal attention weights, they are passed from top to bottom to features at all levels and finally output to the multi-scale self-attention decoding network. The specific steps are as follows:

[0022] Step 2-0: Define input features and After the attention Z h The calculated result is:

[0023]

[0024] Where C is the number of channels, and for The corresponding learning weight matrix, h represents the hth attention head. After the multi-head attention H calculation, we get:

[0025]

[0026] Where E is the number of attention heads, W ois the weight matrix of multi-head output, and cat is the concatenation operation.

[0027] Step 2-1: In-scale encoder

[0028] Multi-layer features extracted from the backbone network The n-th layer feature F in n , calculate the corresponding position encoding result H n and W n represents the height and width of scale n, N is the number of scales, and then F is calculated according to equation (2) n The self-attention of is:

[0029] I s (F n ,p n )=H(F n +p n ,F n +p n ,F n ) (3)

[0030] to I s (F n ,p n ) performs Feed-Forward and normalization to obtain the output features of scale n After performing the same operation on each scale, the final output features of the encoder within the scale can be obtained

[0031] Step 2-2: Inter-scale encoder

[0032] The characteristics of the encoder output within the scale Calculate the attention of adjacent scales and calculate deep features containing more semantic information from coarse to fine. Compared with shallow features containing rich details The attention weights between them are calculated and the shallow features are modulated to achieve cross-scale global modeling of the input sequence image. The attention calculation between scales follows the principle of equation (1). Then, according to equation (4), and The cross-scale attention is:

[0033]

[0034] right Perform feed-forward and normalization to obtain output features of scale n After calculating all scale features, the output features of the scale encoder are obtained. Perform a rearrangement operation on it to make the output features consistent with the original input features Keep the same number of frames, height and width to get the features

[0035] Step 2-3: FPN

[0036] FPN is used to gradually transfer rich semantic information to shallow features, thereby enhancing the global information in the shallow features. It will be used as the input of the multi-scale self-attention decoding network to achieve the detection of small infrared targets.

[0037] Step 3: Construct a multi-scale self-attention decoding network consisting of query filtering, query iteration and feature solution. The query filtering uses the input feature weighted randomly initialized query to increase the adaptability of the target query to the moving target and background. The query iteration is decoded step by step in a top-down manner to adapt to the morphological changes of the target. The feature solution uses the target query and input features to calculate the features for infrared small target detection. The specific steps are as follows:

[0038] Step 3-1: Query Filter

[0039] The general Transformer decoder uses randomly initialized queries as input and calculates cross attention with the features output by the Transformer encoder to generate the final query, but the randomly initialized original query With multi-frame features It is irrelevant and easily leads to feature degradation due to the motion of the target and the background. Therefore, the present invention introduces grouping query based on the original query. That is, use The global features of the original query are used to calculate the grouping weights to obtain the grouped query, which is then used as the initial query input into the decoder. The specific calculation method is as follows:

[0040] Construct randomly initialized raw queries according to the traditional Transformer encoder query design idea Group query T is the number of frames of input features, R is the number of original queries per frame, C is the number of channels, G is the number of grouped queries per frame, R = tG, t is set to 4. For the i-th frame feature, group query It is calculated by:

[0041]

[0042] In the formula represents the weight of the j-th group convolution, for The jth group of original queries in .

[0043] because The shallowest feature in It contains the richest fine-grained information and the largest span of target and background inter-frame motion displacement, so it uses The grouping weights are calculated to eliminate feature degradation to the greatest extent possible, so the total weight of the i-th frame is The calculation is as follows:

[0044]

[0045] In the formula yes The i-th frame feature in G p represents global pooling, M is a mapping function consisting of two linear transformation layers and an activation function, which maps the features into G×t dimensions.

[0046] Simple use Training will increase the difficulty of training, so the present invention will and As the input of the multi-scale self-attention decoding network, the query dimensions of the two inputs are different. In order to share weights during the training phase and make them mutually constrained, the original query Take the average of each group to get

[0047] Step 3-2: Query iteration

[0048] The decoding structure designed by the present invention follows the principle of top-down, iteratively generates the final target query, and can well adapt to the morphological changes of the target to achieve accurate positioning. The input of each step includes the output of the previous decoding layer and The decoding structure includes self-attention and cross-attention modules as the conventional Transformer decoder. According to formula (2), for the kth iteration of the nth layer, the input query The self-attention calculation result is:

[0049]

[0050] In the formula, Representation query The position encoding is obtained through learning. Combining features Calculate criss-cross attention:

[0051]

[0052] Where p N-n+1 Representation characteristics The position code, A learnable scaled positional encoding of 1×C dimensions Copy T×H N-n+1 ×W N-n+1 After N (N=4) layers of decoding and K (K=3) rounds of iteration, the target query is finally output.

[0053] For the original query and group query After query iteration, the final original target query is generated and grouped target queries In the training phase, the two calculation processes share weights, and in the inference phase, the original query path is removed to reduce the computational overhead of the network.

[0054] Step 3-3: Feature Solving

[0055] Because the bottom-level features It contains richer fine-grained information, as well as larger target morphological changes and inter-frame displacements, so and and Calculate the attention map separately and This allows the model to dynamically adjust attention based on the input features, thereby achieving more accurate pixel-level positioning of the target. The calculation process is as follows:

[0056]

[0057]

[0058] Where E is the number of attention heads, and for and The corresponding learning weights are yes The corresponding learning weight, h represents the hth attention head. and Then they were separated with Splicing to obtain the final features for infrared small target detection as well as

[0059] Step 4: Use the detection head composed of four layers of 3D convolution to receive the features used to perform infrared small target detection and output the final target detection results. The overall training process of the network is constrained by the original loss and the grouping loss. The specific steps are as follows:

[0060] After obtaining the features for infrared small target detection as well as After that, the confidence map is formed by the detection head and the threshold segmentation method is used to form the final detection result. The detection head consists of four layers of 3D convolution. The size of the first layer of 3D convolution kernel is 1×3×3, the size of the second and third layers of convolution kernel is 3×3×3, and the size of the last layer of convolution kernel is 1×1×1. and The original loss can be obtained and packet loss

[0061]

[0062]

[0063] Where D h is the detection head, G m is the true value of the target mask, FL is the focal loss, DICE is the diceloss, then the detection loss for:

[0064]

[0065] Where λ is the balance weight between the original loss and the grouping loss, and its value is 0.1. Therefore, the final loss of the entire network is for:

[0066]

[0067] Step 5: Experiments were conducted using the public TSIRMT dataset and NUDT-MIRSDT dataset. The two datasets were divided into training and test sets in a ratio of 7:3, and data augmentation strategies were applied, including random cropping, normalization, and scaling. In order to maintain the temporal continuity of multi-frame image object detection, the same augmentation operation was applied to the multi-frame image fragments input to the network. In all experiments, the network took 5 frames of images as input, the weights were initialized by Kaiming, and the AdamW optimizer was used for 30 epochs. The learning rate was increased from 1×10 -4 Start with a polynomial strategy with a decay exponent of 0.9, decaying to 1×10 -7 . Five queries are used per frame and the batch size is set to 1. The network is implemented using PyTorch and runs on a Nvidia GeForce 4080 GPU.

[0068] Figure 3The detection results of various algorithms in different scenarios of TSIRMT dataset and NUDT-MIRSDT dataset are given. Figure 3 (b, d, f) It can be seen that it is difficult for traditional methods to evolve the target shape based on fixed assumptions or manual features and distinguish it from the background, which makes it difficult to detect extended targets of a certain size; Figure 3 (b,c,e,f) It can be seen that the single-frame deep learning method cannot fully utilize the temporal information and has difficulty distinguishing the target from the background with similar spatial features; although other multi-frame deep learning methods use spatiotemporal information to reduce some background clutter, they lack the extraction of local motion features of the target, resulting in its inability to distinguish the target from the background moving ( Figure 3 The detection performance is poor in the case of (b, e, f)). The present invention adopts a multi-level coding structure to integrate multi-level global background semantic information, and uses group query from the shallowest feature map to achieve fine-grained background modeling, which effectively suppresses mobile background interference (such as Figure 3 (a, b, e, f)), the best detection results were obtained.

Claims

1. A method for infrared small target detection based on multi-scale self-attention motion background modeling, characterized in that The method comprises the following steps: Step 1: For the input multi-frame infrared sequence image, it is divided into patches according to the specified scale and embedding mapping is performed. Then Video SwinTransformer is used as the backbone to extract spatiotemporal features and output four levels of features; Step 2: Construct a multi-scale self-attention encoding network consisting of an intra-scale encoder, an inter-scale encoder, and an FPN to extract global multi-scale features of the input sequence image. The intra-scale encoder calculates the global attention weight for a single scale and generates global features. The inter-scale encoder uses the features of adjacent scales to calculate the attention affinity from coarse to fine to measure the feature correlation between scales. After the top-level features of the FPN are modulated by the spatiotemporal attention weights, they are transmitted from top to bottom to features at all levels and finally output to the multi-scale self-attention decoding network. Step 3: Construct a multi-scale self-attention decoding network consisting of query filtering, query iteration and feature solution, where query filtering uses input features to weight randomly initialized queries to increase the adaptability of target queries to moving targets and backgrounds, query iteration gradually decodes in a top-down manner to adapt to the morphological changes of the target, and feature solution uses target queries and input features to calculate features for infrared small target detection; Step 4: Use the detection head composed of four layers of 3D convolution to receive the features used to perform infrared small target detection and output the final target detection result.

2. The infrared small target detection method based on multi-scale self-attention motion background modeling according to claim 1 is characterized in that The specific steps of step 1 are as follows: First, Video Swin Transformer is used as the backbone to extract the spatiotemporal features of the input sequence images. T×H×W×3 As the network input, T represents the number of frames of the input infrared sequence image, H and W represent the height and width of the image respectively. Then it is divided into T / 2×H / 4×W / 4 patches according to the scale of 2×4×4×3. Each patch contains 96-dimensional features, which are then mapped to dimension C after embedding. The network uses 3D non-overlapping windows and translation windows to divide patches to reduce the complexity of Transformer calculation and extract the global features of the image sequence. Finally, 4 layers of features F1, F2, F3 and F4 are output, with dimensions T×H / 8×W / 8×2C, T×H / 16×W / 16×4C, T×H / 32×W / 32×4C and T×H / 32×W / 32×4C respectively. The output features at all levels will be used as the input of the multi-scale self-attention network.

3. The infrared small target detection method based on multi-scale self-attention motion background modeling according to claim 2 is characterized in that The specific steps of step 2 are as follows: Step 2-0: Define input features and After the attention Z h The calculated result is: Where C is the number of channels, and for The corresponding learning weight matrix, h represents the hth attention head, after multi-head attention H calculation, is obtained: Where E is the number of attention heads, W o is the weight matrix of multi-head output, and cat is the concatenation operation; Step 2-1: In-scale encoder Multi-layer features extracted from the backbone network The n-th layer feature F in n , calculate the corresponding position encoding result H n and W n represents the height and width of scale n, N is the number of scales, and then F is calculated according to equation (2) n The self-attention of is: I s (F n ,p n )=H(F n +p n ,F n +p n ,F n ) (3) to I s (F n ,p n ) performs Feed-Forward and normalization to obtain the output features of scale n After performing the same operation on each scale, the final output feature of the encoder within the scale is obtained Step 2-2: Inter-scale encoder The characteristics of the encoder output within the scale Calculate the attention of adjacent scales and calculate deep features containing more semantic information from coarse to fine. Compared with shallow features containing rich details The attention weights between them are calculated and the shallow features are modulated to achieve cross-scale global modeling of the input sequence image. The attention calculation between scales follows the principle of equation (1). Then, according to equation (4), and The cross-scale attention is: right Perform feed-forward and normalization to obtain output features of scale n After calculating all scale features, the output features of the scale encoder are obtained. Perform a rearrangement operation on it to make the output features consistent with the original input features Keep the same number of frames, height and width to get the features Step 2-3: FPN FPN is used to gradually transfer rich semantic information to shallow features, thereby enhancing the global information in the shallow features. It will be used as the input of the multi-scale self-attention decoding network to achieve the detection of small infrared targets.

4. The infrared small target detection method based on multi-scale self-attention motion background modeling according to claim 1 is characterized in that The specific steps of step 3 are as follows: Step 3-1: Query Filter Construct randomly initialized raw queries according to the traditional Transformer encoder query design idea Group query R is the number of original queries per frame, G is the number of grouped queries per frame, and for the features of the i-th frame, grouped queries It is calculated by: In the formula represents the weight of the j-th group convolution, for The jth group of original queries in ; use Calculate the grouping weights to eliminate feature degradation to the greatest extent possible. The total weight of the i-th frame is The calculation is as follows: In the formula yes The i-th frame feature in G p Represents global pooling, M is a mapping function consisting of two layers of linear transformation layers and an activation function, mapping the features to G×t dimensions; Will and As the input of the multi-scale self-attention decoding network, the original query Take the average of each group to get Step 3-2: Query iteration According to formula (2), for the kth iteration of the nth layer, the input query The self-attention calculation result is: In the formula, Representation query Position encoding, combined with features Calculate cross attention: Where p N-n+1 Representation characteristics The position code, A learnable scaled positional encoding of 1×C dimensions Copy T×H N-n+1 ×W N-n+1 After N layers of decoding and K rounds of iteration, the target query is finally output. For the original query and group query After query iteration, the final original target query is generated and grouped target queries Step 3-3: Feature solving The lowest level features and and Calculate the attention map separately and The calculation process is as follows: Where E is the number of attention heads, and for and The corresponding learning weights are yes The corresponding learning weight, h represents the hth attention head, is obtained and Then they were separated with Splicing to obtain the final features for infrared small target detection as well as

Images