A spatio-temporal state-aware infrared dim small target detection method based on background alignment

Abstract

The application discloses a kind of spatio-temporal state perception infrared weak small target detection methods based on background alignment, comprising: (1) construct by input end, residual network feature extraction module, background alignment feature refinement module, deep shallow constraint interactive module, spatio-temporal state change perception attention module and output end constitute weak small target detection model;(2) construct space-based complex dynamic background infrared moving target dataset, according to proportion segmentation is training set and test set;(3) construct loss function, using training set to the weak small target detection model constructed is trained;(4) detection stage, the low signal-to-noise ratio infrared image sequence to be detected is input to the weak small target detection model trained, and the corresponding infrared dim weak moving target detection result is output.The application realizes target enhancement and background suppression by making full use of multi-frame spatio-temporal information, effectively improves the detection precision and stability of low signal-to-noise ratio space-based infrared weak small target under the premise of ensuring calculation efficiency.

Description

Technical Field

[0001] This invention relates to the fields of image processing and computer vision, and in particular to a spatiotemporal state-aware infrared weak target detection method based on background alignment. Background Technology

[0002] Space-based infrared detection systems have significant application value in fields such as space target monitoring, with infrared detection of small targets being one of their key supporting technologies. With the development of low-Earth orbit optical small satellite constellations, space-based platforms offer advantages in wide-area coverage and high revisit rates. However, limited by imaging aperture and platform size, target radiation energy is extremely weak under long-distance observation conditions, typically occupying only a few pixels and lacking stable geometric and textural features. Simultaneously, the complex and variable background of the Earth's surface, clouds, and celestial bodies, combined with sensor noise, results in a high degree of overlap between targets and background in the spatiotemporal domain in dynamic scenes with low signal-to-noise ratios. Satellite orbital motion and attitude disturbances further exacerbate target energy diffusion and imaging blurring.

[0003] For example, Chinese patent document CN120471928A discloses an infrared weak target detection method. The method involves data enhancement of two acquired infrared weak target images A and B, and feature extraction is performed on each image to obtain feature layers A1, A2, A3 and B1, B2, B3 respectively. The final feature layer C3 is obtained through methods such as stitching, local contrast calculation, and asymmetric fusion. Anchor-free detection is then performed on feature layer C3 to obtain the target detection result.

[0004] Chinese patent document with publication number CNCN118570709A discloses an infrared video weak target detection method and system based on YOLOv5 and time codec structure. It uses images of multiple consecutive frames, extracts spatiotemporal features using a 3D convolutional network, and adds a time codec structure to process temporal information, so as to detect infrared weak targets more accurately.

[0005] Existing infrared weak target detection methods mostly start from a single perspective, such as spatial local comparison, short-time multi-frame difference or long-time modeling. Although they enhance the target response to a certain extent, they face problems such as severe interference from sudden noise and limited target detection stability under complex dynamic backgrounds and extremely low signal-to-noise ratio conditions.

[0006] Therefore, how to achieve stable and reliable detection of small space-based infrared targets while ensuring computational efficiency, and simultaneously modeling spatial differences, short-term transient changes and long-term evolution characteristics, remains a key challenge that needs to be addressed in this field. Summary of the Invention

[0007] To address the issues of false alarms in complex dynamic space-based scenarios and the lack of collaborative utilization of spatiotemporal feature information in existing algorithms, this invention provides a spatiotemporal state-aware infrared weak target detection method based on background alignment. By fully utilizing multi-frame spatiotemporal information, it achieves target enhancement and background suppression, effectively improving the detection accuracy and stability of low signal-to-noise ratio space-based infrared weak targets while ensuring computational efficiency.

[0008] A spatiotemporal state-aware infrared target detection method based on background alignment includes the following steps: (1) Construct a weak target detection model consisting of an input end, a residual network feature extraction module, a background alignment feature refinement module, a depth constraint interaction module, a spatiotemporal state change perception attention module, and an output end; The input module is used to read the infrared image sequence in time. The residual network feature extraction module is used to extract multi-scale feature representations of the input image. The background alignment feature refinement module receives the multi-scale feature representations and outputs temporal difference features. The depth constraint interaction module receives the temporal difference features and outputs multi-scale spatiotemporal difference features. The spatiotemporal state change perception attention module receives multi-scale spatiotemporal difference features and outputs spatiotemporal enhancement features at different scales. The spatiotemporal enhancement features at different scales are fused through upsampling and fusion convolution to generate a predicted mask image. (2) Construct a space-based complex dynamic background infrared moving target dataset and divide it into a training set and a test set according to the proportion; (3) Construct a loss function and use the training set to train the constructed weak target detection model; (4) During the detection phase, the low signal-to-noise ratio infrared image sequence to be detected is input into the trained weak target detection model, and the corresponding infrared dark and weak moving target detection results are output.

[0009] Furthermore, the residual network feature extraction module extracts four feature representations of the input image at different resolutions: original resolution features, 1 / 2 scale features, 1 / 4 scale features, and 1 / 8 scale features.

[0010] Furthermore, the background alignment feature refinement module comprises three parts, and the working process of each part is as follows: The first part involves obtaining the initial motion vector field by modeling the 1 / 8 scale feature layer through correlation, and jointly optimizing the background motion information from the two dimensions of spatial consistency and temporal consistency to suppress abnormal motion and enhance the continuity of background motion. The second part includes a hierarchical feature-guided refinement module. It takes the optimized background motion information as input and propagates it from the 1 / 8 scale to the 1 / 4 scale, 1 / 2 scale and the original resolution feature layer through progressive upsampling. At each scale, the upsampled background motion information is fused with the feature layer of the corresponding resolution. A residual correction mechanism is introduced to refine the background motion progressively and output a refined background motion vector field at each scale. The third part includes an alignment and temporal difference module, which takes the background motion vector field refined at each scale and the corresponding historical features as input, and uses the background motion information to perform spatial alignment compensation on the historical features to obtain the aligned historical features. Then, the current features and the aligned historical features are differentially processed in the time dimension to output the temporal difference features.

[0011] Furthermore, the aforementioned deep and shallow constraint interaction module comprises three parts, and the working process of each part is as follows: The first part is the reparameterized dilated temporal difference convolution, which takes the temporal difference features output by the background alignment feature refinement module as input, performs multi-branch temporal difference modeling at various scales, extracts cross-frame change information, obtains temporal enhancement features, and reduces computational complexity in the inference stage through structural reparameterization. The second part is the multi-level interaction module, which takes temporal enhancement features as input and builds a cross-level interaction mechanism between feature layers of different resolutions. Low-resolution deep features are used to generate gating information to apply global semantic constraints, while high-resolution shallow features are used to extract local difference information. The current scale features are fused under the modulation of gating information to obtain multi-scale interactive features. The third part is reparameterized central difference convolution, which takes multi-scale interactive features as input, introduces a central difference convolution structure at each scale, models spatially abrupt regions through central difference convolution, and transforms the multi-branch structure into a single convolution structure through reparameterization during the inference stage, outputting multi-scale spatiotemporal difference features.

[0012] Furthermore, the spatiotemporal state change perception attention module is built on an attention architecture and consists of three parts, each of which operates as follows: The first part introduces multi-scale spatial center difference convolution and multi-scale temporal difference convolution at each scale to jointly model the input multi-scale spatiotemporal difference features in order to extract spatial abrupt change information and cross-frame change information, thereby obtaining query features. The second part involves constructing key features by linearly mapping the input multi-scale spatiotemporal difference features using a globally learnable temporal transformation matrix. The third part uses query features and key features as input to construct spatiotemporal attention weights, which are then applied to multi-scale spatiotemporal difference features to output the final spatiotemporal enhanced features.

[0013] Furthermore, in step (2), a space-based complex dynamic background infrared moving target dataset is constructed, specifically including: The infrared imaging results of a real satellite platform are used as background images. The target template and target trajectory are simulated to construct M sets of infrared weak target image sequences. Each image has a resolution of R1×R2. The infrared weak targets in the images are labeled at the pixel level to generate a binary mask image.

[0014] Furthermore, in step (3), the loss function is constructed as follows: ; ; ; in, Indicates detection loss; The loss is used to finally output the predicted confidence map after fusing features at all scales; For the first The loss of the scale feature branch outputs the prediction confidence map, { =1,2,3} represent the original resolution layer, the 1 / 2 resolution layer, and the 1 / 4 resolution layer, respectively; These are the weighting coefficients. Indicates the loss of motor compensation; The L1 loss function calculates the mean of the pixel-wise absolute errors of the two tensors; SSIM is the structural similarity index, used to measure the structural similarity between the aligned image and the reference image, and the larger the value, the higher the structural consistency. , These represent the first channel tensors in the aligned historical features and the original features, respectively; and These are the weighting coefficients.

[0015] Furthermore, in step (3), an end-to-end iterative optimization method is adopted, and the model parameters are updated through the backpropagation algorithm until the model converges.

[0016] Compared with the prior art, the present invention has the following beneficial effects: 1. To address the issue of complex dynamic backgrounds in space-based infrared video, this invention performs consistent modeling and joint compensation of background motion across multiple frames at a low-resolution feature layer, and propagates background information step-by-step to a high-resolution layer through cross-scale progressive feature refinement. This module effectively suppresses pseudo-motion interference caused by platform motion, viewpoint changes, and background dynamics, reducing the false alarm rate and improving the detectability of small targets.

[0017] 2. This invention captures cross-frame change information of the target through multi-scale spatiotemporal difference, and combines deep and shallow feature interaction with multi-scale constraints to achieve progressive enhancement of the response of weak targets. At the same time, it greatly improves inference efficiency through the reparameter mechanism.

[0018] 3. By using a spatiotemporal state change perception attention module to collaboratively model local spatial changes and short-term dynamic features, and combining this with global temporal state evolution constraints, the sensitivity of target feature responses is enhanced. Compared to methods that rely solely on local enhancement or global temporal modeling, this approach effectively suppresses background disturbances and achieves a synergistic improvement in the ability to discriminate weak targets and suppress false alarms in low signal-to-noise ratio scenarios.

[0019] 4. This invention achieves high-precision detection of weak infrared targets while ensuring a compact model structure and a small number of parameters. It balances detection performance and computational efficiency, has good engineering practicality and scalability, and is suitable for space-based infrared edge application scenarios. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart of a spatiotemporal state perception infrared weak target detection method based on background alignment, according to an embodiment of the present invention.

[0022] Figure 2 This is a background example diagram for an embodiment of the present invention.

[0023] Figure 3 Overall model structure diagram in an embodiment of the present invention.

[0024] Figure 4 This is a structural diagram of the background alignment feature refinement module in an embodiment of the present invention.

[0025] Figure 5 This is a detection effect diagram of a target with a complex dynamic background and weak infrared light in an embodiment of the present invention. Detailed Implementation

[0026] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] It should be noted that, unless otherwise specified, the features in the following embodiments and implementation methods can be combined with each other.

[0028] like Figure 1 As shown, a spatiotemporal state-aware infrared target detection method based on background alignment includes the following steps: Step 1: Construct a space-based complex dynamic background infrared moving target dataset and divide it into training and test sets according to the proportions.

[0029] 1.1 Obtain the infrared imaging results of the actual satellite platform as the background image. Some example background images are shown below. Figure 2 As shown, the target template and target trajectory are simulated simultaneously to construct M sets of infrared weak target image sequences, totaling P images. Each image has a resolution of R1×R2, and infrared weak targets are labeled on each frame to generate a binary mask image. In this embodiment, P=46989, N=160, R1=512, and R2=256. 1.2 The M sets of image sequences are proportionally segmented. K sets of infrared weak target image sequences and their corresponding labels are used as the training set, and MK sets of infrared weak target image sequences and their corresponding labels are used as the validation set. In this embodiment, M=160 and K=128.

[0030] Step 2: Construct a spatiotemporal state-aware infrared weak target detection model based on motion alignment, such as... Figure 3 As shown, it mainly consists of an input end, a residual network feature extraction module, a background alignment feature refinement module, a depth constraint interaction module, a spatiotemporal state change perception attention module, and an output end.

[0031] 2.1 Sliding window sampling is used for the infrared image sequence, and N=10 consecutive frames are selected as the model input to ensure temporal continuity.

[0032] 2.2 The input end reads the infrared image sequence in a time sequence to ensure the temporal order of the input images. The input images are preprocessed, including random cropping, random enhancement, and uniform input size.

[0033] 2.3 The residual network feature extraction module performs preliminary processing on the input frame information, extracts multi-scale feature representations, and obtains four feature layers with different resolutions, namely the original resolution and its 1 / 2, 1 / 4 and 1 / 8 downsampled features.

[0034] 2.4, Background Alignment Feature Refinement Module, such as Figure 1 As shown in BAFR, where BC-STDR is the background consistency spatiotemporal dynamic refinement module, such as Figure 4 As shown, HFR is the hierarchical feature-guided refinement module, and ATD is the alignment time-series difference module.

[0035] 2.4.1 First, the 1 / 8 scale feature layer is used to perform preliminary correlation modeling to obtain initial motion vector fields Vb and Vf. Then, the background motion information is optimized to obtain motion vector fields V_t and V_s from the two dimensions of spatial consistency and temporal consistency. Based on this, the corresponding weights Gs and Gt are generated through gated convolution, and V_t and V_s are weighted and fused to output the refined motion vector fields V0_2 and V12.

[0036] 2.4.2 The refined motion vector fields V0_2 and V1_2 are processed by HFR with the optimized background motion information as input. They are propagated from the 1 / 8 scale to the 1 / 4 scale, 1 / 2 scale and the original resolution feature layer through progressive upsampling. At each scale, the upsampled background motion information is fused with the feature layer of the corresponding resolution. A residual correction mechanism is introduced to refine the background motion progressively, and the refined background motion vector fields at each scale are output. 2.4.3. Using ATD with the background motion vector field refined at each scale and the corresponding historical features as input, the historical features are spatially aligned and compensated using the background motion information to obtain the aligned historical features. Then, the dynamic features are extracted using temporal difference convolution with weights of [-1,-1,2], and the temporal difference features are output.

[0037] 2.5, the depth-shallow constraint interaction module includes: firstly, using the temporal difference features output by the background alignment feature refinement module as input, and then using time dilation rates of 1, 2, and 3 during the training phase. The 3×1×1 convolutions capture cross-frame changes, and during the inference phase, they are merged into 7×1×1 convolutions to obtain temporal enhancement features at each scale. Then, the features at each scale are processed through a multi-level interaction module, with the temporal enhancement features as input, to build a cross-level interaction mechanism between feature layers of different resolutions. Low-resolution deep features are used to generate gating information to apply global semantic constraints, while high-resolution shallow features are used to extract local difference information. Under the modulation of gating information, the current scale features are fused to obtain multi-scale interactive features. Finally, using the multi-scale interactive features as input, a central difference convolution structure is introduced at each scale. During the training phase, (3,1,1), (3,3,3), and (3,5,5) multi-scale central difference convolutions are used. During the inference phase, they are reparameterized into single convolutions. The central difference convolutions are used to model spatially abrupt change regions, and during the inference phase, the multi-branch structure is equivalently converted into a single convolution structure through reparameterization, outputting multi-scale spatiotemporal difference features.

[0038] 2.6 The spatiotemporal state change perception attention module first captures local spatial changes and short-term dynamic features through central difference convolutions (1,1,1), (1,3,3), and (1,5,5) and temporal difference convolutions (1,1,1), (3,1,1), and (5,1,1), thereby constructing query features. Then, it constructs key features by linearly mapping the input features through the introduction of a globally learnable temporal transformation matrix. Finally, using the query features and key features as input, it constructs spatiotemporal attention weights and applies them to the multi-scale spatiotemporal difference features, outputting the final spatiotemporal enhanced features.

[0039] 2.7 The spatiotemporal enhancement features at different scales are fused through upsampling and 1×1×1 convolution to generate the final prediction confidence map.

[0040] Step 3: Train the model using the training dataset constructed in Step 1.

[0041] 3.1 The training loss function includes detection loss and motion compensation loss: ; ; ; in, =0.3, =0.15, =0.85.

[0042] 3.2. Train the model constructed in step 2. The training environment is NVIDIA GeForce RTX 4090 GPU, the model training is Epoch=38, the initial learning rate Lr=0.001, the learning rate optimizer is Adam, the batch size=4, and training starts from 0.

[0043] 3.2 Retain the optimal weights obtained from training in step 3.1 for model detection and evaluation.

[0044] Step 4: Input the test dataset constructed in Step 1 into the model weights obtained after training in Step 3, and fix them for target detection processing of infrared image sequences.

[0045] 4.1 Input the test dataset constructed in step 1 into the detection model loaded with model weights, perform forward inference on the infrared image sequence, and obtain the corresponding target prediction results.

[0046] 4.2 Generate a detection map or confidence map of the infrared faint moving target based on the prediction results output by the model, and perform binarization segmentation using a threshold of 0.5 to realize automatic detection of the target region in the input image.

[0047] The experimental results of this embodiment on a real infrared image sequence are as follows: Figure 5 As shown, it can be observed that even when the target signal in the input image is extremely weak, the model proposed in this invention can still detect the target stably and accurately. Furthermore, this method has good suppression capabilities against complex background interference, effectively reducing the occurrence of missed detections and false alarms.

[0048] The embodiments described above provide a detailed explanation of the technical solutions and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A spatiotemporal state-aware infrared target detection method based on background alignment, characterized in that, Includes the following steps: (1) Construct a weak target detection model consisting of an input end, a residual network feature extraction module, a background alignment feature refinement module, a depth constraint interaction module, a spatiotemporal state change perception attention module, and an output end; The input module is used to read the infrared image sequence in time. The residual network feature extraction module is used to extract multi-scale feature representations of the input image. The background alignment feature refinement module receives the multi-scale feature representations and outputs temporal difference features. The depth constraint interaction module receives the temporal difference features and outputs multi-scale spatiotemporal difference features. The spatiotemporal state change perception attention module receives multi-scale spatiotemporal difference features and outputs spatiotemporal enhancement features at different scales. The spatiotemporal enhancement features at different scales are fused through upsampling and fusion convolution to generate a predicted mask image. (2) Construct a space-based complex dynamic background infrared moving target dataset and divide it into a training set and a test set according to the proportion; (3) Construct a loss function and use the training set to train the constructed weak target detection model; (4) During the detection phase, the low signal-to-noise ratio infrared image sequence to be detected is input into the trained weak target detection model, and the corresponding infrared dark and weak moving target detection results are output.

2. The spatiotemporal state-aware infrared weak target detection method based on background alignment according to claim 1, characterized in that, The residual network feature extraction module extracts four feature representations of the input image at different resolutions: original resolution features, 1 / 2 scale features, 1 / 4 scale features, and 1 / 8 scale features.

3. The spatiotemporal state-aware infrared weak target detection method based on background alignment according to claim 1, characterized in that, The background alignment feature refinement module consists of three parts, and the working process of each part is as follows: The first part involves obtaining the initial motion vector field by modeling the 1 / 8 scale feature layer through correlation, and jointly optimizing the background motion information from the two dimensions of spatial consistency and temporal consistency to suppress abnormal motion and enhance the continuity of background motion. The second part includes a hierarchical feature-guided refinement module, which takes the optimized background motion information as input and propagates it from the 1 / 8 scale to the 1 / 4 scale, 1 / 2 scale and the original resolution feature layer through progressive upsampling. At each scale, the upsampled background motion information is fused with the feature layer of the corresponding resolution, and a residual correction mechanism is introduced to refine the background motion step by step, outputting a refined background motion vector field at each scale. The third part includes an alignment and temporal difference module, which takes the background motion vector field refined at each scale and the corresponding historical features as input, and uses the background motion information to perform spatial alignment compensation on the historical features to obtain the aligned historical features. Then, the current features and the aligned historical features are differentially processed in the time dimension to output the temporal difference features.

4. The spatiotemporal state-aware infrared weak target detection method based on background alignment according to claim 1, characterized in that, The aforementioned deep and shallow constraint interaction module consists of three parts, and the working process of each part is as follows: The first part is the reparameterized dilated temporal difference convolution, which takes the temporal difference features output by the background alignment feature refinement module as input, performs multi-branch temporal difference modeling at various scales, extracts cross-frame change information, obtains temporal enhancement features, and reduces computational complexity in the inference stage through structural reparameterization. The second part is the multi-level interaction module, which takes temporal enhancement features as input and builds a cross-level interaction mechanism between feature layers of different resolutions. Low-resolution deep features are used to generate gating information to apply global semantic constraints, while high-resolution shallow features are used to extract local difference information. The current scale features are fused under the modulation of gating information to obtain multi-scale interactive features. The third part is reparameterized central difference convolution, which takes multi-scale interactive features as input, introduces a central difference convolution structure at each scale, models spatially abrupt regions through central difference convolution, and transforms the multi-branch structure into a single convolution structure through reparameterization during the inference stage, outputting multi-scale spatiotemporal difference features.

5. The spatiotemporal state-aware infrared weak target detection method based on background alignment according to claim 1, characterized in that, The spatiotemporal state change perception attention module consists of three parts, and the working process of each part is as follows: The first part introduces multi-scale spatial center difference convolution and multi-scale temporal difference convolution at each scale to jointly model the input multi-scale spatiotemporal difference features in order to extract spatial abrupt change information and cross-frame change information, thereby obtaining query features. The second part involves constructing key features by linearly mapping the input multi-scale spatiotemporal difference features using a globally learnable temporal transformation matrix. The third part uses query features and key features as input to construct spatiotemporal attention weights, which are then applied to multi-scale spatiotemporal difference features to output the final spatiotemporal enhanced features.

6. The spatiotemporal state-aware infrared weak target detection method based on background alignment according to claim 1, characterized in that, In step (2), a space-based complex dynamic background infrared moving target dataset is constructed, specifically including: The infrared imaging results of a real satellite platform are used as background images. The target template and target trajectory are simulated to construct M sets of infrared weak target image sequences. Each image has a resolution of R1×R2. The infrared weak targets in the images are labeled at the pixel level to generate a binary mask image.

7. The spatiotemporal state-aware infrared weak target detection method based on background alignment according to claim 1, characterized in that, In step (3), the loss function is constructed as follows: ； ； ； in, Indicates detection loss; The loss is used to finally output the predicted confidence map after fusing features at all scales; For the first The loss of the scale feature branch outputs the prediction confidence map, { =1,2,3} represent the original resolution layer, the 1 / 2 resolution layer, and the 1 / 4 resolution layer, respectively; These are the weighting coefficients. Indicates the loss of motor compensation; The L1 loss function calculates the mean of the pixel-wise absolute errors of the two tensors; SSIM is the structural similarity index, used to measure the structural similarity between the aligned image and the reference image, and the larger the value, the higher the structural consistency. , These represent the first channel tensors in the aligned historical features and the original features, respectively; and These are the weighting coefficients.

8. The spatiotemporal state-aware infrared weak target detection method based on background alignment according to claim 1, characterized in that, In step (3), an end-to-end iterative optimization method is adopted, and the model parameters are updated through the backpropagation algorithm until the model converges.

Images