A method and device for detecting small infrared targets based on spatiotemporal background reconstruction

Abstract

The present invention discloses a method and device for detecting infrared small targets based on spatiotemporal background reconstruction. The method comprises: (1) constructing continuous image frames of a thermal infrared image sequence to be detected into a spatiotemporal tensor; (2) designing a multi-scale feature extraction and aggregation module to realize effective background feature extraction; (3) designing a background reconstruction module based on a variational autoencoder to realize the reconstruction of the background tensor; (4) designing an adaptive background reconstruction loss function for unsupervised learning; (5) calculating the residual of the spatiotemporal tensor of the thermal infrared image sequence to be detected and the reconstructed background tensor to obtain a target tensor and reconstruct the target tensor into a target detection result sequence T to realize infrared small target detection. The present invention utilizes the spatiotemporal information of the infrared image sequence to learn the background features and reconstruct the background in an unsupervised learning manner to realize infrared small target detection based on spatiotemporal background reconstruction. The method can effectively improve the detection performance of infrared small targets.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a method and device for detecting small infrared targets based on spatiotemporal background reconstruction. Background Art

[0002] With the development of deep learning technology, neural networks can extract the difference between infrared background and infrared small targets. Generally, infrared weak small target detection methods based on deep learning can be further divided into detection-based methods and segmentation-based methods. For detection-based methods, Liu et al. proposed an end-to-end deep learning solution for small target detection; Du et al. proposed an end-to-end small target detection framework based on spatial-temporal feature extraction and introduced the inter-frame energy accumulation (IFEA) enhancement mechanism; Bai et al. designed a cross-connected bidirectional pyramid network; Wang et al. proposed a single-frame coarse-fine internal attention perception network (IAANet) for infrared small target detection; Chen et al. proposed a spatial-temporal network (SSTNet) with slice enhancement for detecting moving infrared targets; Zhang et al. proposed a multi-scale progressive fusion filter network to achieve weak infrared small target detection. However, since infrared weak small targets usually occupy only a few pixels, the existing detection-based methods are prone to target loss problems.

[0003] Meanwhile, segmentation-based methods have attracted the attention of researchers, which can achieve pixel-level detection and localization. In order to take advantage of the convolutional neural network, Wang et al. proposed an MDvsFA-cGAN framework to achieve infrared dim small target detection with high accuracy and low false alarm rate through adversarial learning paradigm; Dai et al. proposed an asymmetric context information modulation module to aggregate shallow and deep features for infrared dim small target detection. In addition, Dai et al. designed a model-driven deep network combining traditional methods and discriminant networks; Ciocarlan et al. designed a Deep-NFA framework for small target detection, which achieved accurate target prediction; considering the size and insignificant features of infrared dim small targets, Li et al. designed a medium density adaptive feature fusion network (MDAFNet) to meet the requirements of accurate detection and real-time performance.

[0004] Based on the above network results, the detection performance of infrared dim targets continues to improve, but the problem of deep feature loss still exists, and in scenes with significant changes, the stability and robustness of the detection method may be reduced. As a data-driven method, most current deep learning-based methods still face problems such as large data requirements and insufficient generalization capabilities. Summary of the invention

[0005] In view of the deficiencies in the prior art, the purpose of the present invention is to provide an infrared small target detection method and device based on spatiotemporal background reconstruction. In order to more comprehensively mine and learn spatiotemporal feature information, the present invention designs a spatiotemporal SwinTransformer block structure. On this basis, a multi-scale feature extraction and aggregation module is established for multi-scale global and local feature extraction; in addition, a variational autoencoder-type background reconstruction module is designed for background reconstruction, which includes a multi-head gating mechanism to realize the organic fusion of shallow features and deep features, thereby improving the accuracy of background reconstruction; through an unsupervised learning method, a comprehensive and integrated loss function is designed for network training, and the excellent performance of the infrared small target detection method based on spatiotemporal background reconstruction is comprehensively verified in the embodiments.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] On the one hand, the present invention discloses an infrared small target detection method based on spatiotemporal background reconstruction, which comprises the following steps:

[0008] Step 1): For the thermal infrared image sequence to be detected, its continuous image frames are constructed as a spatiotemporal tensor

[0009] Step 2): design a multi-scale feature extraction and aggregation network for extracting spatiotemporal feature information of the spatiotemporal tensor, wherein the multi-scale feature extraction and aggregation network adopts a pyramid structure, the pyramid structure includes multiple feature extraction layers and local feature information extraction layers, and each feature extraction layer includes several spatiotemporal Swin Transformer modules;

[0010] Step 3): Design a background reconstruction network based on a variational autoencoder to reconstruct the background tensor of the spatiotemporal tensor The background reconstruction network includes a background feature enhancement module, a feature fusion module and a background reconstruction module based on a variational autoencoder;

[0011] Step 4): designing an adaptive background reconstruction loss function, wherein the adaptive background reconstruction loss function includes background reconstruction loss and multi-scale structural similarity loss; performing unsupervised training on the multi-scale feature extraction and aggregation network and the background reconstruction network based on the adaptive background reconstruction loss function;

[0012] Step 5): The spatiotemporal tensor of the thermal infrared image sequence to be detected Re-enter the trained network in step 4) to calculate the space-time tensor With the reconstructed background tensor The residual between them is the target tensor. The target tensor is reconstructed into the infrared small target detection result sequence T to realize infrared small target detection.

[0013] The present invention also discloses an infrared small target detection device based on spatiotemporal background reconstruction for implementing the method, which comprises:

[0014] A space-time tensor construction module constructs continuous image frames in the infrared image sequence to be detected into a space-time tensor;

[0015] A multi-scale feature extraction and aggregation network, used to extract the spatiotemporal feature information of the spatiotemporal tensor, wherein the multi-scale feature extraction and aggregation network adopts a pyramid structure, which includes a plurality of feature extraction layers and a local feature information extraction layer, and each feature extraction layer includes a plurality of spatiotemporal Swin Transformer modules;

[0016] Background reconstruction network, used to reconstruct the background tensor of the spatiotemporal tensor The background reconstruction network includes a background feature enhancement module, a feature fusion module and a background reconstruction module based on a variational autoencoder;

[0017] Loss function design module, which designs adaptive background reconstruction loss functions for unsupervised learning, including background reconstruction loss and multi-scale structural similarity loss, to improve the performance of the network model;

[0018] The target tensor output module calculates the residual between the original spatiotemporal tensor to be detected and the reconstructed background tensor to obtain the target tensor;

[0019] The target detection result output module is used to output the infrared small target detection result map of each frame of infrared image in the infrared image sequence.

[0020] Compared with the prior art, the present invention has the following beneficial effects:

[0021] 1) Tensors can fully express the rich spatiotemporal information in infrared image sequences. The present invention proposes an infrared small target detection method based on spatiotemporal background reconstruction. The designed network can better utilize the global and local spatiotemporal context information in the infrared sequence, which is helpful for the extraction of background features and the reconstruction of the background. In addition, the present invention proposes an unsupervised learning method, which greatly reduces the heavy labeling work, improves the accuracy of the reconstructed background, and realizes efficient infrared small target detection.

[0022] 2) For feature extraction, the present invention proposes a multi-scale feature extraction module based on the spatiotemporal Swin Transformer block, which is used to effectively extract global and local background features while suppressing target features; for background reconstruction, the present invention designs a reconstruction module based on a variational autoencoder with a multi-head gating mechanism, which can achieve satisfactory background reconstruction effect; experimental results show that the network structure designed by the present invention can achieve satisfactory infrared small target detection performance under sufficient data volume and unsupervised learning conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a flow chart of the infrared small target detection method based on spatiotemporal background reconstruction of the present invention;

[0024] Figure 2 It is a structural schematic diagram of an infrared small target detection device based on spatiotemporal background reconstruction in the present invention;

[0025] Figure 3 Schematic diagram of the multi-scale feature extraction and aggregation module in the present invention, (a) is the designed upsampling structure, (b) is the designed point convolution structure;

[0026] Figure 4 Schematic diagram of the spatiotemporal Swin Transformer block structure designed for the present invention, (a) is the spatiotemporal multi-head self-attention calculation with a regular window, (b) is the spatiotemporal multi-head self-attention calculation with a moving window mechanism;

[0027] Figure 5 Schematic diagram of the moving window mechanism in the present invention, (a) is the input spatiotemporal tensor, (b) is a schematic diagram of regular window block division, and (c) is a schematic diagram of moving window block division;

[0028] Figure 6 Schematic diagram of the background reconstruction module based on the variational autoencoder in the present invention, (a) is the designed upsampling structure, (b) is the designed shallow feature and deep feature fusion structure;

[0029] Figure 7 Schematic diagram of the multi-head gating mechanism designed for the present invention;

[0030] Figure 8 This is a sample frame image of a thermal infrared image sequence used for experimental testing;

[0031] Fig. 9 The original image of an example frame of a thermal infrared image and the thermal infrared small target detection result diagram detected by the method of the present invention, MDWCM, RCTVW, STT-TRNR, DNANet and AGPCNet. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solutions and advantages of the present invention clearer, the present invention is described in detail below in conjunction with specific embodiments. Specific embodiments are described below to simplify the present invention. However, it should be recognized that the present invention is not limited to the illustrated embodiments, and various modifications of the present invention are possible without departing from the basic principles, and these equivalent forms also fall within the scope defined by the claims attached to the present application.

[0033] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0034] like Figure 1 As shown, the basic steps of the infrared small target detection method based on spatiotemporal background reconstruction in this embodiment of the present invention mainly include:

[0035] Step 1: For a thermal infrared image sequence to be detected, construct its continuous image frames into a spatiotemporal tensor , used as training samples for subsequent background feature extraction and reconstruction;

[0036] Specifically, from the original thermal infrared image sequence to be detected, continuous image frames are obtained each time, and they are stacked in sequence according to the time order of the image frames to construct a space-time tensor As the input sample of the subsequent network, the spatiotemporal tensor The number of frames, height, width, and channels are F, H, W, and C respectively;

[0037] Specifically in the embodiment, H and W are both 256, F is 16, C is 1, and the constructed space-time tensor The dimensions are 16×256×256×1.

[0038] Step 2: Design a multi-scale feature extraction and aggregation network to extract the spatiotemporal feature information of the spatiotemporal tensor, such as Figure 3 As shown, the multi-scale feature extraction and aggregation network adopts a pyramid structure, which includes multiple feature extraction layers and local feature information extraction layers, and each feature extraction layer includes several spatiotemporal Swin Transformer modules.

[0039] Specifically in this embodiment, the multiple feature extraction layers include a top layer, a middle layer and a bottom layer, and the top layer, the middle layer and the bottom layer are respectively composed of 1, 2, and 3 spatiotemporal Swin Transformer modules in cascade, and the top layer, the middle layer and the bottom layer respectively obtain multi-scale feature maps At the same time, shallow features are obtained

[0040] The local feature information extraction layer combines 3D convolution, batch normalization and activation function to extract local feature information from the spatiotemporal tensor. Extract enhanced local feature information Respectively and Perform downsampling and upsampling to obtain and Then point-by-point convolution is performed to achieve feature integration; the multi-scale feature aggregation process is expressed as follows:

[0041]

[0042] in is the aggregated feature, g(·)=ReLU(BN(Conv3D(·))) represents local feature extraction, Indicates that i ST-STB cascades are used for feature extraction, Down(·) indicates downsampling calculation, Up(·) indicates upsampling calculation, and f 1×1 (·) indicates point convolution calculation.

[0043] The self-attention mechanism helps the model better understand the contextual spatiotemporal information and process sequence data more accurately by calculating the correlation of each element in the sample with respect to other elements; the background component exhibits uniformity and low rank characteristics in both temporal and spatial dimensions, which indicates that there are correlations between different background blocks; the Swin Transformer is based on a multi-head self-attention mechanism, which divides the image into non-overlapping image blocks to capture details and local patterns; the present invention expands the traditional single-frame-based Swin Transformer structure into a spatiotemporal Swin Transformer block (ST-STB).

[0044] like Figure 4 and 5 As shown, the spatiotemporal Swin Transformer module of the present invention includes multiple spatiotemporal Swin Transformer blocks, each of which includes two layers of multi-head self-attention calculation layers connected in series, wherein the first layer of multi-head self-attention calculation layer uses a regular local window to split the input and performs multi-head self-attention calculation on each local window obtained by the segmentation; the second layer of multi-head self-attention calculation layer first uses a moving window mechanism to shift the local window to enhance the information exchange between local windows, and then uses the obtained local window to split the output of the first layer of multi-head self-attention calculation layer and performs multi-head self-attention calculation on each local window obtained by the segmentation.

[0045] In a specific embodiment of the present invention, Figure 4 As shown, the feature extraction layer of the present invention (assuming it includes n spatiotemporal Swin Transformer blocks) is used for each input sample First convert it to the scale token as a feature; after the feature extraction of the first spatiotemporal Swin Transformer block, the output feature For the subsequent spatiotemporal Swin Transformer block, the input features are The output features are in, represents the input features of the nth spatiotemporal Swin Transformer block; represents the feature calculated by the nth spatiotemporal Swin Transformer block. Specifically in the embodiment, P=2, M=2.

[0046] like Figure 5 As shown, for each spatiotemporal Swin Transformer block, the calculation process of its self-attention mechanism specifically includes: Assuming that the size of the input feature is F in ×H in ×W in , the number of channels is 1, and the size of the moving window is P×M×M; in the nth stage, the two layers of multi-head self-attention calculation layers are marked as the lth layer and the l+1th layer; the lth layer uses a regular local window segmentation to obtain A local window is used to perform spatiotemporal multi-head self-attention calculations. Indicates the rounding up operation; for the l+1th layer, the window partition moves along the time, height and width directions respectively Token units, and calculate the spatiotemporal multi-head self-attention mechanism on the obtained local window.

[0047] The multi-head self-attention mechanism of the two-layer multi-head self-attention calculation layer is the same, specifically:

[0048] The multi-head self-attention mechanism first calculates the query in the local window and each key The similarity between them is then re-weighted by Taking a local window of as an example, the self-attention calculation process SA(·) can be expressed as

[0049]

[0050] in, The scale of the spatiotemporal relative position encoding b is P 2 ×M 2 ×M 2 , d means check and The number of channels, As a scaling factor, express The transpose of , softmax(·) represents the activation function; Through linear transformation get, represents the weight tensor, Indicates characteristics;

[0051] For the nth spatiotemporal Swin Transformer module, the process of two-layer multi-head self-attention calculation is expressed as follows:

[0052]

[0053]

[0054]

[0055]

[0056] in, represents the features obtained by calculating self-attention using a regular local window in layer l, represents the final feature obtained in the lth layer, It represents the feature obtained by calculating the self-attention using the local window of the moving window mechanism in the l+1th layer, represents the final feature obtained after the spatiotemporal Swin Transformer module, LN(·) represents layer normalization calculation, BN(·) represents batch normalization calculation, MLP(·) represents full connection calculation, STW-MSA(·) represents the spatiotemporal multi-head self-attention calculation with regular window of the lth layer, and STSW-MSA(·) represents the spatiotemporal multi-head self-attention calculation with moving window mechanism of the l+1th layer.

[0057] Based on the above-mentioned spatiotemporal Swin Transformer structure, a multi-scale information extraction and aggregation structure is established as follows: a pyramid-like structure is used to learn the spatiotemporal tensor The top, middle and bottom layers are composed of 1, 2 and 3 spatiotemporal Swin Transformer modules in cascade, respectively. The top, middle and bottom layers capture more refined multi-scale feature maps from the input. At the same time, shallow features are obtained

[0058] In addition, the local feature information extraction layer combines three-dimensional convolution (Conv3D), batch normalization (BN) and activation function (LeakyReLU) to extract local feature information from the spatiotemporal tensor. Extract enhanced local feature information In order to achieve size unification and aggregate multi-scale features, and Perform downsampling and upsampling to obtain and Then point-by-point convolution is performed to achieve feature integration; the multi-scale feature aggregation process is expressed as follows:

[0059]

[0060] in, is the aggregated feature, g(·)=ReLU(BN(Conv3D(·))) represents local feature extraction, Indicates that i ST-STBs are used for feature extraction, Down(·) indicates downsampling calculation, Up(·) indicates upsampling calculation, and f 1×1 (·) indicates point convolution calculation.

[0061] Step 3: Design a background reconstruction network based on variational autoencoder to reconstruct the background tensor of the spatiotemporal tensor The background reconstruction network includes a background feature enhancement module, a feature fusion module and a background reconstruction module based on a variational autoencoder;

[0062] Specifically, for the aggregated features obtained in step 2) It not only includes background features, but also infrared small target features; in order to further enhance the difference between the background feature distribution and the target feature distribution, the background feature enhancement network based on the variational autoencoder includes an encoder network and a decoder network. The encoder network extracts the multi-scale features in step 2) and the aggregated features obtained by the aggregation network. Mapped to the latent space, the mean μ and deviation logσ of the multidimensional Gaussian distribution are obtained, and the decoder network reconstructs the features from the latent space features Constraints are imposed on latent variables to enhance the low-rank nature of the background while ensuring that the decoder reconstructs the features The latent code is represented as μ+γ×σ, where γ is a random value sampled from a normal distribution; the Kullback-Leibler (KL) divergence between the latent variable and the normal distribution is defined as one of the loss terms of the variational autoencoder:

[0063]

[0064] Among them, μ and σ both have C channels, μ c and σ c is the cth channel;

[0065] The background component is low-rank in the original space. Through the multi-scale feature extraction and aggregation module, the low-rank nature of the background feature can remain relatively stable. Therefore, the latent variable also has similar properties. The tensor nuclear norm is used to impose low-rank constraints on the latent variable to enhance the low-rank characteristics of the background component. Therefore, the background low-rank loss is defined as follows:

[0066]

[0067] where k is a positive constant, ‖·‖ tnn is the tensor nuclear norm;

[0068] Therefore, the loss function of the designed variational autoencoder is for

[0069]

[0070] Wherein, α1 represents a weight coefficient; specifically in the embodiment, k=1, α1=1.

[0071] Furthermore, the shallow feature and deep feature fusion module uses a multi-head gating mechanism to achieve the fusion of shallow features and deep features, specifically including:

[0072] For the shallow features obtained in step 2) Apply a skip connection strategy, expressed as f 1×1 (·) indicates point convolution calculation;

[0073] The multi-head gating mechanism plays an important role in feature fusion, such as Figure 7 As shown, With features Splicing is performed, expressed as Among them, concat(·) represents the concatenation operation of the channel dimension; It refers to the input feature of the i-th feature fusion process, that is, the output feature of the i-1-th feature fusion process (when i is greater than 2), where That is, the decoder network reconstructs the features from the latent space features

[0074] Design a multi-head gating mechanism; for the nth gating head, Perform linear transformation and calculate its gating weight, the expression is Among them, W n and b n is a linear transformation parameter that can be learned; the gating weights corresponding to all heads are concatenated and expressed as in, Represent the gating weights of the 1st, 2nd, and Nth heads respectively; the total gating weight is calculated as Among them, W final and b final is a linear transformation parameter that can be learned; thus, the multi-head gating mechanism is used to realize shallow features and deep features The fusion of can be expressed as ⊙ represents element-wise product;

[0075] In the background reconstruction process, the feature map needs to be upsampled to have the same resolution as the original infrared sequence. By alternately performing upsampling operations and feature fusion operations until the size of the spatiotemporal feature map meets the requirements, the entire feature fusion process can be expressed as

[0076]

[0077]

[0078] in, represents the final fusion feature, is the reconstructed background tensor, Represents the variational autoencoder reconstruction The obtained features, VAE(·) represents the variational autoencoder network operation, Up(·) represents the upsampling operation, Fuse(·) represents the fusion designed using a multi-head gating mechanism, ConvT(·) represents three-dimensional transposed convolution, Conv3D(·) represents three-dimensional convolution, and Sigmoid(·) represents the activation function.

[0079] Step 4: designing an adaptive background reconstruction loss function, wherein the adaptive background reconstruction loss function includes background reconstruction loss and multi-scale structural similarity loss; performing unsupervised training on the multi-scale feature extraction and aggregation network and the background reconstruction network based on the adaptive background reconstruction loss function;

[0080] Background reconstruction loss: Considering that small infrared targets have large reconstruction errors in the early stages of training, by reducing the weights of regions with high reconstruction errors, the reconstruction of small infrared targets can be suppressed and the background reconstruction can be promoted. The reconstruction error tensor ε is expressed as Indicates the background tensor reconstructed by the current epoch during training, and then the weight tensor is obtained The smaller elements in correspond to the small infrared target area, so the background reconstruction loss Defined as

[0081]

[0082] in,‖·‖ F represents the Frobenius norm, ln(·) represents the logarithmic operation;

[0083] Multi-scale structural similarity loss: For the j-th frame infrared image D j and The jth front slice of Through Gaussian smoothing and downsampling operations to generate D j and At each scale, the local mean, variance and covariance of the image are calculated to obtain the local structural similarity (SSIM) and contrast similarity (MCS). j and Taking the kth scale as an example, SSIM j, and MCS j, The calculation formula is

[0084]

[0085]

[0086] Where k = 1, 2, ..., K, μ 1,j,k and μ 2,, D j and The local mean at the kth scale, σ 1,j,k and σ 2,,k D j and The local variance at the kth scale, σ 12,j,k Yes D j and The local covariance at the kth scale, c1 and c2 are positive constants;

[0087] Specifically in the embodiment, c1=0.0001, c2=0.0009;

[0088] Therefore, a multi-scale structural similarity loss with K scales is designed as follows:

[0089]

[0090] Among them, ω j, is the weight coefficient;

[0091] Specifically in the embodiment, K=4, ω j,k (j=1,2,3,4) is 0.0448, 0.2856, 0.3001, 0.2363, 0.1333;

[0092] Combining the KL divergence and background low-rank loss constraints in step 3), the total loss function is

[0093]

[0094] Among them, α1, α2, α3 represent weight coefficients;

[0095] Specifically in the embodiment, α1=1, α2=5, α3=10;

[0096] In network training, the SGD algorithm with momentum of 0.9 and weight decay coefficient of 0.00005 is used for optimization. The initial learning rate is lr = 0.0005, and the cosine learning rate is used to dynamically adjust the learning rate. The expression is The coefficient lrf = 0.01, x is the current training iteration round, E is the total number of training iteration rounds, which is set to 10000, and the batch size is set to 8 during training and testing.

[0097] Step 5: Transform the spatiotemporal tensor of the thermal infrared image sequence to be detected Re-enter the trained network in step 4) to calculate the space-time tensor With the reconstructed background tensor The residual between them is the target tensor. The target tensor is reconstructed into the infrared small target detection result sequence T to realize infrared small target detection.

[0098] Specifically, calculate the space-time tensor With the reconstructed background tensor The residual between them is the target tensor. The target tensor is reconstructed into an infrared small target detection result sequence T as the infrared small target detection result of the original infrared thermal infrared image sequence, realizing infrared small target detection based on spatiotemporal background reconstruction.

[0099] Corresponding to the aforementioned embodiment of an infrared small target detection method based on spatiotemporal background reconstruction, the present invention also provides an embodiment of an infrared small target detection device based on spatiotemporal background reconstruction.

[0100] Figure 2 FIG. 1 is a block diagram of an infrared small target detection device based on spatiotemporal background reconstruction according to an exemplary embodiment. Figure 2 As shown, the device comprises:

[0101] A space-time tensor construction module constructs continuous image frames in an infrared image sequence into a space-time tensor;

[0102] Multi-scale feature extraction and aggregation module, which is designed to include spatiotemporal Swin Transformer structure and multi-scale information extraction and aggregation structure to effectively extract spatiotemporal feature information;

[0103] Background reconstruction module, which is designed to include background feature enhancement, feature fusion and background reconstruction functions based on variational autoencoder to achieve reconstruction of background components;

[0104] Loss function design module, which designs adaptive background reconstruction loss functions for unsupervised learning, including background reconstruction loss and multi-scale structural similarity loss, to improve the performance of the network model;

[0105] The target tensor output module calculates the residual between the original spatiotemporal tensor and the reconstructed background tensor to obtain the target tensor;

[0106] The target detection result output module is used to output the infrared small target detection result map of each frame of infrared image in the infrared image sequence.

[0107] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0108] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic. The various modules in the device are a kind of logical function division. There may be other division methods in actual implementation. For example, multiple modules can be combined or integrated into another unit. Another point is that the connection between the modules shown or discussed can be a communication connection through some interfaces, which can be electrical or other forms. Some or all of the modules can be selected according to actual needs to achieve the purpose of the present application. Ordinary technicians in this field can understand and implement it without paying creative labor. The following takes the disclosed real thermal infrared image sequence as an example to illustrate the specific implementation method to reflect the technical effect of the present invention. The specific steps in the embodiment will not be repeated.

[0109] Example

[0110] The accompanying drawings shown in the embodiments of the present invention can make the purpose, technical solutions and advantages of the present invention more clearly described. It should be noted that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention. All equivalent replacements, improvements, etc. made within the method ideas and principles provided by the present invention should be included in the protection scope of the present invention.

[0111] In this embodiment, the effectiveness of the thermal infrared small target detection algorithm will be verified through a public infrared image sequence. The public infrared image sequence contains 1599 frames of infrared images, each of which has a size of 256×256. The infrared image sequence uses the sky-city building as the background, with strong interference structures in the local area. The slow-flying small target is affected by strong clutter and noise. The typical image frame of the sequence is as follows: Figure 8 In order to more accurately and objectively evaluate the superiority of the proposed infrared small target detection method based on spatiotemporal background reconstruction, the latest thermal infrared small target detection methods based on traditional and deep learning are selected, and the target detection result images of the method in the instance frames of the infrared image sequence are qualitatively compared and analyzed; from a quantitative perspective, the detection performance of the algorithm is evaluated by the 3D-ROC evaluation index system, which includes the index AUC for evaluating target detection capabilities. (D,τ) , AUC TD ; AUC is an indicator used to evaluate background suppression ability (F , τ) , AUC BS , AUC SNPR ; AUC, an indicator used to evaluate the comprehensive ability of the detector (D ,F),AUC TDBS , AUC ODP ;

[0112] In order to more objectively verify the effectiveness of the method of the present invention, five latest thermal infrared small target detection methods are selected for comparison with the method of the present invention, including MDWCM, RCTVW, STT-TRNR, DNANet and AGPCNet. Table 1 shows the indicators of the 3D-ROC evaluation system for infrared dim small target detection results of the thermal infrared image sequence using the comparison method and the method of the present invention. The bold values ​​and underlined values ​​represent the corresponding best performance and second best performance, respectively.

[0113] MDWCM is derived from Lu, Ruitao, et al. "Robust infrared small target detection via multidirectional derivative-based weighted contrast measure." IEEE Geoscience and Remote Sensing Letters 19(2020):1-5.

[0114] RCTVW originates from Liu, Ting, et al. "Representative coefficient total variation for efficient infrared small target detection." IEEE Transactions onGeoscience and Remote Sensing (2023).

[0115] STT-TRNR is derived from Yi, Haiyang, et al. "Spatial-temporal tensor ring normregularization for infrared small target detection." IEEE Geoscience andRemote Sensing Letters 20(2023):1-5.

[0116] DNANet is derived from Li, Boyang, et al. "Dense nested attention network for infrared small target detection." IEEE Transactions on Image Processing 32(2022):1745-1758.

[0117] AGPCNet is derived from Zhang, Tianfang, et al. "AGPCNet: Attention-guided pyramidcontext networks for infrared small target detection." arxiv preprint arxiv:2111.03580 (2021).

[0118] Fig. 9 The original image of the infrared image example frame and the thermal infrared small target detection result image detected by the method of the present invention, MDWCM, RCTVW, STT-TRNR, DNANet and AGPCNet. It can be observed from the intuitive infrared weak small target detection result image that the method effectively reconstructs the background component, and the target result image is rarely affected by the background and noise. Fig. 9 It can be seen from the infrared weak target detection result image that RCTVW has a target loss problem, AGPCNet has weak target saliency, and there is clutter interference in the detection results of STT-TRNR and MDWCM, such as Fig. 9 As shown in the area marked by the blue dotted box, DNANet's background suppression ability is insufficient. In contrast, the infrared small target detection method based on spatiotemporal background reconstruction proposed in the present invention can enhance the saliency of small targets and suppress background, noise and other components. In the field of target detection, target enhancement ability and background suppression ability are a pair of contradictory indicators. Improving the target enhancement ability will lead to a weakening of the background suppression ability, and vice versa. Therefore, it is necessary to comprehensively evaluate the performance of the infrared weak small target detection method. According to the indicator results based on the 3D-ROC evaluation system shown in Table 1, the method proposed in the present invention achieves the best performance in comprehensive ability, followed by RCTVW; although MDWCM has excellent target detection ability, its background suppression ability is the weakest, resulting in the worst comprehensive ability. Based on the above qualitative and quantitative analysis, the infrared small target detection method based on spatiotemporal background reconstruction proposed in the present invention has superior target detection ability, background suppression ability and comprehensive effectiveness.

[0119] Table 1 Quantitative indicators of the detection results of thermal infrared image example sequences using MDWCM, RCTVW, STT-TRNR, DNANet, AGPCNet and the method of the present invention

[0120]

[0121]

[0122] The accompanying drawings shown in the embodiments of the present invention can make the purpose, technical solutions and advantages of the present invention more clearly described. It should be noted that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention. All equivalent replacements, improvements, etc. made within the method ideas and principles provided by the present invention should be included in the protection scope of the present invention.

Claims

1. A method for detecting small infrared targets based on spatiotemporal background reconstruction, characterized in that: The steps include: Step 1): For the thermal infrared image sequence to be detected, its continuous image frames are constructed as a spatiotemporal tensor Step 2): design a multi-scale feature extraction and aggregation network for extracting spatiotemporal feature information of the spatiotemporal tensor, wherein the multi-scale feature extraction and aggregation network adopts a pyramid structure, the pyramid structure includes multiple feature extraction layers and local feature information extraction layers, and each feature extraction layer includes several spatiotemporal Swin Transformer modules; Step 3): Design a background reconstruction network based on a variational autoencoder to reconstruct the background tensor of the spatiotemporal tensor The background reconstruction network includes a background feature enhancement module, a feature fusion module and a background reconstruction module based on a variational autoencoder; Step 4): designing an adaptive background reconstruction loss function, wherein the adaptive background reconstruction loss function includes background reconstruction loss and multi-scale structural similarity loss; performing unsupervised training on the multi-scale feature extraction and aggregation network and the background reconstruction network based on the adaptive background reconstruction loss function; Step 5): The spatiotemporal tensor of the thermal infrared image sequence to be detected Re-enter the trained network in step 4) to calculate the space-time tensor With the reconstructed background tensor The residual between them is the target tensor. The target tensor is reconstructed into the infrared small target detection result sequence T to realize infrared small target detection.

2. The infrared small target detection method based on spatiotemporal background reconstruction according to claim 1 is characterized in that: The step 1) comprises: From the original thermal infrared image sequence, continuous image frames are obtained and stacked in order according to the time sequence of the image frames to construct a space-time tensor. F, H, W, C represent the space-time tensor The number of frames, height, width, and number of channels.

3. The infrared small target detection method based on spatiotemporal background reconstruction according to claim 1 is characterized in that: The spatiotemporal Swin Transformer module includes multiple spatiotemporal Swin Transformer blocks, each of which includes two layers of multi-head self-attention calculation layers connected in series, wherein the first layer of multi-head self-attention calculation layer uses a regular local window to split the input and performs multi-head self-attention calculation on each local window obtained by the segmentation; the second layer of multi-head self-attention calculation layer first uses a moving window mechanism to shift the local window to enhance the information exchange between the local windows, and then uses the obtained local window to split the output of the first layer of multi-head self-attention calculation layer and performs multi-head self-attention calculation on each local window obtained by the segmentation.

4. The infrared small target detection method based on spatiotemporal background reconstruction according to claim 3 is characterized in that: The calculation process of the spatiotemporal Swin Transformer block is specifically as follows: Assume that the size of the input feature is F in ×H in ×W in , the number of channels is 1, and the size of the moving window is P×M×M; the two layers of multi-head self-attention calculation layers are marked as the lth layer and the l+1th layer; the lth layer uses a regular local window to segment the input features and obtain A local window is used to perform spatiotemporal multi-head self-attention calculations. Indicates the rounding up operation; for the l+1th layer, the window partition moves along the time, height and width directions respectively Units, calculate the spatiotemporal multi-head self-attention mechanism calculation on the obtained local window; The multi-head self-attention mechanism of the two-layer multi-head self-attention calculation layer is: First, the query in the local window is calculated and each key The similarity between them is then re-weighted The self-attention calculation process SA(·) is expressed as in, represents the relative position encoding in time and space, and d represents and The number of channels, As a scaling factor, express The transpose of , softmax(·) represents the activation function; Through linear transformation get, represents the weight tensor, Represents input features; For the nth spatiotemporal Swin Transformer module, the process of two-layer multi-head self-attention calculation is expressed as follows: in, represents the features obtained by calculating self-attention using a regular local window in layer l, represents the final feature obtained in the lth layer, It represents the feature obtained by calculating the self-attention using the local window of the moving window mechanism in the l+1th layer, represents the final feature obtained after the spatiotemporal Swin Transformer module, LN(·) represents layer normalization calculation, BN(·) represents batch normalization calculation, MLP(·) represents full connection calculation, STW-MSA(·) represents the spatiotemporal multi-head self-attention calculation with regular window of the lth layer, and STSW-MSA(·) represents the spatiotemporal multi-head self-attention calculation with moving window mechanism of the l+1th layer.

5. The infrared small target detection method based on spatiotemporal background reconstruction according to claim 1 is characterized in that: The multiple feature extraction layers include a top layer, a middle layer and a bottom layer, and the top layer, the middle layer and the bottom layer are respectively composed of 1, 2, and 3 spatiotemporal SwinTransformer modules in cascade, and the top layer, the middle layer and the bottom layer respectively obtain multi-scale feature maps At the same time, shallow features are obtained ; The local feature information extraction layer combines 3D convolution, batch normalization and activation function to extract local feature information from the spatiotemporal tensor. Extract enhanced local feature information Respectively and After downsampling and upsampling, we get and Then point-by-point convolution is performed to achieve feature integration; the multi-scale feature aggregation process is expressed as follows: in is the aggregated feature, g(·)=ReLU(BN(Conv3D(·))) represents local feature extraction, Indicates that i spatiotemporal Swin Transformer modules are cascaded for feature extraction, Down(·) indicates downsampling calculation, Up(·) indicates upsampling calculation, and f 1×1 (·) indicates point convolution calculation.

6. The infrared small target detection method based on spatiotemporal background reconstruction according to claim 5 is characterized in that: The step 3) is specifically as follows: The background feature enhancement network based on variational autoencoder includes an encoder network and a decoder network. The encoder network extracts the multi-scale features in step 2) and the aggregated features obtained by the aggregation network. Mapped to the latent space, the mean μ and deviation logσ of the multidimensional Gaussian distribution are obtained, and the decoder network reconstructs the features from the latent space features Constraints are imposed on latent variables to enhance the low-rank nature of the background while ensuring that the decoder reconstructs the features accuracy; The Kullback-Leibler divergence between the latent variable and the normal distribution is defined as one of the loss terms of the variational autoencoder: Among them, μ and σ both have C channels, μ c and σ c represents the cth channel; The tensor nuclear norm is used to impose low-rank constraints on latent variables to enhance the low-rank characteristics of the background component; thus, the background low-rank loss is defined as follows: where k is a positive constant, ‖·‖ tnn is the tensor nuclear norm.

7. The infrared small target detection method based on spatiotemporal background reconstruction according to claim 6 is characterized in that: The shallow feature and deep feature fusion module uses a multi-head gating mechanism to achieve the fusion of shallow features and deep features, specifically including: For the shallow features obtained in step 2) Apply a skip connection strategy, expressed as f 1×1 (·) indicates point convolution calculation; Will With features Splicing is performed, expressed as Among them, concat(·) represents the concatenation operation of the channel dimension; Refers to the input features of the i-th feature fusion process, where That is, the decoder network reconstructs the features from the latent space features Design a multi-head gating mechanism; for the nth gating head, Perform linear transformation and calculate its gating weight, the expression is Among them, W n and b n are the linear transformation parameters that can be learned; The gating weights corresponding to all heads are concatenated and expressed as in, Represent the gating weights of the 1st, 2nd, and Nth heads respectively; the total gating weight is calculated as Among them, W final and b final is a linear transformation parameter that can be learned; thus, the multi-head gating mechanism is used to realize shallow features and deep features The fusion of Among them, ⊙ represents the element-wise product; In the background reconstruction process, the feature map needs to be upsampled to have the same resolution as the original infrared sequence. By alternately performing upsampling operations and feature fusion operations until the resolution of the spatiotemporal feature map meets the requirements, the entire feature fusion process is expressed as in, represents the final fusion feature, is the reconstructed background tensor, Represents the variational autoencoder reconstruction The obtained features, VAE(·) represents the variational autoencoder network operation, Up(·) represents the upsampling operation, Fuse(·) represents the fusion designed using a multi-head gating mechanism, ConvT(·) represents three-dimensional transposed convolution, Conv3D(·) represents three-dimensional convolution, and Sigmoid(·) represents the activation function.

8. The infrared small target detection method based on spatiotemporal background reconstruction according to claim 7 is characterized in that: The step 4) is specifically as follows: Background reconstruction loss Defined as: in,‖·‖ F represents the Frobenius norm, ln(·) represents the logarithmic operation; Represents the background tensor reconstructed in the current training iteration during the training process. is the weight tensor corresponding to the reconstruction error tensor ε, Multi-scale structural similarity loss: For the j-th frame infrared image D j and The jth front slice of Through Gaussian smoothing and downsampling operations to generate D j and At each scale, the local mean, variance and covariance of the image are calculated to obtain the local structural similarity SSIM and contrast similarity MCS; for D j and The k-th scale, SSIM j,k and MCS j,k The calculation formula is Among them, μ 1,j,k and μ 2,j,k D j and The local mean at the kth scale, σ 1,j,k and σ 2,j,k D j and The local variance at the kth scale, σ 12,j,k Yes D j and The local covariance at the kth scale, c1 and c2 are positive constants; thus, a multi-scale structural similarity loss with K scales is designed. as follows: Among them, ω j,k is the weight coefficient; Combining the KL divergence and background low-rank loss constraints in step 3), the total loss function of adaptive background reconstruction is: Among them, α1, α2, and α3 represent weight coefficients.

9. An infrared small target detection device based on spatiotemporal background reconstruction for implementing the method of claim 1, characterized in that: include: A space-time tensor construction module constructs continuous image frames in the infrared image sequence to be detected into a space-time tensor; A multi-scale feature extraction and aggregation network, used to extract the spatiotemporal feature information of the spatiotemporal tensor, wherein the multi-scale feature extraction and aggregation network adopts a pyramid structure, which includes a plurality of feature extraction layers and a local feature information extraction layer, and each feature extraction layer includes a plurality of spatiotemporal Swin Transformer modules; Background reconstruction network, used to reconstruct the background tensor of the spatiotemporal tensor The background reconstruction network includes a background feature enhancement module, a feature fusion module and a background reconstruction module based on a variational autoencoder; Loss function design module, which designs adaptive background reconstruction loss functions for unsupervised learning, including background reconstruction loss and multi-scale structural similarity loss, to improve the performance of the network model; The target tensor output module calculates the residual between the original spatiotemporal tensor to be detected and the reconstructed background tensor to obtain the target tensor; The target detection result output module is used to output the infrared small target detection result map of each frame of infrared image in the infrared image sequence.

Images