An infrared small target detection method and device based on a space-time information completion model

Abstract

The application discloses an infrared small target detection method and device based on a space-time information completion model. The method comprises the following steps: (1) constructing a three-dimensional space-time tensor by using a target detection result prior and an infrared image to be detected; (2) suppressing low-frequency signals of the three-dimensional space-time tensor, and establishing a tensor completion model based on low-rank sparsity; (3) defining a multi-mode tensor kernel norm of a background tensor; (4) introducing a sparsity reweighting strategy, and establishing a low-rank sparse tensor completion model based on space-time information; (5) solving an infrared small target detection model based on the space-time information completion model; and (6) designing a space-time information fusion strategy, fusing information of a target tensor, obtaining an infrared small target detection result of an infrared image sequence to be detected, and realizing infrared small target detection based on the space-time information completion model. The low-rank sparse tensor completion model based on space-time information and the solving algorithm can effectively improve the detection performance of the infrared small target.

Description

Technical Field

[0001] This invention relates to the field of image processing, and in particular to an infrared small target detection method and apparatus based on a spatiotemporal information completion model. Background Technology

[0002] The primary goal of infrared small target detection technology is to achieve accurate and efficient small target detection in infrared images under complex scenes. Infrared images typically possess characteristics different from visible light images, primarily reflecting the thermal distribution of targets. Therefore, infrared images have a unique advantage in detecting concealed or nocturnal targets at night or under adverse weather conditions. Infrared small target detection tasks generally face the following challenges: Small targets typically refer to small, weak targets, such as drones and vehicles, and the temperature difference between small targets and the surrounding environment is small, making them difficult to detect and locate accurately in infrared images; the background of infrared images can be affected by various factors, including surface temperature distribution, weather conditions, and environmental interference, making accurate detection of small targets from complex backgrounds even more difficult; furthermore, infrared small target detection needs to fully consider spatiotemporal information, as the target's trajectory and temporal changes are crucial for separating small targets from the background.

[0003] With the development of tensor theory, tensor analysis has been widely used for infrared small target detection. The main drawbacks of current tensor analysis-based infrared small target detection methods include:

[0004] (1) The constructed tensor structure cannot preserve the spatiotemporal consistency and spatial correlation within the infrared image sequence well. The tensor structure based on a single frame image ignores the temporal information in the infrared image sequence. The tensor structure based on image blocks destroys the spatial structure of the image to some extent, which may cause data distribution deviation between real image data and tensor data.

[0005] (2) The distribution assumptions of target, background and noise components fail to accurately capture the distribution characteristics of the corresponding components, which will affect the performance of the established tensor analysis-based model and reduce the effectiveness of infrared small target detection results;

[0006] (3) Large-scale tensor structures will increase the computational complexity of infrared small target detection models based on tensor analysis, affecting the timeliness of small target detection methods.

[0007] Therefore, it is necessary to improve the existing infrared small target detection model based on tensor analysis, and to study aspects such as spatiotemporal tensor construction and detection model establishment, so as to further improve the overall performance of infrared small target detection technology. Summary of the Invention

[0008] To address the shortcomings of existing technologies, the present invention aims to provide an infrared small target detection method and apparatus based on a spatiotemporal information completion model. This method fully utilizes the spatiotemporal information carried by infrared image sequences, innovates the tensor construction approach, and uses prior target detection results to assist in the detection of infrared image frames for small infrared targets. The prior target detection results and the images to be detected are constructed into a three-dimensional spatiotemporal tensor according to frame order. Based on this, a multi-mode expansion kernel norm of the background tensor is defined through multi-mode expansion, achieving low-rank estimation of the infrared image background. Simultaneously, a low-rank sparse tensor completion model based on spatiotemporal information is established, and an optimized solution framework based on the ADMM algorithm is used to improve the tensor completion model. The method solves the model, infers target information, and fills in the missing parts of the target tensor to obtain a complete target tensor. In addition, a spatiotemporal information fusion strategy is introduced, which uses some information of the target tensor obtained from the current three-dimensional spatiotemporal tensor as a priori information for the target detection result to construct the next three-dimensional spatiotemporal tensor for small target detection. Subsequently, the target tensor obtained by the tensor completion model and the target tensor obtained by the current three-dimensional spatiotemporal tensor are fused with image information to finally obtain the accurate small target detection result of the infrared image sequence to be detected. The embodiments effectively verify that the method proposed in this invention has excellent target detection capability, background suppression capability, and comprehensive detection performance.

[0009] To achieve the above objectives, the present invention provides the following technical solution:

[0010] This invention discloses an infrared small target detection method based on a spatiotemporal information completion model, characterized by comprising the following steps:

[0011] Step 1): Utilize prior knowledge from target detection results and the infrared image to be detected The carried temporal and spatial information is used to construct a three-dimensional spatiotemporal tensor from the prior target detection result and the infrared image to be detected in frame order. , This represents the sub-tensor corresponding to the prior detection result of the target. This represents the sub-tensor corresponding to the infrared image to be detected. This represents the index value of the constructed three-dimensional spacetime tensor. This indicates an operation that merges tensors along the third dimension.

[0012] Step 2): Utilizing the characteristics of background, target, and noise, from the three-dimensional spacetime tensor Establish a tensor initialization and completion model based on low-rank sparsity to suppress The low-frequency signal is used to obtain the high-frequency projection transform tensor. ,Depend on Establish a tensor completion model based on low-rank sparsity;

[0013] Step 3): Based on the background tensor The low-rank property of the background tensor Perform multi-mode expansion and define the background tensor. Multimodal tensor norm ;

[0014] Step 4): Combining Steps 1) to 3), a sparsity reweighting strategy is introduced into the low-rank sparse tensor completion model to establish a low-rank sparse tensor completion model based on spatiotemporal information.

[0015] Step 5): Design an optimization solution framework based on the ADMM algorithm to solve the low-rank sparse tensor completion model based on spatiotemporal information, and obtain the high-frequency projection transformation tensor. target tensor ;

[0016] Step 6): Design a spatiotemporal information fusion strategy, utilizing the current high-frequency projection transformation tensor The corresponding target tensor and the subsequent high-frequency projection transformation tensor The corresponding target tensor By analyzing image frames The corresponding target detection results are fused to obtain image frames. Infrared small target detection results This allows for the detection of small infrared targets in each frame of the infrared image sequence to be detected, thus achieving infrared small target detection based on a spatiotemporal information completion model.

[0017] The present invention also discloses an infrared small target detection device based on a spatiotemporal information completion model for implementing the method, comprising:

[0018] The three-dimensional spatiotemporal tensor construction module uses the prior target detection results and the infrared image to be detected to construct a three-dimensional spatiotemporal tensor, laying the data foundation for the spatiotemporal information completion model;

[0019] The module for establishing a low-rank sparse tensor completion model suppresses low-frequency signals in the three-dimensional spatiotemporal tensor and establishes a low-rank sparse tensor completion model based on the characteristics of background, target and noise.

[0020] The low-rank background estimation norm design module performs multi-modal expansion on the background tensor and defines the multi-modal tensor kernel norm of the background tensor.

[0021] The module for establishing a low-rank sparse tensor completion model based on spatiotemporal information consists of a three-dimensional spatiotemporal tensor and a low-rank background estimation norm. A sparsity reweighting strategy is introduced into the low-rank sparse tensor completion model to establish a low-rank sparse tensor completion model based on spatiotemporal information.

[0022] The optimization model module is designed, and an optimization solution framework based on the ADMM algorithm is designed to solve the low-rank sparse tensor completion model based on spatiotemporal information to obtain the target tensor.

[0023] The spatiotemporal information fusion strategy design module fuses the target tensor of the current spatiotemporal tensor and the target tensor of the next spatiotemporal tensor to obtain the small target detection results of the corresponding infrared image.

[0024] The target detection result output module is used to output the infrared small target detection result map for each frame of infrared image.

[0025] Compared with the prior art, the beneficial effects of the present invention include:

[0026] 1) This invention proposes a novel tensor construction method that constructs a three-dimensional spatiotemporal tensor from the prior target detection result and the infrared image frame to be detected. The prior target detection result is used to assist the detection of small infrared targets in the image frame. The prior information contained in this tensor construction method helps the model pay more attention to possible target regions, thereby reducing false detections and false negatives and improving the accuracy and efficiency of detection. On this basis, a multi-mode tensor kernel norm of the background tensor is proposed, which helps to make low-rank estimation of background components and improve the robustness and generalization performance of background modeling. At the same time, a low-rank sparse tensor completion model based on spatiotemporal information is established. By comprehensively utilizing the existing prior detection result and the temporal-spatial information of the image sequence, the target information is inferred and the missing parts of the target tensor are filled in, finally obtaining a complete target detection result tensor. This model can more comprehensively and accurately track the state of the target in different times and spaces and improve the detection performance of small targets.

[0027] 2) This invention designs a spatiotemporal information fusion strategy, which fuses the spatiotemporal information of the preceding and following image frames to improve detection performance. Specifically, it uses partial information of the target tensor corresponding to the current three-dimensional spatiotemporal tensor as a priori information for target detection, and constructs the next three-dimensional spatiotemporal tensor for small target detection. The target tensor obtained by the next three-dimensional spatiotemporal tensor through the tensor completion model is fused with the target tensor obtained by the current three-dimensional spatiotemporal tensor through the tensor completion model to obtain the final small target detection result of the infrared image sequence to be detected, thereby improving the overall performance of infrared small target detection. Attached Figure Description

[0028] Figure 1This is a schematic diagram illustrating the process of constructing a three-dimensional spacetime tensor from the target detection prior frame and the image frame to be detected in this invention;

[0029] Figure 2 This is a schematic diagram of the spatiotemporal information fusion strategy designed in this invention;

[0030] Figure 3 This is a schematic diagram of the infrared small target detection device based on the spatiotemporal information completion model in this invention;

[0031] Figure 4 Example frame images of a thermal infrared image sequence used for experimental testing;

[0032] Figure 5 The thermal infrared small target detection results are obtained by applying the method proposed in this invention to an example frame of a thermal infrared image sequence.

[0033] Figure 6 The original image of a thermal infrared image example frame, the thermal infrared small target detection result image after rcDoGF and GCM detection, and the corresponding three-dimensional grayscale image are shown.

[0034] Figure 7 The image shows the results of small thermal infrared target detection and the corresponding 3D grayscale image of a sample frame of thermal infrared image after IPCE, RCTVW and STT-TRNR detection.

[0035] Figure 8 The image shows the thermal infrared small target detection results and corresponding 3D grayscale image of a sample thermal infrared image frame detected by STTM-TSNM, DGRL and AGPCNet.

[0036] Figure 9 The image shows the results of thermal infrared small target detection using IAANet and the method of this invention, along with the corresponding 3D grayscale image, for an example frame of a thermal infrared image. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of this invention clearer, the invention is described in detail below with reference to specific embodiments. Specific embodiments are described below to simplify the invention. However, it should be understood that the invention is not limited to the described embodiments, and various modifications are possible without departing from the basic principles; these equivalent forms also fall within the scope defined by the appended claims.

[0038] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0039] The basic steps of the infrared small target detection method based on the spatiotemporal information completion model of the present invention in this embodiment mainly include:

[0040] Step 1: Utilize prior knowledge from target detection results and the infrared image to be detected The carried temporal and spatial information is used to construct a three-dimensional spatiotemporal tensor from the prior target detection result and the infrared image to be detected in frame order. ;

[0041] Specifically, such as Figure 1 As shown, by utilizing the temporal and spatial information of thermal infrared images, the current target image is identified as the first... Frame infrared image Compared to its previous Frame infrared image Prior to target detection results and subsequent tests to be conducted Frame infrared image Stacked sequentially according to the time order of the image frames. Prior to frame target detection results and Frame of infrared image to be detected Constructed as a three-dimensional spacetime tensor , Indicates prior knowledge based on target detection results. The corresponding subtensor, Indicates the infrared image to be detected The corresponding subtensor, This represents the index value of the constructed three-dimensional spacetime tensor. This represents the operation of merging tensors along the third dimension; a three-dimensional spacetime tensor. The size is , express The width of the front slice, express The height of the front slice, express The thickness, and ;in, Indicates that it is located at The first frame image before, Indicates that it is located at The second frame of the infrared image, Indicates that it is located at The first Frame infrared image, Indicates that it is located at The first frame of the infrared image after that, Indicates that it is located at The second frame of the infrared image, Indicates that it is located at The next Frame infrared image;

[0042] Specifically, in the embodiments... The three-dimensional spacetime tensor constructed by taking 3 as its value. The size is 256 256 7.

[0043] Step 2: Utilizing the characteristics of background, target, and noise, from the three-dimensional spacetime tensor Establish a tensor initialization and completion model based on low-rank sparsity to suppress The low-frequency signal is used to obtain the high-frequency projection transform tensor. ,Depend on Establish a tensor completion model based on low-rank sparsity;

[0044] Specifically, the three-dimensional spacetime tensor constructed in step 1 This preserves the low-rank properties of the background component, the sparsity of the target component, and the characteristics of the noise component, thereby enhancing the three-dimensional spatiotemporal tensor. Modeled as a low-rank background tensor sparse target tensor and noise component tensor A linear combination of these elements is used to establish a tensor initial completion model based on low-rank sparseness, as shown in equation (1):

[0045] (1)

[0046] in, express Norm, and Represents positive numbers. Represents the rank estimator operator;

[0047] Considering the impact of low-frequency signals in complex scenes on the detection of small infrared targets, the three-dimensional spatiotemporal tensor is... Projecting onto the frequency domain and suppressing low-frequency signals is achieved through the following process: via Fourier transform. Three-dimensional spacetime tensor neutron tensor Project each front slice into the frequency domain; design a Gaussian filter. By adjusting variance The process involves suppressing low-frequency signals and enhancing high-frequency signals in each frontal slice; finally, the filtered data is projected into the spatial domain to obtain the high-frequency projection transform tensor. For high-frequency projection transformation tensors The tensor initial completion model (1) based on low-rank sparse tensor is improved into a tensor completion model based on low-rank sparse tensor, as shown in formula (2):

[0048] (2)

[0049] in, Describing the optimal approximation norm Norm;

[0050] Specifically, in the embodiments... variance Set it to 40.

[0051] Step 3: Based on the background tensor The low-rank property of the background tensor Perform multi-mode expansion and define the background tensor. Multimodal tensor norm ;

[0052] Specifically, for low-rank background tensors background tensor Perform multi-modal expansion to obtain three background tensors with different expansion modes. , and , respectively represented as , , , This indicates that the tensor will be processed according to the pattern - Expanding operators; , and All maintain low-rank properties, for , and The background tensor is obtained by weighting the tensor norms. Multimodal tensor norm As shown in formula (3):

[0053] (3)

[0054] in, Indicates the weighting coefficient. Represents the tensor nuclear norm;

[0055] Specifically, in the embodiments... All are taken as .

[0056] Step 4: Combining Steps 1 to 3, introduce a sparsity reweighting strategy into the low-rank sparse tensor completion model to establish a low-rank sparse tensor completion model based on spatiotemporal information.

[0057] Specifically, solving the tensor completion model (2) in step 2 is an optimization iterative process. To accelerate the iterative solution speed of the model, a sparsity reweighting strategy is introduced into the tensor completion model (2). The reweighted weight tensors From the current iteration round The obtained target tensor If confirmed, then the next iteration round... In the middle, the reweighted weight tensor of the target As shown in formula (4):

[0058] (4)

[0059] in, Represents positive numbers. This represents the operation of calculating the reciprocal of an element. This indicates the operation of calculating the absolute value;

[0060] Combining the tensor completion model (2) in step 2) and the background tensor in step 3). The multimodal tensor norm is used to establish a low-rank sparse tensor completion model based on spatiotemporal information, as shown in formula (5):

[0061] (5)

[0062] in, It represents the Hadamardi (or Hadama) stack.

[0063] Step 5: Design an optimization solution framework based on the ADMM algorithm to solve the low-rank sparse tensor completion model based on spatiotemporal information, and obtain the high-frequency projection transformation tensor. target tensor ;

[0064] Specifically, the ADMM algorithm is used to solve the low-rank sparse tensor completion model based on spatiotemporal information (5). To ensure that each variable to be solved in (5) is solvable, auxiliary variables are introduced respectively. Formula (6) is obtained as follows:

[0065] (6)

[0066] The corresponding augmented Lagrangian function is

[0067] (7)

[0068] in, For Lagrange multipliers, This is a penalty term; each variable is solved through alternating iterations, and the subproblems corresponding to each variable are as follows:

[0069] 1) The iterative formula is

[0070] (8)

[0071] in, Indicates the number of iterations. Indicates the first Round iterations obtained , This represents the singular value decomposition of the tensor. For singular value tensors, and The unitary tensor after singular value decomposition of the tensor. The transpose operator for tensors. To represent the singular value contraction operator, we have: “ "For tensor product operators, ;

[0072] 2) The iterative formula is

[0073] (9)

[0074] in, Indicates the first Round iterations obtained ;

[0075] 3) The iterative formula is

[0076] (10)

[0077] in, Indicates the first Round iterations obtained ;

[0078] 4) The iterative formula is

[0079] (11)

[0080] The equivalent formula of formula (11) is:

[0081] (12)

[0082] in, Indicates the first Round iterations obtained , This indicates that the tensor will be processed according to the pattern - Folding operators;

[0083] 5) The iterative formula is

[0084] (13)

[0085] in, Indicates the first Round iterations obtained , For soft threshold shrinkage operator, , For symbolic functions, Soft threshold;

[0086] 6) The iterative formula is

[0087] (14)

[0088] in, Indicates the first Round iterations obtained ;

[0089] 7) The iterative formula is

[0090] (15)

[0091] in, Indicates the first Round iterations obtained ;

[0092] 8) Lagrange multipliers The iterative formula is

[0093] (16)

[0094] (17)

[0095] (18)

[0096] (19)

[0097] in, They represent the first Round iterations obtained ;

[0098] 9) Penalty Items The iterative formula is

[0099] (20)

[0100] in, Indicates the first Round iterations obtained , Indicates the update coefficients. express The maximum value;

[0101] The iteration stopping condition is: 1) When the relative error Less than the error When the iteration terminates; 2) when the target tensor is reweighted. When the norm no longer changes, stop iterating; 3) When the norm obtained in two consecutive iterations and satisfy 4) Stop iteration when the results of two consecutive iterations are obtained. and satisfy Stop iterating when the time comes;

[0102] This completes the iterative solution of the low-rank sparse tensor completion model (5) based on spatiotemporal information in step 5, yielding... The corresponding target tensor ;

[0103] Specifically, in the embodiments... , , , , , , , , ;

[0104] Step 6: Design a spatiotemporal information fusion strategy, utilizing the current high-frequency projection transformation tensor The corresponding target tensor and the subsequent high-frequency projection transformation tensor The corresponding target tensor By analyzing image frames The corresponding target detection results are fused to obtain image frames. Infrared small target detection results This allows us to obtain the infrared small target detection results for each frame in the infrared image sequence to be detected, thus realizing infrared small target detection based on the spatiotemporal information completion model.

[0105] like Figure 2 As shown, specifically, in the target tensor In the middle, the first Zhang Zhidi Zhang frontal slice As an infrared image The test results prior ; combination and the image to be detected The next three-dimensional spacetime tensor is constructed by stacking the frames sequentially. , Infrared image The corresponding three-dimensional spacetime tensor index; By projecting the vector into the frequency domain and suppressing low-frequency signals, a high-frequency projection transform tensor is obtained. Therefore, based on step 5, the low-rank sparse tensor completion model based on spatiotemporal information is solved, as shown in formula (21):

[0106] (twenty one)

[0107] Solving for the next high-frequency projection transformation tensor yields the solution. The corresponding target tensor Extract the target tensor The Middle Zhang Zhidi Zhang's frontal slice is used to construct the next three-dimensional spacetime tensor. Target detection priors; extraction The Middle Zhang Zhidi Zhang frontal slice and with The Zhang Zhidi Zhang frontal slice The images are fused together to obtain the image to be detected. Target detection result sequence:

[0108] (twenty two)

[0109] And there are

[0110] (twenty three)

[0111] in," "Indicates a fusion operation, Indicates the result after fusion The target detection results;

[0112] “ The fusion operation can be selected from matrix element-wise addition, matrix element-wise multiplication, matrix element-wise division, etc. Specifically, in this embodiment, the fusion operation... "Take the corresponding elements of the matrix and add them together."

[0113] For the infrared image sequence to be detected, repeat the process of formulas (21) to (23) in step 6 to obtain the target detection result corresponding to each frame image; for the first construction of the three-dimensional space-time tensor, the prior of the target detection result used in this embodiment is calculated by 4DST-BTMD, which comes from Luo, Yuan, et al. "4DST-BTMD: An Infrared Small Target Detection Method based on 4D Data-Sphered Space." IEEE Transactions on Geoscience and Remote Sensing (2023).; the prior of the target detection result for the first construction of the three-dimensional space-time tensor can also be obtained by other methods, which are not limited in this invention.

[0114] The infrared image to be detected is obtained from formula (23). The corresponding infrared small target detection result image The infrared small target detection result image is obtained from each frame of infrared image, and finally the infrared small target detection based on the spatiotemporal information completion model is realized.

[0115] Corresponding to the aforementioned embodiment of an infrared small target detection method based on a spatiotemporal information completion model, the present invention also provides an embodiment of an infrared small target detection device based on a spatiotemporal information completion model.

[0116] Figure 3 This is a block diagram illustrating an infrared small target detection device based on posterior information according to an exemplary embodiment, such as... Figure 3 As shown, the device includes:

[0117] The three-dimensional spatiotemporal tensor construction module uses the prior target detection results and the infrared image to be detected to construct a three-dimensional spatiotemporal tensor, laying the data foundation for the spatiotemporal information completion model;

[0118] The module for establishing a low-rank sparse tensor completion model suppresses low-frequency signals in the three-dimensional spatiotemporal tensor and establishes a low-rank sparse tensor completion model based on the characteristics of background, target and noise.

[0119] The low-rank background estimation norm design module performs multi-modal expansion on the background tensor and defines the multi-modal tensor kernel norm of the background tensor.

[0120] The module for establishing a low-rank sparse tensor completion model based on spatiotemporal information consists of a three-dimensional spatiotemporal tensor and a low-rank background estimation norm. A sparsity reweighting strategy is introduced into the low-rank sparse tensor completion model to establish a low-rank sparse tensor completion model based on spatiotemporal information.

[0121] The optimization model module is designed, and an optimization solution framework based on the ADMM algorithm is designed to solve the low-rank sparse tensor completion model based on spatiotemporal information to obtain the target tensor.

[0122] The spatiotemporal information fusion strategy design module fuses the target tensor of the current spatiotemporal tensor and the target tensor of the next spatiotemporal tensor to obtain the small target detection results of the corresponding infrared image.

[0123] The target detection result output module is used to output the infrared small target detection result map for each frame of infrared image.

[0124] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0125] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the description of the method embodiments. The device embodiments described above are merely illustrative; the various modules in the device represent a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another unit. Furthermore, the connections between the displayed or discussed modules may be communication connections through some interfaces, which may be electrical or other forms. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort. The following uses a publicly available real thermal infrared image sequence as an example to illustrate specific implementation methods to demonstrate the technical effects of the present invention; specific steps in the embodiments will not be repeated.

[0126] Example

[0127] The accompanying drawings illustrating the embodiments of the present invention will make the objectives, technical solutions, and advantages of the present invention clearer. It should be noted that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention. All equivalent substitutions and improvements made within the framework of the methods and principles provided by the present invention should be included within the protection scope of the present invention.

[0128] In this embodiment, the effectiveness of the thermal infrared small target detection algorithm will be verified using a publicly available infrared image sequence. The publicly available infrared image sequence is described as follows:

[0129] This thermal infrared image sequence contains 60 real infrared frames, each 256×256 pixels in size. The background is dominated by dense forests and linear, bright structures, containing a large amount of strong clutter and noise. The flying object is slowly flying above the forest, passing through the forest and clouds. There are also bright cloud textures. Against this complex air-to-ground background, a drone is slowly flying through the clouds. Figure 4 Here is an example frame from this infrared image sequence. Figure 5 The image shows the target detection results obtained by applying the infrared small target detection method based on the spatiotemporal information completion model of this invention to instance frames of an infrared image sequence. The results show that complex background textures and noise components are suppressed, and the salience of small targets is enhanced. To better evaluate the effectiveness of the infrared small target detection method based on the spatiotemporal information completion model proposed in this invention, from a qualitative perspective, the target detection results of instance frames of the infrared image sequence and their corresponding 3D grayscale images are compared and analyzed. From a quantitative perspective, the detection performance of the algorithm is evaluated using the 3D-ROC evaluation index system, which includes three types of indicators, including the target detection capability index (AUC). (D, τ) and AUC TD Background suppression capability (AUC) (F, τ) AUC BS and AUC SNPR The comprehensive effectiveness evaluation index of the detector (AUC) and the detector TDBS AUC (D,F) and AUC ODP The 3D-ROC evaluation index system comes from Chang, Chein-I. "An effective evaluation tool for hyperspectral target detection: 3D receiver operating characteristic curve analysis." IEEE Transactions on Geoscience and Remote Sensing 59.6 (2020): 5131-5153.

[0130] Table 1 presents the 3D-ROC evaluation results of small target detection in this thermal infrared image sequence using rcDoGF, GCM, IPCE, RCTVW, STT-TRNR, STTM-TSNM, DGRL, AGPCNet, IAANet, and the method of this invention, respectively. The bold and underlined values ​​represent the corresponding optimal and second-best AUC values, respectively. Specifically, rcDoGF and GCM are infrared small target detection methods based on local contrast measurement; DGRL is an infrared small target detection method based on single-frame low-rank sparse matrix factorization; IPCE is an infrared small target detection method based on single-frame low-rank sparse tensor factorization; RCTVW, STT-TRNR, and STTM-TSNM are infrared small target detection methods based on multi-frame low-rank sparse tensor factorization; and AGPCNet and IAANet are infrared small target detection methods based on deep learning.

[0131] rcDoGF comes from Li, Yongsong, et al. "Infrared Small Target Detection Basedon 1-D Difference of Guided Filtering." IEEE Geoscience and Remote SensingLetters 20 (2023): 1-5.

[0132] GCM comes from Tang, Ye, Kun Xiong, and Chunxi Wang. "Fast Infrared SmallTarget Detection Based on Global Contrast Measure Using Dilate Operation." IEEE Geoscience and Remote Sensing Letters 20 (2023): 1-5.

[0133] IPCE comes from Zhang, Chunmin, et al. "Infrared small target detection via interpatch correlation enhancement and joint local visual saliency prior." IEEE Transactions on Geoscience and Remote Sensing 60 (2021): 1-14.

[0134] RCTVW is from Liu, Ting, et al. "Representative Coefficient Total Variation for Efficient Infrared Small Target Detection." IEEE Transactions on Geoscience and Remote Sensing (2023).

[0135] STT-TRNR is from Yi, Haiyang, et al. "Spatial-Temporal Tensor Ring Norm Regularization for Infrared Small Target Detection." IEEE Geoscience and Remote Sensing Letters 20 (2023): 1-5.

[0136] STTM-TSNM is from Pang, Dongdong, et al. "Tensor Spectral k-support Norm Minimization for Detecting Infrared Dim and Small Target against Urban Backgrounds." IEEE Transactions on Geoscience and Remote Sensing (2023).

[0137] DGRL is from Zhou, Fei, et al. "Robust infrared small target detection via jointly sparse constraint of l 1 / 2-metric and dual-graph regularization." Remote Sensing 12.12 (2020): 1963.

[0138] AGPCNet is from Zhang, Tianfang, et al. "AGPCNet: Attention-guided pyramid context networks for infrared small target detection." arXiv preprint arXiv:2111.03580 (2021).

[0139] IAANet comes from Wang, Kewei, et al. "Interior attention-aware network for infrared small target detection." IEEE Transactions on Geoscience and RemoteSensing 60 (2022): 1-13.

[0140] Table 1. Quantitative indicators of detection results using rcDoGF, GCM, IPCE, RCTVW, STT-TRNR, STTM-TSNM, DGRL, AGPCNet, IAANet, and the method of this invention for example thermal infrared image sequences.

[0141]

[0142] Figure 6 The original image of a thermal infrared image example frame, the thermal infrared small target detection results after rcDoGF and GCM detection, and the corresponding 3D grayscale image are shown. Figure 7 The image shows the results of small thermal infrared target detection and the corresponding 3D grayscale image of a sample frame of thermal infrared image after IPCE, RCTVW and STT-TRNR detection. Figure 8 The image shows the thermal infrared small target detection results and corresponding 3D grayscale image of a sample thermal infrared image frame detected by STTM-TSNM, DGRL and AGPCNet. Figure 9 This image shows the results of detecting small thermal infrared targets using IAANet and the method of this invention, along with the corresponding 3D grayscale image, from an example frame of a thermal infrared image. Figures 6 to 9 Qualitative results show that the deep learning-based infrared small target detection methods AGPCNet and IAANet exhibit severe false alarms, especially AGPCNet. For traditional infrared small target detection methods, IPCE suffers from missed target detections. Other comparative methods can detect infrared small targets, but according to the 3D detection results, background and noise clutter remain, such as GCM, rcDoGF, and DGRL. Conversely, the infrared small target detection method based on the spatiotemporal information completion model proposed in this invention can effectively suppress background and noise and enhance target saliency. Quantitative analysis based on the statistical indicators shown in Table 1 shows that IPCE performs the worst, with an AUC of... (D,F) It achieved a score of 0.000, placing GCM second in overall performance, with an AUC of [missing value]. ODPAchieving a score of 1.880, the method proposed in this invention surpasses all comparable algorithms in all eight 3D-ROC evaluation metrics, demonstrating superior target detection capability, background suppression capability, and overall detection performance. Based on both qualitative and quantitative analysis, the thermal infrared small target detection method proposed in this invention exhibits excellent target detection capability, background suppression capability, and overall effectiveness.

[0143] The accompanying drawings illustrating the embodiments of the present invention will make the objectives, technical solutions, and advantages of the present invention clearer. It should be noted that the specific embodiments described herein are merely illustrative and not intended to limit the scope of the invention. All equivalent substitutions and improvements made within the framework of the methods and principles provided by the present invention should be included within the protection scope of the present invention.

Claims

1. A method for detecting small infrared targets based on a spatiotemporal information completion model, characterized in that, Includes the following steps: Step 1): Utilize prior knowledge from target detection results and the infrared image to be detected The carried temporal and spatial information is used to construct a three-dimensional spatiotemporal tensor from the prior target detection result and the infrared image to be detected in frame order. , This represents the sub-tensor corresponding to the prior detection result of the target. This represents the sub-tensor corresponding to the infrared image to be detected. This represents the index value of the constructed three-dimensional spacetime tensor. This indicates an operation that merges tensors along the third dimension. Step 2): Utilizing the characteristics of background, target, and noise, from the three-dimensional spacetime tensor Establish a tensor initialization and completion model based on low-rank sparsity to suppress The low-frequency signal is used to obtain the high-frequency projection transform tensor. ,Depend on Establish a tensor completion model based on low-rank sparsity; Step 3): Based on the background tensor The low-rank property of the background tensor Perform multi-mode expansion and define the background tensor. Multimodal tensor norm ; Step 4): Combining Steps 1) to 3), a sparsity reweighting strategy is introduced into the low-rank sparse tensor completion model to establish a low-rank sparse tensor completion model based on spatiotemporal information. Step 5): Design an optimization solution framework based on the ADMM algorithm to solve the low-rank sparse tensor completion model based on spatiotemporal information, and obtain the high-frequency projection transformation tensor. Target Tensor ; Step 6): Design a spatiotemporal information fusion strategy, utilizing the current high-frequency projection transformation tensor The corresponding target tensor and the subsequent high-frequency projection transformation tensor The corresponding target tensor By analyzing image frames The corresponding target detection results are fused to obtain image frames. Infrared small target detection results This allows for the detection of small infrared targets in each frame of the infrared image sequence, thus achieving infrared small target detection based on a spatiotemporal information completion model.

2. The infrared small target detection method based on a spatiotemporal information completion model according to claim 1, characterized in that, Step 1) specifically refers to: Using the temporal and spatial information of thermal infrared images, the current target to be detected is... Frame infrared image Compared to its previous Frame infrared image Prior to target detection results and subsequent tests to be conducted Frame infrared image Stacked sequentially according to the time order of the image frames. Prior to frame target detection results and Frame of infrared image to be detected Constructed as a three-dimensional spacetime tensor , 3D spacetime tensor The size is , express The width of the front slice, express The height of the front slice, express The thickness, and .

3. The infrared small target detection method based on a spatiotemporal information completion model according to claim 1, characterized in that, Step 2) specifically refers to: The three-dimensional spacetime tensor constructed in step 1) This preserves the low-rank properties of the background component, the sparsity of the target component, and the characteristics of the noise component, thereby enhancing the three-dimensional spatiotemporal tensor. Modeled as a low-rank background tensor sparse target tensor and noise component tensor A linear combination of these elements is used to establish a tensor initial completion model based on low-rank sparseness, as shown in equation (1): （1） in, express Norm, and Represents positive numbers. Represents the rank estimator operator; Considering the impact of low-frequency signals in complex scenes on the detection of small infrared targets, the three-dimensional spatiotemporal tensor is... Projecting onto the frequency domain and suppressing low-frequency signals is achieved through the following process: via Fourier transform. Three-dimensional spacetime tensor neutron tensor Project each front slice into the frequency domain; design a Gaussian filter. By adjusting variance The process involves suppressing low-frequency signals and enhancing high-frequency signals in each frontal slice; finally, the filtered data is projected into the spatial domain to obtain the high-frequency projection transform tensor. For high-frequency projection transformation tensors The tensor initial completion model (1) based on low-rank sparse tensor is improved into a tensor completion model based on low-rank sparse tensor, as shown in formula (2): （2） in, Describing the optimal approximation norm Norm.

4. The infrared small target detection method based on a spatiotemporal information completion model according to claim 1, characterized in that, Step 3) specifically refers to: For low-rank background tensor background tensor Perform multi-modal expansion to obtain three background tensors with different expansion modes. , and , respectively represented as , , , This indicates that the tensor will be processed according to the pattern - Expanding operators; , and All maintain low-rank properties, for , and The background tensor is obtained by weighting the tensor norms. Multimodal tensor norm As shown in formula (3): （3） in, Indicates the weighting coefficient. This represents the tensor nuclear norm.

5. The infrared small target detection method based on a spatiotemporal information completion model according to claim 3, characterized in that, Step 4) specifically refers to: Solving the tensor completion model (2) in step 2) is an iterative optimization process. To accelerate the iterative solution of the model, a sparsity reweighting strategy is introduced into the tensor completion model (2). The reweighted weight tensors From the current iteration round The obtained target tensor If confirmed, then the next iteration round... In the middle, the reweighted weight tensor of the target As shown in formula (4): （4） in, Represents positive numbers. This represents the operation of calculating the reciprocal of an element. This indicates the operation of calculating the absolute value; Combining the tensor completion model (2) in step 2) and the background tensor in step 3). The multimodal tensor norm is used to establish a low-rank sparse tensor completion model based on spatiotemporal information, as shown in formula (5): （5） in, It represents the Hadamardi (or Hadama) stack.

6. The infrared small target detection method based on a spatiotemporal information completion model according to claim 5, characterized in that, Step 5) specifically refers to: The ADMM algorithm is used to solve the low-rank sparse tensor completion model based on spatiotemporal information (5). In order to make each variable to be solved in (5) solvable, auxiliary variables are introduced respectively. Formula (6) is obtained as follows: （6） The corresponding augmented Lagrangian function is （7） in, For Lagrange multipliers, The penalty term is used; each variable is solved through alternating iterations, and the subproblems corresponding to each variable are as follows: 1) The iterative formula is （8） in, Indicates the number of iterations. Indicates the first Round iterations obtained , This represents the singular value decomposition of the tensor. For singular value tensors, and The unitary tensor after singular value decomposition of the tensor. The transpose operator for tensors. To represent the singular value contraction operator, we have: " "For tensor product operators, ; 2) The iterative formula is （9） in, Indicates the first Round iterations obtained ; 3) The iterative formula is （10） in, Indicates the first Round iterations obtained ; 4) The iterative formula is （11） The equivalent formula of formula (11) is: （12） in, Indicates the first Round iterations obtained , This indicates that the tensor will be processed according to the pattern - Folding operators; 5) The iterative formula is （13） in, Indicates the first Round iterations obtained , For soft threshold shrinkage operators, , For symbolic functions, Soft threshold; 6) The iterative formula is （14） in, Indicates the first Round iterations obtained ; 7) The iterative formula is （15） in, Indicates the first Round iterations obtained ; 8) Lagrange multipliers The iterative formula is （16） （17） （18） （19） in, They represent the first Round iterations obtained ; 9) Penalty Items The iterative formula is （20） in, Indicates the first Round iterations obtained , Indicates the update coefficients. express The maximum value; The iteration stopping condition is: 1) When the relative error Less than the error When the iteration terminates; 2) when the target tensor is reweighted. Stop iterating when the norm no longer changes; 3) When the norm of two consecutive iterations is no longer changing. and satisfy 4) Stop iteration when the results of two consecutive iterations are obtained. and satisfy Stop iterating when the time comes; This completes the iterative solution of the low-rank sparse tensor completion model (5) based on spatiotemporal information in step 5), yielding... The corresponding target tensor .

7. The infrared small target detection method based on a spatiotemporal information completion model according to claim 1, characterized in that, Step 6) specifically refers to: In the target tensor In the middle, the first Zhang Zhidi Zhang frontal slice As an infrared image The test results prior ; combination and the image to be detected The next three-dimensional spacetime tensor is constructed by stacking the frames sequentially. , Infrared image The corresponding three-dimensional spacetime tensor index; By projecting the vector into the frequency domain and suppressing low-frequency signals, a high-frequency projection transform tensor is obtained. Therefore, based on step 5), the low-rank sparse tensor completion model based on spatiotemporal information is solved, as shown in formula (21): （21） Solving for the next high-frequency projection transformation tensor yields the solution. Corresponding target tensor Extract the target tensor The Middle Zhang Zhidi Zhang's frontal slice is used to construct the next three-dimensional spacetime tensor. Target detection priors; extraction The Middle Zhang Zhidi Zhang frontal slice and with The Zhang Zhidi Zhang frontal slice The images are fused together to obtain the image to be detected. Target detection result sequence: （22） And there are （23） in," "Indicates a fusion operation, Indicates the result after fusion The target detection results; For the infrared image sequence to be detected, repeat the process of formulas (21) to (23) in step 6) to obtain the target detection result corresponding to each frame of the image; The infrared image to be detected is obtained from formula (23). The corresponding infrared small target detection result image The infrared small target detection result image is obtained from each frame of infrared image, and finally the infrared small target detection based on the spatiotemporal information completion model is realized.

8. The infrared small target detection method based on a spatiotemporal information completion model according to claim 7, characterized in that, The fusion operation is to add corresponding elements of a matrix, multiply corresponding elements of a matrix, or divide corresponding elements of a matrix.

9. An infrared small target detection device based on a spatiotemporal information completion model implementing the method of claim 1, characterized in that, include: The three-dimensional spatiotemporal tensor construction module uses the prior target detection results and the infrared image to be detected to construct a three-dimensional spatiotemporal tensor, laying the data foundation for the spatiotemporal information completion model; The module for establishing a low-rank sparse tensor completion model suppresses low-frequency signals in the three-dimensional spatiotemporal tensor and establishes a low-rank sparse tensor completion model based on the characteristics of background, target and noise. The low-rank background estimation norm design module performs multi-modal expansion on the background tensor and defines the multi-modal tensor kernel norm of the background tensor. The module for establishing a low-rank sparse tensor completion model based on spatiotemporal information consists of a three-dimensional spatiotemporal tensor and a low-rank background estimation norm. A sparsity reweighting strategy is introduced into the low-rank sparse tensor completion model to establish a low-rank sparse tensor completion model based on spatiotemporal information. The optimization model module is designed, and an optimization solution framework based on the ADMM algorithm is designed to solve the low-rank sparse tensor completion model based on spatiotemporal information to obtain the target tensor. The spatiotemporal information fusion strategy design module fuses the target tensor of the current spatiotemporal tensor and the target tensor of the next spatiotemporal tensor to obtain the small target detection results of the corresponding infrared image. The target detection result output module is used to output the infrared small target detection result map for each frame of infrared image.

Images