An infrared dim small target super-resolution reconstruction and detection method based on spatiotemporal energy aggregation constraint

Abstract

The application discloses an infrared dim small target super-resolution reconstruction and detection method based on space-time energy aggregation constraint, which comprises the following steps: constructing an infrared dim small target super-resolution reconstruction network framework, wherein the framework core baseline is a lightweight BasicVSR++ model with ROI perception, and the input low-resolution infrared video sequence is pretreated; the core structure of the lightweight BasicVSR++ baseline model with ROI perception is reconstructed; a target region local contrast LCM constraint loss function is designed; a feature extraction and detection head structure is built based on a lightweight detection baseline network; the detection network completes the positioning and detection process of the infrared dim small target, and the super-resolution image sequence is used for visual analysis and performance verification. The application constructs the lightweight BasicVSR++ model with ROI perception as the super-resolution core baseline, replaces the original SpyNet optical flow module with a low-level deformable convolution TDAN alignment module, and realizes the accurate alignment of multi-frame features of the textureless infrared dim small target.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method for super-resolution reconstruction and detection of infrared weak targets based on spatiotemporal energy aggregation constraints. Background Technology

[0002] In the field of computer vision technology, target detection is a research hotspot with significant research value in areas such as imaging guidance, airspace defense, and security monitoring. Deep learning algorithms are currently the core component of mainstream detection methods. The core workflow of Infrared Small Target Detection (IRSTD) involves using a series of algorithms to amplify the differences between the target and the background from low-resolution, low-signal-to-noise-ratio, and complex background infrared images or video sequences, identifying the location of infrared small targets with small pixel proportions, weak grayscale features, and no obvious texture structure. Based on the amount of information utilized by the detection algorithm, infrared small target detection can be divided into single-frame detection and multi-frame detection. For single-frame detection, the network has advantages in real-time performance and low structural complexity; however, because it only uses one frame of image for detection, the problems of false alarms and missed detections are difficult to solve.

[0003] In recent years, various algorithms have developed rapidly in the field of single-frame detection. However, in the field of multi-frame detection, the development of algorithms has been relatively slow due to the high complexity and computational resource consumption of multi-frame detection networks, insufficient visualization and interpretability, and the loss of target features during the learning process of deep learning networks. Compared with single-frame detection, multi-frame detection can utilize information in the temporal dimension to reduce false alarms and is more in line with the actual needs of anti-drone missions in air-to-ground scenarios, thus possessing higher research value.

[0004] In the field of single-frame detection, there are two main types of methods: one is to manually design models to separate the target from the background, such as filtering and local information algorithms. However, these algorithms are extremely sensitive to preset model parameters and are difficult to adapt to detection tasks in complex scenes. The other type is to use deep learning networks to learn target or background features and then distinguish them, such as UIU and DNANet. These algorithms require a large amount of labeled data and suffer from serious false alarms and missed detections in low signal-to-noise ratio and low-resolution images. They are also difficult to effectively utilize infrared video sequences obtained from real sampling equipment and cannot meet the requirements of high-precision detection.

[0005] In multi-frame IRSTD tasks, the mainstream methods currently fall into two categories. One type utilizes an end-to-end multi-frame IRSTD detection network to fuse spatiotemporal features to learn the motion patterns of the target, thereby completing the detection task, such as TDCNet. However, this method has high computational resource requirements, insufficient interpretability and visualization, difficulty in aligning features with complex motion or pseudo-deformation, and is prone to motion blur. The other type uses super-resolution image synthesis technology to synthesize a super-resolution image from multiple frame sequences to enhance details before performing the detection task. This method is suitable for scenarios where the original data is a low-resolution image with a low signal-to-noise ratio, and has strong interpretability.

[0006] Currently, the joint super-resolution reconstruction and object detection task still faces many challenges. First, the current state-of-the-art (SOTA) super-resolution reconstruction models are all based on visible light video, favoring overall image clarity and smoothness, and are not fully suited to the IRSTD task, which only provides grayscale information and focuses on the difference between the target and the background. Second, during multi-frame alignment and information fusion, due to fewer target features and complex background interference, alignment inaccuracies can occur, resulting in ghosting or image distortion, which is a fatal blow to the detection network. Third, to achieve better reconstruction and detection results, multi-frame super-resolution networks typically have high computational costs, impacting real-time performance requirements.

[0007] A method for detecting weak spatial targets based on super-resolution feature enhancement is disclosed in Chinese patent document CN113223059B. This method includes: 1. performing illumination compensation on a spatial video stream to obtain a spatial video image; 2. using a foreground detection algorithm based on background modeling and inter-frame difference to obtain a grayscale feature map of the foreground target, extracting the foreground target contour using morphological filtering, and calculating a set of predicted bounding boxes for the foreground target; 3. using a multi-target tracker based on the KM algorithm and Kalman filtering to track the predicted bounding boxes and filter out suspected target bounding boxes; 4. training a super-resolution neural network model on a spatial target super-resolution dataset, and using the super-resolution model to perform super-resolution enhancement on the foreground image region mapped by the suspected target bounding boxes during the detection phase; 5. training a neural network model based on the YOLOv4 framework on a spatial target detection dataset, and using a detector model to detect targets in the super-resolution enhanced foreground target region during the detection phase to obtain the target detection result. However, this method for detecting weak spatial targets based on super-resolution feature enhancement has poor performance in infrared low-visibility scenes.

[0008] A method and system for detecting small infrared targets using YOLOv8, which integrates super-resolution reconstruction, is disclosed in Chinese patent document CN120673287A. This method and system involves real-time acquisition of infrared image data; preprocessing the acquired infrared image data; improving the RRDB network structure in the Real-ESRGAN model generator; processing the super-resolution reconstructed infrared dataset through a data augmentation module, and then replacing the stride convolutional downsampling module of the original YOLOv8 backbone and neck networks with a fine-grained supplementary feature aggregation module; feature maps from the backbone network enter the neck network, are processed by a context information augmentation module, and then reach the head network; the results of infrared super-resolution reconstruction and small target detection are displayed through a front-end interface, and the algorithm model library and data sample library are iteratively updated through feedback. This improved YOLOv8 infrared small target detection method and system, which integrates super-resolution reconstruction, addresses the problems existing in the current technology and achieves high-precision, low-false-judgment detection of small infrared targets in complex backgrounds, thereby improving detection efficiency and reliability. However, it completely ignores the inter-frame temporal energy correlation, resulting in prominent issues of false alarms and missed detections of dynamic clutter and weak targets at long distances.

[0009] A method for detecting and tracking infrared weak targets based on spatiotemporal trajectory fitting is disclosed in Chinese patent document CN121883364A. This method involves constructing an infrared weak target trajectory fitting dataset (Data-match); building an infrared weak target detection network model (Model-Det); repeatedly training and validating Model-Det using Data-match; obtaining infrared weak target detection candidate boxes (Bboxes); inputting them into a Kalman filter unit to generate a target trajectory (Trajectory-A) for each sequence; building an infrared weak target trajectory segmentation network model (Model-Seg); repeatedly training and validating Model-Seg using Data-match; obtaining the target trajectory (Trajectory-B) for each sequence; and taking the intersection of Trajectory-A and Trajectory-B to obtain the final infrared weak target detection and tracking result and target trajectory output. The above invention reduces data dependence and can achieve stable and accurate detection of infrared weak targets in different scenarios. However, this method for detecting and tracking infrared weak targets based on spatiotemporal trajectory fitting suffers from severe loss of features of distant infrared weak targets with very few pixels.

[0010] To address the shortcomings of the existing technologies, providing a super-resolution reconstruction and detection method for infrared weak targets based on spatiotemporal energy aggregation constraints is a worthwhile research topic. Summary of the Invention

[0011] The purpose of this invention is to overcome the shortcomings of inaccurate motion estimation and insufficient detail enhancement in super-resolution networks, and to provide a super-resolution reconstruction and detection method for infrared weak targets based on spatiotemporal energy aggregation constraints. This method achieves synergistic optimization of super-resolution reconstruction and target detection, thereby improving the accuracy and robustness of infrared weak target detection.

[0012] The objective of this invention is achieved through the following technical solution: A super-resolution reconstruction and detection method for infrared weak targets based on spatiotemporal energy aggregation constraints includes the following steps: Step 1: Construct a super-resolution reconstruction network framework for infrared weak targets. The core baseline of the framework is a lightweight BasicVSR++ model with ROI awareness, and the input low-resolution infrared video sequence is preprocessed. Step 2: Reconstruct the core structure of the lightweight BasicVSR++ baseline model with ROI awareness; Step 3: Design the local contrast LCM constraint loss function for the target region; Step 4: Construct a feature extraction and detection head structure based on a lightweight detection baseline network; Step 5: The detection network completes the localization and detection process of infrared weak targets. The super-resolution image sequence is used for visualization analysis and performance verification. The overall architecture takes into account both lightweight design and temporal modeling capabilities, adapts to low signal-to-noise ratio and textureless infrared weak target scenes, and solves the alignment offset, ghosting and other problems of traditional multi-frame super-resolution networks.

[0013] Preferably, the network framework in step one retains the core second-order grid bidirectional recursive propagation mechanism to extract the global temporal context. It is divided into two parts: a super-resolution module and a detection module, which are independent and can be cascaded and optimized with spatiotemporal aggregation constraints. The input low-resolution infrared video sequence is preprocessed to obtain a continuous and standardized infrared video frame sequence. At the same time, the positions of weak infrared targets in the sample set are marked, and target ROI mask maps and detection labels are generated. The super-resolution and detection modules are independent of each other and can be optimized separately, which reduces the training difficulty and facilitates iterating the super-resolution network or the detection network separately, making it more flexible for engineering implementation.

[0014] Preferably, the specific method for constructing the ROI-aware lightweight BasicVSR++ model and preprocessing the video sequence in step one is as follows: S1. Perform channel clipping and branch simplification on the native BasicVSR++, compress the number of main convolution channels, remove redundant loop propagation structures, adapt to infrared single-channel grayscale image input, and build a lightweight baseline. S2. Embed an ROI-aware branch at the front end of the baseline, and use the target ROI mask as a guide to perform weight adaptive enhancement on the target region during the feature extraction stage. S3. Perform inter-frame denoising, grayscale normalization, and size unification on the input infrared video sequence. Mark the target position manually or semi-automatically, generate the target ROI mask map and detection label, and simplify the redundant structure through channel pruning to achieve network lightweighting, reduce edge hardware computing power consumption, meet the real-time requirements of infrared detection equipment, and adaptively amplify the feature weight of small targets and suppress background interference in the ROI perception branch to solve the pain points of low pixel ratio of infrared weak targets and easy feature dilution and loss by the background.

[0015] Preferably, in step two, when reconstructing the core structure of the lightweight BasicVSR++ baseline model, during the multi-frame feature alignment stage, the SpyNet optical flow module (Light TDAN) of the original BasicVSR++ network is replaced by a low-level deformable convolutional TDAN alignment module to achieve accurate multi-frame feature alignment of textureless infrared weak targets. In the spatiotemporal feature fusion stage, the original residual fusion network of BasicVSR++ is abandoned and replaced by a lightweight temporal attention fusion module with local windows. After completing bidirectional feature propagation and convergence, temporal attention aggregation is performed in the local window containing the target. This reduces the computational cost while completing the fusion of spatiotemporal features from multiple frames. The deformable convolution of TDAN replaces the optical flow module. Adaptive offset guided by ROI is used to achieve accurate alignment of textureless infrared targets, overcoming the matching failure and ghosting problems of SpyNet in low-texture infrared scenes.

[0016] Preferably, the specific construction method of the TDAN alignment module is as follows: The TDAN alignment module uses a two-layer deformable convolution to build a low-level lightweight structure. The first layer completes coarse-grained motion estimation between adjacent frames and the current frame. The second layer combines ROI masks to complete fine-grained motion offset correction of the target region and outputs a multi-frame feature map with accurate alignment, thus solving the alignment deviation problem of the SpyNet optical flow module in textureless scenes. The specific construction method of the local window lightweight temporal attention fusion module is as follows: The local window lightweight temporal attention fusion module adopts a non-overlapping local window partitioning strategy. Temporal attention calculation is performed within a local window composed of partial feature maps. Combined with ROI masks, the weight of windows containing targets is enhanced. While reducing the amount of computation, it completes the accurate aggregation of spatiotemporal features of multiple frames. The two-layer hierarchical TDAN alignment architecture first performs global coarse matching, and then corrects the target region offset based on ROI masks, which greatly improves the alignment accuracy of weak moving targets at long distances. Pure background windows directly take the feature mean and skip attention calculation. The amount of computation in the fusion stage is reduced to less than 40% of the native network, which is suitable for airborne and ground-based small infrared detection terminals.

[0017] Preferably, in step three, when designing the local contrast LCM constraint loss function for the target region, the local contrast metric is used as a core component of the loss function of the super-resolution image synthesis network. Combined with general pixel-level loss and edge gradient loss, a joint training loss specific to the super-resolution module is constructed. The LCM constraint loss forces the local contrast between the target region and its neighboring background to remain unchanged or positively enhanced during the super-resolution reconstruction process, locking in and preserving the core radiation features and saliency of weak infrared targets. The super-resolution network is trained separately to obtain the super-resolution reconstructed image. The addition of a contrast constraint loss for infrared tasks changes the traditional super-resolution approach of only optimizing global pixel errors, allowing the reconstruction process to serve downstream target detection.

[0018] Preferably, after the LCM constraint loss function is introduced, the overall super-resolution module joint training loss function is: , This is the pixel loss term, used to constrain the pixel restoration accuracy of the super-resolution image. These are the weights for the gradient loss mask, ensuring that the gradient information of the target is not lost during training. For the mask gradient loss term, The contrast loss weighting coefficient is... This is a local contrast constraint loss term for LCM, ensuring that the target object is always a fixed brightness higher than the background, regardless of whether the background is flat or bright. To ensure that the target is always brighter than the background, regardless of whether the background is flat or bright, the mask gradient loss only constrains the gradient of the target area, preserves the pulse-like edge features of small infrared targets, and prevents the target from becoming a light spot after reconstruction.

[0019] Preferably, in step four, when building the feature extraction and detection head structure, a classification loss and regression loss specific to the detection task are designed. High-resolution infrared image samples and corresponding detection labels are used. The detection module is independently trained using super-resolution image sequences to complete parameter optimization and convergence of the detection module. The detection network is trained based on the enhanced high-contrast super-resolution image. Compared with training directly using the original image, the detection network is more likely to distinguish between the target and the complex background, reducing the false alarm rate.

[0020] Positive and beneficial effects: 1. The infrared weak target super-resolution reconstruction and detection method based on spatiotemporal energy aggregation constraints has achieved excellent results in ground-to-air long-range infrared weak target detection scenarios. A lightweight BasicVSR++ model with ROI awareness is constructed as the super-resolution core baseline. The low-level deformable convolution TDAN alignment module is used to replace the original SpyNet optical flow module to achieve accurate alignment of multi-frame features of textureless infrared weak targets. A local window lightweight temporal attention fusion module is designed to replace the original bidirectional propagation spatiotemporal feature fusion module to complete the low-complexity accurate aggregation of multi-frame spatiotemporal features. An ROI-aware branch is embedded in the front end of the baseline to achieve targeted enhancement and priority extraction of target region features.

[0021] 2. This method for super-resolution reconstruction and detection of infrared weak targets based on spatiotemporal energy aggregation constraints integrates lightweight architecture modification, precise alignment optimization of textureless scenes, and ROI directional feature enhancement. It makes good use of the advantages of each module and makes up for the shortcomings of existing technologies, so as to adapt to the super-resolution scene of infrared weak targets.

[0022] 3. This infrared weak target super-resolution reconstruction and detection method based on spatiotemporal energy aggregation constraints overcomes the problems of feature matching failure and alignment deviation causing ghosting in infrared textureless scenes by using the multi-frame alignment module based on Light TDAN through adaptive receptive field and offset learning of deformable convolution. The lightweight temporal attention fusion module based on local windows replaces the original global feature interaction with non-overlapping block local computation, solving the problem of excessive computation and difficulty in meeting the real-time requirements of edge devices in traditional multi-frame super-resolution networks. The feature enhancement branch based on ROI perception solves the problems of low pixel ratio, weak texture features, and easy loss of core features during propagation of infrared small targets by adaptively allocating target region weights.

[0023] 4. This infrared weak target super-resolution reconstruction and detection method based on spatiotemporal energy aggregation constraint solves the problem that traditional visible light super-resolution models tend to have a clear and smooth overall image but poor adaptability to infrared weak target detection tasks by designing a local contrast LCM constraint loss function for the target region. Experiments show that the main reason for the decrease in detection performance after infrared super-resolution reconstruction is not the insufficient global pixel accuracy of the overall image, but the weakening of the local contrast between the target and the adjacent background during the super-resolution process, which leads to the loss of the core radiation features of the target and the difficulty for the detection network to distinguish the target from the complex background.

[0024] 5. This infrared weak target super-resolution reconstruction and detection method based on spatiotemporal energy aggregation constraints uses LCM metric as the core component of the loss function of the super-resolution image synthesis network. This forces the local contrast between the target region and its neighboring background to remain unchanged or be positively enhanced during the super-resolution reconstruction process. This replaces the traditional super-resolution loss design that only focuses on global pixel consistency, avoiding the problem of target features being diluted by the background and target saliency decreasing after super-resolution reconstruction. At the same time, it locks and preserves the core radiation features of infrared weak targets, allowing the super-resolution reconstruction process to fully serve the downstream detection task, and greatly improving the adaptability of super-resolution results to infrared weak target detection tasks. Attached Figure Description

[0025] Figure 1 This is an implementation block diagram of the training phase of the present invention; Figure 2 This is an implementation block diagram of the testing phase of the present invention; Figure 3 This is a block diagram of the Light TDAN module of the present invention; Figure 4 This is a block diagram of the local window lightweight temporal attention fusion module of the present invention; Figure 5 This is a schematic diagram illustrating the calculation logic of the LCM constraint loss in the target region of this invention. Detailed Implementation

[0026] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention. Example 1

[0027] like Figures 1 to 5 As shown, a method for super-resolution reconstruction and detection of infrared weak targets based on spatiotemporal energy aggregation constraints includes the following steps: Step 1: Construct a super-resolution reconstruction network framework for infrared weak targets. The core baseline of the framework is a lightweight BasicVSR++ model with ROI awareness, and the input low-resolution infrared video sequence is preprocessed. Step 2: Reconstruct the core structure of the lightweight BasicVSR++ baseline model with ROI awareness; Step 3: Design the local contrast LCM constraint loss function for the target region; Step 4: Construct a feature extraction and detection head structure based on a lightweight detection baseline network; Step 5: The detection network completes the localization and detection process of infrared weak targets. The super-resolution image sequence is used for visualization analysis and performance verification. The overall architecture takes into account both lightweight design and temporal modeling capabilities, adapts to low signal-to-noise ratio and textureless infrared weak target scenes, and solves the alignment offset, ghosting and other problems of traditional multi-frame super-resolution networks.

[0028] The network framework in Step 1 retains the core second-order grid bidirectional recursive propagation mechanism to extract the global temporal context. It is divided into two parts: a super-resolution module and a detection module, which are independent and can be cascaded and optimized with spatiotemporal aggregation constraints. The input low-resolution infrared video sequence is preprocessed to obtain a continuous and standardized infrared video frame sequence. At the same time, the positions of weak infrared targets in the sample set are marked, and target ROI mask maps and detection labels are generated. The super-resolution and detection modules are independent of each other and can be optimized separately, which reduces the training difficulty and facilitates iterating the super-resolution network or the detection network separately, making it more flexible for engineering implementation.

[0029] The specific methods for constructing the ROI-aware lightweight BasicVSR++ model and preprocessing the video sequence in step one are as follows: S1. Perform channel clipping and branch simplification on the native BasicVSR++, compress the number of main convolution channels, remove redundant loop propagation structures, adapt to infrared single-channel grayscale image input, and build a lightweight baseline. S2. An ROI-aware branch is embedded at the baseline front end. Guided by the target ROI mask, the target region is adaptively enhanced in the feature extraction stage. Based on the native BasicVSR++, a lightweight modification is made for the infrared single-channel grayscale image. The backbone convolution channels are cropped from 64 to 32, the input channel is set to 1, and the number of parameters is controlled within 5M. An ROI-aware branch is embedded at the feature extraction front end. The ROI mask is used as input. Channel alignment is completed through 1×1 convolution, and the spatial weight matrix is ​​output. Gated weighting F_out=F_in×(1+M) is performed on the backbone features to achieve adaptive enhancement of target region features and suppression of background region weights, so as to avoid the dilution of small target features by the background. S3. Perform denoising, normalization, and size unification processing on the low-resolution infrared video in sequence. Use a 3-frame sliding window temporal mean filter to remove thermal noise, and a 3×3 Gaussian kernel spatial filter to remove salt-and-pepper noise; normalize the grayscale values ​​from [0,255] to [0,1] to improve training convergence; adjust all frames to a preset fixed size to ensure network input consistency. Based on manually labeled bounding boxes of the targets, a hierarchical ROI mask map is generated, with the same size as the infrared frame. The mask value of the target core area is 1, the mask value of the adjacent background area extending outward by 2 times is 0.5, and the mask value of the far-field background area is 0, providing accurate spatial labels for subsequent modules. Through channel pruning, the redundant structure is simplified to achieve network lightweighting, reduce the computing power consumption of edge hardware, and meet the real-time requirements of infrared detection equipment. The ROI perception branch adaptively amplifies the feature weight of small targets and suppresses background interference, solving the pain points of low pixel ratio of small infrared targets and easy feature dilution and loss due to background. Example 2

[0030] like Figure 3 and Figure 4 As shown, in step two, when reconstructing the core structure of the lightweight BasicVSR++ baseline model, in the multi-frame feature alignment stage, the SpyNet optical flow module (Light TDAN) of the original BasicVSR++ network is replaced by a low-level deformable convolutional TDAN alignment module to achieve accurate multi-frame feature alignment of textureless infrared weak targets. In the spatiotemporal feature fusion stage, the original residual fusion network of BasicVSR++ is abandoned and replaced by a lightweight temporal attention fusion module with local windows. After completing bidirectional feature propagation and convergence, temporal attention aggregation is performed in the local window containing the target. This reduces the computational load while completing the fusion of spatiotemporal features from multiple frames. The deformable convolution of TDAN replaces the optical flow module and relies on adaptive offset to achieve accurate alignment of textureless infrared targets, overcoming the matching failure and ghosting problems of SpyNet in low-texture infrared scenes.

[0031] Light TDAN alignment module construction: A lightweight TDAN alignment module is built using two deformable convolutional layers to replace the native SpyNet optical flow module. All convolutional layers are configured with 3×3 kernels, stride 1, padding 1, and 32 output channels. Figure 3 As shown, the module takes N consecutive frames of low-resolution feature maps as input, uses the middle frame as the reference frame, the first layer completes coarse-grained motion estimation, and outputs the initial offset and coarsely aligned feature map; the second layer combines ROI mask to focus on the target region and completes fine-grained motion offset correction.

[0032] A lightweight temporal attention fusion module with local windows is designed to replace the bulky residual convolutional fusion layer of the native BasicVSR++, achieving low-computational spatiotemporal feature aggregation. While retaining the bidirectional recursive propagation of the second-order grid in BasicVSR++, the network enters this fusion module after the current frame receives and aggregates the temporal features from both forward and backward propagation.

[0033] like Figure 4As shown, the aligned and stitched multi-frame feature maps are divided into non-overlapping local windows of 16×16. Based on the ROI mask, windows containing the target with a mask mean > 0.1 undergo full temporal attention calculation, while pure background windows with a mask mean ≤ 0.1 directly take the mean of multi-frame features. Windows containing the target are compressed using a 1×1 convolutional channel, and temporal weights are generated by a fully connected layer. After softmax normalization, they are weighted and fused. All window features are stitched together in their original positions to output the final single-frame fused feature map. This design preserves the long temporal awareness capabilities of BasicVSR++ while reducing the computational cost of the fusion stage to less than 40% of the original module. The two-layer hierarchical TDAN alignment architecture first performs global coarse matching, and then corrects the target region offset based on the ROI mask, which greatly improves the alignment accuracy of weak moving targets at long distances. The pure background window directly takes the feature mean, skips the attention calculation, and reduces the computation amount in the fusion stage to less than 40% of the native network, making it suitable for airborne and ground-based small infrared detection terminals. Example 3

[0034] like Figure 1 , Figure 2 , Figure 5 As shown, in step three, when designing the local contrast LCM constraint loss function for the target region, the local contrast metric is used as a core component of the loss function of the super-resolution image synthesis network. Combined with the general pixel-level loss and edge gradient loss, a joint training loss specific to the super-resolution module is constructed. The LCM constraint loss forces the local contrast between the target region and its neighboring background to remain unchanged or positively enhanced during the super-resolution reconstruction process, locking in and preserving the core radiation features and saliency of weak infrared targets. The super-resolution network is trained separately to obtain the super-resolution reconstructed image. The addition of a contrast constraint loss for infrared tasks changes the traditional super-resolution approach of only optimizing global pixel errors, allowing the reconstruction process to serve downstream target detection.

[0035] After the LCM-constrained loss function is introduced, the joint training loss function of the overall super-resolution module is: , Basic reconstruction losses: As a pixel-level fundamental loss, it ensures the overall brightness and pixel fidelity of the infrared image, overcoming the convergence defect of L1 / L2 loss. The formula is: , in For super-resolution image reconstruction, For true high-resolution images, To avoid an extremely small constant with a square root of zero, this loss forms the foundation of super-resolution reconstruction, ensuring the pixel accuracy of the overall image.

[0036] Core feature loss: Mask gradient loss (L Masked_GradDesigned for the high-frequency pulse characteristics of small infrared targets, this method calculates gradient differences only for the expanded target mask region. While suppressing background noise and sharpening, it forcibly recovers the steep edge gradient of the target. The formula is: , in This is the expanded target mask (the target and its surrounding area are 1s, and the rest are 0s). For element-wise multiplication, For gradient calculation operators, this loss avoids the target being reconstructed as a blurry light spot, and preserves its core gradient features.

[0037] Physical constraint loss: Anti-interference contrast loss (L) Contrast To overcome the limitation of traditional local contrast algorithms failing at bright background edges, this method forces the network to maintain the thermal radiation difference between the target and its neighborhood through target-background difference. The formula is as follows: , Where N is the number of targets in a single frame image, and T i For the core region pixels of the i-th target, B i Let be the pixels in the neighboring background region of the i-th target. This is a mean calculation operator; regardless of whether the background is a flat area or a bright edge, this loss forces the target area of ​​the reconstructed image to be brighter than the neighboring background, thus achieving effective recovery of the target in complex backgrounds.

[0038] Network training optimization: Based on paired samples from low-to-high-resolution infrared images, the reconstructed super-resolution network was trained end-to-end. The optimizer used was Adam W with a weight decay coefficient of 1e-4 to avoid overfitting. The initial learning rate was 1e-4, which was gradually reduced to 1e-6 using a cosine annealing learning rate decay strategy to ensure convergence stability in the later stages of training. The training batch size was 16, and the total training epochs were 200. Accuracy validation was performed on the validation set every 10 epochs. PSNR and SSIM were used as super-resolution evaluation metrics, and the model weights with the best metrics on the validation set were saved as the final inference weights.

[0039] like Figure 1 and Figure 2As shown, in step four, when building the feature extraction and detection head structure, a classification loss and regression loss specific to the detection task are designed. High-resolution infrared image samples and corresponding detection labels are used. The detection module is independently trained using super-resolution image sequences to optimize and converge the parameters of the detection module. Specifically, the low-resolution infrared video frame sequence preprocessed in step one is input into the trained super-resolution network in groups of N consecutive frames. After precise alignment of multi-frame features by the LightTDAN module and feature aggregation by the local window temporal attention fusion module, a 4x high-resolution reconstruction is achieved through an upsampling layer. The super-resolution infrared video sequence is output frame by frame. The reconstructed image retains the core radiation features of the target, and the contrast between the target and the background is effectively enhanced. The high-resolution frame sequence after super-resolution reconstruction is directly input into the detection module. This module, with lightweight metrics as its core, quickly completes the localization and recognition of weak infrared targets, realizing the integrated implementation of super-resolution reconstruction and detection. The detection network is trained based on the enhanced high-contrast super-resolution image. Compared with training directly using the original image, the detection network can more easily distinguish the target from the complex background, reducing the false alarm rate.

[0040] The above is only used to illustrate the technical solution of the present invention and not to limit it. Any other modifications or equivalent substitutions made by those skilled in the art to the technical solution of the present invention, as long as they do not depart from the spirit and scope of the technical solution of the present invention, should be covered within the scope of the claims of the present invention.

Claims

1. A method for super-resolution reconstruction and detection of infrared weak targets based on spatiotemporal energy aggregation constraints, characterized in that, Includes the following steps: Step 1: Construct a super-resolution reconstruction network framework for infrared weak targets. The core baseline of the framework is a lightweight BasicVSR++ model with ROI awareness, and the input low-resolution infrared video sequence is preprocessed. Step 2: Reconstruct the core structure of the lightweight BasicVSR++ baseline model with ROI awareness; Step 3: Design the local contrast LCM constraint loss function for the target region; Step 4: Construct a feature extraction and detection head structure based on a lightweight detection baseline network; Step 5: The detection network completes the localization and detection process of infrared weak targets, and the super-resolution image sequence is used for visualization analysis and performance verification.

2. The method for super-resolution reconstruction and detection of infrared weak targets based on spatiotemporal energy aggregation constraints according to claim 1, characterized in that: The network framework in step one retains the core second-order grid bidirectional recursive propagation mechanism to extract the global temporal context. It is divided into two parts: a super-resolution module and a detection module with mutually independent and cascaded optimized spatiotemporal aggregation constraints. The input low-resolution infrared video sequence is preprocessed to obtain a continuous and standardized infrared video frame sequence. At the same time, the positions of weak infrared targets in the sample set are marked, and target ROI mask maps and detection labels are generated.

3. The method for super-resolution reconstruction and detection of infrared weak targets based on spatiotemporal energy aggregation constraints according to claim 2, characterized in that: The specific method for constructing the ROI-aware lightweight BasicVSR++ model and preprocessing the video sequence in step one is as follows: S1. Perform channel clipping and branch simplification on the native BasicVSR++, compress the number of main convolution channels, remove redundant loop propagation structures, adapt to infrared single-channel grayscale image input, and build a lightweight baseline. S2. Embed an ROI-aware branch at the front end of the baseline, and use the target ROI mask as a guide to perform weight adaptive enhancement on the target region during the feature extraction stage. S3. Perform inter-frame denoising, grayscale normalization, and size unification on the input infrared video sequence. Mark the target position manually or semi-automatically and generate a target ROI mask map and detection label.

4. The method for super-resolution reconstruction and detection of infrared weak targets based on spatiotemporal energy aggregation constraints according to claim 1, characterized in that: In step two, when reconstructing the core structure of the lightweight BasicVSR++ baseline model, during the multi-frame feature alignment stage, the SpyNet optical flow module of the original BasicVSR++ network is replaced by a low-level deformable convolutional TDAN alignment module to achieve accurate multi-frame feature alignment of textureless infrared weak targets. In the spatiotemporal feature fusion stage, the original residual fusion network of BasicVSR++ is abandoned and replaced by a lightweight temporal attention fusion module with local window. After completing bidirectional feature propagation and convergence, temporal attention aggregation is performed within the local window containing the target.

5. The method for super-resolution reconstruction and detection of infrared weak targets based on spatiotemporal energy aggregation constraints according to claim 4, characterized in that: The specific construction method of the TDAN alignment module is as follows: The TDAN alignment module uses a two-layer deformable convolution to build a low-level lightweight structure. The first layer completes coarse-grained motion estimation between adjacent frames and the current frame. The second layer combines ROI masks to complete fine-grained motion offset correction of the target region and outputs a multi-frame feature map with accurate alignment, avoiding the alignment deviation of the SpyNet optical flow module in textureless scenes. The specific construction method of the local window lightweight temporal attention fusion module is as follows: The local window lightweight temporal attention fusion module adopts a non-overlapping local window partitioning strategy, performs temporal attention calculation within a local window composed of partial feature maps, and combines ROI masks to enhance the weight of windows containing targets.

6. The method for super-resolution reconstruction and detection of infrared weak targets based on spatiotemporal energy aggregation constraints according to claim 1, characterized in that: In step three, when designing the target region local contrast LCM constraint loss function, the local contrast metric is used as the core component of the loss function of the super-resolution image synthesis network. Combined with the general pixel-level loss and edge gradient loss, a joint training loss specific to the super-resolution module is constructed. The LCM constraint loss forces the local contrast between the target region and its neighboring background to remain unchanged or positively enhanced during the super-resolution reconstruction process, locking in and preserving the core radiation features and saliency of the weak infrared target. The super-resolution network is trained separately to obtain the super-resolution reconstructed image.

7. The method for super-resolution reconstruction and detection of infrared weak targets based on spatiotemporal energy aggregation constraints according to claim 6, characterized in that: After the LCM constraint loss function is introduced, the overall super-resolution module joint training loss function is: For pixel loss terms, These are the mask gradient loss weight coefficients. For the mask gradient loss term, The contrast loss weighting coefficient is... This is the local contrast constraint loss term for LCM.

8. The method for super-resolution reconstruction and detection of infrared weak targets based on spatiotemporal energy aggregation constraints according to claim 1, characterized in that: In step four, when building the feature extraction and detection head structure, a classification loss and regression loss specific to the detection task are designed. High-resolution infrared image samples and corresponding detection labels are used. The detection module is independently trained using super-resolution image sequences to complete the parameter optimization and convergence of the detection module.

Images