Spatial and temporal feature enhancement method for infrared video denoising

Abstract

The invention provides a spatial-temporal feature enhancement method for infrared video denoising, and the method comprises the steps: carrying out the convolution operation of a to-be-processed infrared video frame by frame, and obtaining a preliminary infrared video feature; performing global local feature enhancement processing and down-sampling operation on the preliminary infrared video features based on a preset first global local feature enhancement module to obtain shallow infrared video features; performing global local feature enhancement processing on the shallow infrared video features based on a preset second global local feature enhancement module to obtain intermediate infrared video features; performing up-sampling operation on the intermediate infrared video features and performing global local feature enhancement processing based on a preset third global local feature enhancement module to obtain initial de-noised video features; and performing convolution operation on the initial de-noised video features to obtain a de-noised infrared video. According to the method, the global features and the local features of the infrared video are fully utilized, and the denoising effect is improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method for enhancing spatiotemporal features for infrared video denoising. Background Art

[0002] Since the 21st century, electronics, computer science, and microsystems have made tremendous progress. Imaging systems, which integrate these technologies, have seen even greater development. Imaging systems are widely used in various fields, including industry, medicine, and security. However, due to inherent issues such as the stereoscopic projection characteristics of infrared radiation, variability in material emissivity, and inconsistent focal plane array response, infrared images are generally subject to severe noise contamination. This noise not only significantly degrades image quality but also severely impacts the accuracy and reliability of downstream tasks. This has become a major bottleneck restricting the widespread application of infrared technology, particularly in critical applications such as object detection in autonomous driving and lesion identification in medical diagnosis, which require high image quality.

[0003] Solving the problem of infrared imaging noise from a hardware perspective presents significant challenges. Improving detector performance requires overcoming technical limitations in multiple fields, including materials science and microelectronics, resulting in difficult, time-consuming, and costly R&D. Furthermore, hardware optimization often targets only specific types of noise, making it difficult to comprehensively suppress noise from multiple sources. In contrast, deep learning-based denoising solutions offer significant advantages: they are low-cost, flexible, and capable of addressing all types of noise throughout the entire imaging process.

[0004] Since DnCNN introduced deep neural networks to the field of image denoising, deep learning methods have achieved breakthrough progress in this task. However, existing research still has several key limitations: First, most methods focus on denoising single-frame images in visible light; furthermore, these methods ignore the rich temporal information in video sequences; second, when processing video streams, traditional attention mechanisms have high computational complexity when modeling long-sequence dependencies, making them difficult to meet the requirements of real-time processing. Summary of the Invention

[0005] The present invention proposes a spatiotemporal feature enhancement method for infrared video denoising, so as to overcome or partially overcome the above problems.

[0006] The present invention provides a method for enhancing spatiotemporal features for infrared video denoising, based on a pre-built spatiotemporal feature enhancement network for infrared video denoising, and the method comprises:

[0007] S1, perform convolution operation on the infrared video to be processed frame by frame to obtain preliminary infrared video features;

[0008] S2. Performing global and local feature enhancement processing and downsampling operations on the preliminary infrared video features based on a preset first global and local feature enhancement module to extract a first intra-frame global feature of each video frame and a first local region refined feature between different video frames from the preliminary infrared video features to obtain shallow infrared video features;

[0009] S3, performing global local feature enhancement processing on the shallow infrared video feature based on a preset second global local feature enhancement module, so as to extract a second intra-frame global feature of each video frame and a second local region refined feature between different video frames from the shallow infrared video feature, and obtain an intermediate infrared video feature;

[0010] S4, performing an upsampling operation on the intermediate infrared video features and performing global-local feature enhancement processing based on a preset third global-local feature enhancement module to extract a third intra-frame global feature of each video frame and a third local region refined feature between different video frames from the intermediate infrared video features to obtain an initial denoised video feature;

[0011] S5. Perform a convolution operation on the initial denoised video features to obtain a denoised infrared video frame corresponding to each video frame in the infrared video to be processed, so as to obtain a denoised infrared video.

[0012] Furthermore, while performing the upsampling operation on the intermediate infrared video features and performing global-local feature enhancement processing based on the preset third global-local feature enhancement module, the method further includes:

[0013] Performing a convolution operation on the shallow infrared video global local features obtained after the first global local feature enhancement module performs global local feature enhancement processing on the preliminary infrared video features to obtain the side source infrared video features;

[0014] The side source infrared video features are input into the third global local feature enhancement module, and the output features of the third global local feature enhancement module are used as initial denoising video features.

[0015] Furthermore, before performing the convolution operation on the infrared video to be processed frame by frame, the method further includes:

[0016] S01. Obtain an infrared video sample pair, where the infrared video sample pair includes an infrared video sample and a denoised label sample corresponding to the infrared video sample;

[0017] S02, performing the operations of steps S1 to S5 on the infrared video sample, and obtaining shallow infrared video features of the infrared video sample in step S2, obtaining deep infrared video features of the infrared video sample by upsampling the intermediate infrared video features in step S4, and obtaining a denoised infrared video of the infrared video sample in step S5;

[0018] S03, performing an auxiliary convolution operation on the shallow infrared video features of the infrared video sample to obtain shallow auxiliary infrared denoising video features corresponding to the infrared video sample, and performing an auxiliary convolution operation on the deep infrared video features of the infrared video sample to obtain deep auxiliary infrared denoising video features corresponding to the infrared video sample;

[0019] S04. Calculating a network loss based on the denoised infrared video and denoised label samples of the infrared video sample; calculating a progressive self-distillation loss based on the denoised infrared video, denoised label samples, shallow infrared video features, deep infrared video features, and shallow auxiliary infrared denoising video features and deep auxiliary infrared denoising video features corresponding to the infrared video sample; and adding the network loss and the progressive self-distillation loss to obtain a total loss;

[0020] S05. Adjust the model parameters of the preset spatiotemporal feature enhancement network model for infrared video denoising that executes steps S1 to S5 based on the total loss until the total loss reaches a preset loss threshold.

[0021] Furthermore, the step of calculating the progressive self-distillation loss based on the denoised infrared video of the infrared video sample, the denoised label sample, the shallow infrared video features, the deep infrared video features, and the shallow auxiliary infrared denoising video features and the deep auxiliary infrared denoising video features corresponding to the infrared video sample includes:

[0022] The progressive self-distillation loss is calculated based on the progressive self-distillation loss calculation model, and the progressive self-distillation loss calculation model is:

[0023] Loss PSD (O1,O2,D,T)=Loss STC (O1,T)+Loss STC (O2,T)+Loss STC (O1,D)+Loss STC (O2,D)+Loss STC (M1,M2)

[0024] Where O1 is the shallow-layer auxiliary infrared denoising video feature; O2 is the deep-layer auxiliary infrared denoising video feature; D is the denoised infrared video; T is the denoised label sample; M1 is the shallow-layer infrared video feature; M2 is the deep-layer infrared video feature; Loss PSD() is the progressive self-distillation loss calculation model, Loss STC () is the preset loss calculation model.

[0025] Furthermore, the Loss PSD () is the time-space consistency progressive self-distillation loss calculation model, Loss STC () is the spatiotemporal consistency loss calculation model; where,

[0026] The spatiotemporal consistency loss calculation model is:

[0027]

[0028] Where V1 and V2 are infrared video features at two different processing stages; N is the number of infrared video samples; F is the number of frames of each infrared video sample; L spatial () is the space loss calculation model; L frequency () is the frequency loss calculation model; L temporal () is the temporal consistency loss calculation model;

[0029] The space loss calculation model is:

[0030] L spatial =||V1-V2|| fro

[0031] Where, || || fro is the Frobenius norm;

[0032] The frequency loss calculation model is:

[0033] L frequency =|||FT(V1)-FT(V2)||2·M

[0034] Where FT() is Fourier transform; M is the frequency mask; || ||2 is the Euclidean norm;

[0035] The temporal consistency loss calculation model is:

[0036]

[0037] Where Cat() represents the concatenation matrix.

[0038] Furthermore, the first global-local feature enhancement module, the second global-local feature enhancement module, and the third global-local feature enhancement module are a network structure model of the structure, which is used to perform the following operations on the input video features:

[0039] GLEM(F i )=GMB(Con 3×3,5×5,7×7 (Fi ))+LMB(GMB(Con 3×3,5×5,7×7 (F i )))

[0040] Where GMB is the global Mamba module, which is used to extract global feature pairs within the video frame through long-range dependency modeling; LMB is the local Mamba module, which is used to achieve fine reconstruction of local areas of multiple video frames; Con 3×3,5×5,7×7 It is a multi-scale convolution layer with convolution kernel sizes of 3×3, 5×5 and 7×7 respectively.

[0041] Furthermore, the global Mamba module is used to perform the following operations:

[0042] Convert the input high-dimensional video features into space-first one-dimensional sequence features;

[0043] The spatial-priority one-dimensional sequence features are input into a preset first Mamba module, and the first Mamba module processes the input high-dimensional video features to obtain the intra-frame global features of each video frame.

[0044] Furthermore, the local Mamba module is used to perform the following operations:

[0045] Convert the pixel coordinates of each frame of the input high-dimensional video features into a unique index based on the Hilbert space filling curve;

[0046] Performing a space filling scan on the high-dimensional video feature according to the unique index based on the Hilbert space filling curve to form a one-dimensional sequence feature with position priority in which pixel positions of each video frame are adjacent;

[0047] The position-prioritized one-dimensional sequence features are input into a preset second Mamba module, and the second Mamba module processes the input high-dimensional video features to obtain refined features of local areas between different video frames.

[0048] This application provides a method for enhancing spatiotemporal features for infrared video denoising. This method uses a U-shaped network to denoise infrared videos, avoiding information loss during the sampling process. A global-local feature enhancement module is also used to extract global features within each video frame and refined features of local regions between different frames. This fully processes feature information at different levels within the infrared video, improving denoising efficiency and effectiveness.

[0049] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are specifically listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Various other advantages and benefits will become apparent to those skilled in the art by reading the detailed description of the preferred embodiment below. The accompanying drawings are only for the purpose of illustrating the preferred embodiment and are not to be considered as limiting the present invention. In the accompanying drawings:

[0051] Figure 1 Flowchart of a method for enhancing spatiotemporal features for infrared video denoising according to an embodiment of the present invention;

[0052] Figure 2 This is a schematic diagram of the structure of a network model for enhancing spatiotemporal features of infrared video denoising according to an embodiment of the present invention;

[0053] Figure 3 Schematic diagram of a global scanning method according to an embodiment of the present invention;

[0054] Figure 4 A schematic diagram of a local scanning method according to an embodiment of the present invention;

[0055] Figure 5 Schematic diagram of the structure of the spatiotemporal feature enhancement network model for infrared video denoising in the model training stage according to an embodiment of the present invention;

[0056] Figure 6 Comparison of the reconstruction results of the spatiotemporal feature enhancement method for infrared video denoising provided by an embodiment of the present invention and other SOTA methods on different test videos;

[0057] Figure 7 This is a diagram of the reconstruction results and difference diagram of the spatiotemporal feature enhancement method for infrared video denoising provided by an embodiment of the present invention in different frames of the same video. DETAILED DESCRIPTION

[0058] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0059] It should be noted that the spatiotemporal feature enhancement method for infrared video denoising in the embodiment of the present invention is performed based on a pre-built spatiotemporal feature enhancement network for infrared video denoising, wherein: Figure 1 A temporal and spatial feature enhancement method for infrared video denoising according to an embodiment of the present invention is shown. Figure 1 It can be seen that the spatiotemporal feature enhancement method for infrared video denoising of the present invention includes the following steps:

[0060] S1, perform convolution operation on the infrared video to be processed frame by frame to obtain preliminary infrared video features;

[0061] In an embodiment of the present invention, the infrared video to be processed includes multiple video frames. The multiple video frames of the infrared video are input into the convolutional layer at the input end of the infrared video denoising spatiotemporal feature enhancement network to perform preliminary feature extraction frame by frame to obtain preliminary infrared video features:

[0062] F1=ConV1(F in ) (1)

[0063] S2. Performing global and local feature enhancement processing and downsampling operations on the preliminary infrared video features based on a preset first global and local feature enhancement module to extract a first intra-frame global feature of each video frame and a first local region refined feature between different video frames from the preliminary infrared video features to obtain shallow infrared video features;

[0064] In an embodiment of the present invention, preliminary infrared video features are input into a preset first global-local feature enhancement module for further processing. This global-local feature enhancement module uses a dual Mamba structure for efficient spatiotemporal modeling: the global Mamba module extracts global features within the frame through long-range dependency modeling, while the local Mamba module based on space-filling curve theory realizes refined reconstruction of local areas in multiple frames. The specific implementation process will be described in subsequent embodiments.

[0065] Step 2 is the downsampling process in the embodiment of the present invention. It should be noted that, in order to simplify the description, the present invention is attached. Figure 2 The downsampling process in [1] only involves a global-local feature extraction and downsampling process. For more complex network functions, multiple first global-local feature enhancement modules and downsampling convolution layers can be cascaded to perform cascaded global-local feature extraction and downsampling convolution processing.

[0066] S3, performing global local feature enhancement processing on the shallow infrared video feature based on a preset second global local feature enhancement module, so as to extract a second intra-frame global feature of each video frame and a second local region refined feature between different video frames from the shallow infrared video feature, and obtain an intermediate infrared video feature;

[0067] In an embodiment of the present invention, it is an intermediate processing module connecting the downsampling process and the upsampling process in the network model. The module uses a preset second global local feature enhancement module to process the shallow infrared video features finally output in the downsampling process to obtain intermediate infrared video features, and in step S4, the intermediate infrared video features are input into the upsampling processing process.

[0068] S4, performing an upsampling operation on the intermediate infrared video features and performing global-local feature enhancement processing based on a preset third global-local feature enhancement module to extract a third intra-frame global feature of each video frame and a third local region refined feature between different video frames from the intermediate infrared video features to obtain an initial denoised video feature;

[0069] It should be noted that step 4 in this embodiment is different from the Figure 2 The side source feature input between the first global local feature enhancement module and the third global local feature enhancement module at the top in the downsampling process and the upsampling process is omitted, and only the features are upsampled step by step to obtain the final initial denoised video features.

[0070] Similar to the downsampling process in step S2, in the upsampling process in the embodiment of the present invention, multiple upsampling convolution modules and the third global-local feature enhancement module can also be cascaded to perform cascade upsampling and global-local feature extraction processing.

[0071] S5. Perform a convolution operation on the initial denoised video features to obtain a denoised infrared video frame corresponding to each video frame in the infrared video to be processed, so as to obtain a denoised infrared video.

[0072] The embodiment of the present invention reshapes the feature sequence into the original video features through the last convolutional layer in the network model, and outputs a real noise-free video.

[0073] Furthermore, Figure 2 In another specific embodiment of the present invention shown, the first global local feature enhancement module is also input into the third global local feature enhancement module after convolution processing, so that the video features obtained in the downsampling process can guide the video feature processing in the upsampling process, thereby avoiding the loss of key features in the downsampling process.

[0074] Specifically, while performing an upsampling operation on the intermediate infrared video features and performing global local feature enhancement processing based on a preset third global local feature enhancement module in step S4, the method further includes: performing a convolution operation on the shallow infrared video global local features obtained after the first global local feature enhancement module performs global local feature enhancement processing on the preliminary infrared video features to obtain side source infrared video features; inputting the side source infrared video features into the third global local feature enhancement module, and using the output features of the third global local feature enhancement module as the initial denoising video features.

[0075] In the embodiment of the present invention, the first global-local feature enhancement module, the second global-local feature enhancement module and the third global-local feature enhancement module are structured as a network structure model. The model innovatively adopts a dual Mamba structure for efficient spatiotemporal modeling for the spatiotemporal feature enhancement method of infrared video denoising: the global Mamba module extracts global features within the frame through long-range dependency modeling, while the local Mamba module based on the space filling curve theory realizes the refined reconstruction of local areas of multiple frames, thereby improving the denoising effect of the network model.

[0076] Furthermore, in the embodiment of the present invention, the first global-local feature enhancement module, the second global-local feature enhancement module, and the third global-local feature enhancement module are network structure models with the same structure but different numbers of channels, and are used to perform the following operations on the input video features:

[0077] GLEM(F i )=GMB(Con 3×3,5×5,7×7 (F i ))+LMB(GMB(Con 3×3,5×5,7×7 (F i ))) (2)

[0078] Where GLEM is the global local feature enhancement module, GMB is the global Mamba module, which is used to extract global feature pairs within the video frame through long-range dependency modeling; LMB is the local Mamba module, which is used to achieve fine reconstruction of local areas of multiple video frames; Con 3×3,5×5,7×7 It is a multi-scale convolution layer with convolution kernel sizes of 3×3, 5×5 and 7×7 respectively.

[0079] Furthermore, the global Mamba module in the embodiment of the present invention is used to perform the following operations: converting the input high-dimensional video features into space-priority one-dimensional sequence features; inputting the space-priority one-dimensional sequence features into a preset first Mamba module, and processing the first Mamba module to obtain the intra-frame global features of each video frame in the input high-dimensional video features.

[0080] Specifically, for the global Mamba module structure such as Figure 2 As shown in , in order to break through the computational performance bottleneck of the traditional attention mechanism in video modeling, the global Mamba module is first used to establish the dependency relationship between pixels in the video through a line-by-line scanning strategy. Figure 3 As shown in the figure, the X and Y axes represent pixels in specific rows and columns within each frame, while the Z axis represents different frames in the video. This frame-by-frame scanning method effectively models global spatial features within the frame through its excellent ability to model long-range dependencies. This significantly reduces computational complexity while preserving the integrity of video features.

[0081] Specifically, firstly the input high-dimensional video features Convert to a space-first one-dimensional sequence The reconstructed feature sequence V is then fed into the Mamba module for processing. The features are then fed into a channel-wise convolutional layer to preserve fine-grained details.

[0082] Furthermore, Figure 2 The Mamba module is also shown. The Mamba module is mainly used to complete the following operations:

[0083] V l =BM(LN(V l-1 ))+V l-1 (3)

[0084] V l+1 =DWC(LN(V l ))+V l (4)

[0085] Where BM stands for Bi-Mamba module, LN stands for Layer Norm layer, and DWC stands for channel-by-channel convolution. l-1 is the input feature of the Mamba module, V l is the intermediate feature processed by the Mamba module, V l+1 is the output feature of the Mamba module. Since the Mamba module is a prior art, the present invention will not elaborate on it.

[0086] Furthermore, the local Mamba module of the embodiment of the present invention is used to complete the following operations: converting the pixel coordinates of each frame of video in the input high-dimensional video features into a unique index based on the Hilbert space filling curve; performing a space filling scan on the high-dimensional video features according to the unique index based on the Hilbert space filling curve to form a position-priority one-dimensional sequence feature in which the pixel positions of each video frame are adjacent; inputting the position-priority one-dimensional sequence feature into a preset second Mamba module, and processing the second Mamba module to obtain refined features of the local areas between different video frames in the input high-dimensional video features.

[0087] Specifically, if Figure 4 As shown in , to compensate for the local feature degradation that may be caused by global modeling, we introduce the Hilbert space filling curve into video feature modeling. Unlike traditional line-by-line scanning, the Hilbert curve has excellent locality and space filling properties, so that spatially adjacent pixels still maintain similar index positions after one-dimensional mapping, as shown in Figure 3This property enables the network to accurately model local spatiotemporal structures while maintaining computational efficiency. Specifically, we first convert the pixel coordinates of each frame of video into a unique index based on the Hilbert curve. This index is formed by performing a spatial filling scan of the video frame to form a linear sequence, where the position of each pixel reflects its relative spatial position in the video frame. This process can be described by the following formula:

[0088] Index=x·Q·W+y·C+z (5)

[0089] Where W is the width of the input feature, C is the number of channels of the input feature, and through Hilbert scanning, the feature Expand into a one-dimensional sequence Through the Hilbert curve scanning order, the features are input into the Mamba module for processing. The process can be described by the following formula:

[0090]

[0091] Where BM stands for Bi-Mamba module, LN stands for LayerNorm layer, and DWC stands for channel-by-channel convolution. is the input feature of the Mamba module, Intermediate features processed by the Mamba module, The output features of the Mamba module are shown in Figure 2. Finally, we reshape the output sequence back to its original shape. This fractal geometry-based feature reorganization strategy not only theoretically ensures optimal preservation of local information but also enables the network to more accurately capture the fine-grained spatiotemporal features in video data.

[0092] Furthermore, in deep neural networks, the features extracted at different levels are complementary. However, the traditional end-to-end training method only applies supervision signals at the network output. This single supervision method seriously restricts the ability of each level of the network to model noise features.

[0093] In addition, for video denoising tasks, traditional pixel-domain loss functions (such as mean square error and L1 loss) have two inherent defects: first, the pixel-by-pixel difference calculation cannot accurately distinguish between noise and valuable high-frequency details, resulting in insufficient feature extraction in the intermediate layer; second, this supervision method completely ignores the temporal correlation of video sequences, making it difficult for the network to learn stable spatiotemporal representations.

[0094] Furthermore, in order to better overcome the shortcomings of the existing loss function calculation, the method based on the pre-built infrared video denoising spatiotemporal feature enhancement network further includes training the infrared video denoising spatiotemporal feature enhancement network:

[0095] The training of the infrared video denoising spatiotemporal feature enhancement network includes:

[0096] S01. Obtain an infrared video sample pair, where the infrared video sample pair includes an infrared video sample and a denoised label sample corresponding to the infrared video sample;

[0097] S02, performing the operations of steps S1 to S5 on the infrared video sample, and obtaining shallow infrared video features of the infrared video sample in step S2, obtaining deep infrared video features of the infrared video sample by upsampling the intermediate infrared video features in step S4, and obtaining a denoised infrared video of the infrared video sample in step S5;

[0098] S03, performing an auxiliary convolution operation on the shallow infrared video features of the infrared video sample to obtain shallow auxiliary infrared denoising video features corresponding to the infrared video sample, and performing an auxiliary convolution operation on the deep infrared video features of the infrared video sample to obtain deep auxiliary infrared denoising video features corresponding to the infrared video sample;

[0099] S04. Calculating a network loss based on the denoised infrared video and denoised label samples of the infrared video sample; calculating a progressive self-distillation loss based on the denoised infrared video, denoised label samples, shallow infrared video features, deep infrared video features, and shallow auxiliary infrared denoising video features and deep auxiliary infrared denoising video features corresponding to the infrared video sample; and adding the network loss and the progressive self-distillation loss to obtain a total loss;

[0100] S05. Adjust the model parameters of the preset spatiotemporal feature enhancement network model for infrared video denoising that executes steps S1 to S5 based on the total loss until the total loss reaches a preset loss threshold.

[0101] like Figure 5 As shown in Figure 1, this embodiment of the present invention also introduces two auxiliary convolutional layers during the model training phase (enabled only during training and removed during inference, without adding additional inference cost). O1, O2, M1, and M2 represent the output feature maps of each stage in the network. The spatiotemporal consistency progressive self-distillation loss provided by this embodiment of the present invention uses the network's deep features, network outputs, and true value labels as multiple teacher signals.

[0102] Furthermore, the progressive self-distillation loss provided by the embodiment of the present invention not only calculates the network output infrared video and the true label infrared video, but also calculates the loss of the network's intermediate feature maps. It then calculates the loss of the shallow infrared denoised video and the deep infrared denoised video, the true label, and the network's final output. It optimizes the difference between the intermediate output and the network's final output and the true label, gradually distilling a more refined representation. Secondly, the network's deep feature map M2 is used as a supervisory signal to guide the shallow layer M1 in characterization.

[0103] Specifically, in an embodiment of the present invention, the denoised infrared video, denoised label sample, shallow infrared video feature, deep infrared video feature, and shallow infrared denoised video vector and deep infrared denoised video vector corresponding to the infrared video sample are used to calculate the progressive self-distillation loss, including: calculating the progressive self-distillation loss of the infrared video denoising spatiotemporal feature enhancement network based on the progressive self-distillation loss calculation model, wherein the progressive self-distillation loss calculation model is:

[0104] Loss PSD (O1,O2,D,T)=Loss STC (O1,T)+Loss STC (O2,T)+Loss STC (O1,D)+Loss STC (O2,D)+Loss STC (M1,M2) (9)

[0105] Where O1 is the shallow-layer auxiliary infrared denoising video feature; O2 is the deep-layer auxiliary infrared denoising video feature; D is the denoised infrared video; T is the denoised label sample; M1 is the shallow-layer infrared video feature; M2 is the deep-layer infrared video feature; Loss PSD () is the progressive self-distillation loss calculation model, Loss STC () is the preset loss calculation model.

[0106] Furthermore, Loss STC () can be a loss calculation model between other different feature vectors in the prior art. As a preferred embodiment of the present invention, a spatiotemporal consistency loss calculation model is proposed. PSD () is the calculation model of spatiotemporal consistency progressive self-distillation loss.

[0107] The embodiment of the present invention provides a spatio-temporal consistency loss (STCL). The spatio-temporal consistency self-distillation loss function we designed is divided into three main parts: spatial loss, frequency loss, and temporal consistency loss.

[0108] Furthermore, the space loss calculation model is:

[0109] L spatial =||V1-V2|| fro (10)

[0110] In the above formula, || || fro is the Frobenius norm, where V1 and V2 are infrared videos at two different processing stages. In some embodiments, V1 can be expressed as D, i.e., the denoised infrared video D∈R output by the network. N×F×H×W , V1 can be represented as an infrared video T∈R with true labels N×F×H×W , where N is the number of infrared video samples, F is the number of video frames contained in each infrared video sample, and H×W is the size of each video frame.

[0111] Furthermore, the frequency loss calculation model of the embodiment of the present invention is:

[0112] L frequency =|||FT(V1)-FT(V2)|||2·M (11)

[0113] Where FT() is the Fourier transform; M is the frequency mask; || ||2 is the Euclidean norm.

[0114] Furthermore, noise is typically reflected as high-frequency signals in the frequency domain. To effectively suppress low-frequency signals, we designed a frequency mask. This mask is weighted according to the spatial frequency distribution of the image, giving greater weight to high-frequency components far from the center of the frequency domain, thereby strengthening the loss calculation of high-frequency noise.

[0115] The calculation formula for the frequency mask is:

[0116]

[0117] In the above formula (c x ,c y ) is the image center coordinate, and τ is the set frequency cutoff value.

[0118] Furthermore, to maintain the dynamic consistency of the video, the embodiment of the present invention further introduces inter-frame temporal frequency loss. This loss measures the difference between adjacent frames output by the network and adjacent frames of the true label. The temporal consistency loss calculation model of the embodiment of the present invention is:

[0119]

[0120] Where Cat() represents the concatenation matrix. V1 t and V1 t+1 Represents a pair of adjacent frames.

[0121] Furthermore, the total loss of network training is the sum of the distillation loss and the L2 loss of the network's final output and the true label, which can be expressed as follows:

[0122] Loss=a*Loss PSD (O1,O2D,T)+||DT||2 (14)

[0123] α is set to 0.1.

[0124] To further validate the feasibility and superiority of the method described in this embodiment, the proposed network was trained on an Nvidia GPU using the PyTorch software framework. Due to the high I / O load of video data, the Nvidia DALI video processing framework was employed to reduce latency during data preprocessing. The proposed model was trained for 30 epochs using the Adam optimizer with momentum terms of (0.9, 0.999). The learning rate was adjusted using a cosine annealing strategy, initially set to 1e-4 and gradually decayed to 2e-6.

[0125] To further verify the effectiveness of the proposed network, we conducted quantitative comparisons with the current SOTA methods for video denoising and image denoising, and the results are shown in Tables 1 and 2.

[0126] Table 1 Comparison of the network and the current video denoising SOTA method

[0127]

[0128]

[0129] As shown in the table above, comparative experiments demonstrate that the proposed model demonstrates exceptional performance in infrared video denoising. Across three key performance metrics, PSNR, SSIM, and ST-RRED, the proposed method significantly surpasses state-of-the-art image and video denoising methods, fully demonstrating its technical advantages in image quality reconstruction and visual information recovery. Furthermore, while maintaining high performance, the model also maintains exceptionally high computational efficiency, processing a single frame in just 9.47ms, achieving a perfect balance between performance and efficiency.

[0130] Furthermore, in order to provide a more intuitive evaluation, the present invention shows the visualization results of the reconstructed images of some scenes in the test set, where: Figure 6A visual comparison shows the typical limitations of existing methods in video reconstruction quality: the reconstruction results of FastDVDnet and PaCnet show obvious over-smoothing, indicating that they are insufficient in preserving high-frequency details; although BSVD has a certain effect on overall denoising, significant noise still remains in the edge areas, reflecting the limitations of its edge feature extraction capabilities; Basic VSR++, MMNet, and FloRNN exhibit typical temporal artifacts, indicating that they have obvious deficiencies in temporal feature alignment and fusion. In contrast, the method proposed in this paper achieves superior reconstruction results through a carefully designed spatiotemporal feature extraction architecture and training strategy: it not only effectively avoids the generation of temporal artifacts such as ghosting, but also maintains rich texture details, presenting a more natural visual effect.

[0131] In order to verify the temporal consistency of the network, we also compared the reconstruction results of the network in different frames. Figure 7 Results demonstrate that the proposed network exhibits excellent processing capabilities in both spatial and temporal dimensions. In the spatial dimension, the network not only effectively suppresses noise interference but also preserves image detail and texture information. More importantly, the difference images of adjacent reconstructed frames demonstrate that the proposed network achieves excellent temporal consistency in the temporal dimension, accurately capturing and reconstructing the target object's motion trajectory while effectively avoiding the issues of temporal jitter and motion blur common in traditional methods.

[0132] The present invention provides a method for enhancing spatiotemporal features for infrared video denoising. Considering that noise suppression at the hardware level often faces technical bottlenecks and high costs, the present invention proposes a real-time infrared video denoising network based on a progressive spatiotemporal self-distillation training strategy. This network innovatively employs a dual Mamba structure for efficient spatiotemporal modeling: a global Mamba module extracts global features within a frame through long-range dependency modeling, while a local Mamba module based on space-filling curve theory enables refined reconstruction of local regions across multiple frames. To fully utilize the spatiotemporal information of video sequences, a progressive spatiotemporal consistency self-distillation training strategy is proposed. This strategy uses deep network features, network outputs, and true labels as multiple teacher signals, continuously optimizing feature extraction capabilities in both temporal and spatial dimensions without the need for additional teacher networks.

[0133] Those skilled in the art will understand that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by those skilled in the art in the art to which the present invention pertains. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with those in the context of the prior art and, unless specifically defined, will not be interpreted in an idealized or overly formal sense.

[0134] For simplicity of description, the method embodiments are described as a series of actions. However, those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because certain steps can be performed in other orders or simultaneously according to the embodiments of the present invention. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0135] Furthermore, those skilled in the art will appreciate that although some embodiments herein include certain features included in other embodiments but not other features, combinations of features from different embodiments are intended to be within the scope of the present invention and to form different embodiments. For example, any of the claimed embodiments may be used in any combination.

[0136] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for enhancing spatiotemporal features for infrared video denoising, characterized in that: The method comprises: S1, perform convolution operation on the infrared video to be processed frame by frame to obtain preliminary infrared video features; S2. Performing global and local feature enhancement processing and downsampling operations on the preliminary infrared video features based on a preset first global and local feature enhancement module to extract a first intra-frame global feature of each video frame and a first local region refined feature between different video frames from the preliminary infrared video features to obtain shallow infrared video features; S3, performing global local feature enhancement processing on the shallow infrared video feature based on a preset second global local feature enhancement module, so as to extract a second intra-frame global feature of each video frame and a second local region refined feature between different video frames from the shallow infrared video feature, and obtain an intermediate infrared video feature; S4, performing an upsampling operation on the intermediate infrared video features and performing global-local feature enhancement processing based on a preset third global-local feature enhancement module to extract a third intra-frame global feature of each video frame and a third local region refined feature between different video frames from the intermediate infrared video features to obtain an initial denoised video feature; S5. Perform a convolution operation on the initial denoised video features to obtain a denoised infrared video frame corresponding to each video frame in the infrared video to be processed, so as to obtain a denoised infrared video.

2. The method according to claim 1, characterized in that While performing the upsampling operation on the intermediate infrared video features and performing global-local feature enhancement processing based on a preset third global-local feature enhancement module, the method further includes: Performing a convolution operation on the shallow infrared video global local features obtained after the first global local feature enhancement module performs global local feature enhancement processing on the preliminary infrared video features to obtain the side source infrared video features; The side source infrared video features are input into the third global local feature enhancement module, and the output features of the third global local feature enhancement module are used as initial denoising video features.

3. The method according to claim 1 or 2, characterized in that Before performing the convolution operation on the infrared video to be processed frame by frame, the method further includes: S01. Obtain an infrared video sample pair, where the infrared video sample pair includes an infrared video sample and a denoised label sample corresponding to the infrared video sample; S02, performing the operations of steps S1 to S5 on the infrared video sample, and obtaining shallow infrared video features of the infrared video sample in step S2, obtaining deep infrared video features of the infrared video sample by upsampling the intermediate infrared video features in step S4, and obtaining a denoised infrared video of the infrared video sample in step S5; S03, performing an auxiliary convolution operation on the shallow infrared video features of the infrared video sample to obtain shallow auxiliary infrared denoising video features corresponding to the infrared video sample, and performing an auxiliary convolution operation on the deep infrared video features of the infrared video sample to obtain deep auxiliary infrared denoising video features corresponding to the infrared video sample; S04. Calculating a network loss based on the denoised infrared video and denoised label samples of the infrared video sample; calculating a progressive self-distillation loss based on the denoised infrared video, denoised label samples, shallow infrared video features, deep infrared video features, and shallow auxiliary infrared denoising video features and deep auxiliary infrared denoising video features corresponding to the infrared video sample; and adding the network loss and the progressive self-distillation loss to obtain a total loss; S05. Adjust the model parameters of the preset spatiotemporal feature enhancement network model for infrared video denoising that executes steps S1 to S5 based on the total loss until the total loss reaches a preset loss threshold.

4. The method according to claim 3, characterized in that The step of calculating the progressive self-distillation loss based on the denoised infrared video of the infrared video sample, the denoised label sample, the shallow infrared video feature, the deep infrared video feature, and the shallow auxiliary infrared denoising video feature and the deep auxiliary infrared denoising video feature corresponding to the infrared video sample includes: The progressive self-distillation loss is calculated based on the progressive self-distillation loss calculation model, and the progressive self-distillation loss calculation model is: Loss PSD (O1,O2,D,T)=Loss STC (O1,T)+Loss STC (O2,T)+Loss STC (O1,D)+Loss STC (O2,D)+Loss STC (M1,M2) Where O1 is the shallow-layer auxiliary infrared denoising video feature; O2 is the deep-layer auxiliary infrared denoising video feature; D is the denoised infrared video; T is the denoised label sample; M1 is the shallow-layer infrared video feature; M2 is the deep-layer infrared video feature; Loss PSD () is the progressive self-distillation loss calculation model, Loss STC () is the preset loss calculation model.

5. The method according to claim 4, characterized in that The Loss PSD () is the time-space consistency progressive self-distillation loss calculation model, Loss STC () is the spatiotemporal consistency loss calculation model; where, The spatiotemporal consistency loss calculation model is: Where V1 and V2 are infrared video features at two different processing stages; N is the number of infrared video samples; F is the number of frames of each infrared video sample; L spatial () is the space loss calculation model; L frequency () is the frequency loss calculation model; L temporal () is the temporal consistency loss calculation model; The space loss calculation model is: <h2 style=";text-align:left;direction:ltr">L<h2 style=";text-align:left;direction:ltr"> spatial <h2 style=";text-align:left;direction:ltr"> =||V1-V2||<h2 style=";text-align:left;direction:ltr"> fro Where, || || fro is the Frobenius norm; The frequency loss calculation model is: L frequency =|||FT(V1)-FT(V2)||2·M Where FT() is Fourier transform; M is the frequency mask; || ||2 is the Euclidean norm; The temporal consistency loss calculation model is: Where Cat() represents the concatenation matrix.

6. The method according to claim 1, wherein The first global-local feature enhancement module, the second global-local feature enhancement module, and the third global-local feature enhancement module are a network structure model of a structure, which is used to perform the following operations on the input video features: GLEM(F i )=GMB(Con 3×3,5×5,7×7 (F i ))+LMB(GMB(Con 3×3,5×5,7×7 (F i ))) Where GMB is the global Mamba module, which is used to extract global feature pairs within the video frame through long-range dependency modeling; LMB is the local Mamba module, which is used to achieve fine reconstruction of local areas of multiple video frames; Con 3×3,5×5,7×7 It is a multi-scale convolution layer with convolution kernel sizes of 3×3, 5×5 and 7×7 respectively.

7. The method according to claim 6, characterized in that The global Mamba module is used to complete the following operations: Convert the input high-dimensional video features into space-first one-dimensional sequence features; The spatial-priority one-dimensional sequence features are input into a preset first Mamba module, and the first Mamba module processes the input high-dimensional video features to obtain the intra-frame global features of each video frame.

8. The method according to claim 6, characterized in that The local Mamba module is used to complete the following operations: Convert the pixel coordinates of each frame of the input high-dimensional video features into a unique index based on the Hilbert space filling curve; Performing a space filling scan on the high-dimensional video feature according to the unique index based on the Hilbert space filling curve to form a one-dimensional sequence feature with position priority in which pixel positions of each video frame are adjacent; The position-prioritized one-dimensional sequence features are input into a preset second Mamba module, and the second Mamba module processes the input high-dimensional video features to obtain refined features of local areas between different video frames.

Images