Image and video de-noising method and system based on time domain attention mechanism

Abstract

The application discloses an image and video de-disturbance method and system based on a time domain attention mechanism. The method comprises the following steps: inputting a continuous multi-frame turbulent degradation image sequence; performing inter-frame registration and alignment to compensate for geometric distortion; extracting spatial features through a multi-scale feature extraction network; sequentially enhancing spatial feature representation through a spatial attention module, performing learnable frequency domain filtering through a frequency domain attention module, and fusing time sequence features through a time domain attention module; and finally integrating features to reconstruct clear images after turbulent flow. The application innovatively introduces a time domain attention mechanism, effectively utilizes the time sequence correlation between video frames, and solves the inconsistency problem between frames in the existing single-frame processing method in video restoration. Experiments show that the method is significantly better than the existing mainstream method in terms of objective indicators such as PSNR and SSIM, and can effectively enhance the time sequence continuity and stability of the restored video.

Description

Technical Field

[0001] This invention relates to the fields of image processing and computer vision technology, and in particular to an image and video de-perturbation method and system based on a temporal attention mechanism. Background Technology

[0002] In the Earth's troposphere, atmospheric fluids exhibit various flow types at different velocities, including laminar, transitional, and turbulent flow. Atmospheric turbulence originates from the interaction of inertial and viscous forces within the flow field. This interaction causes random fluctuations in the refractive index of the air, which in turn disturbs the phase of the light beam during propagation, ultimately resulting in severe blurring, tilting, and geometric distortion in the images captured by imaging equipment. This phenomenon poses a significant challenge to applications that rely on high-quality image input, such as remote sensing, astronomical observation, and long-distance photography.

[0003] Currently, the main technical approaches to overcoming the effects of atmospheric turbulence fall into two categories: hardware and software. On the hardware side, adaptive optics technology uses a closed-loop system consisting of a wavefront sensor, controller, and corrector to correct wavefront distortion in real time. However, this approach suffers from high system costs, limited high-frequency correction capabilities, and the need for post-processing. On the software side, image restoration technology is one of the mainstream methods. Early methods, such as lucky imaging, fuse images by selecting "lucky frames," but their quality is unstable and limited by diffraction blur. Traditional algorithms based on inverse filtering or elastic registration are often computationally complex and prone to amplifying noise. With the development of deep learning, convolutional neural network-based methods have made progress in single-image restoration, but the limited receptive field of these networks makes it difficult to model long-range dependencies, easily leading to detail loss and over-smoothing. Generative adversarial networks can generate images rich in detail, but the lack of physical constraints can lead to distortion. In recent years, although some research has attempted to introduce multi-frame information, most methods are not specifically designed for turbulence restoration and fail to effectively utilize inter-frame temporal correlations, resulting in significant inter-frame inconsistencies and disjointedness in the restored video.

[0004] Therefore, a prominent problem in the existing technology is the lack of a turbulent image and video restoration scheme that can effectively integrate spatiotemporal information and improve the quality of a single frame while ensuring the temporal continuity of the video. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide an image and video de-perturbation method and system based on a temporal attention mechanism.

[0006] This invention aims to address the problems of inter-frame inconsistency and fragmentation in existing single-frame image restoration methods when processing video, as well as the insufficient utilization of spatiotemporal features in existing multi-frame methods. By constructing a neural network architecture that integrates spatial, frequency, and temporal attention, this invention seeks to achieve the following objectives:

[0007] By aligning and fusing multiple consecutive frames of images, geometric distortion and pixel displacement caused by atmospheric turbulence are effectively compensated.

[0008] By explicitly modeling the temporal dependencies between frames through the designed temporal attention module, the coherence and stability of the restored video are enhanced, and the visual disjointedness is eliminated.

[0009] While significantly improving objective evaluation indicators such as peak signal-to-noise ratio and structural similarity, it achieves restoration results with better visual perception quality and better detail preservation.

[0010] It effectively improves the stability of the restored video and reduces the distortion rate, thereby ensuring the smoothness of the final video output.

[0011] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0012] A method for image and video deperturbation based on a temporal attention mechanism, implemented by a deep learning neural network, includes the following steps:

[0013] Step 1: Input a sequence of consecutive frames of images degraded by atmospheric turbulence;

[0014] Step 2: Perform inter-frame registration and alignment processing on the image sequence to compensate for geometric distortion and pixel displacement caused by turbulence, and obtain aligned multi-frame images;

[0015] Step 3: Input the aligned multi-frame images into a multi-scale feature extraction network, and extract spatial features at different scales through the parallel branch structure of the multi-scale feature extraction network;

[0016] Step 4: The multi-scale spatial features extracted in Step 3 are first input into the spatial attention module. An attention map is generated by calculating the cross-channel covariance, and the spatial feature representation is enhanced by combining a gating mechanism.

[0017] Step 5: Input the features processed by the spatial attention module into the frequency domain attention module. The frequency domain attention module performs the following sub-steps:

[0018] Step 51: Map the input features from the spatial domain to the frequency domain using a Fast Fourier Transform;

[0019] Step 52: Adaptively filter the frequency domain features using a learnable Fourier filter; wherein the parameters of the Fourier filter are optimized during neural network training using a backpropagation algorithm to adaptively enhance the frequency components beneficial to image reconstruction and suppress the unfavorable frequency components introduced by turbulence degradation.

[0020] Step 53: Map the filtered frequency domain features back to the spatial domain using inverse fast Fourier transform;

[0021] Step 6: Input the features processed by the frequency domain attention module into the temporal domain attention module, and generate features with enhanced temporal consistency by fusing the features of the current frame with the temporal features of adjacent frames;

[0022] Step 7: Integrate the features processed in Step 6 through a refinement module to reconstruct a clear image free of turbulence effects;

[0023] Step 8: During the training phase, the neural network is optimized using a joint loss function, which includes pixel-level L1 loss, perceptual loss based on the VGG-19 network, and local pixel block frequency domain loss.

[0024] Preferably, the inter-frame registration and alignment in step 2 is achieved by calculating the local block matching error between adjacent frames and the reference frame, and by estimating the pixel-level displacement field using an elastic deformation model.

[0025] Preferably, the multi-scale feature extraction network in step 3 contains three parallel branches: the first branch maintains the resolution of the input image; the second branch downsamples the input by 2 times and increases the channel size; the third branch downsamples the input by 4 times and further increases the channel size; each branch extracts features through convolution operations, and finally the feature maps output by all branches are fused.

[0026] Preferably, the learnable Fourier filter is configured as follows:

[0027] Receive the frequency domain feature map obtained by the Fast Fourier Transform as input;

[0028] A set of learnable parameter weights is applied to the input frequency domain feature map through element-wise multiplication. The parameter weights are two-dimensional tensors with the same size as the frequency domain feature map, and each weight value corresponds to the importance coefficient of its spatial frequency coordinate position.

[0029] Among them, the learnable parameter weights are optimized through the backpropagation algorithm during the neural network training process;

[0030] The output of the Fourier filter is the frequency domain feature after adaptive filtering, which is then sent to the subsequent inverse fast Fourier transform layer to be converted back to the spatial domain.

[0031] Preferably, the temporal attention module in step 6 achieves temporal fusion in the following way: the features of the current frame are concatenated with the features of adjacent frames, temporal context information is extracted through convolution to generate an attention map, and the features are weighted and fused based on the attention map.

[0032] Preferably, in the processing steps 4, 5, and 6, a block processing strategy is adopted for the feature maps input to the spatial attention module, frequency domain attention module, and temporal attention module. The block processing strategy includes: dividing the input feature map into multiple non-overlapping local blocks, performing attention calculation on each block separately, and merging the blocks into a complete feature map after processing. The processed feature map is then sent to step 7 for final image reconstruction.

[0033] The present invention also provides an image and video deperturbation system based on a temporal attention mechanism, used to implement the aforementioned image and video deperturbation method based on a temporal attention mechanism, the system comprising:

[0034] The frame alignment module is used to perform inter-frame registration and alignment processing on the input image sequence to compensate for geometric distortion and pixel displacement caused by turbulence, and obtain aligned multi-frame images;

[0035] The multi-scale feature extraction module is used to extract spatial features at different scales from aligned multi-frame images. The multi-scale feature extraction module contains multiple parallel branch structures to process inputs at different scales respectively.

[0036] The spatial attention module is used to generate attention maps by calculating cross-channel covariance and to enhance spatial feature representation by combining gating mechanisms.

[0037] The frequency domain attention module is used to map input features from the spatial domain to the frequency domain through fast Fourier transform, adaptively filter the frequency domain features using a learnable Fourier filter, and then map the filtered frequency domain features back to the spatial domain through inverse fast Fourier transform.

[0038] The temporal attention module is used to generate features with enhanced temporal consistency by fusing features of the current frame with temporal features of adjacent frames.

[0039] The image reconstruction module integrates the features processed by the temporal attention module to reconstruct a clear image free of turbulence effects;

[0040] The loss calculation module is used to calculate the joint loss function, which includes pixel-level L1 loss, perceptual loss based on the VGG-19 network, and local pixel block frequency domain loss.

[0041] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image and video de-perturbation method based on the temporal attention mechanism described above.

[0042] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the program to implement the image and video de-perturbation method based on the temporal attention mechanism.

[0043] Beneficial effects:

[0044] Compared with the prior art, the present invention has the following significant advantages:

[0045] 1. This invention introduces a temporal attention module to explicitly model and fuse the spatiotemporal features between consecutive frames, effectively alleviating the inter-frame inconsistencies and visual disjointedness caused by single-frame processing methods. Experimental results show that this method outperforms the comparison algorithms in both distortion rate and stability, significantly improving the smoothness and viewing quality of the restored video.

[0046] 2. By alternating learning of spatial attention and frequency domain attention modules, the network can extract and clean up features more fully in both the spatial and frequency domains. Quantitative experiments on algorithmic and physical simulation datasets show that this method achieves optimal performance in both peak signal-to-noise ratio and structural similarity, significantly outperforming existing mainstream methods such as TSR-WGAN and Restormer, demonstrating its superior performance in single-frame deblurring and detail restoration.

[0047] 3. The learnable frequency domain filter designed in this invention, inspired by classical image processing theory, can adaptively optimize filtering parameters, enhance beneficial image frequency components (such as low-frequency contour information), suppress unfavorable high-frequency noise introduced by turbulence, and avoid the problem of amplifying noise in traditional inverse filtering and other methods, making the restoration process more robust and adaptable.

[0048] 4. The multi-scale feature extraction and block processing strategy adopted effectively reduces computational complexity while ensuring high performance and facilitates parallel processing, providing feasibility for processing high-resolution video streams. Ablation experiments demonstrate that performance steadily improves with the increase of the number of input frames, showcasing the good scalability of the algorithm framework.

[0049] In summary, this invention not only produces clearer and more coherent video sequences from a subjective visual perspective, but also leads in all aspects of objective evaluation indicators, providing an efficient and reliable solution for the problem of restoring atmospheric turbulence images and videos. Attached Figure Description

[0050] Figure 1 : Diagram of the overall network architecture.

[0051] Figure 2: Structural diagrams of spatial, frequency, and temporal attention modules, where (a), (b), and (c) represent the spatial attention module, frequency domain attention module, and temporal attention module, respectively.

[0052] Figure 3 : A schematic diagram of block splitting and merging operations, where (a) and (b) represent block splitting and block merging, respectively.

[0053] Figure 4 The results of different algorithms on the simulation data restoration process are shown in (a), (b), (c), (d), (e), (f), (g), (h), and (i), respectively: degraded image, TSR-WGAN algorithm, NAFNet algorithm, Restormer algorithm, FFTformer algorithm, ASF-Transformer algorithm, DATUM algorithm, the algorithm of this invention, and clear image.

[0054] Figure 5 The restoration effects of different algorithms on physical simulation data are shown in (a), (b), (c), (d), (e), (f), (g), (h), and (i), respectively: degraded image, TSR-WGAN algorithm, NAFNet algorithm, Restormer algorithm, FFTformer algorithm, ASF-Transformer algorithm, DATUM graphing method, the algorithm of this invention, and clear image. Detailed Implementation

[0055] The present invention will be further described below with reference to the accompanying drawings and embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0056] Example 1

[0057] This embodiment provides a method and system for image and video de-perturbation based on a temporal attention mechanism, the specific scheme of which includes:

[0058] I. Methodology and System Overview

[0059] Figure 1 The overall neural network architecture of the image and video de-perturbation method of the present invention is shown, and its data processing flow is as follows:

[0060] 1. Input and frame alignment:

[0061] The network receives a continuous sequence of turbulent degradation images, i.e., frame 1...target frame...frame t. Subsequently, the frame alignment module registers and aligns these input frames to compensate for geometric distortions and pixel displacements caused by turbulence, providing a spatially consistent sequence for subsequent processing.

[0062] 2. Multi-scale feature extraction:

[0063] The aligned frame sequence is then fed into a multi-scale feature extraction module. This module extracts spatial features at different scales through a parallel branching structure to capture rich information from details to the global picture.

[0064] 3. Multi-dimensional attention enhancement:

[0065] The encoded features are then purified and enhanced sequentially through three core attention modules:

[0066] Spatial-Aware Transformer Block (SATB);

[0067] Frequency-Aware Transformer Block (FATB);

[0068] Time-Domain Attention Block (TDAB).

[0069] 4. Image Reconstruction:

[0070] The image reconstruction module integrates the features extracted from all the aforementioned processing stages, maps them back to the image space through operations such as upsampling and convolution, and finally outputs the restored clear frame.

[0071] 5. Loss Calculation:

[0072] During the training phase, the loss calculation module receives the restored frame and the real clear image output by the network, calculates the joint loss function, and uses the gradient of its output to optimize all parameters of the entire network through the backpropagation algorithm.

[0073] The corresponding system includes, in sequence: frame alignment module, multi-scale feature extraction module, spatial attention module, frequency domain attention module, temporal domain attention module, image reconstruction module, and loss calculation module.

[0074] II. Input and Frame Alignment Processing

[0075] First, a sequence of consecutive multi-frame images degraded by atmospheric turbulence is input. To compensate for geometric distortion and pixel displacement caused by turbulence, a frame alignment module is used to perform inter-frame registration and alignment processing on the image sequence. This processing is achieved by calculating the local block matching error between adjacent frames and the target frame, and estimating the pixel-level displacement field using an elastic deformation model. Specifically, this is achieved by minimizing the following mean square error function:

[0076] ;

[0077] in, Represents pixels ( x , y The small spatial neighborhood, dx and dy For different offsets, This is a turbulence-degraded image. The final result is an aligned multi-frame image, providing spatially consistent input for subsequent processing.

[0078] III. Multi-scale Feature Extraction

[0079] The aligned multi-frame images are input into the multi-scale feature extraction module. For example... Figure 1 As shown, this module includes convolutional blocks and downsampling blocks, with a core parallel branch structure designed to extract spatial features across different receptive fields. Specifically, the structure comprises three parallel branches: the first branch maintains the spatial resolution of the input image and extracts features through a convolutional block containing two 3×3 convolutional layers, each followed by a GELU activation function and layer normalization (norm), outputting C channels; the second branch first downsamples the input by a factor of 2 using a convolutional layer with a stride of 2, increasing the number of channels to 2C, and then extracts features through a similar convolutional block; the third branch downsamples the input by a factor of 4 using two convolutional layers with a stride of 2, increasing the number of channels to 4C, and then extracts features through a convolutional block. Finally, the feature maps output from all three branches are fused. Before fusion, the features from the second and third branches are bilinearly upsampled to restore the original input resolution. Fusion involves concatenating channels followed by a 1×1 convolutional layer for channel compression and information integration, outputting a fused multi-scale feature map.

[0080] IV. Spatial Attention Processing

[0081] The feature map X output by the multi-scale feature extraction module is input into the spatial attention module. The detailed structure of this module is as follows: Figure 2 As shown in (a), its core lies in generating an attention map by calculating the cross-channel cross-covariance to implicitly encode global context information, thereby replacing the computationally intensive Query-Key dot product operation in the standard self-attention mechanism.

[0082] The forward propagation process of the SATB module is as follows:

[0083] 1) Feature projection and normalization:

[0084] The input feature X first passes through a layer normalization (Norm) layer. Then, it is copied and fed into three parallel branches for processing:

[0085] The query is obtained through a 1×1 convolution (Conv1×1) and a 3×3 convolution (DConv3×3). ), key ) and Value Three sets of feature vectors.

[0086] 2) Cross-covariance attention:

[0087] generated and It is fed into a multi-head attention mechanism. This mechanism calculates... and The dot product of the values, adjusted by a learnable scaling parameter α, is then input into the Softmax function to generate the attention weight map. This process can be represented as: Here, α is a learnable scaling parameter used to control the size of the dot product in the Softmax() function. This operation effectively captures the global spatial dependencies of features.

[0088] 3) Connection between projection and residual:

[0089] The output of the attention mechanism is projected through a 1×1 convolution and then fused with the module’s initial input X through a residual connection to obtain intermediate features.

[0090] 4) Gated feedforward network:

[0091] The aforementioned intermediate features are then passed through another normalization layer (Norm) before being fed into a gated-depthn convolutional feedforward network. This network contains two parallel paths:

[0092] Path 1: Pass through depthwise convolution (Dconv) and GELU activation function in sequence.

[0093] Path 2: Perform depthwise convolution followed by 1×1 convolution in sequence;

[0094] The outputs of the two paths are multiplied element-wise to form a gating mechanism.

[0095] 5) Output:

[0096] The output of the gated feedforward network is finally projected through another 1×1 convolution and added to its input through a residual connection to obtain the final output feature.

[0097] In summary, the SATB module significantly improves the network's ability to model the spatial structure of images through an innovative cross-covariance attention mechanism and a gated feedforward network that incorporates deep convolution.

[0098] V. Frequency Domain Attention Processing

[0099] The output features of the spatial attention module are input into the frequency domain attention module (its structure is shown below). Figure 2 (b) This module first maps spatial features to the frequency domain using a Fast Fourier Transform (FFT). Then, it adaptively filters the frequency domain features using a learnable Fourier filter. This filter is implemented as follows: ;in Indicates Fourier transform, This represents the inverse Fourier transform. , These are the query and key vectors obtained from the input features through convolution. The learnable Fourier filter is implemented as an optimizable two-dimensional parameter tensor (or weight matrix) with the same size as the frequency domain feature map. This tensor is applied to the input frequency domain features through element-wise multiplication, where each parameter value in the tensor represents the importance coefficient of its corresponding spatial frequency coordinate position (u, v), used to adaptively enhance or suppress that frequency component. All parameters of the filter are optimized during neural network training using a backpropagation algorithm. The filtered frequency domain features are mapped back to the spatial domain through an inverse fast Fourier transform (IFFT), completing the frequency domain cleanup. The learnable Fourier filter in this embodiment is as follows: Figure 2 As shown in (b), this design aims to replace traditional spatial domain methods for processing feature maps. This filter not only allows for more precise control over the retention and removal of information, but also adaptively optimizes its filtering parameters through training to achieve better image restoration results.

[0100] Inspired by the JPEG compression algorithm, this Fourier filter employs a mechanism similar to the Discrete Cosine Transform (DCT) to transform the image from the spatial domain to the frequency domain. DCT converts pixel values ​​in an image into frequency components, which characterize different details and texture information within the image. In JPEG compression, higher frequency components typically correspond to the details of the image. By quantizing these frequency components, not only is the quantization process of JPEG simulated, but the most beneficial frequency domain information for image restoration is selectively retained while removing frequency components that contribute less to the final image. In this framework, the feature maps obtained by the multi-head attention mechanism are not fed into parallel linear transform layers like in the spatial attention module. Instead, the feature maps processed by deep convolution are retained and fused with the features processed by the Fourier transform. This fusion process ultimately restores the information to the spatial domain through deconvolution, effectively and adaptively deciding which low-frequency and high-frequency information to retain. This strategy, which comprehensively considers both the frequency and spatial domains, significantly improves the quality of image restoration and the integrity of information.

[0101] VI. Temporal Attention Fusion

[0102] The features processed by the frequency domain attention module are input into the time domain attention module (its structure is shown in...). Figure 2 (c) Unlike the two attention modules mentioned above, the temporal attention module focuses on multi-frame characteristics, aiming to help the network more effectively fuse features between different frames with the extracted features, thereby strengthening the temporal correlation between frames in the video sequence. This is because the strategy of alternating spatial and frequency domain learning helps the network extract latent image features better. .

[0103] Specifically, in this module , The image is obtained by performing a 1×1 convolution and a 3×3 convolution on the input image X. Based on the above convolution, a second convolution is performed with the input image X to obtain... This secondary convolution operation is designed to enhance the fusion between Q and multi-frame features and previously extracted deep features, so that the attention mechanism can not only capture detailed spatial information, but also sensitively capture dynamic changes in time.

[0104] Through calculation and The similarity is used to generate attention weights, and these weights are then fused together using the following formula: , where α is a learnable scaling parameter used to adjust the intensity of the attention distribution.

[0105] When acquiring the attention map, this module will fuse the features. and The dot product is fed into the Softmax() function to calculate the attention weights. and Adjusting based on similarity The weights are determined. Simultaneously, a parallel computing method is employed, dividing the number of channels into multiple parts to focus on different features of the image.

[0106] This design helps the network to pay extra attention to information between consecutive frames while preserving deep features, thereby improving the smoothness and coherence of image sequences. In this way, TDAB can effectively enhance the processing capability of temporal information in image processing tasks and optimize the overall image performance.

[0107] Finally, the features are further processed through the following gated units that include GELU activation and layer normalization: ,in G ge The GELU activation function is used, where L represents layer normalization, ⊙ represents element-wise multiplication, and the output features with enhanced temporal consistency are connected through residual connections.

[0108] Through the above processing methods, TDAB enables the model to focus on important temporal or spatial ranges, automatically learning which temporal or spatial points are most critical for image restoration tasks. Introducing an attention mechanism helps the model utilize information from multiple frames more effectively, improving the accuracy and quality of the restoration results. Simultaneously, TDAB allows the model to dynamically focus on the correlations between different regions during image restoration, without being limited by fixed window sizes or receptive fields. This allows the model to more comprehensively understand the semantic content and contextual information of the image, thereby more accurately selecting appropriate restoration strategies and regions.

[0109] VII. Image Reconstruction

[0110] The temporal fusion features processed by the temporal attention module are fed into a refinement module. This module integrates all the features extracted in the preceding processing stages and maps them back to the image space to reconstruct a high-resolution image free of turbulence effects. The refinement module is implemented through convolutional operations. Finally, the last convolutional layer directly outputs a restored image with the same size as the original input image. All parameters of this module, along with other parts of the network, are optimized end-to-end using a joint loss function to ensure its effective decoding of high-level features into visually clear and realistic images.

[0111] 8. Training Strategy and Loss Function

[0112] During the network training phase, an end-to-end optimization approach is adopted, using a joint loss function to comprehensively train the neural network. This joint loss function consists of three parts:

[0113] (1) Pixel loss (L1 Loss)

[0114] Pixel loss is used to constrain the difference in pixel values ​​between the restored image and the true sharp image, promoting accurate reconstruction of overall color and brightness. This loss function calculates the average absolute error between the predicted image and the true image, as expressed below:

[0115] ;

[0116] in, n The total number of pixels in the image. , They represent the first i The predicted and actual values ​​of each pixel.

[0117] (2) Perceptual Loss

[0118] Perceptual loss extracts high-level features from an image using a pre-trained deep network (such as VGG-19). The calculation of perceptual loss involves two inputs: a clean image and a noisy image. Features are extracted from both, and then the Euclidean or Manhattan distance is calculated. The ultimate goal is to minimize this distance. The formula for calculating perceptual loss is as follows: Where x is the input image and y is the target image. and They represent their respective positions in the pre-trained neural network. i Layer feature representation, N It refers to the number of feature layers.

[0119] (3) Local pixel block frequency domain loss (PatchFFT Loss):

[0120] Fourier processing of images can enhance image sharpness while eliminating noise. In this embodiment, patchfftloss is used to monitor the changes in the frequency domain of the generated image, which can be expressed by the following formula: , where u and v represent spatial frequency coordinates, and h and w represent the height and width of the image, respectively. Re () indicates the real part. Im () denotes the imaginary part. This loss function ensures that the network pays attention to frequency, thus obtaining clearer output results.

[0121] The final joint loss is a weighted sum of the three losses mentioned above. The network parameters are optimized using a gradient descent algorithm to minimize this joint loss. In this embodiment, the loss weight coefficients are set to... =1, =0.01, =0.1.

[0122] IX. Block Processing Strategy

[0123] In the spatial, frequency, and temporal attention modules described above, a patching strategy was employed to efficiently handle spatial variation ambiguity. The core operation of this strategy is as follows: Figure 3 As shown, the process includes two key steps: patch division and patch merging. Patch division involves dividing the input feature map into multiple non-overlapping local patches, performing self-attention and other computations on each patch separately, and then merging them into a complete feature map. This strategy significantly reduces computational complexity and allows the model to focus on specific degradations in local regions.

[0124] Specifically, the block segmentation operation P() primarily divides the feature map of the entire image into local regions. These regions are designed to have sufficient self-attention capability and filtering effect, allowing the algorithm to focus on processing local details. This is particularly crucial for handling various turbulence degradations or local blurring. Through this local self-attention, the algorithm can more accurately adapt to the spatial variability in the image, thereby more effectively restoring the sharpness of local regions. The block merging operation P()... -1 The () function is responsible for recombining the processed local blocks into a complete feature map. This step not only ensures the overall coherence and consistency of the image, but also, by considering the relationships between adjacent blocks, allows the locally processed blocks to seamlessly integrate into the overall image, avoiding edge effects that may occur after local optimization. Furthermore, this block segmentation and merging strategy significantly reduces computational load. Because only small regions are processed each time, rather than the entire image, the algorithm can handle details in high-resolution images while maintaining low computational complexity. Simultaneously, this method also enables parallel computing, further improving processing speed.

[0125] X. Experimental Results and Analysis

[0126] To verify the effectiveness of this invention, experiments were conducted on algorithm simulation datasets, physics simulation datasets, and real-world datasets.

[0127] Quantitative experimental results show that the proposed method achieves optimal PSNR and SSIM metrics on both datasets (Table 1). On the algorithm simulation dataset, PSNR reaches 38.78 dB and SSIM reaches 0.9910; on the physical simulation dataset, PSNR reaches 32.84 dB and SSIM reaches 0.9680.

[0128] Table 1. Comparison of quantitative performance of different algorithms:

[0129]

[0130] The video temporal consistency evaluation results (Table 2) show that the proposed method performs well in terms of distortion rate (0.9956) and stability (0.8161), proving that it can effectively improve the temporal coherence of video.

[0131] Table 2 Comparison of Video Smoothness Evaluation

[0132]

[0133] Ablation experiments demonstrate that model performance continuously improves with increasing input frame count, with both the temporal attention module and the multi-scale structure making significant contributions to performance. As shown in Table 3, performance steadily improves with increasing input frame count, proving that fusing more temporal information helps the model better estimate and eliminate turbulence degradation. Simultaneously, processing time increases linearly, providing multiple options between performance and efficiency. Table 4 shows that introducing the temporal attention module (TDAB) results in stable improvements in both PSNR and SSIM, demonstrating the module's crucial role in modeling temporal dependencies and improving inter-frame consistency. Table 5 validates the effectiveness of the multi-scale structure; adding more branches yields better performance metrics, highlighting the key role of integrating multi-scale features in mitigating turbulence degradation.

[0134] Table 3. Quantitative comparison of training results with different numbers of input frames.

[0135]

[0136] Table 4 Performance Comparison Based on TDAB

[0137]

[0138] Table 5. Impact of different network scales on training results

[0139]

[0140] Figure 4The visualization of the restoration effects of different algorithms on the simulated data is shown in the figure. It can be seen from the figure that (b), (c), (d), (e), and (g) cannot effectively remove the blurring in the simulated image and their performance in restoring the zebra stripes is not satisfactory. Among them, the restored image of (b) has obvious noise in addition to blurring. Compared with other algorithms, (f) restores zebra outline and stripe features more accurately, but it cannot show the grass details below the image. The algorithm (h) proposed in this embodiment can not only present more refined zebra details, but also notice the small features that other algorithms cannot notice, such as grass texture, so that the restoration effect is closer to the clean image.

[0141] Figure 5 The restoration effects of different algorithms on physical simulation data are demonstrated. The degraded image (a) exhibits some distortion and blurring compared to the clean image (i). In the restored image, (b) is not close enough to the clean image, and there is a color difference in the sand in front of the building. (c), (d), and (e) are improvements compared to (b), but the building still shows some blurring. (f) further improves the detail restoration capability based on the aforementioned algorithms. For example, as can be seen in (f), compared to the previous algorithms, it can restore some details, such as the steel wire in the lower right corner of the loess area. (g) Although it is not as good as the other algorithms in terms of numerical indicators, its visualization results show that it has certain advantages in detail feature restoration. The image restored by the algorithm proposed in this embodiment is shown in (h). While ensuring the overall quality, it performs good feature restoration on small objects such as the minute and hour hands of the clock in the middle of the image, making it visually closer to the clean image.

[0142] Example 2

[0143] This embodiment provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, performs the following steps:

[0144] Read instructions and data from the storage medium;

[0145] Call the frame alignment program to register the input turbulence video sequence;

[0146] Call the multi-scale feature extraction program to generate multi-scale features;

[0147] Invoke the spatial attention processing program to execute the calculations of the SATB module;

[0148] Invoke the frequency domain attention processing program to perform frequency domain filtering in the FATB module;

[0149] Invoke the temporal attention processing program to perform temporal fusion of the TDAB module;

[0150] Call the image reconstruction program and output the restored, clear image;

[0151] In training mode, the loss calculation program is invoked to calculate the gradient based on the joint loss function and update the network model parameters.

[0152] This embodiment may be one or more physical media such as solid-state drives (SSDs), flash memory chips (Flash Memory), or read-only memory (ROM), wherein the stored code is used to fully implement the method flow described.

[0153] Example 3

[0154] This embodiment provides an electronic device, including:

[0155] Memory, used to store executable instructions;

[0156] The processor is configured to execute instructions in the memory to implement the image and video de-perturbation method.

[0157] The electronic device is a server or workstation equipped with parallel computing acceleration units such as GPU or NPU;

[0158] The processor is connected to the computing acceleration unit, which carries out the forward inference and backward propagation calculations of the neural network system.

[0159] The memory includes volatile memory and non-volatile storage devices, used to store training datasets, model parameters, and intermediate calculation results.

[0160] This embodiment provides a specific hardware platform example whose operation process is fully compatible with the system embodiment and storage medium embodiment.

[0161] In summary, this invention, through its innovative multi-dimensional attention mechanism, collaboratively processes data across the spatial, frequency, and temporal domains, effectively solving the image degradation problem caused by atmospheric turbulence and demonstrating superior performance in both subjective visual aspects and objective metrics.

Claims

1. An image and video de-noising method based on a time domain attention mechanism, characterized in that, The method is implemented by a deep learning neural network, comprising the following steps: Step 1: input a continuous multi-frame image sequence degraded by atmospheric turbulence; Step 2: inter-frame registration and alignment processing is performed on the image sequence to compensate for geometric distortion and pixel displacement caused by turbulence, thereby obtaining a plurality of aligned images; Step 3: input the plurality of aligned images into a multi-scale feature extraction network, and extract spatial features of different scales through the parallel branch structure of the multi-scale feature extraction network; Step 4: input the multi-scale spatial features extracted in step 3 into a spatial attention module, generate an attention map by calculating the cross-channel covariance, and enhance the spatial feature representation by combining the gating mechanism; Step 5: input the features processed by the spatial attention module into a frequency domain attention module, and the frequency domain attention module performs the following sub-steps: Step 51: map the input features from the spatial domain to the frequency domain through fast Fourier transform; Step 52: use a learnable Fourier filter to perform adaptive filtering on the frequency domain features; wherein the parameters of the Fourier filter are optimized through the back propagation algorithm during the neural network training process, to adaptively enhance the frequency components beneficial to image reconstruction, and suppress the adverse frequency components introduced by turbulence degradation; Step 53: map the filtered frequency domain features back to the spatial domain through inverse fast Fourier transform; Step 6: input the features processed by the frequency domain attention module into a time domain attention module, and generate time sequence consistency enhanced features by fusing the current frame features and the time sequence features of adjacent frames; Step 7: integrate the features processed in step 6 through a refinement module to reconstruct a clear image that removes the effects of turbulence; Step 8: in the training stage, the neural network is optimized using a joint loss function, which includes a pixel-level L1 loss, a perception loss based on a VGG-19 network, and a local pixel block frequency domain loss.

2. The image and video de-noising method based on time domain attention mechanism according to claim 1, characterized in that, The inter-frame registration and alignment in step 2 is achieved by calculating the local block matching error between adjacent frames and the reference frame, and estimating the pixel-level displacement field using an elastic deformation model.

3. The image and video de-noising method based on time domain attention mechanism according to claim 1, characterized in that, The multi-scale feature extraction network in step 3 includes three parallel branches: the first branch maintains the resolution of the input image; the second branch performs 2 times down-sampling on the input and expands the channel size; the third branch performs 4 times down-sampling on the input and further expands the channel size; each branch extracts features through convolution operation, and finally fuses the feature maps output by all branches.

4. The image and video de-noising method based on time domain attention mechanism according to claim 1, characterized in that, The learnable Fourier filter is configured to: receive the frequency domain feature map obtained by fast Fourier transform as input; apply a set of learnable parameter weights to the input frequency domain feature map through element-level multiplication, wherein the parameter weights are a two-dimensional tensor with the same size as the frequency domain feature map, and each weight value corresponds to the importance coefficient of its spatial frequency coordinate position; wherein the learnable parameter weights are optimized through the back propagation algorithm during the neural network training process; the output of the Fourier filter is the frequency domain feature after adaptive filtering, which is sent to the subsequent inverse fast Fourier transform layer to convert back to the spatial domain.

5. The image and video de-noising method based on time domain attention mechanism according to claim 1, characterized in that, The time domain attention module in step 6 realizes time sequence fusion by the following manner: splicing the features of the current frame and the features of the adjacent frame, generating an attention map by convolution to extract time sequence context information, and performing weighted fusion on the features according to the attention map.

6. The image and video de-noising method based on time domain attention mechanism according to claim 1, characterized in that, In the processing of steps 4, 5 and 6, the feature maps input to the spatial attention module, the frequency domain attention module and the time domain attention module are all processed by a block processing strategy. The block processing strategy includes: dividing the input feature map into a plurality of non-overlapping local blocks, performing attention calculation on each block respectively, and then merging the blocks into a complete feature map after processing. The processed feature map is sent to step 7 for final image reconstruction.

7. An image and video de-noising system based on time domain attention mechanism, for implementing the method of any one of claims 1-6, characterized in that, The system comprises: a frame alignment module for performing inter-frame registration and alignment processing on the input image sequence to compensate for geometric distortion and pixel displacement caused by turbulence, and obtaining a plurality of aligned images; a multi-scale feature extraction module for extracting spatial features of different scales from the aligned images, the multi-scale feature extraction module comprising a plurality of parallel branch structures for processing different scales of input respectively; a spatial attention module for generating an attention map by calculating cross-channel covariance and enhancing spatial feature representation by combining a gating mechanism; a frequency domain attention module for mapping input features from the spatial domain to the frequency domain by fast Fourier transform, adaptively filtering the frequency domain features by using a learnable Fourier filter, and then mapping the filtered frequency domain features back to the spatial domain by inverse fast Fourier transform; a time domain attention module for generating features with enhanced temporal consistency by fusing the features of the current frame and the temporal features of the adjacent frames; an image reconstruction module for integrating the features processed by the time domain attention module to reconstruct a clear image without turbulence effects; a loss calculation module for calculating a joint loss function, the joint loss function including a pixel-level L1 loss, a perception loss based on a VGG-19 network, and a local pixel block frequency domain loss.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the image and video de-disturbance method based on the time domain attention mechanism as claimed in any one of claims 1-6.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory, wherein the computer program comprises instructions that, when executed by the processor, cause the electronic device to perform the method of any one of claims 1-8. The processor executes the program to implement the image and video de-disturbance method based on the time domain attention mechanism as claimed in any one of claims 1-6.

Images