Image video de-disturbing method and system based on time domain attention mechanism

Abstract

The invention discloses an image video de-disturbing method and system based on a time domain attention mechanism. The method comprises the following steps: inputting a continuous multi-frame turbulence degraded image sequence; performing inter-frame registration and alignment to compensate for geometric distortion; extracting spatial features through a multi-scale feature extraction network; enhancing spatial feature representation through the spatial attention module, performing learnable frequency domain filtering through the frequency domain attention module and fusing time sequence features through the time domain attention module in sequence; and finally, integrating the features to reconstruct a clear image after turbulence is removed. According to the method, a time domain attention mechanism is innovatively introduced, the time sequence correlation between video frames is effectively utilized, and the problem of inter-frame inconsistency existing in video restoration of an existing single-frame processing method is solved. Experiments show that the method is obviously superior to the existing mainstream method in objective indexes such as PSNR and SSIM, and the time sequence continuity and stability of the restored video can be effectively enhanced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing and computer vision, and particularly relates to an image and video de-disturbance method and system based on a time domain attention mechanism. BACKGROUND

[0002] In the troposphere, atmospheric fluid exhibits various flow types such as laminar flow, transitional flow and turbulent flow at different flow rates. Atmospheric turbulence is caused by the interaction of inertial force and viscous force in the flow field, which leads to random fluctuations in the refractive index of air, and further causes disturbances to the phase of the light beam during propagation, ultimately resulting in serious blurring, tilting and geometric distortion of the images captured by imaging devices. This phenomenon poses a significant challenge to applications such as remote sensing, astronomical observation, long-distance photography, and other applications that rely on high-quality image input.

[0003] Currently, the technical approaches to overcoming the effects of atmospheric turbulence mainly include hardware and software. In terms of hardware, adaptive optics technology uses a closed-loop system composed of a wavefront sensor, a controller and a corrector to correct wavefront distortion in real time, but it has problems such as high cost, limited high-frequency correction capability, and often still requires post-processing. In terms of software, image restoration technology is one of the current mainstream methods. Early methods such as lucky imaging fuse by selecting "lucky frames", but their quality is unstable and limited by diffraction blur. Traditional algorithms based on inverse filtering or elastic registration are often computationally complex and prone to amplifying noise. With the development of deep learning, methods based on convolutional neural networks have made progress in single-image restoration, but the limited receptive field of such networks makes it difficult to model long-range dependencies, leading to loss of details and over-smoothing. Although generative adversarial networks can generate images with rich details, the lack of physical constraints can lead to distortion. In recent years, although some studies have attempted to incorporate multi-frame information, most of these methods are not specifically designed for turbulence restoration, and they fail to effectively utilize the temporal correlation between frames, resulting in significant inconsistencies and fragmentation in the restored video.

[0004] Therefore, one of the outstanding problems in the prior art is the lack of a turbulence image and video restoration scheme that can effectively fuse spatio-temporal information, improve single-frame quality while ensuring temporal coherence of the video. SUMMARY

[0005] The present application aims to overcome the shortcomings of the prior art and provide an image and video de-disturbance method and system based on a time domain attention mechanism.

[0006] The present application aims to solve the problems of inter-frame inconsistency and fragmentation in existing single-frame image restoration methods when processing video, as well as the insufficient utilization of spatio-temporal features in existing multi-frame methods. By constructing a neural network architecture that integrates spatial, frequency and temporal attention, the present application aims to achieve the following objectives: By aligning and fusing continuous multiple frames of images, geometric distortion and pixel displacement caused by atmospheric turbulence are effectively compensated. By designing a time domain attention module to explicitly model the inter-frame timing dependency, the coherence and stability of the restored video are enhanced, and the visual fragmentation is eliminated. While significantly improving objective evaluation indicators such as peak signal-to-noise ratio and structural similarity, the restored results have better visual perception quality and better detail preservation. The stability of the restored video is effectively improved and the distortion rate is reduced, thereby ensuring the smoothness of the final video output.

[0007] To achieve the above purpose, the technical scheme adopted by the present application is: An image / video de-disturbance method based on a time domain attention mechanism, the method is realized by a deep learning neural network, comprising the following steps: Step 1: input a continuous multiple frames of image sequences degraded by atmospheric turbulence; Step 2: inter-frame registration and alignment processing is performed on the image sequence to compensate for geometric distortion and pixel displacement caused by turbulence, and multiple frames of aligned images are obtained; Step 3: input the multiple frames of aligned images into a multi-scale feature extraction network, and extract spatial features of different scales through the parallel branch structure of the multi-scale feature extraction network; Step 4: the multi-scale spatial features extracted in step 3 are first input into a spatial attention module, an attention map is generated by calculating the cross-channel covariance, and the spatial feature representation is enhanced by combining the gating mechanism; Step 5: the features processed by the spatial attention module are input into a frequency domain attention module, and the frequency domain attention module performs the following sub-steps: Step 51: map the input features from the spatial domain to the frequency domain through fast Fourier transform; Step 52: use a learnable Fourier filter to adaptively filter the frequency domain features; wherein the parameters of the Fourier filter are optimized through the back propagation algorithm during the neural network training process, to adaptively enhance the frequency components beneficial to image reconstruction, and suppress the adverse frequency components introduced by turbulence degradation; Step 53: map the filtered frequency domain features back to the spatial domain through inverse fast Fourier transform; Step 6: input the features processed by the frequency domain attention module into the time domain attention module, and generate features with enhanced temporal consistency by fusing the current frame features and the timing features of adjacent frames; Step 7: integrate the features processed in step 6 through a refinement module to reconstruct clear images without turbulence effects. Step 8: During the training phase, the neural network is optimized using a joint loss function, which includes pixel-level L1 loss, perceptual loss based on the VGG-19 network, and local pixel block frequency domain loss.

[0008] Preferably, the inter-frame registration and alignment in step 2 is achieved by calculating the local block matching error between the adjacent frames and the reference frame, and estimating the pixel-level displacement field using an elastic deformation model.

[0009] Preferably, the multi-scale feature extraction network in step 3 comprises three parallel branches: the first branch maintains the resolution of the input image; the second branch downsamples the input by a factor of 2 and expands the channel size; the third branch downsamples the input by a factor of 4 and further expands the channel size; each branch extracts features through a convolution operation, and finally the feature maps output by all branches are fused.

[0010] Preferably, the learnable Fourier filter is configured as follows: Receive the frequency domain feature map obtained by fast Fourier transform as input; Apply a set of learnable parameter weights to the input frequency domain feature map through element-wise multiplication, where the parameter weight is a two-dimensional tensor of the same size as the frequency domain feature map, and each weight value corresponds to the importance coefficient of its spatial frequency coordinate position; Among them, the learnable parameter weights are optimized through the back-propagation algorithm during the neural network training process; The output of the Fourier filter is the frequency domain feature after adaptive filtering, and is sent to the subsequent inverse fast Fourier transform layer to be converted back to the spatial domain.

[0011] Preferably, the temporal attention module in step 6 realizes temporal fusion by concatenating the features of the current frame with the features of the adjacent frames, extracting temporal context information through convolution to generate an attention map, and performing weighted fusion of the features based on the attention map.

[0012] Preferably, during the processing of steps 4, 5, and 6, a block processing strategy is adopted for the feature maps input to the spatial attention module, the frequency domain attention module, and the time domain attention module; the block processing strategy includes: dividing the input feature map into multiple non-overlapping local blocks, performing attention calculation on each block separately, and then merging the blocks into a complete feature map after processing; the processed feature map is sent to step 7 for final image reconstruction.

[0013] The present invention also provides an image and video de-disturbance system based on a temporal attention mechanism, which is used to implement the image and video de-disturbance method based on a temporal attention mechanism. The system includes: a frame alignment module, configured to perform inter-frame registration and alignment processing on the input image sequence to compensate for geometric distortion and pixel displacement caused by turbulence, and obtain a plurality of aligned images; a multi-scale feature extraction module, configured to extract spatial features of different scales from the plurality of aligned images, and the multi-scale feature extraction module comprises a plurality of parallel branch structures for processing input of different scales respectively; a spatial attention module, configured to generate an attention map by calculating cross-channel covariance, and enhance spatial feature representation by combining a gating mechanism; a frequency domain attention module, configured to map input features from a spatial domain to a frequency domain by fast Fourier transform, perform adaptive filtering on frequency domain features by using a learnable Fourier filter, and then map the filtered frequency domain features back to the spatial domain by inverse fast Fourier transform; a time domain attention module, configured to generate features with enhanced temporal consistency by fusing current frame features and temporal features of adjacent frames; an image reconstruction module, configured to integrate features processed by the time domain attention module to reconstruct a clear image without turbulence effects; a loss calculation module, configured to calculate a joint loss function, and the joint loss function comprises a pixel-level L1 loss, a perception loss based on a VGG-19 network, and a local pixel block frequency domain loss.

[0014] The application also provides a computer readable storage medium having a computer program stored thereon, and the program is executed by a processor to implement the image and video de-disturbance method based on the time domain attention mechanism.

[0015] The application also provides an electronic device comprising a memory, a processor and a computer program stored on the memory, and the processor implements the image and video de-disturbance method based on the time domain attention mechanism when executing the program.

[0016] Advantages: Compared with the prior art, the application has the following remarkable advantages: 1. By introducing the time domain attention module, the application explicitly models and fuses the spatial and temporal features between consecutive multiple frames, which can effectively alleviate the inter-frame inconsistency and visual fragmentation caused by single-frame processing methods. Experimental results show that the method is superior to the comparative algorithm in terms of distortion rate and stability, two video smoothing indicators, and significantly improves the fluency and visual quality of the recovered video.

[0017] 2.Through the alternating learning of the spatial attention and the frequency domain attention module, the network can more fully extract and purify features in the spatial domain and the frequency domain. Quantitative experiments on algorithm simulation and physical simulation datasets show that the method achieves the optimal performance in two key indicators, peak signal-to-noise ratio and structural similarity, and significantly outperforms existing mainstream methods such as TSR-WGAN and Restormer, proving its excellent performance in single-frame deblurring and detail recovery.

[0018] 3.The learning frequency domain filter designed by the present application can adaptively optimize filter parameters, enhance beneficial image frequency components (such as low-frequency contour information), and suppress adverse high-frequency noise introduced by turbulence, thus avoiding the problem of amplifying noise in traditional inverse filtering methods, and making the restoration process more robust and adaptive.

[0019] 4.The multi-scale feature extraction and block processing strategy adopted effectively reduces the computational complexity while ensuring high performance, and facilitates parallel processing, providing feasibility for processing high-resolution video streams. The ablation experiment shows that the performance steadily improves with the increase of the number of input frames, demonstrating the good scalability of the algorithm framework.

[0020] In summary, the present application not only produces clearer and more coherent video sequences in subjective vision, but also leads in objective evaluation indicators, providing an efficient and reliable solution to the problem of atmospheric turbulence image video restoration. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 Fig. 1 is a schematic diagram of the overall architecture of the network.

[0022] Figure 2 Fig. 2 is a structural diagram of the spatial, frequency domain, and time domain attention modules, wherein (a), (b), and (c) are the spatial attention module, the frequency domain attention module, and the time domain attention module, respectively.

[0023] Figure 3 Fig. 3 is a schematic diagram of block segmentation and merging operations, wherein (a) and (b) are block segmentation and block merging, respectively.

[0024] Figure 4 Fig. 4 is the restoration processing result of algorithm simulation data by different algorithms, wherein (a), (b), (c), (d), (e), (f), (g), (h), and (i) are the degraded image, the TSR-WGAN algorithm, the NAFNet algorithm, the Restormer algorithm, the FFTformer algorithm, the ASF-Transformer algorithm, the DATUM algorithm, the present application algorithm, and the clear image, respectively.

[0025] Figure 5: The restoration effects of different algorithms on physical simulation data, wherein (a), (b), (c), (d), (e), (f), (g), (h), (i) are respectively a degraded image, a TSR-WGAN algorithm, a NAFNet algorithm, a Restormer algorithm, a FFTformer algorithm, an ASF-Transformer algorithm, a DATUM algorithm, an algorithm of the present application, and a clear image. DETAILED DESCRIPTION

[0026] The application will be further described below in conjunction with the drawings and embodiments. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative labor fall within the protection scope of the present application.

[0027] Embodiment 1 The embodiment provides an image and video de-disturbance method and system based on a time-domain attention mechanism, and the specific scheme comprises the following steps: I. Method flow and system overview Figure 1 The overall neural network architecture of the image and video de-disturbance method of the present application is shown, and the data processing flow is as follows: 1. Input and frame alignment The network receives a continuous turbulent degraded image sequence, i.e., frame 1... target frame... frame t. Subsequently, the frame alignment module aligns and aligns these input frames to compensate for the geometric distortion and pixel displacement caused by turbulence, providing a spatially consistent sequence for subsequent processing.

[0028] 2. Multi-scale feature extraction The aligned frame sequence is sent to the multi-scale feature extraction module. The module extracts spatial features of different scales through a parallel branch structure to capture rich information from details to the global.

[0029] 3. Multi-dimensional attention enhancement The encoded features are sequentially purified and enhanced through three core attention modules: a spatial attention module (Spatial-Aware Transformer Block, SATB); a frequency domain attention module (Frequency-Aware Transformer Block, FATB); a time-domain attention module (Time-Domain Attention Block, TDAB).

[0030] 4. Image reconstruction: The image reconstruction module integrates the features extracted in all the aforementioned processing stages, maps them back to the image space through upsampling and convolution operations, and finally outputs the restored clear frames.

[0031] 5. Loss calculation: During the training stage, the loss calculation module receives the restored frames output by the network and the real clear images, calculates the joint loss function, and outputs the gradient for optimizing all the parameters of the entire network through the backpropagation algorithm.

[0032] The corresponding system sequentially includes a frame alignment module, a multi-scale feature extraction module, a spatial attention module, a frequency domain attention module, a time domain attention module, an image reconstruction module, and a loss calculation module.

[0033] II. Input and frame alignment processing First, a continuous multi-frame image sequence degraded by atmospheric turbulence is input. To compensate for the geometric distortion and pixel displacement caused by turbulence, a frame alignment module is used to perform inter-frame registration and alignment processing on the image sequence. This processing is achieved by calculating the local block matching error between adjacent frames and the target frame, and estimating the pixel-level displacement field using an elastic deformation model. Specifically, it is achieved by minimizing the following mean square error function: ;

[0034] wherein, represents the small spatial neighborhood of pixel ( x , y ), dx and dy are different offsets, is the turbulence-degraded image. Finally, the aligned multi-frame image is obtained, providing a spatially consistent input for subsequent processing.

[0035] III. Multi-scale feature extraction The aligned multi-frame image is input into the multi-scale feature extraction module. As Figure 1As shown, this module consists of convolutional and downsampling blocks. Its core is a parallel branch structure designed to extract spatial features at different receptive fields. Specifically, this structure consists of three parallel branches: the first branch maintains the spatial resolution of the input image and extracts features through a convolutional block consisting of two layers of 3×3 convolutions, each followed by a GELU activation function and layer normalization. The output has C channels. The second branch first downsamples the input by a factor of 2 using a convolutional layer with a stride of 2, expanding the number of channels to 2C, and then extracts features through a similar convolutional block. The third branch downsamples the input by a factor of 4 using two convolutional layers with a stride of 2, expanding the number of channels to 4C, and then extracts features through a convolutional block. Finally, the feature maps output by all three branches are fused. Before fusion, the features of the second and third branches are bilinearly upsampled to restore to the original input resolution. Fusion uses channel concatenation followed by a 1×1 convolutional layer for channel compression and information integration, outputting a combined multi-scale feature map.

[0036] 4. Spatial Attention Processing The feature map X output by the multi-scale feature extraction module is input to the spatial attention module. The detailed structure of this module is as follows Figure 2 As shown in (a), its core is to generate an attention map by calculating the cross-channel covariance to implicitly encode global context information, thereby replacing the computationally intensive Query-Key dot product operation in the standard self-attention mechanism.

[0037] The forward propagation process of the SATB module is as follows: 1) Feature projection and normalization: The input feature X first passes through a normalization layer. Then, it is copied and fed into three parallel branches for processing: Through a 1×1 convolution (Conv1×1) and a 3×3 convolution (DConv3×3) operation, the query (Query, ), Key, ) and Value, ) three sets of eigenvectors.

[0038] 2) Cross-covariance attention: Generated and is fed into the multi-head attention mechanism. This mechanism calculates and The dot product of is adjusted by a learnable scaling parameter α and then input into the Softmax function to generate the attention weight map. The process can be expressed as: ; where a is a learnable scaling parameter to control the magnitude of dot product in the Softmax() function. This operation effectively captures the global spatial dependency of features.

[0039] 3) Projection and residual connection: The output of the attention mechanism is projected by a 1x1 convolution and then fused with the original input X through a residual connection to obtain the intermediate feature.

[0040] 4) Gated feed-forward network: The intermediate feature is fed into a gated-deep convolution feed-forward network after passing through another layer normalization (Norm) layer. The network contains two parallel paths: Path one: sequentially passing through a deep convolution (Dconv) and a GELU activation function (GELU); Path two: sequentially passing through a deep convolution and a 1x1 convolution; The outputs of the two paths are element-wise multiplied to form a gating mechanism.

[0041] 5) Output: The output of the gated feed-forward network is finally projected by another 1x1 convolution and added to its input through a residual connection to obtain the final output feature.

[0042] In summary, the SATB module significantly improves the network's ability to model image spatial structures through the innovative cross-covariance attention mechanism and the gated feed-forward network combined with deep convolution.

[0043] Five. Frequency domain attention processing The output feature of the spatial attention module is input into the frequency domain attention module (its structure is shown in Figure 2 (b) in the middle). This module first maps the spatial feature to the frequency domain through the fast Fourier transform (FFT). Then, a learnable Fourier filter is used to adaptively filter the frequency domain feature. The filter is implemented in the following way: ; where denotes the Fourier transform, denotes the inverse Fourier transform, , Query and Key vectors are obtained by input features through convolution operation respectively. The learnable Fourier filter is implemented as an optimizable two-dimensional parameter tensor (or weight matrix) with the same size as the frequency domain feature map. The tensor is applied to the input frequency domain feature through element-level multiplication, where each parameter value in the tensor represents the importance coefficient of its corresponding spatial frequency coordinate position (u, v) for adaptively enhancing or suppressing the frequency component. All parameters of the filter are optimized during the neural network training process through the back propagation algorithm. The filtered frequency domain feature is mapped back to the spatial domain through inverse fast Fourier transform (IFFT), completing the purification process in the frequency domain. The learnable Fourier filter in the present embodiment, as shown in (b) of FIG. 1, is designed to replace the traditional method of processing feature maps in the spatial domain. The filter not only can more accurately control the retention and rejection of information, but also can adaptively optimize its filtering parameters through training to achieve better image restoration results. Figure 2

[0044] Inspired by the JPEG compression algorithm, the Fourier filter adopts a mechanism similar to the discrete cosine transform (DCT) to convert the image from the spatial domain to the frequency domain. DCT can convert the pixel values in the image into frequency components, which represent different detail and texture information in the image. In the JPEG compression process, higher frequency components usually correspond to the detail part of the image. By quantizing these frequency components, not only is the quantization process of JPEG simulated, but also the frequency domain information that is most beneficial to image restoration can be selectively retained, while the frequency components that contribute less to the final image are removed. In this technical framework, the feature map obtained by the multi-head attention mechanism is not sent to the parallel linear transformation layer as in the spatial attention module, but the feature map processed by the deep convolution is retained and fused with the feature after Fourier transform. This fusion process eventually restores the information to the spatial domain through inverse convolution technology, effectively and adaptively deciding which low and high frequency information should be retained. This strategy of considering both the frequency domain and the spatial domain greatly improves the quality of image restoration and the integrity of information.

[0045] VI. Temporal attention fusion The features processed by the frequency domain attention module are input into the temporal attention module (see the structure of FIG. 1 (c)). Figure 2 Unlike the above two attention modules, since the strategy of alternating learning in the spatial and frequency domains can help the network better extract the potential features of the image, the temporal attention module focuses on the multi-frame characteristics, aiming to help the network more effectively fuse the features between different frames with the extracted features to strengthen the temporal correlation between frames in the video sequence. The module first generates features through the following convolution operation: . ​

[0046] Specifically, in this module 、 is obtained by a 1x1 convolution and a 3x3 convolution on the input image X respectively, is obtained by a second convolution on the basis of the above-mentioned convolution The design of this second convolution operation aims to enhance the fusion between Q and the multi-frame features and the previously extracted deep features, so that the attention mechanism can not only capture detailed spatial information, but also sensitively capture dynamic changes in time.

[0047] The similarity between and is calculated to generate attention weights, and weighted fusion is performed, which is expressed by the formula: where α is a learnable scaling parameter used to adjust the strength of the attention distribution.

[0048] When obtaining the attention map, the module multiplies and point by point and sends it into the Softmax() function to calculate the attention weight. The weight of is adjusted according to the similarity between and . At the same time, the parallel computing method is adopted, and the channel number is divided into multiple parts, focusing on different features of the image.

[0049] This design helps the network to pay additional attention to the information between consecutive frames on the basis of maintaining deep features, thereby improving the smoothness and coherence of the image sequence. In this way, TDAB can effectively enhance the processing ability of the time dimension information in image processing tasks, and optimize the overall performance of the image.

[0050] Finally, the features are further processed by the following gating unit containing GELU activation and layer normalization: where G ge is the GELU activation function, L represents layer normalization, and is the element-wise multiplication. The output is the feature with enhanced temporal consistency through residual connection.

[0051] Through the above processing method, TDAB enables the model to focus on important time or spatial ranges, and automatically learns which time or spatial points are most critical for image inpainting tasks. The introduction of attention mechanism helps the model to more effectively utilize the information of multiple frames of images, improving the accuracy and quality of the inpainting results. At the same time, TDAB allows the model to dynamically focus on the relevance between different regions when inpainting images, without being limited by fixed window size or receptive field, enabling the model to more comprehensively understand the semantic content and contextual information of the image, and thus more accurately select appropriate inpainting strategies and regions.

[0052] VII. Image Reconstruction The time-series fusion features processed by the time-domain attention module are sent to a refinement module. The function of this module is to integrate all the features extracted in the previous processing stages and map them back to the image space to reconstruct high-definition images that exclude turbulent effects. The refinement module is implemented through convolution operations. Finally, the same size as the original input image is directly output through the last convolution layer. All parameters of this module are optimized end-to-end with the joint loss function together with other parts of the network to ensure that it can effectively decode high-level features into visually clear and realistic images.

[0053] VIII. Training Strategy and Loss Function During the network training phase, an end-to-end optimization method is used to comprehensively train the neural network using a joint loss function. This joint loss function consists of three parts: (1) Pixel Loss (L1 Loss) Pixel loss is used to constrain the difference in pixel values between the restored image and the real clear image, promoting accurate reconstruction of overall color and brightness. The loss function calculates the average of the absolute error between the predicted image and the real image, as follows: ; where, n is the total number of pixels in the image, , represent the predicted value and the true value of the i th pixel, respectively.

[0054] (2) Perceptual Loss Perceptual loss extracts high-level features of images through a pre-trained deep network (such as VGG-19). The calculation of perceptual loss involves two inputs, the clean image and the noisy image. By extracting features from both and calculating the Euclidean distance or Manhattan distance, the goal is to minimize the distance between the two. The calculation formula of perceptual loss is as follows: where x is the input image, y is the target image, and represent their feature representations at the i th layer of the pre-trained neural network, N is the number of feature layers.

[0055] (3) Local Pixel Block Frequency Domain Loss (PatchFFT Loss): The Fourier processing of the image can eliminate noise while enhancing image sharpness. The patchfftloss used in the present embodiment is used to monitor the changes in the generated image in the frequency domain, which can be expressed by the following formula: where u and v represent the spatial frequency coordinates, and h and w represent the height and width of the image, respectively. Re represents the real part, Im represents the imaginary part. This loss function ensures that the network pays attention to the frequency, thereby obtaining a clearer output result.

[0056] The final joint loss is the weighted sum of the above three losses. The network parameters are optimized by the gradient descent algorithm to minimize the above joint loss. In the present embodiment, the loss weight coefficients are set as = 1, = 0.01, = 0.1.

[0057] Nine. Patch processing strategy In the processing of the above spatial, frequency and time domain attention modules, the patching strategy is used to efficiently process spatial variability blur. The core operation of this strategy is shown in Figure 3 , which includes two key steps of patch division and patch merging. That is, the input feature map is divided into multiple non-overlapping local patches for self-attention calculation, and then merged into a complete feature map after processing. This strategy significantly reduces the computational complexity and helps the model focus on the specific degradation of the local area.

[0058] Specifically, the main function of the patch division operation P() is to divide the feature map of the entire image according to the local area. These areas are designed to have sufficient self-attention ability and filtering effect, so that the algorithm can focus on the processing of local details, which is particularly important for processing various turbulent degradation or local blur. Through this local self-attention, the algorithm can more accurately adapt to the spatial variability in the image, thereby more effectively restoring the clarity of the local area. The patch merging operation P -1 () is responsible for recombining the processed local patches into a complete feature map. This step not only ensures the coherence and consistency of the entire image, but also seamlessly integrates the locally processed patches into the overall image by considering the relationship between adjacent patches, avoiding the edge effect that may occur after local optimization. In addition, using this patch division and merging strategy can significantly reduce the computational load. Because only small blocks are processed each time instead of the entire image, this allows the algorithm to handle details in high-resolution images while maintaining low computational complexity. At the same time, this method also makes parallel computing possible, further improving the processing speed.

[0059] X. Experimental results and analysis To verify the effectiveness of the present application, experiments were conducted on algorithm simulation data sets, physical simulation data sets and real-world data sets.

[0060] The quantitative experimental results show that the PSNR and SSIM indicators of the present method on the two data sets are optimal (Table 1). On the algorithm simulation data set, the PSNR reaches 38.78 dB and the SSIM reaches 0.9910; on the physical simulation data set, the PSNR reaches 32.84 dB and the SSIM reaches 0.9680.

[0061] Table 1 Comparison of quantitative performance of different algorithms

[0062] The video timing consistency evaluation results (Table 2) show that the present method performs excellently in distortion rate (0.9956) and stability (0.8161) indicators, proving that it can effectively improve the temporal coherence of the video.

[0063] Table 2 Comparison of video smoothness evaluation

[0064] The ablation experiment proves that as the number of input frames increases, the model performance continues to improve, and both the temporal attention module and the multi-scale structure contribute significantly to the performance. As shown in Table 3, the performance steadily improves with the increase of the number of input frames, proving that integrating more temporal information helps the model better estimate and eliminate turbulence degradation. At the same time, the processing time grows linearly, providing multiple choices between performance and efficiency. Table 4 shows that after introducing the temporal attention module (TDAB), both PSNR and SSIM are steadily improved, proving the key role of this module in modeling temporal dependence and improving inter-frame consistency. Table 5 verifies the effectiveness of the multi-scale structure, and adding more branches can obtain better performance indicators, highlighting the key role of integrating multi-scale features in reducing turbulence degradation.

[0065] Table 3 Quantitative comparison of training results with different number of input frames

[0066] Table 4 Performance comparison based on TDAB

[0067] Table 5 Influence of different scale network structures on training results

[0068] Figure 4The visual effects of different algorithms for restoring algorithm simulation data are shown in the figures. It can be seen from the figures that (b), (c), (d), (e) and (g) cannot remove the blur in the algorithm simulation image well, and the performance of restoring the zebra body pattern is not satisfactory. In addition to the blur, (b) also has obvious noise. (f) is more accurate in restoring the zebra contour and stripes than the other algorithms, but cannot reflect the details of the grass below the image. The algorithm (h) proposed in this embodiment can not only present more detailed zebra features, but also pay attention to the small features that other algorithms ignore, such as grass texture, so that the restoration effect is closer to the clean image.

[0069] Figure 5 The restoration effects of different algorithms on physical simulation data are shown. The degraded image (a) presents a certain degree of distortion and blur compared with the clean image (i). In the restored image, the result of (b) is not close enough to the clean image, and there is a color difference in the sand in front of the building. (c), (d) and (e) are improved compared with (b), but the building still presents a certain degree of blur. (f) further improves the detail restoration capability based on the previous algorithms. For example, in (f), compared with the previous algorithms, it can restore some details such as the steel wire in the lower right corner of the yellow land. (g) is not as good as the other algorithms in numerical indicators, but its visual results show that it has certain advantages in detail feature restoration. The algorithm restoration image (h) proposed in this embodiment ensures the overall quality while repairing the features of small objects such as the minute and hour hands of the clock in the image well, and is visually closer to the clean image.

[0070] Embodiment 2 The embodiment provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the following steps: reading instructions and data from the storage medium; calling a frame alignment program to register the input turbulent video sequence; calling a multi-scale feature extraction program to generate multi-scale features; calling a spatial attention processing program to perform calculation of the SATB module; calling a frequency domain attention processing program to perform frequency domain filtering of the FATB module; calling a time domain attention processing program to perform time sequence fusion of the TDAB module; calling an image reconstruction program to output a restored clear image; in the training mode, calling a loss calculation program to calculate the gradient and update the network model parameters according to the joint loss function.

[0071] The embodiment can be one or more solid state disks (SSD), flash memory, read-only memory (ROM), or other physical media, in which the stored code is used to fully implement the method flow described.

[0072] Embodiment 3 The embodiment provides an electronic device, comprising: a memory for storing executable instructions; a processor configured to execute the instructions in the memory to implement the image / video de-denoising method; The electronic device is a server or workstation with a parallel computing acceleration unit such as GPU or NPU; The processor is connected with the computing acceleration unit, and the forward inference and back propagation calculation of the neural network system are borne by the computing acceleration unit; The memory includes volatile memory and non-volatile storage device, for storing training data set, model parameter and intermediate calculation result.

[0073] The embodiment provides a specific hardware platform instance, and the running flow is fully compatible with the system embodiment and the storage medium embodiment.

[0074] In summary, the application effectively solves the image degradation problem caused by atmospheric turbulence through the innovative multi-dimensional attention mechanism, which cooperatively processes in three dimensions of space, frequency and time domain, and exhibits superior performance in subjective vision and objective indicators.

Claims

1. An image and video denoising method based on temporal attention mechanism, characterized in that: The method is implemented by a deep learning neural network and includes the following steps: Step 1: Input a sequence of multiple consecutive frames of images degraded by atmospheric turbulence; Step 2: performing inter-frame registration and alignment processing on the image sequence to compensate for geometric distortion and pixel displacement caused by turbulence, and obtaining aligned multi-frame images; Step 3: Input the aligned multi-frame images into a multi-scale feature extraction network, and extract spatial features of different scales through the parallel branch structure of the multi-scale feature extraction network; Step 4: The multi-scale spatial features extracted in step 3 are first input into the spatial attention module, which generates an attention map by calculating the cross-channel covariance and combines the gating mechanism to enhance the spatial feature representation; Step 5: Input the features processed by the spatial attention module into the frequency domain attention module, which performs the following sub-steps: Step 51: Map the input features from the spatial domain to the frequency domain through fast Fourier transform; Step 52: Adaptively filtering the frequency domain features using a learnable Fourier filter; wherein the parameters of the Fourier filter are optimized by a back-propagation algorithm during the neural network training process to adaptively enhance frequency components that are beneficial to image reconstruction and suppress unfavorable frequency components introduced by turbulence degradation; Step 53: Map the filtered frequency domain features back to the spatial domain through inverse fast Fourier transform; Step 6: Input the features processed by the frequency domain attention module into the time domain attention module, and generate features with enhanced temporal consistency by fusing the current frame features with the temporal features of adjacent frames; Step 7: A refinement module integrates the features processed in step 6 to reconstruct a clear image without turbulence effects. Step 8: During the training phase, the neural network is optimized using a joint loss function, which includes pixel-level L1 loss, perceptual loss based on the VGG-19 network, and local pixel block frequency domain loss.

2. The image and video de-perturbation method based on the temporal attention mechanism according to claim 1 is characterized in that: The inter-frame registration and alignment in step 2 is achieved by calculating the local block matching error between adjacent frames and the reference frame and estimating the pixel-level displacement field using an elastic deformation model.

3. The image and video de-perturbation method based on the temporal attention mechanism according to claim 1 is characterized in that: The multi-scale feature extraction network in step 3 contains three parallel branches: the first branch maintains the resolution of the input image; the second branch downsamples the input by a factor of 2 and expands the channel size; the third branch downsamples the input by a factor of 4 and further expands the channel size; each branch extracts features through convolution operations, and finally the feature maps output by all branches are fused.

4. The image and video de-perturbation method based on the temporal attention mechanism according to claim 1 is characterized in that: The learnable Fourier filter is configured as follows: Receive the frequency domain feature map obtained by fast Fourier transform as input; Apply a set of learnable parameter weights to the input frequency domain feature map through element-wise multiplication, where the parameter weight is a two-dimensional tensor of the same size as the frequency domain feature map, and each weight value corresponds to the importance coefficient of its spatial frequency coordinate position; Among them, the learnable parameter weights are optimized through the back-propagation algorithm during the neural network training process; The output of the Fourier filter is the frequency domain feature after adaptive filtering, and is sent to the subsequent inverse fast Fourier transform layer to be converted back to the spatial domain.

5. The image and video de-perturbation method based on the temporal attention mechanism according to claim 1 is characterized in that: The temporal attention module in step 6 implements temporal fusion by concatenating the features of the current frame with the features of the adjacent frames, extracting temporal context information through convolution to generate an attention map, and performing weighted fusion of features based on the attention map.

6. The image and video de-perturbation method based on the temporal attention mechanism according to claim 1, characterized in that: In the processing of steps 4, 5, and 6, a block processing strategy is adopted for the feature maps input to the spatial attention module, the frequency domain attention module, and the time domain attention module; The block processing strategy includes: dividing the input feature map into multiple non-overlapping local blocks, performing attention calculation on each block separately, and then merging the blocks into a complete feature map after processing; The processed feature map is sent to step 7 for final image reconstruction.

7. An image and video de-perturbation system based on a temporal attention mechanism, used to implement the method according to any one of claims 1 to 6, characterized in that: The system comprises: The frame alignment module is used to perform inter-frame registration and alignment processing on the input image sequence to compensate for the geometric distortion and pixel displacement caused by turbulence, and obtain aligned multi-frame images; The multi-scale feature extraction module is used to extract spatial features of different scales from the aligned multi-frame images. The multi-scale feature extraction module contains multiple parallel branch structures to process inputs of different scales respectively. The spatial attention module is used to generate attention maps by calculating cross-channel covariance and enhance spatial feature representation in combination with a gating mechanism; The frequency domain attention module is used to map the input features from the spatial domain to the frequency domain through the fast Fourier transform, adaptively filter the frequency domain features using a learnable Fourier filter, and then map the filtered frequency domain features back to the spatial domain through the inverse fast Fourier transform; The temporal attention module is used to generate features with enhanced temporal consistency by fusing the features of the current frame with the temporal features of adjacent frames; The image reconstruction module is used to integrate the features processed by the temporal attention module to reconstruct a clear image without turbulence effects; The loss calculation module is used to calculate the joint loss function, which includes pixel-level L1 loss, perceptual loss based on the VGG-19 network, and local pixel block frequency domain loss.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the image and video de-perturbation method based on the temporal attention mechanism as described in any one of claims 1 to 6 is implemented.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that: When the processor executes the program, the image and video de-perturbation method based on the temporal attention mechanism is implemented as described in any one of claims 1-6.

Images