A gradient-guided lightweight atmospheric turbulence video rectification system and method

Abstract

This invention discloses a lightweight atmospheric turbulence video correction system and method based on gradient guidance. The system includes: a multi-frame input module for acquiring multiple consecutive frames of atmospheric turbulence degradation video frames and forming a multi-frame input tensor; a gradient guidance module for generating global frame-level weights and spatial pixel-level weights for each video frame, and combining them to generate pixel-level fusion weights; a weighted fusion module for performing pixel-by-pixel soft weighted fusion of multiple video frames based on the pixel-level fusion weights to obtain a multi-frame fused image; and a residual enhancement module for performing detail residual enhancement on the multi-frame fused image to obtain a corrected video image. This invention can fully utilize the complementary sharpness information between multiple video frames, reduce the block effect and inter-frame flicker caused by traditional hard-select fusion, and reduce network computational complexity while ensuring image detail restoration, making it suitable for resource-constrained edge computing platforms.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a gradient-guided lightweight atmospheric turbulence video correction system and method. Background Technology

[0002] Atmospheric turbulence can cause spatial random pixel displacement, geometric distortion, and blurring in videos acquired by optical imaging systems, severely impacting the accuracy of target detection and recognition in applications such as video surveillance, autonomous driving, and long-distance observation. With the rapid adoption of edge computing in fields such as intelligent security and unmanned systems, achieving real-time, high-quality atmospheric turbulence video correction on resource-constrained embedded edge platforms has become a key research direction for improving the robustness of long-distance optical vision perception systems.

[0003] To achieve atmospheric turbulence video correction, existing technologies mainly employ deep learning restoration methods and traditional processing methods. However, for example, the turbulence image restoration method based on high- and low-frequency synthesis reconstruction provided by patent CN116029933A uses a dual U-Net cascade structure to reconstruct high-frequency information and clear images separately. This not only leads to a significant increase in the number of parameters and computation due to the dual-level network, making it difficult to implement real-time inference on edge devices, but it is also essentially a single-frame restoration, ignoring the temporal complementary information of multiple frames in the video sequence, thus limiting the restoration effect under strong turbulence. The atmospheric turbulence video correction method provided by patent CN121392716A, which uses multi-scale feature fusion, relies on a SpyNet optical flow estimation network, deformable attention module, bidirectional gated recurrent unit, and multi-head temporal channel self-attention module. Although the restoration quality is high, the network structure is bloated, the number of parameters is huge, and the pixel-by-pixel offset prediction overhead and bidirectional temporal modeling require a large amount of memory buffer, making it impossible to run in real time on resource-constrained edge devices such as embedded NPUs. In addition, while the classic lucky imaging method has the advantages of being lightweight and not requiring explicit alignment, it relies on manually designed sharpness metrics for "hard selection" stitching, which has problems such as block artifacts and poor scene adaptability. Furthermore, it has not yet been effectively transformed into a deep network paradigm that can be learned end-to-end.

[0004] Therefore, it is necessary to propose a gradient-guided lightweight atmospheric turbulence video correction method to solve the problems of high computational complexity, difficulty in real-time operation on edge devices, poor lightweight network restoration quality, and easy generation of ghosting and block effects in existing atmospheric turbulence correction techniques. Summary of the Invention

[0005] In view of this, the present invention provides a gradient-guided lightweight atmospheric turbulence video correction system and method to solve the technical problem that existing technologies are unable to achieve high-quality restoration and correction of atmospheric turbulence videos with low computational cost and low memory overhead on embedded edge platforms with extremely limited computing resources.

[0006] To achieve the above-mentioned technical objectives, the present invention adopts the following technical solution: In a first aspect, the present invention provides a gradient-guided lightweight atmospheric turbulence video correction system, comprising: The multi-frame input module is used to acquire a series of consecutive atmospheric turbulence degradation video frames, and to stitch the multi-frame atmospheric turbulence degradation video frames along the channel dimension to form a multi-frame input tensor. The gradient guidance module, connected to the multi-frame input module, is used to generate global frame-level weights and spatial pixel-level weights corresponding to each atmospheric turbulence degradation video frame based on the multi-frame input tensor, and to generate pixel-level fusion weights based on the global frame-level weights and spatial pixel-level weights. The weighted fusion module, connected to the multi-frame input module and the gradient guidance module, is used to perform pixel-by-pixel soft weighted fusion of multiple atmospheric turbulence degradation video frames based on the pixel-level fusion weights to obtain a multi-frame fused image. The residual enhancement module, connected to the weighted fusion module, is used to generate detail residuals based on the multi-frame fused image, and to superimpose the detail residuals with the multi-frame fused image to obtain the corrected video image.

[0007] Furthermore, the gradient guidance module includes a global score branch and a spatial weight branch set in parallel; The global scoring branch is used to extract global features based on the multi-frame input tensor and generate global frame-level weights corresponding to each atmospheric turbulence degradation video frame. The spatial weight branch is used to extract gradient features for each atmospheric turbulence degradation video frame, and to jointly process the gradient features with the corresponding video frame image features to generate corresponding spatial pixel-level weights. The gradient guidance module generates the pixel-level fusion weights based on the global frame-level weights and the spatial pixel-level weights.

[0008] Furthermore, the global scoring branch includes a downsampling convolution unit, a global feature aggregation unit, and a global weight mapping unit connected in sequence, used to perform spatial resolution compression and global feature aggregation on the multi-frame input tensor, and to map and generate global frame-level weights for each atmospheric turbulence degradation video frame. The spatial weight branch includes a spatial gradient extraction unit, a feature channel stitching unit, and a spatial weight mapping unit connected in sequence. It is used to extract the gradient features of each atmospheric turbulence degradation video frame and generate spatial pixel-level weights based on the joint features of the gradient features and the corresponding video frame image features.

[0009] Furthermore, the gradient guidance module also includes a weight combination and normalization unit, which combines the global frame-level weights and the spatial pixel-level weights element-wise to obtain initial fusion weights corresponding to each atmospheric turbulence degradation video frame and each pixel position, and normalizes the initial fusion weights in the frame dimension to generate the pixel-level fusion weights, so as to achieve normalized weight allocation between different atmospheric turbulence degradation video frames corresponding to the same pixel position.

[0010] Furthermore, the residual enhancement module includes a feature expansion unit, a residual feature extraction unit, a refinement unit, a residual mapping unit, and a residual superposition unit connected in sequence. The feature expansion unit is used to expand the feature dimensions of the multi-frame fused image; The residual feature extraction unit is used to extract texture and structural detail features from the expanded features; The thinning unit is used to perform convolutional mapping on the texture and structural detail features to adjust the feature responses of the texture and structural detail features and obtain the thinned detail features; The residual mapping unit is used to map the refined detail features into detail residuals corresponding to the multi-frame fused image; The residual overlay unit is used to overlay the detail residual with the multi-frame fused image element by element to obtain the corrected video image.

[0011] Furthermore, the gradient guidance module, weighted fusion module, and residual enhancement module together constitute a differentiable end-to-end deep neural network; The gradient guidance module and the residual enhancement module each include learnable parameters, while the weighted fusion module is a forward fusion node without learnable parameters.

[0012] Secondly, this invention discloses a gradient-guided lightweight atmospheric turbulence video correction method, implemented using the system described above, comprising: Acquire multiple consecutive frames of atmospheric turbulence degradation video frames, and stitch these multiple frames along the channel dimension to form a multi-frame input tensor; The multi-frame input tensor is input to a gradient guidance module that includes a global score branch and a spatial weight branch. The global frame-level sharpness score of each frame is predicted through the global score branch, and the spatial pixel-level weight map of each frame is predicted through the spatial weight branch in combination with a fixed gradient operator and a learnable convolution. The global frame-level weights and spatial pixel-level weights are combined element-wise and normalized in the frame dimension to generate a pixel-level fusion weight map corresponding to each video frame. Based on each video frame and its corresponding pixel-level fusion weight map, a pixel-wise weighted summation is performed to obtain a multi-frame fusion image. The multi-frame fused image is input into a lightweight residual enhancement network to obtain the corresponding detail residuals. The detail residuals are then added element-wise to the multi-frame fused image to obtain the corrected video image.

[0013] Thirdly, the present invention discloses a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the atmospheric turbulence video correction method described above.

[0014] Compared with existing technologies, the gradient-guided lightweight atmospheric turbulence video correction method proposed in this invention has the following advantages: (1) Real-time processing at the edge with extremely low computational cost: The entire network of this invention is constructed using depthwise separable convolution and one-dimensional channel attention, without any computationally intensive modules such as optical flow estimation, deformable convolution, recurrent networks or self-attention. The total number of parameters is controlled in the range of 15K to 25K. After integer quantization, the single-frame inference latency on the embedded NPU is less than 30 milliseconds, which can meet the real-time deturbulence processing requirements of video streams above 30fps. This solves the core problem that existing deep learning methods cannot run in real time on edge devices due to excessive computational overhead.

[0015] (2) Adaptive temporal fusion without explicit alignment: This invention replaces the traditional optical flow registration or deformable alignment module with a one-dimensional channel attention mechanism. It dynamically learns the contribution weights of each frame feature in the channel dimension through global average pooling and one-dimensional convolution. This mechanism only adds a few parameters to achieve an adaptive fusion effect similar to "frame selection". It has good adaptability to different turbulence intensities and scene content, and avoids the failure problem of explicit alignment in large displacement scenarios.

[0016] (3) Center frame residual learning ensures restoration quality and training efficiency: This invention uses the center frame of the input sequence as the basis, and the network only outputs the degradation residual for compensation. This strategy allows the network to focus on learning the turbulent degradation components, avoiding structural distortion and artifacts that may be introduced when reconstructing a complete image from scratch. The center frame naturally preserves the real lighting and color distribution of the scene, and the restoration result is faithful to the original input. At the same time, the simplified learning objective significantly accelerates the training convergence speed. Attached Figure Description

[0017] Figure 1 A schematic diagram of the structure of the gradient-guided lightweight atmospheric turbulence video correction system provided by the present invention; Figure 2 This is a schematic diagram of the gradient guidance module provided by the present invention; Figure 3 The diagram shows a differentiable end-to-end deep neural network structure composed of the gradient guidance module, weighted fusion module, and residual enhancement module provided by this invention. Figure 4a This is a schematic diagram of the original turbulence degradation frame in a weak turbulence scenario provided by the present invention; Figure 4b This is a screenshot of the video frame after correction using this system. Figure 5a This is a schematic diagram of the original turbulence degradation frame in a strong turbulence scenario provided by the present invention; Figure 5b This is a screenshot of the video frame after correction using this system. Figure 6 This is a flowchart illustrating the gradient-guided lightweight atmospheric turbulence video correction method provided by the present invention. Detailed Implementation

[0018] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0019] Please see Figure 1 This embodiment provides a gradient-guided lightweight atmospheric turbulence video correction system 100, comprising: The multi-frame input module 101 is used to acquire a series of atmospheric turbulence degradation video frames and stitch the multi-frame atmospheric turbulence degradation video frames along the channel dimension to form a multi-frame input tensor. The gradient guidance module 102 is connected to the multi-frame input module 101 and is used to generate global frame-level weights and spatial pixel-level weights corresponding to each atmospheric turbulence degradation video frame based on the multi-frame input tensor, and generate pixel-level fusion weights based on the global frame-level weights and spatial pixel-level weights. The weighted fusion module 103 is connected to the multi-frame input module 101 and the gradient guidance module 102, and is used to perform pixel-by-pixel soft weighted fusion of multiple atmospheric turbulence degradation video frames based on the pixel-level fusion weights to obtain a multi-frame fused image. The residual enhancement module 104 is connected to the weighted fusion module 103 and is used to generate detail residuals based on the multi-frame fused image, and to superimpose the detail residuals with the multi-frame fused image to obtain the corrected video image.

[0020] The system provided in this embodiment acquires continuous atmospheric turbulence degradation video frames through a multi-frame input module and constructs a multi-frame input tensor. A gradient-guided module determines the sharpness weights of the video frames at both the global frame level and the local spatial pixel level, combining these two types of weights to form pixel-level fusion weights. A weighted fusion module then performs pixel-by-pixel soft-weighted fusion on the multi-frame video frames, fully utilizing the complementary sharpness information between different video frames. Finally, a residual enhancement module compensates for detail loss and slight blur in the fusion result, yielding the corrected video image. This system can reduce complex inter-frame alignment and high computational network dependencies while mitigating the block artifacts and inter-frame flicker caused by traditional hard-select fusion, improving the detail recovery capability, correction quality, and real-time processing adaptability of atmospheric turbulence videos on edge devices.

[0021] As a specific implementation, this system acquires T consecutive frames (typically T=6) of RGB turbulence degradation images through a multi-frame input module, and stitches them along the channel dimension to form an input tensor with 3×T channels. Where B is the batch size, and H and W are the spatial dimensions. This tensor is directly used as network input, without the need for explicit splitting into independent frames, or explicit inter-frame alignment operations such as optical flow or deformable convolution; let the image of frame t be denoted as . Then X is The channel splicing.

[0022] In a preferred embodiment, the gradient guidance module includes a global score branch and a spatial weight branch configured in parallel; The global scoring branch is used to extract global features based on the multi-frame input tensor and generate global frame-level weights corresponding to each atmospheric turbulence degradation video frame. The spatial weight branch is used to extract gradient features for each atmospheric turbulence degradation video frame, and to jointly process the gradient features with the corresponding video frame image features to generate corresponding spatial pixel-level weights. The gradient guidance module generates the pixel-level fusion weights based on the global frame-level weights and the spatial pixel-level weights.

[0023] Furthermore, the global scoring branch includes a downsampling convolution unit, a global feature aggregation unit, and a global weight mapping unit connected in sequence, used to perform spatial resolution compression and global feature aggregation on the multi-frame input tensor, and to map and generate global frame-level weights for each atmospheric turbulence degradation video frame. The spatial weight branch includes a spatial gradient extraction unit, a feature channel stitching unit, and a spatial weight mapping unit connected in sequence. It is used to extract the gradient features of each atmospheric turbulence degradation video frame and generate spatial pixel-level weights based on the joint features of the gradient features and the corresponding video frame image features.

[0024] In one specific embodiment, the receptive field of the global scoring branch covers the entire image, and its main task is to determine the overall degradation level of the entire frame in order to suppress the overall contribution of frames with severe turbulence.

[0025] like Figure 2 As shown, the global score branch first performs two levels of convolutional downsampling with a stride of 2. The spatial resolution is compressed to 1 / 4, then adaptive average pooling is used to obtain global feature vectors, and finally T global sharpness scores are output through a fully connected layer. Frame-level weights are obtained after Softmax normalization. This can be expressed as a formula:

[0026] in, and For the t-th frame and the t-th frame respectively The unnormalized global sharpness score of the frame image prediction output. The total number of frames in the continuous atmospheric turbulence degradation image input. is the base of the natural logarithm.

[0027] The spatial weight branch is guided by the gradient magnitude. First, the input tensor... Split into frames and calculate the grayscale image for each frame. The spatial gradient magnitude is extracted using the fixed Sobel operator and expressed as:

[0028] in, For Sobel core, Represents convolution. To ensure numerical stability, small constants are used.

[0029] Then the original RGB frame The gradient magnitude map Gt is concatenated with the gradient magnitude map Gt along the channel dimension to form a four-channel input. , [Through a lightweight learnable convolutional layer] The stitched features are modulated pixel by pixel, and a spatial weight map is generated using the Sigmoid function, i.e.:

[0030] This spatial weight branch preserves the prior sensitivity of gradient magnitude to edge and texture sharpness, while enabling the network to simultaneously perceive color and texture information, avoiding color shift caused by pure grayscale gradients, thus more accurately distinguishing sharp and degraded areas within the same frame.

[0031] This embodiment constructs a dynamic fusion mechanism that balances global anti-interference and local detail preservation through the coarse-grained collaborative guidance of global and local scoring branches. The global scoring branch efficiently compresses spatial resolution through two convolutional downsampling steps of 2, rapidly perceives the global receptive field of the entire image using global pooling and fully connected layers, and directly suppresses the overall weights of large-area, drastically degraded frames at the macroscopic level using Softmax normalization. The local branch introduces the Sobel gradient magnitude as an explicit edge prior and concatenates it with the original RGB image to form a 4-channel input. Convolutional layers and sigmoid activation achieve joint modulation of color and gradient, accurately capturing clear edges and texture details in the microscopic dimension and effectively avoiding color shift caused by single-channel grayscale gradients. Finally, through element-wise multiplication of global frame-level weights and local pixel-level weights and frame dimension normalization, a dual suppression and selection effect is achieved, which suppresses severely degraded frames globally and selects good frames locally. This significantly improves the edge sharpness and color fidelity of the fused image while maintaining extremely low computational overhead.

[0032] In some embodiments, the gradient guidance module further includes a weight combination and normalization unit, which is used to multiply the global frame-level weights and the spatial pixel-level weights element-wise to obtain a multi-scale initial weight map, and to normalize the multi-scale initial weight map in the frame dimension to generate the pixel-level fusion weights, with the sum of the fusion weights of all video frames at the same pixel position constrained to a constant value.

[0033] Finally, the global frame-level weights and spatial pixel-level weights are multiplied element-wise and normalized along the frame dimension to obtain the fused weight map for each pixel in each frame:

[0034] in, and These are the predictions made by the global score branch. Frame and the Global frame-level weights of a frame image. and The predictions of the spatial weighted branches are respectively the first... Frame and the Frame image in pixel coordinates Spatial pixel-level weights at each location.

[0035] This system uses the aforementioned dual-level weighting mechanism to suppress overall blurred frames from a global perspective, while simultaneously optimizing the clear pixels of each frame at a local scale.

[0036] In a preferred embodiment, after obtaining the pixel-level fusion weights, the input tensor is split into frames by the weighted fusion module. Based on the pixel-level fusion weights, each frame is weighted with its corresponding pixel-level weight map, and the summation along the frame dimension yields the preliminary clear image F after fusion, expressed by the formula:

[0037] in, For the first Atmospheric turbulence degradation video frame spatial pixel coordinates The corresponding pixel-level fusion weights; For the input of the first Atmospheric turbulence degradation video frame spatial pixel coordinates The original pixel value at that location.

[0038] In this weighted fusion module, the fusion process is a purely linear operation with no learnable parameters, resulting in extremely low computational cost. Unlike the hard-select stitching method of traditional lucky imaging, the soft-weighted fusion used in this system is naturally smooth and free of block artifacts, and can simultaneously utilize complementary information from different frames.

[0039] Since the fused image may have slight average blur due to incomplete alignment of multiple frames, in order to solve this problem, as a preferred embodiment, the present invention employs an ultra-lightweight residual enhancement network R for compensation.

[0040] The residual enhancement module includes a feature expansion unit, a residual feature extraction unit, a refinement unit, a residual mapping unit, and a residual overlay unit connected in sequence. The feature expansion unit is used to expand the feature dimensions of the multi-frame fused image; The residual feature extraction unit is used to extract texture and structural detail features from the expanded features; The thinning unit is used to perform convolutional mapping on the texture and structural detail features to adjust the feature responses of the texture and structural detail features and obtain the thinned detail features; The residual mapping unit is used to map the refined detail features into detail residuals corresponding to the multi-frame fused image; The residual overlay unit is used to overlay the detail residual with the multi-frame fused image element by element to obtain the corrected video image.

[0041] As a specific implementation, the fused image F is taken as input, the output detail residual is added element-wise to the fused image to obtain the final clear image, which can be expressed by the formula:

[0042] in, This indicates that the residual enhancement module uses multi-frame fused images. The detailed residual map is used as input for prediction.

[0043] The residual enhancement module in this embodiment includes a sequentially connected layer of convolutional expansion channels, a residual block, a layer of convolutional thinning features, and a layer of output convolution.

[0044] By introducing a residual learning mechanism to construct a residual enhancement module, the decoupling of low-frequency structure restoration and high-frequency detail compensation in the atmospheric turbulence correction process is achieved. Specifically, after multi-frame soft weighted fusion, the macroscopic geometric topology and basic structural information of the multi-frame input images have been effectively restored. However, due to the inherent smoothing effect of the weighted summation operation, the high-frequency edges and subtle textures of the image are prone to slight blurring.

[0045] The residual enhancement module of this invention focuses only on fitting the minute detail residuals between the fused image and the clear, realistic image, without reconstructing the well-restored low-frequency background and main structure. This "incremental" learning paradigm effectively avoids the structural distortions and artifacts that are easily generated when traditional end-to-end networks reconstruct the global image, accurately compensating for high-frequency textures while maintaining a highly faithful representation of the main physical structure. Simultaneously, the identity skip connections in the residual structure allow the backpropagation gradient to directly reach the previous stage, greatly reducing the search range of the model's solution space and lowering the optimization difficulty, thereby significantly improving the stability and convergence speed of neural network training.

[0046] In a preferred embodiment, the gradient guidance module, weighted fusion module, and residual enhancement module together constitute a differentiable end-to-end deep neural network, such as... Figure 3 As shown; The gradient guidance module and the residual enhancement module each include learnable parameters, while the weighted fusion module is a forward fusion node without learnable parameters.

[0047] The system in this embodiment is differentiable throughout its process, from gradient extraction to weight prediction, and from weighted fusion to residual enhancement, supporting end-to-end training. The total number of network parameters is kept below 50K, and all operations are standard convolution, pooling, and element-wise operations. There are no computationally intensive modules such as optical flow estimation, deformable convolution, recurrent networks, or self-attention, which fully adapts to the integer quantization and parallel acceleration requirements of embedded NPUs, enabling real-time atmospheric turbulence video correction on edge devices such as the RK3588.

[0048] To verify the actual application effect of this system, such as Figure 4a and Figure 4b As shown, Figure 4a and Figure 4b The study showcases a comparison of algorithm performance in weak turbulence scenarios on a real dataset. It can be seen that, in weak atmospheric turbulence scenarios, after correction by this system, the overall image structure remains largely consistent with the original scene. The horizontal texture and edge contours of building doors within the selected area are clearer, and the boundaries of target structures such as vehicle outlines are more stable. This indicates that, under conditions of low turbulence degradation, this system can effectively utilize complementary and clear information between multiple frames to restore local details while avoiding obvious stitching marks and structural distortion. Figure 5a and Figure 5b The presentation showcases a comparison of algorithm performance under strong turbulence scenarios on a real dataset. It reveals that in the original image, target edges, road structures, and distant pedestrians are more significantly blurred and disturbed. After correction by our system, the outlines of the selected areas and structures such as vehicles and road boundaries are much clearer. Compared to weak turbulence scenarios, the improvement under strong turbulence conditions better demonstrates the effectiveness of our system in obtaining complementary and clear information from different video frames through multi-frame pixel-level soft weighted fusion, combined with residual enhancement to compensate for details in the fused image.

[0049] like Figure 6 As shown, this embodiment of the invention also provides a gradient-guided lightweight atmospheric turbulence video correction method, implemented using the system described above, including: Step S101: Obtain multiple consecutive atmospheric turbulence degradation video frames, and stitch the multiple atmospheric turbulence degradation video frames along the channel dimension to form a multi-frame input tensor; Step S102: Input the multi-frame input tensor into a gradient guidance module that includes a global score branch and a spatial weight branch. Predict the global frame-level sharpness score of each frame through the global score branch. Predict the spatial pixel-level weight map of each frame through the spatial weight branch, which combines a fixed gradient operator with a learnable convolution. Step S103: Combine the global frame-level weights and the spatial pixel-level weights element by element and normalize them in the frame dimension to generate a pixel-level fusion weight map corresponding to each video frame. Perform pixel-by-pixel weighted summation based on each video frame and its corresponding pixel-level fusion weight map to obtain a multi-frame fusion image. Step S104: Input the multi-frame fused image into a lightweight residual enhancement network to obtain the corresponding detail residual, and add the detail residual to the multi-frame fused image element by element to obtain the corrected video image.

[0050] The gradient-guided lightweight atmospheric turbulence video correction method proposed in this invention first obtains a sequence of consecutive multi-frame atmospheric turbulence-degraded video frames of the target scene, and performs frame-by-frame enhanced alignment and shared feature extraction to obtain corresponding multi-frame feature maps. Second, the feature maps are concatenated along the channel dimension, and adaptive weighted fusion is performed based on a temporal channel attention mechanism to fully exploit temporal correlations and enhance the weights of key features. Finally, residual reconstruction is performed based on the fused features to generate a residual image of the center frame, which is then pixel-level fused with the original center frame to output the final target video frame after atmospheric turbulence removal. This method effectively solves the problem of difficult precise coordination of temporal information under extreme atmospheric turbulence by combining frame-by-frame pre-alignment with temporal channel adaptive fusion depth. It can accurately correct image geometric distortion and eliminate jitter blur, improving the clarity, high-frequency edge details, and temporal smoothness of the restored video frame.

[0051] In a preferred embodiment, the method further includes: jointly updating the learnable parameters in the gradient guidance module and the residual enhancement module based on the training error corresponding to the corrected video image, so that the gradient guidance, weight generation, weighted fusion and residual enhancement processes are jointly optimized in a unified training process.

[0052] This embodiment also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the gradient-guided lightweight atmospheric turbulence video correction method described in any of the above technical solutions.

[0053] The computer-readable storage medium and computing device provided in the above embodiments of the present invention can be implemented with reference to the content specifically described above for a gradient-guided lightweight atmospheric turbulence video correction method, and have similar beneficial effects to the gradient-guided lightweight atmospheric turbulence video correction method described above, which will not be repeated here.

[0054] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A lightweight atmospheric turbulence video correction system based on gradient guidance, characterized in that, include: The multi-frame input module is used to acquire a series of consecutive atmospheric turbulence degradation video frames, and to stitch the multi-frame atmospheric turbulence degradation video frames along the channel dimension to form a multi-frame input tensor. The gradient guidance module, connected to the multi-frame input module, is used to generate global frame-level weights and spatial pixel-level weights corresponding to each atmospheric turbulence degradation video frame based on the multi-frame input tensor, and to generate pixel-level fusion weights based on the global frame-level weights and spatial pixel-level weights. The weighted fusion module, connected to the multi-frame input module and the gradient guidance module, is used to perform pixel-by-pixel soft weighted fusion of multiple atmospheric turbulence degradation video frames based on the pixel-level fusion weights to obtain a multi-frame fused image. The residual enhancement module, connected to the weighted fusion module, is used to generate detail residuals based on the multi-frame fused image, and to superimpose the detail residuals with the multi-frame fused image to obtain the corrected video image.

2. The lightweight atmospheric turbulence video correction system based on gradient guidance according to claim 1, characterized in that, The gradient guidance module includes a global score branch and a spatial weight branch set in parallel; The global scoring branch is used to extract global features based on the multi-frame input tensor and generate global frame-level weights corresponding to each atmospheric turbulence degradation video frame. The spatial weight branch is used to extract gradient features for each atmospheric turbulence degradation video frame, and to jointly process the gradient features with the corresponding video frame image features to generate corresponding spatial pixel-level weights. The gradient guidance module generates the pixel-level fusion weights based on the global frame-level weights and the spatial pixel-level weights.

3. The gradient-guided lightweight atmospheric turbulence video correction system according to claim 2, characterized in that, The global scoring branch includes a downsampling convolution unit, a global feature aggregation unit, and a global weight mapping unit connected in sequence. These units are used to perform spatial resolution compression and global feature aggregation on the multi-frame input tensor, and to map and generate global frame-level weights for each atmospheric turbulence degradation video frame.

4. The gradient-guided lightweight atmospheric turbulence video correction system according to claim 2, characterized in that, The spatial weight branch includes a spatial gradient extraction unit, a feature channel stitching unit, and a spatial weight mapping unit connected in sequence. It is used to extract the gradient features of each atmospheric turbulence degradation video frame and generate spatial pixel-level weights based on the joint features of the gradient features and the corresponding video frame image features.

5. The gradient-guided lightweight atmospheric turbulence video correction system according to claim 2, characterized in that, The gradient guidance module further includes a weight combination and normalization unit, which combines the global frame-level weights and the spatial pixel-level weights element-wise to obtain initial fusion weights corresponding to each atmospheric turbulence degradation video frame and each pixel position, and normalizes the initial fusion weights in the frame dimension to generate the pixel-level fusion weights, so as to achieve normalized weight allocation between different atmospheric turbulence degradation video frames corresponding to the same pixel position.

6. The gradient-guided lightweight atmospheric turbulence video correction system according to claim 1, characterized in that, The residual enhancement module includes a feature expansion unit, a residual feature extraction unit, a refinement unit, a residual mapping unit, and a residual overlay unit connected in sequence. The feature expansion unit is used to expand the feature dimensions of the multi-frame fused image; The residual feature extraction unit is used to extract texture and structural detail features from the expanded features; The thinning unit is used to perform convolutional mapping on the texture and structural detail features to adjust the feature responses of the texture and structural detail features and obtain the thinned detail features; The residual mapping unit is used to map the refined detail features into detail residuals corresponding to the multi-frame fused image; The residual overlay unit is used to overlay the detail residual with the multi-frame fused image element by element to obtain the corrected video image.

7. The gradient-guided lightweight atmospheric turbulence video correction system according to claim 1, characterized in that, The gradient guidance module, weighted fusion module, and residual enhancement module together constitute a differentiable end-to-end deep neural network. The gradient guidance module and the residual enhancement module each include learnable parameters, while the weighted fusion module is a forward fusion node without learnable parameters.

8. A gradient-guided lightweight atmospheric turbulence video correction method, implemented using the system described in any one of claims 1-7, characterized in that, include: Acquire multiple consecutive frames of atmospheric turbulence degradation video frames, and stitch these multiple frames along the channel dimension to form a multi-frame input tensor; The multi-frame input tensor is input to a gradient guidance module that includes a global score branch and a spatial weight branch. The global frame-level sharpness score of each frame is predicted through the global score branch, and the spatial pixel-level weight map of each frame is predicted through the spatial weight branch in combination with a fixed gradient operator and a learnable convolution. The global frame-level weights and spatial pixel-level weights are combined element-wise and normalized in the frame dimension to generate a pixel-level fusion weight map corresponding to each video frame. Based on each video frame and its corresponding pixel-level fusion weight map, a pixel-wise weighted summation is performed to obtain a multi-frame fusion image. The multi-frame fused image is input into a lightweight residual enhancement network to obtain the corresponding detail residuals. The detail residuals are then added element-wise to the multi-frame fused image to obtain the corrected video image.

9. The gradient-guided lightweight atmospheric turbulence video correction method according to claim 8, characterized in that, The method further includes: Based on the training error corresponding to the corrected video image, the learnable parameters in the gradient guidance module and the residual enhancement module are jointly updated so that the gradient guidance, weight generation, weighted fusion and residual enhancement processes are jointly optimized in a unified training process.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the atmospheric turbulence video correction method as described in any one of claims 8-9.

Images