Motion information enabled space-time consistent turbulence degraded image restoration method

Abstract

The invention provides a motion information enabled space-time consistent turbulence degraded image restoration method, which is used for solving the problem that the existing turbulence degraded image restoration algorithm based on data driving does not consider the large-range pixel deviation caused by the movement of a moving target in long integral time. The pixel deviation and space-time related pixel-by-pixel turbulence degradation jointly act on the optical imaging process, and the final imaging effect is affected. According to the method, a covariant deformable convolution alignment unit is adopted to learn a turbulence degradation sampling offset and an inter-frame alignment sampling offset on three divided short-time turbulence degradation continuous frames, so that pixel-by-pixel turbulence degradation feature extraction is carried out while inter-frame alignment is carried out, and three-frame registration turbulence degradation features are obtained; a coupling relation between large-range pixel deviation caused by movement of a moving target in long integral time and space-time related pixel-by-pixel turbulence degradation can be accurately modeled so as to adapt to turbulence non-rigid distortion and high-order blurring of a turbulence degradation image.

Description

Technical Field

[0001] The present invention relates to a method for restoring a turbulently degraded image, and more particularly to a method for restoring a spatio-temporally consistent turbulently degraded image empowered by motion information. Background Art

[0002] At present, quite a lot of achievements have been made in the technology of restoring turbulently degraded images. Traditional image deconvolution algorithms based on wavefront detection usually assume that the point spread function (PSF) is known in advance or the PSF is relatively simple, so they are only applicable to the restoration of blurred images under the conditions of known blur and low-frequency random media. However, the wavefront detection data is inevitably interfered by noise, and the PSF at this time only contains the data of the wavefront detector and does not contain the information of the observed image. Therefore, it is necessary to add the prior information constraint of the PSF and the device information of the observed image to estimate the PSF. There are many methods to implement the image deconvolution algorithm, such as blind deconvolution method, maximum likelihood estimation method, regularization method, wavelet transform method, etc. However, when these methods for implementing the image deconvolution algorithm are applied to the task of restoring turbulently degraded images, noise amplification is likely to occur, and there are problems such as strong dependence on the degradation model, high computational complexity, and limited restoration effect.

[0003] Due to the highly chaotic characteristics of atmospheric turbulence itself, traditional image deconvolution algorithms based on wavefront detection are difficult to accurately characterize the strongly spatio-temporally correlated turbulent degradation process affected by multiple factors and serve the task of restoring turbulently degraded images. In the era of greatly improved computing power, data-driven neural representation algorithms can rely on their powerful non-linear feature representation ability to perform image restoration, and certain progress has also been made in their application to the task of restoring turbulently degraded images, thus emerging data-driven algorithms for restoring turbulently degraded images. However, for ground-based large-aperture observation devices applied to different astronomical observation tasks, a relatively long integration time is often required to achieve energy accumulation. During the long integration time, moving targets will have orbital motion, and at the same time, the significant influence of the temporal correlation of atmospheric conditions on the light field propagation process should be considered. However, in the data-driven algorithm for restoring turbulently degraded images, when modeling the spatio-temporally correlated turbulent degradation process, only the turbulence suppression is regarded as a many-to-one restoration problem, and the large-range pixel deviation caused by the motion of moving targets during the long integration time is not considered. This pixel deviation and the spatio-temporally correlated pixel-by-pixel turbulent degradation jointly act on the optical imaging process and have a complex coupling relationship with the final turbulent imaging degradation effect, affecting the final imaging effect. Summary of the Invention

[0004] The object of the present invention is to solve the technical problem that existing data-driven turbulence-degraded image restoration algorithms do not consider the large-range pixel deviation caused by the movement of moving targets within a long integration time, resulting in the combined effect of pixel deviation and spatio-temporally correlated pixel-by-pixel turbulence degradation on the optical imaging process, which affects the final imaging effect, and to provide a spatio-temporally consistent turbulence-degraded image restoration method empowered by motion information.

[0005] To achieve the above object, the technical solution provided by the present invention is as follows: A spatio-temporally consistent turbulence-degraded image restoration method empowered by motion information, characterized in that it includes the following steps: Step 1, constructing a training set and a test set; the training set includes a plurality of consecutive turbulence-degraded images and corresponding plurality of consecutive clear images, and the test set includes a plurality of consecutive turbulence-degraded images; Step 2, respectively dividing the plurality of consecutive turbulence-degraded images and the plurality of consecutive clear images in the training set into a plurality of three-frame short-time turbulence-degraded consecutive frames and corresponding plurality of three-frame short-time clear consecutive frames, and taking the middle frame of each three-frame short-time clear consecutive frame as the ground truth frame of the corresponding three-frame short-time turbulence-degraded consecutive frame; Step 3, constructing a covariant deformable convolution alignment unit; the covariant deformable convolution alignment unit includes a left branch, a right branch and a residual convolution module; the input end of the left branch is used to receive the middle frame of the three-frame short-time turbulence-degraded consecutive frames, and the input end of the right branch is used to receive the three-frame short-time turbulence-degraded consecutive frames to respectively learn the turbulence-degraded sampling offset and the inter-frame alignment sampling offset; the input end of the residual convolution module is connected to the output ends of the left branch and the right branch for performing pixel-by-pixel turbulence-degraded feature extraction while achieving inter-frame alignment; Step 4, respectively inputting the middle frames of the plurality of three-frame short-time turbulence-degraded consecutive frames and the plurality of three-frame short-time turbulence-degraded consecutive frames into the left branch and the right branch of the covariant deformable convolution alignment unit, so that the residual convolution module outputs corresponding three-frame registered turbulence-degraded features; Step 5, respectively obtaining the features of each three-frame registered turbulence-degraded feature in the implicit space and the features of the ground truth frame of each three-frame short-time turbulence-degraded consecutive frame in the implicit space, and respectively denoting them as three-frame degraded implicit features and ground truth implicit features; Step 6, generating a random Gaussian noise ε1 with the same size as the ground truth implicit feature, and adding the random Gaussian noise ε1 to the ground truth implicit feature to obtain the noise-added implicit features at different time steps; Step 7, on the premise of the two-way optical flow consistency constraint, inputting the noise-added implicit features, the ground truth implicit features, the three-frame degraded implicit features at different time steps corresponding to the training set and the corresponding time steps into a noise prediction network for iterative training to obtain a trained noise prediction network; Step 8: Divide multiple consecutive turbulent degradation images in the test set into multiple three-frame short-term turbulent degradation consecutive frames, and obtain the corresponding three-frame degradation implicit features according to the methods in Steps 4 to 5; generate random Gaussian noise ε2 with the same size as the three-frame degradation implicit features, and input the corresponding three-frame degradation implicit features and random Gaussian noise ε2 in the test set into the trained noise prediction network. The three-frame degradation implicit features are forwarded under the constraint of bidirectional optical flow consistency, so that the trained noise prediction network outputs predicted noise; Step 9: Perform reverse denoising and decoding on the predicted noise obtained in Step 8 to obtain the restored turbulent degradation image.

[0006] Further, in Step 7, on the premise of the bidirectional optical flow consistency constraint, the noisy implicit features, ground-truth implicit features, three-frame degradation implicit features corresponding to different time steps in the training set, and the corresponding time steps are fed into the noise prediction network for iterative training specifically as follows: Step a1: Feed the noisy implicit features, ground-truth implicit features, three-frame degradation implicit features corresponding to different time steps in the training set, and the corresponding time steps into the noise prediction network, obtain the feature optical flow from the middle frame to the subsequent frame and the feature optical flow from the middle frame to the previous frame of each three-frame degradation implicit feature corresponding to the time step, and record them as the forward optical flow and the backward optical flow respectively; Step b1: Calculate the forward optical flow offset and the backward optical flow offset of each three-frame degradation implicit feature, and further obtain the consistency error between the forward optical flow offset and the corresponding backward optical flow offset, and judge whether it is less than or equal to the set threshold. If so, transfer the noisy implicit features, ground-truth implicit features, and three-frame degradation implicit features at the current time step to the next time step under the action of the optical flow function; otherwise, transfer the noisy implicit features, ground-truth implicit features, and three-frame degradation implicit features at the previous time step to the next time step under the action of the optical flow function.

[0007] Further, in Step 3, the turbulent degradation sampling offset is expressed as follows:

[0008] where, is the turbulent degradation sampling offset, and respectively represent the offset domains of per-pixel turbulent degradation in the two-dimensional plane x direction and y direction, represents the real number field, H represents the height of the turbulent degradation image, and W represents the width of the turbulent degradation image; The inter-frame alignment sampling offset is expressed as follows:

[0009] Among them, is the inter-frame alignment sampling offset, and respectively represent and aligned to in the two-dimensional plane x direction and y the offset field in the direction.

[0010] Furthermore, in step 5, specifically obtaining the features of each three-frame registered turbulent degradation feature in the implicit space and the features of the ground truth frame of each three-frame short-term turbulent degradation continuous frame in the implicit space is as follows: Step a2, construct an autoencoder model, which is an autoencoder model learned on a de-motion-blur dataset and includes an encoder and a decoder; Step b2, respectively obtain the features of each three-frame registered turbulent degradation feature in the implicit space and the features of the ground truth frame of each three-frame short-term turbulent degradation continuous frame in the implicit space through the encoder of the autoencoder model.

[0011] Furthermore, in step 7, when iteratively training the noise prediction network, optimize the noise prediction network jointly through a noise prediction loss function, a structural loss function, and a perceptual loss function.

[0012] Furthermore, specifically optimizing the noise prediction network jointly through a noise prediction loss function, a structural loss function, and a perceptual loss function in step 7 is as follows: Step a3, adopt a noise prediction loss function to minimize the distance between the output of the noise prediction network during iterative training and the random Gaussian noise ε1; Step b3, calculate the clean ground truth images at different time steps:

[0013] Among them, is the noise-added implicit feature at time step i, is the noise scheduling parameter at time step i; Step c3, decode the clean ground truth images at different time steps using the decoder of the autoencoder model; Step d3, adopt a structural loss function to minimize the distance between the calculated clean ground truth images at different time steps and the middle frame of the three-frame short-term clear continuous frames; Step e3, adopt a perceptual loss function to optimize the calculated clean ground truth images Minimize the distance between the middle frame of three short - time clear consecutive frames for optimization.

[0014] Further, in step 7, the noise prediction loss function is the CharbonnierLoss function; The structure loss function is a multi - scale structure function; The perceptual loss function is a pre - trained VGG19 network.

[0015] Further, in step 6, adding random Gaussian noise ε1 to the true - value implicit feature to obtain the noisy implicit features at different time steps is specifically as follows: Adding random Gaussian noise ε1 to the true - value implicit feature through the noise scheduling parameters at different time steps to obtain the noisy implicit features at different time steps. Its specific expression is as follows:

[0016] Among them, is the noisy implicit feature at time step i, is the noise scheduling parameter at time step i, is the true - value implicit feature.

[0017] Further, in step 9, use the DDIM sampling method to perform reverse denoising on the predicted noise obtained in step 8.

[0018] Further, step 9 is specifically as follows: Use the DDIM sampling method to perform reverse denoising on the predicted noise obtained in step 8 to obtain the implicit feature of the restored turbulent - degraded image; then decode the implicit feature of the restored turbulent - degraded image through the decoder of the auto - encoder model to obtain the restored turbulent - degraded image.

[0019] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. A spatio - temporally consistent turbulent - degraded image restoration method empowered by motion information provided by the present invention divides multiple consecutive turbulent - degraded images and multiple consecutive clear images respectively, and uses a covariant deformable convolution alignment unit to learn the turbulent - degradation sampling offset and the inter - frame alignment sampling offset on three short - time consecutive frames of turbulent - degraded images after division. Furthermore, while performing inter - frame alignment, pixel - by - pixel turbulent - degradation feature extraction is carried out to obtain three frames of registered turbulent - degradation features. Performing subsequent feature learning on the three frames of registered turbulent - degradation features can accurately model the large - range pixel deviation caused by the motion of moving objects during long integration times and the coupling relationship between spatio - temporally correlated pixel - by - pixel turbulent degradation, so as to adapt to the turbulent non - rigid distortion and high - order blur of turbulent - degraded images, obtain a high - quality moving - object turbulent - degraded image restoration method, and effectively ensure the local feature consistency of turbulent - degraded images.

[0020] 2. A spatio-temporally consistent turbulent degraded image restoration method empowered by motion information provided by the present invention uses bidirectional optical flow consistency constraints for feature filtering during the forward propagation process of the diffusion model, and only allows the feature regions with smaller consistency errors between the forward optical flow offset and the backward optical flow offset to propagate forward in the diffusion model, effectively ensuring the content consistency between long-time continuous frames.

[0021] 3. The related basic theories and key technologies of the present invention will promote the technological development of large ground-based optical telescopes in China for multiple ground-based observation tasks. The application of this invention in the field of direct imaging of exoplanets will help astronomers directly detect Earth-like planets in the habitable zones of solar-like stars, and can be used to answer the basic scientific question of "whether humans are alone in the universe". The introduction of the continuous motion characteristics of moving targets by this invention can also provide a powerful tool for the study of exoplanet orbital dynamics. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a schematic diagram of the principle of the covariant deformable convolution alignment unit in step 3 of the embodiment of a spatio-temporally consistent turbulent degraded image restoration method empowered by motion information of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0023] To make the advantages and features of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0024] A spatio-temporally consistent turbulent degraded image restoration method empowered by motion information of the present invention models the turbulent degraded images with spatio-temporal variations of moving targets during long integration times based on a diffusion model with text implicit conditions, and introduces the spatial motion information of exoplanets during long integration times into the sampling process of the diffusion model to accurately restore the turbulent degraded samples, which specifically includes the following steps: Step 1, construct a training set and a test set.

[0025] The training set includes multiple consecutive turbulent degraded images of the collected moving target and corresponding multiple consecutive clear images. The multiple consecutive clear images are multiple temporally related clear images collected when the moving target is stationary, and they correspond to the multiple turbulent degraded images one by one. The test set includes multiple consecutive turbulent degraded images of the collected moving target.

[0026] Step 2, image division.

[0027] Denote the number of turbulent degraded images in the training set as N, then N consecutive turbulent degraded images can be expressed as . The N consecutive turbulent degraded images Divided into multiple three-frame short-term turbulent degradation continuous frames. When N is an integer multiple of 3, multiple three-frame short-term turbulent degradation continuous frames can be expressed as , when N is not an integer multiple of 3, the extra consecutive turbulent degradation images at the back can be discarded. Among them, the middle frame of is the previous frame is , the middle frame of is the previous frame is , and so on, the middle frame of is the previous frame is .

[0028] Correspondingly, the number of clear images is also N. Then N consecutive clear images can be expressed as . Divide N consecutive clear images into multiple three-frame short-term clear continuous frames in the above way. Then multiple three-frame short-term clear continuous frames can be expressed as . Among them, the middle frame of is the previous frame is , the middle frame of is the previous frame is , and so on, the middle frame of is the previous frame is .

[0029] Take the middle frame of each three-frame short-term clear continuous frame as the true value frame of the corresponding three-frame short-term turbulent degradation continuous frame. That is, take as 's true value frame, take as 's true value frame, and so on, take as 's true value frame.

[0030] Step 3, construct a covariant deformable convolution alignment unit.

[0031] The covariant deformable convolution alignment unit includes a left branch, a right branch, and a residual convolution module . As Figure 1 shown, the input end of the left branch is used to receive the middle frame of the three-frame short-term turbulent degradation continuous frame , that is, receive of , in The turbulence degradation sampling offset of three frames of short-term turbulence degradation consecutive frames is learned on ; The input end of the right branch is used to receive three frames of short-term turbulence degradation continuous frames , that is, receiving of , in Learning the inter-frame alignment sampling offset of three short-term turbulence-degraded consecutive frames The left branch and the right branch cooperate to achieve dual-path covariant learning.

[0032] in, , and They represent the pixel-by-pixel turbulence degradation in the two-dimensional plane. x Direction and y The offset domain of the direction, represents the real number domain, H represents the height of the turbulence-degraded image, and W represents the width of the turbulence-degraded image. The sampling position is adjusted, that is, the sampling points of the convolution kernel can dynamically adjust the sampling position according to the actual content of the input data to adapt to the turbulent non-rigid distortion and high-order blur of the turbulence-degraded image. , and Respectively and Align to In two-dimensional plane x Direction and y The offset field for the direction.

[0033] Considering the complex process of the final imaging caused by the large-scale pixel deviation and pixel-by-pixel turbulence degradation on three consecutive frames of short-term turbulence degradation when the moving target moves, the residual convolution module is used in the present invention. The input end is connected to the output end of the left branch and the right branch, using the residual module While aligning the frames, pixel-by-pixel turbulence degradation features are extracted to obtain corresponding multiple three-frame registered turbulence degradation features.

[0034] Step 4: The middle frame The input is learned in the left branch of the deformable convolution alignment unit. The input covariant deformable convolution alignment unit is learned in the right branch. At this time, the residual convolution module Output the three-frame registered turbulence degradation features corresponding to each three-frame short-time turbulence-degraded continuous frame.

[0035] Step 5: Learn on the existing de - motion - blur dataset to obtain an auto - encoder model, which includes an encoder and a decoder. Through the encoder of the auto - encoder model, obtain the features of each three - frame registered turbulent degradation feature in the implicit space and the features of the true - value frame of each three - frame short - time turbulent degradation continuous frame in the implicit space, and denote them as three - frame degradation implicit features and true - value implicit features respectively. The three - frame registered turbulent degradation features will act together with the time encoding on the spatial layer of the subsequent noise prediction network.

[0036] Step 6: Forward noise addition.

[0037] Generate random Gaussian noise ε1 with the same size as the true - value implicit feature. Through the noise scheduling parameters at different time steps, add the random Gaussian noise ε1 to the true - value implicit feature to obtain the noise - added implicit features at different time steps. Its specific expression is as follows:

[0038] where, is the noise - added implicit feature at time step i, is the noise scheduling parameter at time step i, is the true - value implicit feature.

[0039] Step 7: Iterative training of the noise prediction network.

[0040] On the premise of the two - way optical flow consistency constraint, send the noise - added implicit features, true - value implicit features, three - frame degradation implicit features at different time steps corresponding to the training set, and the corresponding time steps into the noise prediction network for iterative training. Through the two - way optical flow consistency constraint, the features can be transmitted forward in the network, and then a trained noise prediction network can be obtained.

[0041] Specifically, send the noise - added implicit features, true - value implicit features, three - frame degradation implicit features at different time steps corresponding to the training set, and the corresponding time steps into the noise prediction network to obtain the feature optical flow from the middle frame to the next frame and the feature optical flow from the middle frame to the previous frame , and denote them as forward optical flow and backward optical flow respectively. Assume that between two frames corresponding to time t and t + 1, a certain feature region moves from position p to position , then the forward optical flow offset is , that is, . Correspondingly, between two frames corresponding to time t and t - 1, a certain feature region moves back from position to position p , then the backward optical flow offset is .

[0042] Calculate the consistency error between the forward optical flow offset and the corresponding backward optical flow offset of each three-frame degraded implicit feature through the following formula :[[]]

[0043] If the forward optical flow offset and the backward optical flow offset are exactly the same, its consistency error should be 0. In actual calculation, a set threshold is usually set. If the consistency error is less than or equal to the set threshold, it proves that the optical flow of this feature area is reliable, and the noisy implicit feature, true value implicit feature, and three-frame degraded implicit feature at the current time step are transferred to the next time step under the action of the optical flow function. Otherwise, if the consistency error is greater than the set threshold, it proves that the optical flow of this feature area is unreliable, and the noisy implicit feature, true value implicit feature, and three-frame degraded implicit feature at the previous time step are transferred to the next time step under the action of the optical flow function. Specifically, the present invention sets a mask with the same size as the three-frame degraded implicit feature. The 1 on the mask represents the reliable area, and 0 represents the unreliable area. After passing through the mask image, only the features of the feature area with reliable optical flow are transferred forward. Its expression is as follows:

[0044] Among them, represents the noisy implicit feature, true value implicit feature, and three-frame degraded implicit feature of the j th frame image at time t , represents the noisy implicit feature, true value implicit feature, and three-frame degraded implicit feature of the j th frame image at time t+1 ; M is the value of the mask. When the consistency error is less than or equal to the set threshold, M = 1. When the consistency error is greater than the set threshold, M = 0.

[0045] The present invention adopts the above-mentioned bidirectional optical flow consistency constraint to ensure the content consistency between long-term continuous frames. Among them, the set threshold is adjusted according to the training process and can be appropriately relaxed in dynamic scenarios.

[0046] Preferably, when iteratively training the noise prediction network in this embodiment, the noise prediction network is optimized by jointly using the noise prediction loss function, the structure loss function, and the perceptual loss function. Specifically as follows: (1) Adopt the noise prediction loss function to the output of the noise prediction network during iterative training Minimize the distance between the random Gaussian noise ε1; in this embodiment, the noise prediction loss function is the CharbonnierLoss loss function.

[0047] (2) Calculate the clean ground truth images at different time steps :

[0048] (3) Use the decoder of the autoencoder model to decode the clean ground truth images at different time steps for decoding.

[0049] (4) Use the structural loss function to minimize the distance between the decoded clean ground truth images at different time steps and the middle frame of three short-term clear and continuous frames; in this embodiment, the structural loss function is the multi-scale structural function MS-SSIM.

[0050] (5) Use the perceptual loss function to minimize the distance between the decoded clean ground truth images at different time steps and the middle frame of three short-term clear and continuous frames; in this embodiment, the perceptual loss function is the pre-trained VGG19 network.

[0051] Step 8: Divide multiple consecutive turbulent degradation images in the test set into multiple three-frame short-term turbulent degradation continuous frames, and input the middle frame of multiple three-frame short-term turbulent degradation continuous frames and multiple three-frame short-term turbulent degradation continuous frames into the left and right branches of the covariant deformable convolution alignment unit respectively, so that the residual convolution module outputs the corresponding three-frame registered turbulent degradation features; obtain the features of each three-frame registered turbulent degradation feature in the implicit space through the encoder of the autoencoder model, that is, obtain the corresponding three-frame degradation implicit features in the test set.

[0052] Generate random Gaussian noise ε2 with the same size as the three-frame degradation implicit features, and input the corresponding three-frame degradation implicit features and random Gaussian noise ε2 in the test set into the trained noise prediction network together, so that the trained noise prediction network outputs the predicted noise. Among them, the corresponding three-frame degradation implicit features in the test set are also passed forward using the same bidirectional optical flow consistency constraint in step 7, that is, if its corresponding consistency error is less than or equal to its set threshold, then the three-frame degradation implicit features at the current time step are passed to the next time step under the action of the optical flow function, otherwise, the three-frame degradation implicit features at the previous time step are passed to the next time step under the action of the optical flow function.

[0053] Step 9: Use the DDIM sampling method to perform reverse denoising on the predicted noise obtained in Step 8 to obtain the implicit features of the restored turbulence-degraded image. Then, decode the implicit features of the restored turbulence-degraded image through the decoder of the autoencoder model to obtain the restored turbulence-degraded image.

[0054] As described above, it is only used to illustrate the technical solution of the present invention and is not a limitation thereof. For those of ordinary skill in the art, the specific technical solution recorded in the above embodiments can be modified, or some of the technical features can be equivalently replaced. However, these modifications or replacements do not cause the essence of the corresponding technical solution to deviate from the scope of the technical solution protected by the present invention.

Claims

1. A method for restoring a spatiotemporally consistent turbulent degraded image empowered by motion information, characterized in that, It includes the following steps: Step 1, constructing a training set and a test set; the training set includes a plurality of consecutive turbulent degradation images and corresponding plurality of consecutive clear images, and the test set includes a plurality of consecutive turbulent degradation images; Step 2, respectively dividing the plurality of consecutive turbulent degradation images and the plurality of consecutive clear images in the training set into a plurality of three-frame short-time turbulent degradation consecutive frames and corresponding plurality of three-frame short-time clear consecutive frames, and taking the middle frame of each three-frame short-time clear consecutive frame as the ground truth frame of the corresponding three-frame short-time turbulent degradation consecutive frame; Step 3, constructing a covariant deformable convolution alignment unit; the covariant deformable convolution alignment unit includes a left branch, a right branch, and a residual convolution module; the input end of the left branch is used to receive the middle frame of the three-frame short-time turbulent degradation consecutive frames, and the input end of the right branch is used to receive the three-frame short-time turbulent degradation consecutive frames to respectively learn the turbulent degradation sampling offset and the inter-frame alignment sampling offset; the input end of the residual convolution module is connected to the output ends of the left branch and the right branch, and is used to perform pixel-by-pixel turbulent degradation feature extraction while performing inter-frame alignment; Step 4, respectively inputting the middle frames of the plurality of three-frame short-time turbulent degradation consecutive frames and the plurality of three-frame short-time turbulent degradation consecutive frames into the left branch and the right branch of the covariant deformable convolution alignment unit, so that the residual convolution module outputs corresponding three-frame registered turbulent degradation features; Step 5, respectively obtaining the features of each three-frame registered turbulent degradation feature in the implicit space and the features of the ground truth frame of each three-frame short-time turbulent degradation consecutive frame in the implicit space, and respectively denoting them as three-frame degradation implicit features and ground truth implicit features; Step 6, generating a random Gaussian noise ε1 with the same size as the ground truth implicit feature, and adding the random Gaussian noise ε1 to the ground truth implicit feature to obtain the noisy implicit features at different time steps; Step 7, on the premise of the two-way optical flow consistency constraint, sending the noisy implicit features, ground truth implicit features, three-frame degradation implicit features at different time steps corresponding to the training set and the corresponding time steps into a noise prediction network for iterative training to obtain a trained noise prediction network; Step 8, dividing the plurality of consecutive turbulent degradation images in the test set into a plurality of three-frame short-time turbulent degradation consecutive frames, and obtaining corresponding three-frame degradation implicit features according to the methods of Steps 4 to 5; Generating a random Gaussian noise ε2 with the same size as the three-frame degradation implicit feature, and inputting the corresponding three-frame degradation implicit feature and the random Gaussian noise ε2 in the test set into the trained noise prediction network together. The three-frame degradation implicit feature is propagated forward under the two-way optical flow consistency constraint, so that the trained noise prediction network outputs a predicted noise; Step 9, performing reverse denoising and decoding on the predicted noise obtained in Step 8 to obtain the restored turbulent degradation image.

2. The spatio-temporally consistent turbulent degradation image restoration method empowered by motion information according to claim 1, wherein: In Step 7, on the premise of the two-way optical flow consistency constraint, sending the noisy implicit features, ground truth implicit features, three-frame degradation implicit features at different time steps corresponding to the training set and the corresponding time steps into a noise prediction network for iterative training specifically means: Step a1: Input the noisy implicit features, ground-truth implicit features, three-frame degraded implicit features at different time steps corresponding to the training set, and the corresponding time steps into the noise prediction network to obtain the feature optical flow from the middle frame to the subsequent frame and the feature optical flow from the middle frame to the previous frame for each three-frame degraded implicit feature at the corresponding time step, which are respectively denoted as the forward optical flow and the backward optical flow. Step b1: Calculate the forward optical flow offset and the backward optical flow offset for each three-frame degraded implicit feature, and then obtain the consistency error between the forward optical flow offset and the corresponding backward optical flow offset. Determine whether it is less than or equal to the set threshold. If so, transfer the noisy implicit features, ground-truth implicit features, and three-frame degraded implicit features at the current time step to the next time step under the action of the optical flow function; otherwise, transfer the noisy implicit features, ground-truth implicit features, and three-frame degraded implicit features at the previous time step to the next time step under the action of the optical flow function.

3. The method for spatio-temporally consistent turbulent degradation image restoration empowered by motion information according to claim 2, wherein: In step 3, the turbulent degradation sampling offset is expressed as follows: ； Among them, is the turbulent degradation sampling offset, and respectively represent the offset domains of per-pixel turbulent degradation in the two-dimensional plane x direction and y direction, represents the real number field, H represents the height of the turbulent degradation image, and W represents the width of the turbulent degradation image; The inter-frame alignment sampling offset is expressed as follows: ； Among them, is the inter-frame alignment sampling offset, and respectively represent and aligned to in the two-dimensional plane x direction and y direction of the offset field.

4. The method for spatio-temporally consistent turbulent degradation image restoration empowered by motion information according to claim 3, wherein: In step 5, specifically obtaining the features of each three-frame registered turbulent degradation feature in the implicit space and the features of the ground-truth frames of each three-frame short-term turbulent degradation continuous frames in the implicit space is as follows: Step a2: Construct an autoencoder model, which is an autoencoder model learned on a deblurring dataset and includes an encoder and a decoder. Step b2: Use the encoder of the autoencoder model to obtain the features of each three-frame registered turbulent degradation feature in the implicit space and the features of the ground-truth frames of each three-frame short-term turbulent degradation continuous frames in the implicit space respectively.

5. The method for spatio-temporally consistent turbulent degradation image restoration empowered by motion information according to claim 4, wherein: In step 7, when iteratively training the noise prediction network, optimize the noise prediction network jointly through a noise prediction loss function, a structure loss function, and a perceptual loss function.

6. The method for spatio-temporally consistent turbulent degradation image restoration empowered by motion information according to claim 5, wherein: In step 7, specifically optimizing the noise prediction network jointly through a noise prediction loss function, a structure loss function, and a perceptual loss function is as follows: Step a3, using a noise prediction loss function, minimizes and optimizes the distance between the output of the noise prediction network during iterative training and the random Gaussian noise ε1; Step b3, calculate the clean ground truth images at different time steps : ； Among them, is the noisy implicit feature at time step i, is the noise schedule parameter at time step i; Step c3, use the decoder of the auto-encoding model to decode the clean ground-truth images at different time steps for decoding; Step d3, using a structural loss function, minimizes the distance between the calculated clean ground-truth images at different time steps and the middle frame of three short-term clear and continuous frames; Step e3, using a perceptual loss function, minimizes the distance between the calculated clean ground-truth images at different time steps and the middle frame of three short-term clear and continuous frames.

7. The method for spatio-temporally consistent turbulent degradation image restoration empowered by motion information according to claim 6, wherein: In step 7, the noise prediction loss function is the CharbonnierLoss loss function; The structure loss function is a multi-scale structure function; The perceptual loss function is a pre-trained VGG19 network.

8. The method for spatio-temporally consistent turbulent degradation image restoration empowered by motion information according to claim 1, wherein: In step 6, adding the random Gaussian noise ε1 to the ground-truth implicit features to obtain the noisy implicit features at different time steps is specifically as follows: By means of the noise scheduling parameters at different time steps, random Gaussian noise ε1 is added to the true implicit features to obtain the noise-added implicit features at different time steps. The specific expression is as follows: ； Among them, is the noisy implicit feature at time step i, is the noise schedule parameter at time step i, is the true implicit feature.

9. A spatio-temporally consistent turbulent degradation image restoration method empowered by motion information according to any one of claims 1-8, characterized in that: In step 9, the DDIM sampling method is used to perform reverse denoising on the predicted noise obtained in step 8.

10. A method for restoring a spatiotemporally consistent turbulent degraded image empowered by motion information according to claim 9, characterized in that, Step 9 is specifically as follows: The DDIM sampling method is used to perform reverse denoising on the predicted noise obtained in step 8 to obtain the implicit features of the restored turbulent degradation image; then, the decoder of the autoencoder model is used to decode the implicit features of the restored turbulent degradation image to obtain the restored turbulent degradation image.

Images