A motion information-enabled, spatiotemporally consistent turbulence-degraded image restoration method

Abstract

The present invention provides a motion information-enabled spatiotemporally consistent turbulently degraded image restoration method, which is used to solve the technical problem that the existing data-driven turbulently degraded image restoration algorithm does not consider the large-scale pixel deviation caused by the motion of the moving target in a long integration time, resulting in the pixel deviation and the spatiotemporally related pixel-by-pixel turbulent degradation acting together on the optical imaging process, affecting the final imaging effect. The present invention adopts a covariant deformable convolution alignment unit to learn the turbulent degradation sampling offset and the inter-frame alignment sampling offset on the three frames of short-time turbulently degraded continuous frames after division, and then performs pixel-by-pixel turbulent degradation feature extraction while aligning the frames to obtain three-frame registered turbulent degradation features. It can accurately model the coupling relationship between the large-scale pixel deviation caused by the motion of the moving target in a long integration time and the spatiotemporally related pixel-by-pixel turbulent degradation, so as to adapt to the turbulent non-rigid distortion and high-order blur of the turbulently degraded image.

Description

Technical Field

[0001] The present invention relates to a turbulence-degraded image restoration method, and in particular to a motion information-enabled spatiotemporally consistent turbulence-degraded image restoration method. Background Art

[0002] Considerable progress has been made in restoring turbulence-degraded images. Traditional image deconvolution algorithms based on wavefront detection typically assume a priori knowledge of the point spread function (PSF), or a relatively simple PSF. Therefore, they are only suitable for restoring blurred images in low-frequency random media with known blur. However, wavefront detection data is inevitably contaminated by noise, and the PSF in this case only contains the wavefront detector data and no information about the observed image. Therefore, it is necessary to incorporate prior information constraints on the PSF and estimate the PSF using device information from the observed image. Numerous methods exist for implementing image deconvolution, such as blind deconvolution, maximum likelihood estimation, regularization, and wavelet transforms. However, these methods are prone to noise amplification when applied to turbulence-degraded image restoration, suffer from strong dependence on the degradation model, high computational complexity, and limited restoration effectiveness.

[0003] Due to the highly chaotic nature of atmospheric turbulence, traditional wavefront-based image deconvolution algorithms struggle to accurately characterize the strongly spatiotemporally correlated turbulence degradation process, which is influenced by multiple factors, and thus serve the task of restoring turbulently degraded images. In an era of rapidly increasing computing power, data-driven neural representation algorithms can leverage their powerful nonlinear feature representation capabilities for image restoration. Their application to turbulently degraded image restoration has also made some progress, leading to the emergence of data-driven turbulently degraded image restoration algorithms. However, for ground-based large-aperture observation equipment used in various astronomical observation missions, long integration times are often required to achieve energy accumulation. During these long integration times, moving targets exhibit orbital motion, and the significant impact of the temporal correlation of atmospheric conditions on the light field propagation process must also be considered. However, the data-driven turbulence-degraded image restoration algorithm only considers turbulence suppression as a many-to-one restoration problem when modeling the spatiotemporal-correlated turbulence degradation process, without considering the large-scale pixel deviation caused by the motion of the moving target within a long integration time. This pixel deviation acts together with the spatiotemporal-correlated pixel-by-pixel turbulence degradation in the optical imaging process, and has a complex coupling relationship with the final turbulence imaging degradation effect, thus affecting the final imaging effect. Summary of the Invention

[0004] The purpose of the present invention is to solve the technical problem that the existing data-driven turbulence-degraded image restoration algorithm does not take into account the large-scale pixel deviation caused by the motion of the moving target within a long integration time, resulting in the pixel deviation and the time-space related pixel-by-pixel turbulence degradation acting together on the optical imaging process, affecting the final imaging effect, and provide a time-space consistent turbulence-degraded image restoration method enabled by motion information.

[0005] In order to achieve the above object, the technical solution provided by the present invention is as follows:

[0006] A motion information-enabled spatiotemporally consistent turbulence-degraded image restoration method comprises the following steps:

[0007] Step 1: construct a training set and a test set; the training set includes multiple continuous turbulence-degraded images and corresponding multiple continuous clear images, and the test set includes multiple continuous turbulence-degraded images;

[0008] Step 2: Divide the multiple continuous turbulence-degraded images and the multiple continuous clear images in the training set into multiple three-frame short-time turbulence-degraded continuous frames and corresponding multiple three-frame short-time clear continuous frames, and use the middle frame of each three-frame short-time clear continuous frame as the true value frame of the corresponding three-frame short-time turbulence-degraded continuous frame;

[0009] Step 3: construct a covariant deformable convolution alignment unit; the covariant deformable convolution alignment unit includes a left branch, a right branch, and a residual convolution module; the input end of the left branch is used to receive the intermediate frame of three frames of short-term turbulence-degraded continuous frames, and the input end of the right branch is used to receive three frames of short-term turbulence-degraded continuous frames to respectively learn the turbulence degradation sampling offset and the inter-frame alignment sampling offset; the input end of the residual convolution module is connected to the output ends of the left branch and the right branch, and is used to perform pixel-by-pixel turbulence degradation feature extraction while aligning the frames;

[0010] Step 4: input the middle frame of the three-frame short-time turbulence-degraded continuous frames and the three-frame short-time turbulence-degraded continuous frames into the left branch and the right branch of the covariant deformable convolution alignment unit respectively, so that the residual convolution module outputs the corresponding three-frame registered turbulence degradation features;

[0011] Step 5: Obtain the features of each three-frame registered turbulence degradation feature in the implicit space and the features of each three-frame short-time turbulence degradation continuous frame in the implicit space, and record them as three-frame degradation implicit features and true implicit features respectively;

[0012] Step 6: Generate random Gaussian noise ε1 of the same size as the true implicit feature, and add the random Gaussian noise ε1 to the true implicit feature to obtain the noisy implicit feature at different time steps;

[0013] Step 7: Under the premise of bidirectional optical flow consistency constraint, the noisy implicit features, true implicit features, three-frame degraded implicit features and corresponding time steps at different time steps corresponding to the training set are sent to the noise prediction network for iterative training to obtain a trained noise prediction network;

[0014] Step 8: Divide the multiple continuous turbulence-degraded images in the test set into multiple three-frame short-time turbulence-degraded continuous frames, and obtain the corresponding three-frame degradation implicit features according to the methods of steps 4 and 5; generate random Gaussian noise ε2 of the same size as the three-frame degradation implicit features, and input the corresponding three-frame degradation implicit features in the test set and the random Gaussian noise ε2 into the trained noise prediction network together. The three-frame degradation implicit features are forwarded under the bidirectional optical flow consistency constraint, so that the trained noise prediction network outputs the predicted noise;

[0015] Step 9: perform reverse denoising and decoding on the predicted noise obtained in step 8 to obtain a restored turbulence-degraded image.

[0016] Furthermore, in step 7, under the premise of bidirectional optical flow consistency constraint, the noisy implicit features, true implicit features, three-frame degraded implicit features and corresponding time steps at different time steps of the training set are sent to the noise prediction network for iterative training. Specifically:

[0017] Step a1: The noisy implicit features, true implicit features, three-frame degraded implicit features, and corresponding time steps of the training set at different time steps are fed into the noise prediction network. The feature optical flow from the middle frame to the next frame, and the feature optical flow from the middle frame to the previous frame of each three-frame degraded implicit feature at the corresponding time step are obtained, and are recorded as forward optical flow and backward optical flow respectively.

[0018] Step b1, calculate the forward optical flow offset and backward optical flow offset of each three-frame degraded implicit feature, and then obtain the consistency error of the forward optical flow offset and the corresponding backward optical flow offset, and judge whether it is less than or equal to the set threshold. If so, the noisy implicit features, true value implicit features, and three-frame degraded implicit features at the current time step are transferred to the next time step under the action of the optical flow function; otherwise, the noisy implicit features, true value implicit features, and three-frame degraded implicit features at the previous time step are transferred to the next time step under the action of the optical flow function.

[0019] Furthermore, in step 3, the turbulence degradation sampling offset is expressed as follows:

[0020]

[0021] in, is the turbulence degradation sampling offset, and They represent the pixel-by-pixel turbulence degradation in the two-dimensional plane. x Direction and y The offset domain of the direction, represents the real number domain, H represents the height of the turbulence-degraded image, and W represents the width of the turbulence-degraded image;

[0022] The inter-frame alignment sampling offset is expressed as follows:

[0023]

[0024] in, is the inter-frame alignment sampling offset, and Respectively and Align to In two-dimensional plane x Direction and y The offset field for the direction.

[0025] Furthermore, in step 5, the features of each three-frame registered turbulence degradation feature in the implicit space and the features of each three-frame short-time turbulence degradation continuous frame true value frame in the implicit space are obtained respectively as follows:

[0026] Step a2: constructing an automatic encoding and decoding model, wherein the automatic encoding and decoding model is an automatic encoding and decoding model learned on a de-motion blur dataset, and includes an encoder and a decoder;

[0027] In step b2, the encoder of the automatic encoding and decoding model is used to obtain the features of each three-frame registered turbulence degradation feature in the implicit space and the features of each three-frame short-time turbulence degradation continuous frame true value frame in the implicit space.

[0028] Furthermore, in step 7, when the noise prediction network is iteratively trained, the noise prediction network is optimized by using the noise prediction loss function, the structural loss function, and the perceptual loss function.

[0029] Furthermore, in step 7, the noise prediction network is optimized by the noise prediction loss function, the structural loss function, and the perceptual loss function as follows:

[0030] Step a3: Use the noise prediction loss function to calculate the output of the noise prediction network during iterative training. Minimize the distance between the random Gaussian noise ε1;

[0031] Step b3, calculate the clean ground truth image at different time steps :

[0032]

[0033] in, is the noise implicit feature at time step i, is the noise scheduling parameter at time step i;

[0034] Step c3: Use the decoder of the automatic encoding and decoding model to decode the clean true value images at different time steps. Decode;

[0035] Step d3 uses the structural loss function to calculate the clean truth images at different time steps Minimize the distance between the middle frame and the three short-term clear continuous frames;

[0036] Step e3, using the perceptual loss function, calculates the clean truth images at different time steps The distance between the three short-term clear continuous frames is minimized and optimized.

[0037] Furthermore, in step 7, the noise prediction loss function is a CharbonnierLoss loss function;

[0038] The structural loss function is a multi-scale structural function;

[0039] The perceptual loss function is a pre-trained VGG19 network.

[0040] Furthermore, in step 6, random Gaussian noise ε1 is added to the true implicit feature to obtain the noisy implicit features at different time steps:

[0041] By using the noise scheduling parameters at different time steps, random Gaussian noise ε1 is added to the true implicit features to obtain the noisy implicit features at different time steps. The specific expressions are as follows:

[0042]

[0043] in, is the noise implicit feature at time step i, is the noise scheduling parameter at time step i, is the true value implicit feature.

[0044] Furthermore, in step 9, the prediction noise obtained in step 8 is reversely denoised using the DDIM sampling method.

[0045] Furthermore, step 9 is specifically as follows:

[0046] The predicted noise obtained in step 8 is reversely denoised using the DDIM sampling method to obtain the implicit features of the restored turbulence-degraded image; the implicit features of the restored turbulence-degraded image are then decoded by the decoder of the automatic encoding and decoding model to obtain the restored turbulence-degraded image.

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] 1. The present invention provides a motion information-enabled spatiotemporally consistent turbulently degraded image restoration method, which divides multiple continuous turbulently degraded images and multiple continuous clear images respectively, and uses a covariant deformable convolution alignment unit to learn the turbulent degradation sampling offset and the inter-frame alignment sampling offset on the three frames of short-time turbulently degraded continuous frames after division, and then performs pixel-by-pixel turbulent degradation feature extraction while aligning the frames to obtain three-frame registered turbulent degradation features. Subsequent feature learning is performed on the three-frame registered turbulent degradation features, which can accurately model the coupling relationship between the large-scale pixel deviation caused by the motion of the moving target in a long integration time and the spatiotemporal-related pixel-by-pixel turbulent degradation, so as to adapt to the turbulent non-rigid distortion and high-order blur of the turbulently degraded image, obtain a high-quality moving target turbulently degraded image restoration method, and effectively ensure the local feature consistency of the turbulently degraded image.

[0049] 2. The present invention provides a motion information-enabled spatiotemporally consistent turbulence-degraded image restoration method, which uses bidirectional optical flow consistency constraints for feature filtering during the forward propagation of the diffusion model. Only feature areas with smaller consistency errors between the forward optical flow offset and the backward optical flow offset are passed forward in the diffusion model, effectively ensuring content consistency between long-term continuous frames.

[0050] 3. The underlying theories and key technologies of this invention will advance the development of my country's ground-based large-aperture optical telescopes for a variety of ground-based observation missions. Application of this invention to direct exoplanet imaging will help astronomers directly detect Earth-like planets within the habitable zones of Sun-like stars, potentially answering the fundamental scientific question of "Is humanity alone in the universe?" This invention's introduction of continuous motion characteristics for moving objects can also provide a powerful tool for studying the orbital dynamics of exoplanets. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 Schematic diagram of the principle of the covariant deformable convolution alignment unit in step 3 of an embodiment of a motion information-enabled spatiotemporally consistent turbulence-degraded image restoration method of the present invention. DETAILED DESCRIPTION

[0052] In order to make the advantages and features of the present invention more clear, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0053] The present invention provides a motion information-enabled, spatiotemporally consistent turbulence-degraded image restoration method. Based on a diffusion model with implicit textual conditions, the spatiotemporally variable turbulence-degraded image of a moving target within a long integration time is modeled. The spatial motion information of an exoplanet within the long integration time is introduced into the sampling process of the diffusion model to restore the turbulence-degraded sample with high precision. The method specifically includes the following steps:

[0054] Step 1: Construct training and test sets.

[0055] The training set includes multiple consecutive turbulence-degraded images of a moving target and corresponding multiple consecutive clear images. The multiple consecutive clear images are time-sequentially correlated clear images of the moving target captured while it was stationary, and correspond one-to-one with the multiple turbulence-degraded images. The test set includes multiple consecutive turbulence-degraded images of the moving target.

[0056] Step 2: Image segmentation.

[0057] The number of turbulence-degraded images in the training set is recorded as N, and N consecutive turbulence-degraded images can be expressed as . N consecutive turbulence-degraded images Divided into multiple three-frame short-time turbulence degradation continuous frames, when N is an integer multiple of 3, multiple three-frame short-time turbulence degradation continuous frames can be expressed as , when N is not an integer multiple of 3, the redundant continuous turbulence-degraded images can be discarded. The middle frame is , the previous frame is , the next frame is , The middle frame is , the previous frame is , the next frame is , and so on, The middle frame is , the previous frame is , the next frame is .

[0058] Correspondingly, the number of clear images is also N, then N consecutive clear images can be expressed as Divide N consecutive clear images into multiple three-frame short-time clear continuous frames according to the above method, and then the multiple three-frame short-time clear continuous frames can be expressed as .in, The middle frame is , the previous frame is , the next frame is , The middle frame is , the previous frame is , the next frame is , and so on, The middle frame is , the previous frame is , the next frame is .

[0059] The middle frame of each three-frame short-time clear continuous frame is used as the true value frame of the corresponding three-frame short-time turbulence degraded continuous frame. As The true value frame of As The true value frame of As The true value frame.

[0060] Step 3: Construct a covariant deformable convolutional alignment unit.

[0061] The covariant deformable convolution alignment unit includes a left branch, a right branch, and a residual convolution module. .like Figure 1 As shown, the input end of the left branch is used to receive the middle frame of three frames of short-term turbulence degradation continuous frames , that is, receiving of , in The turbulence degradation sampling offset of three frames of short-term turbulence degradation continuous frames is learned on The input of the right branch is used to receive three consecutive frames of short-term turbulence degradation , that is, receiving of , in Learning the inter-frame alignment sampling offset of three frames of short-term turbulence-degraded consecutive frames The left branch and the right branch cooperate to achieve dual-path covariant learning.

[0062] in, , and They represent the pixel-by-pixel turbulence degradation in the two-dimensional plane. x Direction and y The offset domain of the direction, Represents the real number domain, H represents the height of the turbulence degradation image, and W represents the width of the turbulence degradation image. The sampling position is adjusted, that is, the sampling points of the convolution kernel can be dynamically adjusted according to the actual content of the input data to adapt to the turbulence non-rigid distortion and high-order blur of the turbulence-degraded image. , and Respectively and Align to In two-dimensional plane x Direction and y The offset field for the direction.

[0063] Considering the complex process of large-scale pixel deviation and pixel-by-pixel turbulence degradation caused by the three consecutive frames of short-term turbulence degradation when the moving target moves, the residual convolution module is used to generate the final image. The input end is connected to the output end of the left branch and the right branch, using the residual module While aligning the frames, pixel-by-pixel turbulence degradation features are extracted to obtain the corresponding multiple three-frame registered turbulence degradation features.

[0064] Step 4: Intermediate frame The input is learned in the left branch of the deformable convolution alignment unit, The input is learned in the right branch of the covariant deformable convolution alignment unit. At this time, the residual convolution module Output the three-frame registered turbulence degradation features corresponding to each three-frame short-time turbulence degradation continuous frame.

[0065] In step 5, the automatic encoding and decoding model is learned on an existing deblurred image dataset. The automatic encoding and decoding model consists of an encoder and a decoder. The encoder of the automatic encoding and decoding model obtains the implicit space features of each three-frame registered turbulence degradation feature and the implicit space features of each three-frame short-term turbulence degradation continuous frame. These features are recorded as the three-frame degradation implicit features and the true implicit features, respectively. The three-frame registered turbulence degradation features will be used together with the temporal encoding to feed into the spatial layer of the subsequent noise prediction network.

[0066] Step 6: Forward noise addition.

[0067] Generate random Gaussian noise ε1 of the same size as the true implicit feature. Add the random Gaussian noise ε1 to the true implicit feature through the noise scheduling parameters at different time steps to obtain the noisy implicit features at different time steps. The specific expression is as follows:

[0068]

[0069] in, is the noise implicit feature at time step i, is the noise scheduling parameter at time step i, is the true value implicit feature.

[0070] Step 7: Iterative training of the noise prediction network.

[0071] Under the premise of bidirectional optical flow consistency constraints, the noisy implicit features, true value implicit features, three-frame degraded implicit features and corresponding time steps at different time steps corresponding to the training set are sent into the noise prediction network for iterative training. After the bidirectional optical flow consistency constraints, the features can be passed forward in the network, and then a trained noise prediction network is obtained.

[0072] Specifically, the noise implicit features, true implicit features, three-frame degraded implicit features and corresponding time steps of the training set at different time steps are fed into the noise prediction network to obtain the intermediate frame of each three-frame degraded implicit feature at the corresponding time step. To the next frame Feature optical flow, and intermediate frames To previous frame The characteristic optical flow is recorded as forward optical flow and backward optical flow respectively. Assume that between the two frames of images corresponding to time t to t+1, a feature area moves from position p Move to position , then the forward optical flow offset is ,Right now , accordingly, between the two frames of images corresponding to time t to t-1, a feature area changes from position Move back to position p , then the backward optical flow offset is .

[0073] The consistency error between the forward optical flow offset and the corresponding backward optical flow offset of each three-frame degraded implicit feature is calculated by the following formula: :

[0074]

[0075] If the forward optical flow offset and the backward optical flow offset Completely consistent, its consistency error Should be 0. In actual calculation, a threshold is usually set. If the consistency error If the value is less than or equal to the set threshold, it proves that the optical flow of the feature area is reliable, so that the noisy implicit features, true implicit features, and three-frame degraded implicit features at the current time step are transferred to the next time step under the action of the optical flow function. Otherwise, if the consistency error If the value is greater than the set threshold, it is proved that the optical flow of the feature area is unreliable. The noisy implicit features, true implicit features, and three-frame degraded implicit features at the previous time step are transferred to the next time step under the action of the optical flow function. Specifically, the present invention sets a mask with the same size as the three-frame degraded implicit features. The 1 on the mask represents a reliable area, and the 0 represents an unreliable area. After passing through the mask image, only the features of the feature areas with reliable optical flow are transferred forward. Its expression is as follows:

[0076]

[0077] in, Indicates the j Frame image at time t The noise implicit features, true value implicit features, and three-frame degradation implicit features are Indicates the j Frame image at time t+1 The noise implicit features, true value implicit features, and three-frame degradation implicit features when M is the value of the mask. When the consistency error When the consistency error is less than or equal to the set threshold, M=1. When it is greater than the set threshold, M=0.

[0078] The present invention adopts the above-mentioned bidirectional optical flow consistency constraint to ensure the content consistency between long-term continuous frames. The threshold is adjusted according to the training process and can be appropriately relaxed in dynamic scenes.

[0079] Preferably, when the noise prediction network is iteratively trained in this embodiment, the noise prediction network is optimized by using the noise prediction loss function, the structural loss function, and the perceptual loss function. Specifically,

[0080] (1) Using the noise prediction loss function, the output of the noise prediction network during iterative training The distance between the noise prediction function and the random Gaussian noise ε1 is minimized and optimized; in this embodiment, the noise prediction loss function is the CharbonnierLoss loss function.

[0081] (2) Calculate the clean ground truth images at different time steps :

[0082]

[0083] (3) The decoder of the automatic encoding and decoding model is used to decode the clean ground truth images at different time steps. to decode.

[0084] (4) Using a structural loss function, the distance between the decoded clean true value image and the middle frame of the three short-time clear continuous frames at different time steps is minimized and optimized; in this embodiment, the structural loss function is a multi-scale structure function MS-SSIM.

[0085] (5) Using a perceptual loss function, the distance between the decoded clean true value image and the middle frame of the three short-time clear continuous frames at different time steps is minimized and optimized; in this embodiment, the perceptual loss function is a pre-trained VGG19 network.

[0086] Step 8: Divide multiple continuous turbulence-degraded images in the test set into multiple three-frame short-time turbulence-degraded continuous frames, and input the intermediate frames of the multiple three-frame short-time turbulence-degraded continuous frames and the multiple three-frame short-time turbulence-degraded continuous frames into the left branch and the right branch of the covariant deformable convolution alignment unit respectively, so that the residual convolution module outputs the corresponding three-frame registration turbulence degradation features; obtain the features of each three-frame registration turbulence degradation feature in the implicit space through the encoder of the automatic encoding and decoding model, that is, obtain the corresponding three-frame degradation implicit features in the test set.

[0087] Generate random Gaussian noise ε2 of the same size as the three-frame degraded implicit features, and input the corresponding three-frame degraded implicit features in the test set and the random Gaussian noise ε2 into the trained noise prediction network, so that the trained noise prediction network outputs the predicted noise. The corresponding three-frame degraded implicit features in the test set are also forwarded using the same bidirectional optical flow consistency constraint as in step 7. That is, if the corresponding consistency error is less than or equal to the set threshold, the three-frame degraded implicit features at the current time step are transferred to the next time step under the action of the optical flow function. Otherwise, the three-frame degraded implicit features at the previous time step are transferred to the next time step under the action of the optical flow function.

[0088] In step 9, the predicted noise obtained in step 8 is reversely denoised using the DDIM sampling method to obtain the implicit features of the restored turbulence-degraded image. The implicit features of the restored turbulence-degraded image are then decoded using the decoder of the automatic encoding and decoding model to obtain the restored turbulence-degraded image.

[0089] The above description is only used to illustrate the technical solution of the present invention, rather than to limit it. For ordinary professional and technical personnel in this field, the specific technical solutions recorded in the above embodiments can be modified, or some of the technical features therein can be replaced by equivalents. These modifications or replacements do not cause the essence of the corresponding technical solution to deviate from the scope of the technical solution protected by the present invention.

Claims

1. A motion information-enabled, spatiotemporally consistent turbulence-degraded image restoration method, characterized in that: The following steps are involved: Step 1: construct a training set and a test set; the training set includes multiple continuous turbulence-degraded images and corresponding multiple continuous clear images, and the test set includes multiple continuous turbulence-degraded images; Step 2: Divide the multiple continuous turbulence-degraded images and the multiple continuous clear images in the training set into multiple three-frame short-time turbulence-degraded continuous frames and corresponding multiple three-frame short-time clear continuous frames, and use the middle frame of each three-frame short-time clear continuous frame as the true value frame of the corresponding three-frame short-time turbulence-degraded continuous frame; Step 3: construct a covariant deformable convolution alignment unit; the covariant deformable convolution alignment unit includes a left branch, a right branch, and a residual convolution module; the input end of the left branch is used to receive the intermediate frame of three frames of short-term turbulence-degraded continuous frames, and the input end of the right branch is used to receive three frames of short-term turbulence-degraded continuous frames to respectively learn the turbulence degradation sampling offset and the inter-frame alignment sampling offset; the input end of the residual convolution module is connected to the output ends of the left branch and the right branch, and is used to perform pixel-by-pixel turbulence degradation feature extraction while aligning the frames; Step 4: input the middle frame of the three-frame short-time turbulence-degraded continuous frames and the three-frame short-time turbulence-degraded continuous frames into the left branch and the right branch of the covariant deformable convolution alignment unit respectively, so that the residual convolution module outputs the corresponding three-frame registered turbulence degradation features; Step 5: Obtain the features of each three-frame registered turbulence degradation feature in the implicit space and the features of each three-frame short-time turbulence degradation continuous frame in the implicit space, and record them as three-frame degradation implicit features and true implicit features respectively; Step 6: Generate random Gaussian noise ε1 of the same size as the true implicit feature, and add the random Gaussian noise ε1 to the true implicit feature to obtain the noisy implicit feature at different time steps; Step 7: Under the premise of bidirectional optical flow consistency constraint, the noisy implicit features, true implicit features, three-frame degraded implicit features and corresponding time steps at different time steps corresponding to the training set are sent to the noise prediction network for iterative training to obtain a trained noise prediction network; Step 8: Divide the multiple continuous turbulence-degraded images in the test set into multiple three-frame short-time turbulence-degraded continuous frames, and obtain the corresponding three-frame degradation implicit features according to the method of steps 4 to 5; Generate random Gaussian noise ε2 of the same size as the three-frame degraded implicit features, and input the corresponding three-frame degraded implicit features and random Gaussian noise ε2 in the test set into the trained noise prediction network. The three-frame degraded implicit features are forwarded under the bidirectional optical flow consistency constraint, so that the trained noise prediction network outputs the predicted noise; Step 9: perform reverse denoising and decoding on the predicted noise obtained in step 8 to obtain a restored turbulence-degraded image.

2. The method for restoring a spatiotemporally consistent turbulence-degraded image using motion information according to claim 1, characterized in that: In step 7, under the premise of bidirectional optical flow consistency constraint, the noisy implicit features, true implicit features, three-frame degraded implicit features and corresponding time steps at different time steps of the training set are sent to the noise prediction network for iterative training. Specifically: Step a1: The noisy implicit features, true implicit features, three-frame degraded implicit features, and corresponding time steps of the training set at different time steps are fed into the noise prediction network. The feature optical flow from the middle frame to the next frame, and the feature optical flow from the middle frame to the previous frame of each three-frame degraded implicit feature at the corresponding time step are obtained, and are recorded as forward optical flow and backward optical flow respectively. Step b1, calculate the forward optical flow offset and backward optical flow offset of each three-frame degraded implicit feature, and then obtain the consistency error of the forward optical flow offset and the corresponding backward optical flow offset, and judge whether it is less than or equal to the set threshold. If so, the noisy implicit features, true value implicit features, and three-frame degraded implicit features at the current time step are transferred to the next time step under the action of the optical flow function; otherwise, the noisy implicit features, true value implicit features, and three-frame degraded implicit features at the previous time step are transferred to the next time step under the action of the optical flow function.

3. The method for restoring a spatiotemporally consistent turbulence-degraded image using motion information according to claim 2, characterized in that: In step 3, the turbulence degradation sampling offset is expressed as follows: ； in, is the turbulence degradation sampling offset, and They represent the pixel-by-pixel turbulence degradation in the two-dimensional plane. x Direction and y The offset domain of the direction, represents the real number domain, H represents the height of the turbulence-degraded image, and W represents the width of the turbulence-degraded image; The inter-frame alignment sampling offset is expressed as follows: ； in, is the inter-frame alignment sampling offset, and Respectively and Align to In two-dimensional plane x Direction and y The offset field for the direction.

4. The method for restoring a spatiotemporally consistent turbulence-degraded image using motion information as claimed in claim 3, characterized in that: In step 5, the features of each three-frame registered turbulence degradation feature in the implicit space and the features of each three-frame short-time turbulence degradation continuous frame in the implicit space are obtained respectively as follows: Step a2: constructing an automatic encoding and decoding model, wherein the automatic encoding and decoding model is an automatic encoding and decoding model learned on a de-motion blur dataset, and includes an encoder and a decoder; In step b2, the encoder of the automatic encoding and decoding model is used to obtain the features of each three-frame registered turbulence degradation feature in the implicit space and the features of each three-frame short-time turbulence degradation continuous frame true value frame in the implicit space.

5. The method for restoring a spatiotemporally consistent turbulence-degraded image using motion information as claimed in claim 4, characterized in that: In step 7, when the noise prediction network is iteratively trained, the noise prediction network is optimized by the noise prediction loss function, the structural loss function, and the perceptual loss function.

6. The method for restoring a spatiotemporally consistent turbulence-degraded image using motion information according to claim 5, characterized in that: In step 7, the noise prediction network is optimized by the noise prediction loss function, the structural loss function, and the perceptual loss function as follows: Step a3: Use the noise prediction loss function to calculate the output of the noise prediction network during iterative training. Minimize the distance between the random Gaussian noise ε1; Step b3, calculate the clean ground truth image at different time steps : ； in, is the noise implicit feature at time step i, is the noise scheduling parameter at time step i; Step c3: Use the decoder of the automatic encoding and decoding model to decode the clean true value images at different time steps. Decode; Step d3 uses the structural loss function to calculate the clean truth images at different time steps Minimize the distance between the middle frame and the three short-term clear continuous frames; Step e3, using the perceptual loss function, calculates the clean truth images at different time steps The distance between the three short-term clear continuous frames is minimized and optimized.

7. The method for restoring a spatiotemporally consistent turbulence-degraded image using motion information according to claim 6, characterized in that: In step 7, the noise prediction loss function is the CharbonnierLoss loss function; The structural loss function is a multi-scale structural function; The perceptual loss function is a pre-trained VGG19 network.

8. The method for restoring a spatiotemporally consistent turbulence-degraded image using motion information as claimed in claim 1, characterized in that: In step 6, random Gaussian noise ε1 is added to the true implicit feature to obtain the noisy implicit features at different time steps: By using the noise scheduling parameters at different time steps, random Gaussian noise ε1 is added to the true implicit features to obtain the noisy implicit features at different time steps. The specific expressions are as follows: ； in, is the noise implicit feature at time step i, is the noise scheduling parameter at time step i, is the true value implicit feature.

9. The method for restoring a spatiotemporally consistent turbulence-degraded image using motion information according to any one of claims 1 to 8, characterized in that: In step 9, the prediction noise obtained in step 8 is reversely denoised using the DDIM sampling method.

10. The method for restoring a spatiotemporally consistent turbulence-degraded image using motion information according to claim 9, characterized in that: Step 9 is as follows: The predicted noise obtained in step 8 is reversely denoised using the DDIM sampling method to obtain the implicit features of the restored turbulence-degraded image; the implicit features of the restored turbulence-degraded image are then decoded by the decoder of the automatic encoding and decoding model to obtain the restored turbulence-degraded image.

Images