A method for correcting image turbulence distortion

Abstract

The present invention relates to the field of image processing technology, and solves the technical problem that the existing technology cannot improve the prediction accuracy of distortion-corrected images. In particular, it relates to a method for correcting image turbulence distortion, comprising the following steps: S1, constructing a first-level correction network based on the VGG19 network, the first-level correction network including a feature fusion module and a dense connection module; S2, inputting the turbulence-distorted image into the first-level correction network, and outputting an estimated uncertainty map of the turbulence-distorted image on the first-level correction network, that is, a variance map of the turbulence-distorted image on the first-level correction network. The present invention uses a second-level correction network to focus on restoring image areas with greater uncertainty under the guidance of the uncertainty map of the first-level correction network, and finally outputs a turbulence-distorted image. Furthermore, by performing network optimization through network adversarial and loss functions, a better distortion correction effect can be obtained, thereby improving the prediction accuracy of the distortion-corrected image.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method for correcting image turbulence distortion. Background Art

[0002] Random fluctuations in the refractive index under atmospheric turbulence can cause geometric distortion and spatiotemporal blurring in images, significantly degrading the quality of images captured by long-range visible and thermal imaging systems. While adaptive optics-based techniques can compensate for turbulence effects in images, this approach typically requires large, complex, and expensive hardware. Alternatively, deep learning-based image processing methods can be used to compensate for the visual degradation caused by atmospheric turbulence, offering a low-cost and effective approach. Currently, most neural networks use fiducials, semantic maps, or image priors alone to predict the corrected image. However, turbulence effects involve a combination of geometric distortion and spatiotemporal blurring, making it impossible to reliably extract structural information such as fiducials and semantic maps. Using image priors alone cannot completely remove image distortion, resulting in limited prediction performance. Summary of the Invention

[0003] In view of the shortcomings of the existing technology, the present invention provides an image turbulence distortion correction method, which solves the technical problem that the existing technology cannot improve the distortion correction image prediction accuracy.

[0004] To solve the above technical problems, the present invention provides the following technical solution: a method for correcting image turbulence distortion, comprising the following steps:

[0005] S1. Build a first-level correction network based on the VGG19 network. The first-level correction network includes a feature fusion module and a dense connection module.

[0006] S2. Input the turbulence-distorted image into the first-level correction network, and output an estimated uncertainty map of the turbulence-distorted image on the first-level correction network, that is, a variance map of the turbulence-distorted image on the first-level correction network;

[0007] S3. Based on the first-level correction network, a second-level correction network is constructed to focus on restoring the image area with larger difference above the pixel points in the turbulence-distorted image under the guidance of the variance map of the first-level correction network, and finally output the turbulence-distorted corrected image.

[0008] Furthermore, in step S3, constructing a secondary correction network based on the primary correction network includes the following steps:

[0009] S31. Build a PyTorch development environment and select a training set containing turbulent distortion images.

[0010] S32, constructing a secondary correction network based on the uncertainty of the primary correction network, adding a feature fusion module before the shallow feature extraction part of the primary correction network, and constructing a secondary correction network with a similar structure to the primary correction network and applicable to the uncertainty of the primary correction network;

[0011] S33, determining a loss function of a secondary correction network, where the loss function of the secondary correction network includes a generator loss function and a discriminator loss function;

[0012] S34. Use Adam optimizer to optimize the loss function;

[0013] S35. Use the training set to train the secondary correction network.

[0014] Furthermore, in step S32, a secondary correction network is constructed based on the uncertainty of the primary correction network. The specific process includes the following steps:

[0015] S321, using a first-level correction network to perform S operations on the turbulence distortion image to obtain S three-channel distortion correction rough prediction images, where the size of S is set to 10;

[0016] S322, traversing the pixel points on the S distortion-corrected rough prediction images, calculating the standard deviation of the pixel points at corresponding positions in the S distortion-corrected rough prediction images, and obtaining a variance map of the turbulence-distorted image on the first-level correction network;

[0017] S323, constructing a generator based on the primary correction network and the secondary correction network;

[0018] S324, the secondary correction network splices the turbulence distortion image and its variance map on the primary correction network in the channel dimension, and performs downsampling and ReLU activation processing on the spliced ​​vector;

[0019] S325. After the first-level correction network in the generator performs shallow feature extraction of 64 channels through two convolutional layers with a convolution kernel size of 3×3, it passes through three densely connected modules with the same structure to perform local feature fusion and local residual learning between the densely connected layers, and performs global feature fusion and global residual learning between the densely connected modules. Finally, it passes through an upsampling layer and a fully connected layer with a convolution kernel size of 3×3 to output a 3-channel distortion-corrected prediction image.

[0020] S326. A 10-layer neural network is used to construct a discriminator. The real natural image and the distortion-corrected predicted image are input into the discriminator for shallow feature extraction. Then, convolution, batch normalization, and Leaky-ReLU downsampling operations are performed in sequence. Finally, a Dense block is used followed by a Sigmoid function to perform a binary classification and assign probability scores to the real natural image and the distortion-corrected predicted image.

[0021] Furthermore, in step S321, S operations are performed on the turbulence-distorted image, including shallow feature extraction of 64 channels, local feature fusion between densely connected layers, local residual learning, global feature fusion, global residual learning, and upsampling and convolution image magnification operations, and finally S 3-channel distortion-corrected rough prediction images are output.

[0022] Furthermore, in step S325, the dense connection module includes three dense connection layers, each layer consisting of a 3×3 convolution kernel and a ReLU activation function. The layers in the dense connection module are densely connected using a continuous memory mechanism. The output features of the previous module and the features obtained in each layer of the current module are fused together through concat, and then 1×1 convolution is used to reduce the number of channels of concat. Finally, local residual learning is performed on the output features of the previous module.

[0023] Furthermore, in step S33, the generator loss function L G Composed of pixel loss L1 and perceptual loss L perceptual Composition, pixel loss L1 and perceptual loss L perceptual They are:

[0024]

[0025]

[0026] In the above formula, L1 represents pixel loss, x and represent the real natural image and the distortion-corrected predicted image respectively; L perceptual represents the perceived loss, represents the output features of the jth layer obtained after inputting the image into the VGG19 network;

[0027] Generator loss function L G for:

[0028] L G =L1+λL perceptual

[0029] In the above formula, λ represents the weight, and the value of λ is 0.002.

[0030] Furthermore, in step S33, the discriminator loss function L D for:

[0031]

[0032] Where D. represents the probability that the discriminator determines that the input image is a real natural image, x and represent the true natural image and the distortion corrected predicted image respectively.

[0033] Furthermore, in step S34, the Adam optimizer is set as follows: the input image size is set to 256×256, the batch size is 5, the learning rate is set to le-4, and the secondary correction network is trained for a total of 150 steps.

[0034] By means of the above technical solution, the present invention provides a method for correcting image turbulence distortion, which has at least the following beneficial effects:

[0035] The present invention uses a secondary correction network to focus on restoring image areas with greater uncertainty in the turbulence-distorted image under the guidance of the uncertainty mapping diagram of the primary correction network, and finally outputs a turbulence-distorted image. By optimizing the network through network adversarial and loss functions, a better distortion correction effect can be obtained, thereby improving the prediction accuracy of the distortion-corrected image. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0037] Figure 1 This is a flow chart of the image turbulence distortion correction method of the present invention;

[0038] Figure 2 This is a network structure diagram of the generator of the present invention;

[0039] Figure 3 This is a network structure diagram of a densely connected module in a single-stage network in the generator of the present invention;

[0040] Figure 4 This is a network structure diagram of the discriminator of the present invention. DETAILED DESCRIPTION

[0041] To make the above-mentioned objectives, features, and advantages of the present invention more clearly understood, the present invention is further described below in detail with reference to the accompanying drawings and specific embodiments. This will enable a full understanding of how this application uses technical means to solve technical problems and achieve technical effects, and to implement the invention accordingly.

[0042] Those skilled in the art will appreciate that all or part of the steps in the above-mentioned embodiment methods can be accomplished by instructing the relevant hardware through a program. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0043] Please refer to Figures 1-4 , shows a specific implementation method of this embodiment. This embodiment takes into account the uncertainty of parameters and structures in the network model training process, which may lead to poor model training effect. Therefore, according to Monte-Carlo Dropout, the variance map of the turbulence distortion image on the first-level correction network is used to guide the second-level correction network to correct the turbulence distortion image, which can obtain better distortion correction effect.

[0044] Please refer to Figure 1 This embodiment proposes a method for correcting image turbulence distortion, which includes the following steps:

[0045] S1. Build a first-level correction network based on the VGG19 network. The first-level correction network includes a feature fusion module and a dense connection module.

[0046] S2. Input the turbulence-distorted image into the first-level correction network, and output an estimated uncertainty map of the turbulence-distorted image on the first-level correction network, that is, a variance map of the turbulence-distorted image on the first-level correction network;

[0047] S3. Based on the first-level correction network, a second-level correction network is constructed to focus on restoring the image area with larger difference above the pixel points in the turbulence-distorted image under the guidance of the variance map of the first-level correction network, and finally output the turbulence-distorted corrected image.

[0048] See also Figure 2-Figure 4 In order to clearly and completely explain step S3, this embodiment provides method steps for implementing a secondary correction network based on the primary correction network. The specific implementation steps are as follows:

[0049] S31. Build a Pytorch development environment and select a training set containing turbulent distortion images. PyTorch is an open source Python machine learning library based on Torch and used for applications such as natural language processing. At the same time, on the basis of open source, building a Pytorch development environment is a conventional technical operation in this field, and the details are not described in detail. For the selection of training sets, images with turbulent distortion characteristics can be selected from the development environment after the construction is completed, and the number of selected images can be selected according to the requirements of conventional training for the number of samples. In this embodiment, no restrictions are imposed.

[0050] S32. Construct a secondary correction network based on the uncertainty of the primary correction network, add a feature fusion module before the shallow feature extraction part of the primary correction network, and construct a secondary correction network with a similar structure to the primary correction network and applicable to the uncertainty of the primary correction network.

[0051] In order to clearly and completely explain step S32, this embodiment provides method steps for implementing the construction of a secondary correction network based on the uncertainty of the primary correction network. The specific implementation steps are as follows:

[0052] S321, using a first-level correction network to perform S operations on the turbulence distortion image to obtain S three-channel distortion correction rough prediction images, where the size of S is set to 10;

[0053] In step S321, S operations are performed on the turbulence-distorted image, including shallow feature extraction of 64 channels, local feature fusion between densely connected layers, local residual learning, global feature fusion, global residual learning, and upsampling and convolution image magnification operations. Finally, S 3-channel distortion-corrected rough prediction images are output.

[0054] S322, traversing the pixel points on the S distortion-corrected rough prediction images, calculating the standard deviation of the pixel points at corresponding positions in the S distortion-corrected rough prediction images, and obtaining a variance map of the turbulence-distorted image on the first-level correction network;

[0055] S323. Construct a generator based on the primary correction network and the secondary correction network.

[0056] Please refer to Figure 2 , is the network structure diagram of the generator in this embodiment. After the first-level correction network Net1 of the generator performs shallow feature extraction of 64 channels through two convolutional layers with a convolution kernel size of 3×3, it passes through three densely connected modules with the same structure to perform local feature fusion and local residual learning between densely connected layers, and performs global feature fusion and global residual learning between densely connected modules. Finally, it passes through an upsampling layer and a fully connected layer with a convolution kernel size of 3×3 to output a 3-channel distortion-corrected prediction image.

[0057] A feature fusion module is added before the shallow feature extraction part of the first-level correction network, thereby constructing a second-level correction network Net2 with a similar structure to the first-level correction network and applicable to the uncertainty of the first-level correction network. Through the second-level correction network Net2, the image area with larger pixel difference above the turbulence distortion image can be restored under the guidance of the variance map of the first-level correction network.

[0058] Figure 3 The network structure of the densely connected modules in the single-stage network of the generator is shown in Figure 2. Each densely connected module contains three densely connected layers, each consisting of a 3×3 convolution kernel and a ReLU activation function. The layers in each densely connected module are densely connected using a continuous memory mechanism. The output features of the previous module are fused with the features obtained from each layer of the current module through concat. 1×1 convolution is then used to reduce the number of channels in concat. Finally, local residual learning is performed on the output features of the previous module.

[0059] S324, the secondary correction network splices the turbulence distortion image and its variance map on the primary correction network in the channel dimension, and performs downsampling and ReLU activation processing on the spliced ​​vector;

[0060] S325. After the first-level correction network in the generator performs shallow feature extraction of 64 channels through two convolutional layers with a convolution kernel size of 3×3, it passes through three densely connected modules with the same structure to perform local feature fusion and local residual learning between the densely connected layers, and performs global feature fusion and global residual learning between the densely connected modules. Finally, it passes through an upsampling layer and a fully connected layer with a convolution kernel size of 3×3 to output a 3-channel distortion-corrected prediction image.

[0061] S326. A 10-layer neural network is used to construct a discriminator. The real natural image and the distortion-corrected predicted image are input into the discriminator for shallow feature extraction. Then, convolution, batch normalization, and Leaky-ReLU downsampling operations are performed in sequence. Finally, a Dense block is used followed by a Sigmoid function to perform a binary classification and assign probability scores to the real natural image and the distortion-corrected predicted image.

[0062] Please refer to Figure 4, is the network structure diagram of the discriminator of the generative adversarial network in this embodiment. The discriminator consists of a 10-layer neural network. The real natural image and the distortion-corrected predicted image are input into the discriminator for shallow feature extraction, and then convolution, batch normalization and Leaky-ReLU downsampling operations are performed in sequence. Finally, a Dense block is used followed by a Sigmoid function to perform a binary classification. The real natural image and the distortion-corrected predicted image are probability-scored, and the highest score is taken to obtain the final corrected image.

[0063] S33. Determine a loss function of the secondary correction network. The loss function of the secondary correction network includes a generator loss function and a discriminator loss function.

[0064] In step S33, the generator loss function L G Composed of pixel loss L1 and perceptual loss L perceptual Composition, pixel loss L1 and perceptual loss L perceptual They are:

[0065]

[0066]

[0067] In the above formula, L1 represents pixel loss, x and represent the real natural image and the distortion-corrected predicted image respectively; L perceptual represents the perceived loss, represents the output features of the jth layer obtained after inputting the image into the VGG19 network;

[0068] Generator loss function L G for:

[0069] L G =L1+λL perceptual

[0070] In the above formula, λ represents the weight, and the value of λ is 0.002.

[0071] In step S33, the discriminator loss function L D for:

[0072]

[0073] Where D(.) represents the probability that the discriminator determines that the input image is a real natural image, x and represent the true natural image and the distortion corrected predicted image respectively.

[0074] S34. Use the Adam optimizer to optimize the loss function, set the input image size to 256×256, the batch size to 5, the learning rate to le-4, and train the secondary correction network for a total of 150 steps.

[0075] S35. Use the training set to train the secondary correction network. Conventional training techniques are used to train the secondary correction network using the training set. In this embodiment, no limitation is imposed on the training method.

[0076] This embodiment uses a first-level correction network Net1, constructed based on feature fusion and dense connection modules, to first perform S rough predictions on the distortion-corrected image. It then traverses the pixels of these S rough predictions to estimate the uncertainty map of the turbulence-distorted image on the first-level correction network Net1, namely, the variance map of the turbulence-distorted image on the first-level correction network Net1. A second-level correction network Net2, similar in structure to the first-level correction network Net1 and based on feature fusion and dense connection modules, is then constructed. Guided by the variance map of the first-level correction network Net1, the second-level correction network Net2 focuses on restoring the turbulence-distorted image with large pixel deltas. Furthermore, when constructing a neural network using this method, network optimization is performed through network adversarial analysis and loss functions, resulting in better distortion correction results.

[0077] The above embodiments provide a detailed introduction to the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas. At the same time, for those skilled in the art, according to the ideas of the present invention, there may be changes in the specific implementation methods and application scopes. In summary, the contents of this specification should not be understood as limiting the present invention.

Claims

1. A method for correcting image turbulence distortion, characterized in that: The following steps are involved: S1. Build a first-level correction network based on the VGG19 network. The first-level correction network includes a feature fusion module and a dense connection module. S2. Input the turbulence-distorted image into the first-level correction network, and output an estimated uncertainty map of the turbulence-distorted image on the first-level correction network, that is, a variance map of the turbulence-distorted image on the first-level correction network; S3. Based on the first-level correction network, a second-level correction network is constructed to focus on restoring the image area with larger difference above the pixel points in the turbulence-distorted image under the guidance of the variance map of the first-level correction network, and finally output the turbulence-distorted corrected image; Constructing a secondary correction network based on the uncertainty of the primary correction network includes the following steps: S321, using a first-level correction network to perform S operations on the turbulence distortion image to obtain S three-channel distortion correction rough prediction images, where the size of S is set to 10; S322, traversing the pixel points on the S distortion-corrected rough prediction images, calculating the standard deviation of the pixel points at corresponding positions in the S distortion-corrected rough prediction images, and obtaining a variance map of the turbulence-distorted image on the first-level correction network; S323, constructing a generator based on the primary correction network and the secondary correction network; S324, the secondary correction network splices the turbulence distortion image and its variance map on the primary correction network in the channel dimension, and performs downsampling and ReLU activation processing on the spliced ​​vector; S325. After the first-level correction network in the generator performs shallow feature extraction of 64 channels through two convolutional layers with a convolution kernel size of 3×3, it passes through three densely connected modules with the same structure to perform local feature fusion and local residual learning between the densely connected layers, and performs global feature fusion and global residual learning between the densely connected modules. Finally, it passes through an upsampling layer and a fully connected layer with a convolution kernel size of 3×3 to output a 3-channel distortion-corrected prediction image. S326. A 10-layer neural network is used to construct a discriminator. The real natural image and the distortion-corrected predicted image are input into the discriminator for shallow feature extraction. Then, convolution, batch normalization, and Leaky-ReLU downsampling operations are performed in sequence. Finally, a Dense block is used followed by a Sigmoid function to perform a binary classification and assign probability scores to the real natural image and the distortion-corrected predicted image.

2. The image turbulence distortion correction method according to claim 1, characterized in that: In step S3, constructing a secondary correction network based on the primary correction network includes the following steps: S31. Build a PyTorch development environment and select a training set containing turbulent distortion images. S32, constructing a secondary correction network based on the uncertainty of the primary correction network, adding a feature fusion module before the shallow feature extraction part of the primary correction network, and constructing a secondary correction network with a similar structure to the primary correction network and applicable to the uncertainty of the primary correction network; S33, determining a loss function of a secondary correction network, where the loss function of the secondary correction network includes a generator loss function and a discriminator loss function; S34. Use Adam optimizer to optimize the loss function; S35. Use the training set to train the secondary correction network.

3. The image turbulence distortion correction method according to claim 1, characterized in that: In step S321, S operations are performed on the turbulence-distorted image, including shallow feature extraction of 64 channels, local feature fusion between densely connected layers, local residual learning, global feature fusion, global residual learning, and upsampling and convolution image magnification operations. Finally, S 3-channel distortion-corrected rough prediction images are output.

4. The image turbulence distortion correction method according to claim 1, characterized in that: In step S325, the dense connection module includes three dense connection layers, each consisting of a 3×3 convolution kernel and a ReLU activation function. The layers in the dense connection module are densely connected using a continuous memory mechanism. The output features of the previous module and the features obtained from each layer in the current module are fused together through concat, and then 1×1 convolution is used to reduce the number of channels for concat. Finally, local residual learning is performed on the output features of the previous module.

5. The image turbulence distortion correction method according to claim 2, characterized in that: In step S33, the generator loss function Pixel loss and perceptual loss Composition, pixel loss and perceptual loss They are: ； In the above formula, represents pixel loss, and Represent the real natural image and the distortion corrected predicted image respectively; represents the perceived loss, Indicates the first Output features of the layer; Generator loss function for: ； In the above formula, Represents weight, The value of is 0.

002.

6. The image turbulence distortion correction method according to claim 2, characterized in that: In step S33, the discriminator loss function for: ； in, represents the probability that the discriminator determines that the input image is a real natural image, and represent the true natural image and the distortion corrected predicted image respectively.

7. The image turbulence distortion correction method according to claim 2, characterized in that: In step S34, the Adam optimizer is set as follows: the input image size is set to 256×256, the batch size is 5, the learning rate is set to le-4, and the secondary correction network is trained for a total of 150 steps.

Images