A restoration method for turbulence-degraded images based on high- and low-frequency synthesis

Abstract

The present invention discloses a restoration method for reconstructing a turbulent degraded image based on high and low frequency synthesis, including a turbulent degradation analysis module for image, a neural network structure design module, a high and low frequency information separation module, and a neural network de-turbulence module. The turbulent degradation module for image quality obtains the wavefront phase distribution of turbulence and the degraded image under turbulence by modeling an optical imaging system and setting Zernike polynomials under different turbulent intensities (D / r0) of the wavefront; the neural network structure design module adopts a double U-Net network structure; the high and low frequency information separation module separates the high frequency information and the low frequency information of the image through a high-pass filter; the neural network de-turbulence module achieves the restoration effect of removing turbulence after multiple rounds of training. The present invention can achieve a good restoration effect in the field of restoring turbulent degraded images. This method does not require wavefront detection and correction devices, simplifies the experimental system, improves reliability, and reduces costs.

Description

Technical Field

[0001] The present invention relates to the fields of optical system imaging technology, computer technology and atmospheric optics, and in particular to a restoration method for turbulence-degraded images based on high- and low-frequency synthesis and reconstruction. The method mainly achieves restoration of turbulence-degraded images by separating high- and low-frequency information of images and then combining them with neural networks, thereby improving the restoration effect. Background Art

[0002] When using a ground-based optical telescope system to observe distant targets, the presence of atmospheric turbulence causes the observed targets to be blurred, jittered, and distorted. At present, scholars can compensate for the atmosphere in various ways, such as using a wavefront sensor to detect the wavefront phase, and then using a deformable mirror to compensate the wavefront phase to obtain an observed image after turbulence phase compensation. However, this method cannot completely eliminate the impact of atmospheric turbulence on imaging quality; or image post-processing methods can be used, such as single-frame blind convolution restoration methods and multi-frame blind convolution restoration methods. The key to the blind convolution image restoration method is how to reasonably introduce prior information in the restoration process, but it requires multiple iterations and the restoration time is long.

[0003] In recent years, deep learning technology is in a stage of rapid development. It is a data-driven technology. At present, it has made great progress in target tracking, autonomous driving and other aspects, and has great prospects and uses. The biggest feature of deep learning is end-to-end, which makes it more advantageous than other methods in solving various inverse problems. At the same time, deep learning has also been widely used in the field of optical information processing, such as holographic reconstruction, super-resolution imaging, image denoising, and phase extraction. In these applications, neural networks are used to establish a nonlinear mapping relationship between input and output after training with a large amount of data, and to fit the hidden relationship between related or approximate inverse problems to the maximum extent. When atmospheric turbulence causes image degradation, it mainly causes the loss of high-frequency parts such as edge information of the image, which greatly reduces the image quality. Therefore, it is possible to consider processing the high-frequency part of the information separately through neural networks.

[0004] In view of the above problems, the present invention proposes a restoration method based on high- and low-frequency synthesis to reconstruct turbulence-degraded images. First, the Matlab simulation platform is used to simulate an optical imaging system under turbulence intensity D / r 0 =8, the order of Zernike polynomial is 4-60, and then the degraded image and the clear image are separated by high-pass filter to achieve high and low frequency information separation, and then the turbulence degraded image is restored through neural network training. When used in the actual system, expensive wavefront sensor devices are not required and real-time performance can be guaranteed. Summary of the invention

[0005] In order to overcome the above-mentioned deficiencies of the prior art, the present invention provides a restoration method of turbulence-degraded images by synthesizing and reconstructing high and low frequencies to achieve the restoration of atmospheric turbulence-degraded images.

[0006] The technical solution adopted by the present invention is: a restoration method based on high- and low-frequency synthesis to reconstruct turbulence-degraded images. The system used in the method includes: a turbulence-induced image degradation analysis module 1, a neural network structure design module 2, a high- and low-frequency information separation module 3, and a neural network turbulence removal module 4. The method includes the following steps:

[0007] Step 1: using the turbulence-induced image degradation analysis module 1 to obtain a degradation image of the optical imaging system under any turbulence intensity;

[0008] Step 2: The neural network structure design module 2 designs a dual network mode, which serves as a training network for high-frequency information and a training network for high- and low-frequency synthetic information respectively;

[0009] Among them, the first network of the dual network mode is responsible for reconstructing high-frequency information, and the second network is responsible for reconstructing clear pictures. Dual U-Net is used here;

[0010] Step 3: Use the high- and low-frequency information separation module 3 to separate the high-frequency information and low-frequency information of the image;

[0011] Among them, the high- and low-frequency information separation module 3 uses a high-pass filter to separate the high-frequency information from the low-frequency information of the image to obtain high-frequency image information;

[0012] Step 4: The neural network de-turbulence module 4 takes the high-frequency information of the turbulence-degraded image as the input of CNN1, adds its output to the degraded image to obtain a synthetic image, and takes it as the input of CNN2, and its output is a reconstructed clear image;

[0013] Among them, the neural network de-turbulence module 4 uses the high-frequency information of the degraded image as the input of CNN1, and the high-frequency information of the clear image as its label. The output of CNN1 is added and synthesized with the degraded image as the input of CNN2, and the reconstructed image is output.

[0014] Furthermore, the establishment of the turbulence-induced image degradation analysis module 1 includes: establishing a complete imaging system simulation model according to actual needs, obtaining the actual target size and focal length according to the object distance, field of view size and resolution, establishing an optical system model in Matlab, and obtaining an imaging system model of F=16, and then calculating the wavefront phase caused by turbulence according to the turbulence theory. Here, the turbulence intensity D / r is simulated. 0 =8, turbulence phase screen with Zernike polynomial order 4-60, to obtain the degradation picture under the turbulence intensity.

[0015] Furthermore, the neural network structure design module 2 includes: according to the task requirements, it is necessary to design a dual network structure, each network is a U-Net.

[0016] Furthermore, the high- and low-frequency information separation module 3 separates the high- and low-frequency information of the image information through a high-pass filter as the input and label of the network.

[0017] Among them, the high-frequency information and low-frequency information of the image can be separated by using the high-frequency and low-frequency information separation module 3, so as to improve the reconstruction effect for high-frequency image information of different frequencies.

[0018] Furthermore, the neural network de-turbulence module 4 uses the high-frequency information of the degraded image as the input of CNN 1 and the high-frequency information of the clear image as the label. The output of CNN1 is added to the degraded image and then used as the input of CNN2. The clear image is used as the label of CNN2, and several rounds of training are performed until the network converges and obtains good results.

[0019] Among them, the neural network de-turbulence module 4 needs to feed the designed neural network with the prepared data. The output of CNN1 is added to the degraded image as the input of CNN2. The output of CNN2 is the reconstructed image. The two networks each have a loss function. The two loss functions are added and then back-propagated. It is trained for several epochs until the network loss function converges and a good recovery effect is obtained.

[0020] The principle of the present invention is that: from the analysis of the optical imaging system, when observing a distant target, the presence of atmospheric turbulence will cause the target we observe to appear a certain blur, jitter and distortion. In recent years, deep learning technology is in a stage of rapid development. It is a data-driven technology. At present, it has made great progress in target tracking, automatic driving and other aspects, and has great prospects and uses. The biggest feature of deep learning is end-to-end, which makes it more advantageous than other methods in solving various inverse problems. At the same time, deep learning has also been widely used in the field of optical information processing, such as holographic reconstruction, super-resolution imaging, image denoising and phase extraction. In these applications, neural networks are used to establish a nonlinear mapping relationship between input and output after training with a large amount of data, and the hidden relationship between related or approximate inverse problems is fitted to the maximum extent. Since the low-frequency information in the image accounts for the main information, the high-frequency information is often at the edge of the image and the background, but turbulence has the most serious impact on the edge information, so the high-frequency information can be reconstructed through the neural network first, and then the high and low-frequency image information are synthesized and then reconstructed through the neural network to reconstruct a clear image. Specifically, a restoration method for reconstructing turbulence-degraded images based on high- and low-frequency synthesis is provided, and the method includes the following steps: Step 1: Model an optical imaging system in Maltlab, calculate the phase screen of turbulence of a certain intensity according to turbulence theory, and obtain a turbulence-degraded image data set; Step 2: According to task requirements, a dual network structure is designed, one network is responsible for reconstructing high-frequency information, and the other network is responsible for reconstructing the original image after synthesizing high- and low-frequency image information; Step 3: A high-pass filter is used to separate the high- and low-frequency image information of the data set to obtain a high-frequency image; Step 4: The prepared image data set is input into the network for several rounds of training until the network converges and obtains good training results.

[0021] Compared with the existing method, the present invention has the following advantages:

[0022] (1) Compared with the adaptive optical system, the present invention does not require wavefront sensors, deformable mirrors and other wavefront detection and compensation devices, which simplifies the experimental optical path and reduces the experimental cost.

[0023] (2) Compared with traditional image post-processing, the present invention does not require multiple iterations, has a fast restoration time, and has a greatly improved restoration effect.

[0024] (3) The present invention restores the high-frequency information of the image separately, thereby enhancing its noise suppression capability and improving the robustness of the method.

[0025] (4) Compared with the multi-frame restoration method, the present invention only requires one frame of turbulence-degraded image to achieve restoration, which reduces the difficulty of obtaining the experimental data set. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a schematic diagram of the turbulence-induced image degradation analysis module 1 of the present invention.

[0027] Figure 2 It is an architecture diagram of the restoration method of the present invention based on high and low frequency synthesis to reconstruct turbulence-degraded images.

[0028] Figure 3 It is a structural diagram of each sub-network of the present invention, and its network structure is a U-Net.

[0029] Figure 4 This is the restoration result achieved by the present invention. DETAILED DESCRIPTION

[0030] The specific implementation modes of the present invention are described in detail below with reference to the accompanying drawings.

[0031] like Figure 1 As shown in the figure, a telescope imaging system with F=16 is simulated on the Matlab platform, with a focal length of 100mm and a camera pixel size of 5.5μm. The presence of atmospheric turbulence will cause a phase difference in the wavefront. This causes the observed image to degrade. According to the imaging principle of the optical system: the light intensity distribution on the image plane is equal to the convolution of the light intensity distribution on the object plane and the point spread function of the system, which can be expressed by the formula: Where (r, θ) represents the point on the image plane, f(r, θ) represents the original image, and g(r, θ) represents the light intensity distribution on the image plane obtained by the camera, that is, the observed picture. represents convolution, n(r,θ) represents noise, and h(r,θ) represents the point spread of the system. The specific formula is:

[0032]

[0033] Where P(r,θ) represents the aperture function, the value inside the aperture is 1, and the value outside the aperture is 0; represents Fourier transform, represents the wavefront aberration, which can be expressed by Zernike polynomials:

[0034]

[0035] Where ai represents the coefficient of the first Zernike polynomial, z i (r,θ) represents the i-th Zernike polynomial. According to Kolmogorov turbulence theory, we can get the Zernike polynomial coefficient vector A = {a l ,a 2 ,...,a n The covariance matrix C of}, C = [c ij], then

[0036]

[0037] Where: c ij is the covariance coefficient; a i , a j are the i-th and j-th order Zernike polynomial coefficients respectively, D is the aperture size of the optical system, r 0 is the atmospheric coherence length. The coefficient matrix A can be obtained by the Karhumen-Loeve polynomial.

[0038]

[0039] Where: S is a diagonal matrix; V is the coefficient matrix of the Karhumen-Loeve polynomial; B is the phase wavefront, which is a random vector of Gaussian distribution with a mean of zero, and the variance matrix is ​​the coefficient of S. The coefficient matrix A is generated by the above derivation process, and then the atmospheric turbulence distortion wavefront phase difference that conforms to the Kolmogorov turbulence spectrum is obtained by (2). Then, the degraded image data set is obtained according to the imaging formula, and the clear image original data comes from the NWPU-RESISC45 Xi'an Jiaotong University remote sensing data set.

[0040] After that, the design Figure 2 The high- and low-frequency synthesis restoration architecture separates the high-frequency information of the degraded image and the clear image through a high-pass filter and uses them as the input and label of CNN1 respectively. Its loss function loss1 uses L1loss. Then its output is synthesized with the degraded image as the input of CNN2. Considering that the low-frequency information occupies the main part of the image information and the high-frequency information occupies a small part, the degraded image is added and synthesized as the low-frequency information and the high-frequency information. The label of CNN2 uses the original image, and the loss function loss2 still uses L1 Loss. The loss of the entire network is the sum of the two network losses, loss = 0.2*loss1+0.8*loss2. For each sub-network, it is a U-Net, such as Figure 3 As shown in the figure, each sub-network has 5 layers, 3 layers are shown here. The training hyper parameters are as follows: batchsize = 16, initial learning rate is 2e-4, learning rate decay uses cosine annealing, minimum learning rate is 1e-6, optimization algorithm uses Adam, and weight decay is 2e-5. There are 400 epochs in total, and the training time lasts about 4 hours. The recovery results after training are as follows Figure 4 .

[0041] The present invention discloses a restoration method for reconstructing turbulence-degraded images based on high- and low-frequency synthesis. There are many methods and approaches to implement the technical solution. The above is only a preferred implementation scheme of the present invention.

Claims

1. A restoration method for turbulence-degraded images based on high- and low-frequency synthesis, characterized in that: The system used in the method includes: a turbulence-induced image degradation analysis module (1), a neural network structure design module (2), a high- and low-frequency information separation module (3) and a neural network turbulence removal module (4). The method includes the following steps: Step 1: Use the turbulence-induced image degradation analysis module (1) to obtain a degraded image obtained by imaging the optical system under any turbulence intensity; according to the imaging principle of the optical system: the image plane light intensity distribution is equal to the convolution of the object plane light intensity distribution and the system point spread function, the formula is: , where represents a point on the image plane, Represents the object plane light intensity distribution of the original image, Represents the light intensity distribution on the image plane obtained by the camera, that is, the observed picture, represents convolution, represents noise, Represents the point spread of the system, and the specific formula is: （1） In the formula, represents the aperture function, where the value inside the aperture is 1 and the value outside the aperture is 0; represents Fourier transform, , represents the wavefront aberration, which is expressed by Zernike polynomials: （2） In the formula, a i represents the coefficient of the i-th Zernike polynomial, represents the i-th Zernike polynomial. According to Kolmogorov turbulence theory, the Zernike polynomial coefficient matrix A={a l , a2,...,a n }, C=[c ij ], then （3） Where: c ij is the covariance coefficient; a i , a j are the coefficients of the i-th and j-th order Zernike polynomials, D is the aperture size of the optical system, and r0 is the atmospheric coherence length; the coefficient matrix A is obtained by the Karhumen-Loeve polynomial; （4） Where: S is a diagonal matrix; V is the coefficient matrix of the Karhumen-Loeve polynomial; B is the phase wave surface, which is a random vector of Gaussian distribution with a mean of zero and a variance matrix of The coefficient matrix A is generated by the above derivation process, and then the atmospheric turbulence distortion wavefront phase difference that conforms to the Kolmogorov turbulence spectrum is obtained by formula (2), and then the degraded image data set is obtained according to the imaging formula; Step 2: The neural network structure design module (2) adopts a dual network structure design, which serves as a training network for high-frequency information and a training network for high- and low-frequency synthetic information respectively; Step 3: Using the high- and low-frequency information separation module (3) to separate the high-frequency information and the low-frequency information of the image; Step 4: The neural network de-turbulence module (4) takes the high-frequency information of the turbulence-degraded image as the input of CNN1, adds its output to the degraded image to obtain a synthetic image, and takes it as the input of CNN2, whose output is the reconstructed clear image; The establishment of the turbulence-induced image degradation analysis module (1) includes: establishing a complete optical imaging system simulation model according to actual needs, establishing an optical system model in Matlab according to the object distance, the field of view size, the camera target size and the focal length, obtaining the imaging result image under the turbulence intensity D / r0=8 and the Zernike polynomial of 4-60 order, and using the original image as the label of the corresponding degraded image; The neural network structure design module (2) includes: the high-frequency information and low-frequency information of the image need to be separated and trained accordingly, so a dual network structure needs to be designed, where each network is a U-Net, and each network has its own loss function; Among them, the high- and low-frequency information separation module (3) separates the high- and low-frequency information of the image information through a high-pass filter, which is used for network input and label; Among them, the neural network de-turbulence module (4) uses the high-frequency information of the degraded image as the input of CNN1 and the high-frequency information of the clear image as its label. Then, the output of CNN1 is added to the degraded image and synthesized as the input of CNN2. The clear image is used as the label of CNN2, and several rounds of training are performed until the network converges and obtains good results.

Images