Atmospheric turbulence wavefront phase factor estimation method in long-distance imaging

Abstract

The invention provides an atmospheric turbulence wavefront phase factor estimation method in long-distance imaging, and the method comprises the steps: directly constructing a mapping relation model between an optical system detection image and a wavefront phase factor, and learning the mapping relation through a trained deep neural network. And end-to-end output of a real part and an imaginary part of the wavefront phase factor corresponding to each frame of detection image is realized, so that estimation of the atmospheric degradation wavefront phase factor on the transmission path is completed. According to the method, the application scope and research thinking of the turbulence medium light propagation field are expanded, it is guaranteed that the output of the dual-network model is optimally fitted in data space distribution, and it is also guaranteed that the predicted phase factors conform to Kolmogorov turbulence statistics, that is, the scheme is that data driving is combined with physical constraint.

Description

Technical Field

[0001] The invention relates to the field of photoelectric detection and imaging technology, and in particular to a wavefront phase factor estimation method. Background Art

[0002] The performance of optical systems in long-distance imaging scenarios such as airborne slant range or shipborne level imaging can usually be described by parameters such as signal-to-noise ratio, resolution, and contrast, or can be comprehensively characterized by the system's effective range. Among them, imaging resolution is one of the main parameters that indicate the performance of the system. After the detector model is determined, atmospheric turbulence often becomes a key factor affecting the system resolution. The imaging quality is largely limited by the distorted wavefront phase introduced by atmospheric turbulence. Fried (Fried D L.Optical resolution through a randomly inhomogeneous medium for very long and very short exposures[J].JOSA, 1966, 56(10): 1372-1379.) et al. have conducted many in-depth studies on the resolution of optical systems under long-exposure and short-exposure atmospheric turbulence, established a basic imaging physical model, explored the statistical characteristics of the turbulent phase screen, and pointed out the basic research ideas and research directions for removing the influence of atmospheric turbulence after acquiring long-distance detection images.

[0003] Traditional atmospheric turbulence degradation image correction technologies mainly include phase recovery, phase extraction, blind deconvolution, wavefront deconvolution, phase difference, point spread function estimation, etc. Based on the imaging model and Fourier optical principles, the image de-turbulence technology is modeled as an inverse problem of the imaging process, and iterative optimization is performed by adding physical constraints, or directly solving a clear image or a diffraction-limited point spread function, and then performing non-blind deconvolution processing; or first directly estimating the atmospheric distortion wavefront phase from the detected degraded image, and then combining the correspondence between the point spread function and the wavefront phase for non-blind deconvolution processing. It can be seen that the distorted wavefront phase, an intermediate parameter, is a key physical quantity in the process of improving the imaging quality and resolution of long-distance scenes. Therefore, it can be said that the accuracy of wavefront phase estimation directly affects the improvement of the working distance of the long-distance imaging optical system.

[0004] At present, the unfavorable factors or specific manifestations that affect the ability to estimate the atmospheric degradation wavefront phase may include: the arbitrariness of the selection of Zernike mode orders, the limited range of phase peak-valley value estimation, the need for manual intervention for iteration stop, poor adaptability to different turbulence intensities, poor adaptability to imaging background differences, etc. In recent years, with the development of technologies such as deep learning and high-performance computing, artificial intelligence-based methods have been rapidly promoted to estimate the distorted wavefront phase. Deep neural networks can directly output the Zernike mode coefficients of the phase or the two-dimensional distribution of the wavefront phase. This type of method generally does not require a forward analytical model from the wavefront phase to the detection image, and can get rid of the constraints of the physical imaging model to a certain extent. However, at the same time, this type of technology is generally still based on the Zernike mode representation of the wavefront phase. The research focus is on how to improve the prediction accuracy of the Zernike coefficients. For example, the patent number CN202010075311.X, named "A Shack-Hartmann wavefront detector based on deep learning", uses a deep neural network to directly predict the Zernike polynomial coefficients of the degraded wavefront based on the point array image and then reconstruct the turbulent wavefront.

[0005] For actual shipborne horizontal or airborne oblique long-distance optoelectronic detection and imaging scenarios, there will be many limitations in specific imaging environments such as small field of view, low light flux or changing turbulence. Directly estimating the wavefront phase by deep neural networks will inevitably introduce two parts of errors: one is the model error, which comes from the modeling and approximation ability of the selected network architecture for the mapping relationship between phase and observed image; the other is the truncation error, which comes from the number of manually selected Zernike modes representing the phase. In addition, in order to improve the scene adaptability of the deviation phase coefficient estimation, additional optical defocus components or presets are generally required by current deep learning methods. In view of the defects of the above methods or the unfavorable factors of dynamically changing scenes, it is urgent to further expand research methods and research ideas, or improve traditional wavefront phase estimation technology, or construct a new deep neural network mapping model to further improve the wavefront phase prediction accuracy in long-distance imaging. Summary of the invention

[0006] In order to overcome the shortcomings of the prior art, the present invention provides a method for estimating the atmospheric turbulence wavefront phase factor in long-distance imaging. It is a key physical quantity. However, the wavefront phase factor is more often used in theoretical and applied research. Rather than optical phase Based on this, the present invention considers the traditional method of wavefront phase The research on extraction, prediction and estimation of wavefront phase factor The estimation and statistical characteristics of the wavefront phase factor are studied to explore solutions for directly estimating the wavefront phase factor from the detection image obtained from the optical system.

[0007] To overcome the existing long-distance imaging scenarios, the distorted optical wavefront phase To overcome the lack of predictive power of this random quantity, the present invention proposes an atmospheric turbulence wavefront phase factor for long-range imaging. Estimation method. This method directly constructs a mapping relationship model between the optical system detection image and the wavefront phase factor, and uses a trained deep neural network to learn this mapping relationship, achieving end-to-end output of the real and imaginary parts of the wavefront phase factor corresponding to each frame of the detection image, thereby completing the estimation of the atmospheric degradation wavefront phase factor on the transmission path.

[0008] The technical solution adopted by the present invention to solve the technical problem specifically includes the following steps:

[0009] Step 1: Construct a pair of GANs (PGAN) model. The network model is the mapping relationship between the detection image and the real part of the wavefront phase factor and the imaginary part of the wavefront phase factor. The pair of GANs P R and P I They are used to estimate the real part of the wavefront phase factor and imaginary part The dual network parameters are transferred alternately during training and are independent during prediction.

[0010] Step 2: Construct a paired data set of "wavefront phase factor-degraded observation image". The clear scene image in the paired data set is the measured data collected in clear weather, and the degraded observation image in the paired data set is the phase screen based on the power spectrum inversion method. Simulation construction; Power spectral density function PSD n (k) Using the atmospheric refractive index structure parameters When simulating a horizontal scene Take the typical value of the ground plane, when simulating the inclined scene Adopting the atmospheric structure parameter model that varies with altitude Wavefront phase factor in paired dataset The real and imaginary parts of Build;

[0011] Step 3: Iteratively train and test the dual generative adversarial network designed in step 1 on the paired dataset. The generator network GR of the domain and The domain generator network GI and the corresponding The discriminator network DR of the domain and The discriminator networks DI of the two domains are combined for iterative optimization. The specific steps are as follows:

[0012] The dual generative adversarial network P R and P I At the same time, the loss function L adds the wavefront phase factor The constraint relationship will Domain and The domain and the corresponding degraded observation image are paired and synthesized as the training set input, and the neural network method is used for training until the training set and the test set converge to obtain a trained dual generative adversarial network model;

[0013] Step 4: Verify the dual generative adversarial network model trained in step 3. First, use a deep neural network to verify it. Then, verify the statistical characteristics of atmospheric turbulence of the generated wavefront phase factor, referring to the phase structure function of the turbulence phase screen simulation. Test method, using the phase factor structure function that conforms to Kolmogorov statistics As a theoretical value, verify the wavefront phase factor estimated by the theoretical value and the model The degree of conformity of the curve, such as Figure 3 As shown, if the conformity is poor, the network structure in step 1 is re-optimized and designed. The degree of conformity can be determined based on the degree of closeness of the two curves judged by the human eye.

[0014] At this point, the estimation of the atmospheric turbulence wavefront phase factor in long-distance imaging has been completed. When used, the collected single-frame observation images are input into the trained The generator network GR of the domain and The generator network GI of the domain outputs the real and imaginary parts of the wavefront phase factor corresponding to the frame image.

[0015] In the process of constructing the dual generative adversarial network PGAN model in step 1, the generator networks GR and GI both adopt the Unet or ResNet architecture, and the specific design of the loss function L is as follows:

[0016]

[0017] Where: They represent the adversarial loss functions of the PGAN discriminant network DR and DI of the model respectively;

[0018] They represent the content loss functions of the generated networks GR and GI respectively; the design of the third item represents the addition of the constraint relationship between the real part and the imaginary part determined by the definition of the wavefront phase factor; γ1 and γ2 are the regularization parameters of the balance weights.

[0019] In the process of constructing the "wavefront phase factor-degraded observation image" paired data set in step 2, the power spectral density function PSD during phase screen simulation n (k) Using structural parameters including atmospheric refractive index The modified Von Karman spectral model,

[0020]

[0021] Where: k is the spatial frequency, k is the internal scale of turbulence m =5.92 / l0; turbulence outer scale k0 = 2π / L0; l0 is the turbulence inner scale, L0 is the turbulence outer scale. For long-distance horizontal imaging scenes, the atmospheric refractive index structure parameter Take the typical value of ground plane C0=1.7×10 -14 m -2 / 3 ; For the oblique imaging scene, Adopting the parameter model proposed by the International Telecommunication Union that varies with altitude

[0022]

[0023] Where: h is the altitude (m); v RMS is the root mean square wind speed on the vertical path, and e is a natural constant.

[0024] When the dual generative adversarial network PGAN is jointly trained in step 3, the two networks P R and P I Alternately, passing the estimate of the network output output by each iteration and When training alternately, the network P R The loss function degenerates into the following expression:

[0025]

[0026] Similarly, the network's P I The loss function degenerates into the following expression:

[0027]

[0028] When the wavefront phase factor generated by the dual network model in step 4 is verified by the statistical characteristics of atmospheric turbulence, the theoretical value of the phase factor structure function that conforms to Kolmogorov statistics is The expression is:

[0029]

[0030] Where: is the theoretical phase structure function that conforms to Kolmogorov statistics, and r is the spatial interval vector.

[0031] An electronic device comprises: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the method as described above.

[0032] A computer-readable storage medium stores program codes, and the program codes can be called by a processor to execute the method as described above.

[0033] The beneficial effects of the present invention are:

[0034] (1) An alternative scheme is designed to extract the wavefront information of atmospheric turbulence distortion carried in degraded observation images obtained by long-distance imaging. This scheme estimates the real and imaginary parts of the wavefront phase factor from the observation image, rather than the wavefront phase itself. This is because the wavefront phase factor is more directly used in theoretical and applied research. right Direct predictions are made, and theoretical analysis and simulations are performed on its statistical characteristics, which expands the application scope and research ideas in the field of light propagation in turbulent media.

[0035] (2) Due to the wavefront phase factor The real and imaginary domains of are bounded in the range of [-1,1], which is very suitable for treating it as two-dimensional spatial distribution data, so that it can be input into deep neural network modeling and learn prediction models. The model can be applied to turbulent environments of different intensities. Affected by the intensity of turbulence, its value range changes dynamically and randomly, which is not suitable as a direct input to a deep network and cannot be directly predicted. It requires indirect prediction methods such as Zernike mode characterization, but this does not avoid the error in the selection of the mode order.

[0036] (3) The statistical characteristics verification method based on the theoretical value of the wavefront phase factor structure function ensures that the output of the dual network model is not only the best fit in the data space distribution, but also that the predicted phase factor is consistent with the Kolmogorov turbulence statistics, that is, the scheme is data-driven combined with physical constraints. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 The phase factor real part prediction network P of the present invention R right A schematic diagram of an estimated frame, Figure 1 (a) is a schematic diagram of the real part prediction value. Figure 1 (b) is a schematic diagram of the real part reference value.

[0038] Figure 2 is the phase factor imaginary part prediction network P of the present invention I right A schematic diagram of an estimated frame, Figure 2 (a) is a schematic diagram of the imaginary part prediction value. Figure 2 (b) is a schematic diagram of the imaginary part reference value.

[0039] Figure 3 Schematic diagram of the degree of conformity of the phase factor structure function output by the validation set with the theoretical value DETAILED DESCRIPTION

[0040] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0041] The hardware environment used to implement the solution in the embodiment: CPU, Core i5-13600KF, 14 cores, 20 threads, 3.5GHz, 24M cache; memory, Micron 16G, 3200DDR4; GPU, GALAXY RTX4070ti S, 16G graphics card. The software environment used to implement the solution in the embodiment: integrated development environment, PyCharm 2024.2.3; operating system, Microsoft Windows 11.

[0042] A method for estimating atmospheric turbulence wavefront phase factor in long-distance imaging, characterized in that it specifically comprises the following steps:

[0043] Step 1: Construct a dual generative adversarial network (PGAN) model for the mapping relationship between the detection image and the real and imaginary parts of the wavefront phase factor. R ,P I They are used to directly extract the real part of the wavefront phase factor from the observed degraded image simultaneously and imaginary part During training, the dual network parameters are transferred alternately to each other, and they work independently during prediction.

[0044] In the dual generative adversarial network PGAN model, P R and P IThe generator networks GR and GI both use the ResNet101 network architecture. The first layer is a 7×7 convolution layer; followed by 4 stages, each of which contains several residual blocks; followed by a global average pooling layer and a fully connected layer. The function of the fully connected layer is to expand the output of the pooling layer into a vector and map it to the dimension of the number of categories. Each residual block of ResNet101 consists of two 3*3 convolution layers, each of which is followed by batch normalization and ReLU activation functions. There are also batch normalization and ReLU activation functions between residual blocks, but no convolution layers. The first residual block of each stage uses a 1*1 convolution layer to convert the number of input channels to the number of output channels for addition operations with subsequent residual blocks.

[0045] P R and P I The discriminator network DR and DI architecture adopts the conventional structure design of the discriminator in the generative adversarial network.

[0046] The loss function of the dual generative adversarial network PGAN model is specifically designed as follows:

[0047]

[0048] Where: They represent the adversarial loss functions of the PGAN discriminant network DR and DI of the model respectively; They represent the content loss functions of the generated networks GR and GI respectively; the design of the third item represents the addition of the constraint relationship between the real part and the imaginary part determined by the wavefront phase factor definition; γ1 and γ2 are regularization parameters for balancing the weights of the three parts.

[0049] P in the dual network R For example, the adversarial loss of the discriminant network DR is defined using the Wasserstein distance with gradient penalty.

[0050]

[0051] In the formula, It represents the estimation of the real part of the atmospheric degradation wavefront phase factor output by the generator network each time, and i(x) is the degraded observation image input to the network. The content loss of the generator network GR only uses pixel content loss

[0052]

[0053] Where o(x) represents the clear real scene image corresponding to the atmospheric degradation observation image. IThe loss function of the network is defined as P R The two networks are similar. The two networks are trained simultaneously, which improves the efficiency and makes full use of the mathematical constraints.

[0054] Step 2: Construct a paired data set of "wavefront phase factor-degraded observation image". The clear scene image is approximated by the measured data collected when the atmospheric influence is weak. In the embodiment, a total of 10 scenes including road surface, grass, sand, forest and typical targets are collected. The phase screen of the degraded observation image in the data set based on the power spectrum inversion method Simulation construction, turbulence degradation imaging model is i(x) = o(x) × h(x) + n(x), h(x) is the degradation point spread function, which is related to the phase The corresponding relationship is n(x) is random noise; here × represents a two-dimensional convolution operation.

[0055] Power spectral density function PSD during phase screen simulation n (k) Using the atmospheric refractive index structure parameters The modified Von Karman spectral model,

[0056]

[0057] Where: k m =5.92 / l0; k0 = 2π / L0; l0 and L0 are the inner and outer scales of turbulence respectively. For long-distance horizontal imaging scenes, the atmospheric refractive index structure parameter Take the typical value near the ground plane C0=1.7*10 -14 m -2 / 3 ; For the oblique imaging scene, Adopting the parameter model proposed by the International Telecommunication Union that varies with altitude

[0058]

[0059] Where: h is the height above the ground; v RMS is the root mean square wind speed on the vertical path, and its value is related to the near-ground wind speed v g Related, here we use

[0060] According to the above simulation method, 2K phase screen data are simulated for horizontal and oblique scenes respectively, and added to 10 clear images of the scenes respectively to construct a paired data set with a total of 40,000 degraded images and 10 clear images, which are then divided into training set, test set, and validation set according to conventional processing methods.

[0061] Step 3: On the dataset constructed in step 2, the dual generative adversarial network P designed in step 1 is used.R and P I Iterate and alternate training and testing until convergence.

[0062] Will predict Domain and The generator network GR, GI and the discriminator network DR, DI of the domain data are combined for iterative optimization, which is equivalent to combining the two networks P R and P I Training is performed simultaneously and alternately, and the wavefront phase factor is added to the loss function The constraint relationship of each iteration step is thus transmitted through this added constraint term. and Will Domain and The domain and the corresponding degraded observation images are paired and synthesized as the training set input until the training set and the test set converge.

[0063] When training alternately, the network P R The loss function degenerates into the following expression:

[0064]

[0065] Similarly, network P I The loss function degenerates into the following expression:

[0066]

[0067] In the embodiment, parameter γ1 is set to 300, parameter γ2 is set to 0.05, and the three losses have converged after about 600 alternating iterations of training.

[0068] Step 4: Generate adversarial network P trained in step 3 R and P I In addition to the verification method conventionally used in the generative adversarial network model, the embodiment mainly verifies the statistical characteristics of atmospheric turbulence on the generated wavefront phase factor. Test method, using the phase factor structure function that conforms to Kolmogorov statistics As a theoretical value, verify its compatibility with the wavefront phase factor estimated by the model The degree of conformity of the curve is poor, and if the conformity is poor, the network structure in step 1 needs to be re-optimized. The embodiment assumes that the phase factor of the atmospheric turbulence wave front is statistically homogeneous, and thus defines the theoretical value of the phase factor structure function that conforms to Kolmogorov statistics The expression is:

[0069]

[0070] Where: is the theoretical phase structure function that conforms to Kolmogorov statistics, r = x'-x is the space interval vector, r0 is the atmospheric coherence length defined by Fried's constant. Phase factor structure function The specific derivation process is as follows:

[0071] First, derive the phase factor covariance function Assuming the wavefront phase factor is statistically homogeneous, it is defined by the random variable covariance, where <·> represents the ensemble average, is the phase space interval variation. If p(α) is the probability density function of α(r), for a certain space interval vector r, based on The statistical characteristics of α(r) are zero-mean Gaussian random variables, that is,

[0072]

[0073] In the formula, σ 2 is the variance of α(r),

[0074] So, by The definition of Substituting the expression for p(α) into this The definition of , combined with the duality of p(α) and Euler's formula, after calculating this integral, the expression of the phase factor covariance function can be obtained as follows:

[0075]

[0076] At this point, the relationship between the random variable structure function and the covariance function is The phase factor structure function that conforms to Kolmogorov statistics can be obtained The theoretical expression of

[0077]

[0078] The phase factor structure function of the dual network model prediction output needs to be consistent with this Perform compliance test.

[0079] So far, the embodiment has completed the estimation of the atmospheric turbulence wave front phase factor in long-distance imaging. The above description is only a part of the embodiments of the present technology, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should belong to the scope of protection of the present invention.

Claims

1. A method for estimating the atmospheric turbulence wavefront phase factor in long-range imaging, characterized in that The steps include: Step 1: Construct a dual generative adversarial network model. The network model is the mapping relationship between the detection image and the real part of the wavefront phase factor and the imaginary part of the wavefront phase factor. The dual generative adversarial network P R and P I They are used to estimate the real part of the wavefront phase factor and imaginary part The dual network parameters are transferred alternately during training and are independent during prediction. Step 2: Construct a paired data set of "wavefront phase factor-degraded observation image". The clear scene image in the paired data set is the measured data collected in clear weather, and the degraded observation image in the paired data set is the phase screen based on the power spectrum inversion method. Simulation construction; Power spectral density function PSD n (k) Using the atmospheric refractive index structure parameters When simulating a horizontal scene Take the typical value of the ground plane, when simulating the inclined scene Adopting the atmospheric structure parameter model that varies with altitude Wavefront phase factor in paired data set The real and imaginary parts of Build; Step 3: Iteratively train and test the dual generative adversarial network designed in step 1 on the paired dataset. The generator network GR of the domain and The domain generator network GI and the corresponding The discriminator network DR of the domain and The discriminator networks DI of the two domains are combined for iterative optimization. The specific steps are as follows: The dual generative adversarial network P R and P I At the same time, the loss function L adds the wavefront phase factor The constraint relationship will Domain and The domain and the corresponding degraded observation image are paired and synthesized as the training set input, and the neural network method is used for training until the training set and the test set converge to obtain a trained dual generative adversarial network model; Step 4: Verify the dual generative adversarial network model trained in step 3. First, use a deep neural network to verify it. Then, verify the statistical characteristics of atmospheric turbulence of the generated wavefront phase factor, referring to the phase structure function of the turbulence phase screen simulation. Test method, using the phase factor structure function that conforms to Kolmogorov statistics As a theoretical value, verify the wavefront phase factor estimated by the theoretical value and the model The degree of conformity of the curves. If the conformity is poor, the network structure in step 1 is re-optimized and designed. The degree of conformity can be determined based on the subjective judgment of the human eye on the closeness of the two curves. At this point, the estimation of the atmospheric turbulence wavefront phase factor in long-distance imaging has been completed. When used, the collected single-frame observation images are input into the trained The generator network GR of the domain and The generator network GI of the domain outputs the real and imaginary parts of the wavefront phase factor corresponding to the frame image.

2. The method for estimating the atmospheric turbulence wavefront phase factor in long-distance imaging according to claim 1, characterized in that: In the process of constructing the dual generative adversarial network PGAN model in step 1, the generator networks GR and GI both adopt the Unet or ResNet architecture, and the specific design of the loss function L is as follows: Where: They represent the adversarial loss functions of the PGAN discriminant network DR and DI of the model respectively; They represent the content loss functions of the generated networks GR and GI respectively; the design of the third item represents the addition of the constraint relationship between the real part and the imaginary part determined by the definition of the wavefront phase factor; γ1 and γ2 are the regularization parameters of the balance weights.

3. The method for estimating atmospheric turbulence wavefront phase factor in long-distance imaging according to claim 1, characterized in that: In the process of constructing the "wavefront phase factor-degraded observation image" paired data set in step 2, the power spectral density function PSD during phase screen simulation n (k) Using structural parameters including atmospheric refractive index The modified Von Karman spectral model, Where: k is the spatial frequency, k is the internal scale of turbulence m =5.92 / l0; turbulence outer scale k0 = 2π / L0; l0 is the turbulence inner scale, L0 is the turbulence outer scale. For long-distance horizontal imaging scenes, the atmospheric refractive index structure parameter Take the typical value of ground plane C0=1.7×10 -14 m -2 / 3 ; For the oblique imaging scene, Adopting the parameter model proposed by the International Telecommunication Union that varies with altitude Where: h is the altitude (m); v RMS is the root mean square wind speed on the vertical path, and e is a natural constant.

4. The method for estimating the atmospheric turbulence wavefront phase factor in long-distance imaging according to claim 1, characterized in that: When the dual generative adversarial network PGAN is jointly trained in step 3, the two networks P R and P I Alternately, passing the estimate of the network output output by each iteration and When training alternately, the network P R The loss function degenerates into the following expression: Similarly, the network's P I The loss function degenerates into the following expression:

5. The method for estimating atmospheric turbulence wavefront phase factor in long-distance imaging according to claim 1, characterized in that: When the wavefront phase factor generated by the dual network model in step 4 is verified by the statistical characteristics of atmospheric turbulence, the theoretical value of the phase factor structure function that conforms to Kolmogorov statistics is The expression is: Where: is the theoretical phase structure function that conforms to Kolmogorov statistics, and r is the spatial interval vector.

6. An electronic device, characterized in that: include: one or more processors; Memory; One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the method according to any one of claims 1 to 5.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, and the program codes can be called by a processor to execute the method according to any one of claims 1 to 5.

Images