Wavefront phase restoration method for extended target imaging based on defocused grating and neural network
By using defocus gratings and neural networks on a camera, feature images are directly extracted from far-field spot images, solving the problems of complexity and low efficiency of traditional phase difference methods, and achieving efficient and real-time wavefront phase recovery. Simplifies the equipment structure and improves accuracy.
Patent Information
- Application Number
- CN202210628093.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-06-06
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2042-06-06
AI Technical Summary
The traditional phase difference method is difficult to acquire focus and defocus images at the same time under a camera, resulting in complex algorithms and low operating efficiency, unable to meet real-time requirements, and multiple solutions problems, affecting the accuracy and efficiency of wavefront phase recovery.
Using a defocus grating-based design, the feature images are extracted directly from the far-field spot image through a neural network (U-Net), eliminate the extended target information, retain aberration information, and use the nonlinear fitting ability of the neural network for real-time and efficient Wavefront recovery.
It realizes the acquisition of focus and defocus images at the same time under one camera, simplifies the equipment structure, avoids the problem of synchronous sampling of multiple cameras, improves the accuracy and efficiency of wavefront recovery, and the calculation time is less than 8ms, reducing the inference time.
Smart Images

Figure CN114964524B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the technical field of phase difference method and wavefront-free phase inversion method, and in particular to a wavefront phase restoration method for extended target imaging based on a defocused grating and a neural network. Background Art
[0002] Phase inversion technology directly reconstructs the wavefront phase information based on the far-field light intensity distribution. It is insensitive to the environment and does not require a wavefront sensor. The traditional phase inversion method is to establish a relationship between the far-field information and the near-field wavefront, and use the evaluation index that characterizes the difference between the two to iteratively calculate the wavefront aberration multiple times, and then restore the wavefront phase. It has a simple structure and is easy to implement. In 2018, SW Paine and JRFienup et al. from the University of Rochester used the Inception v3 network architecture to estimate the wavefront phase only through a focal plane intensity map. The results showed that this structure can make an effective initial prediction of the wavefront aberration.
[0003] In 2019, Y. Nishizaki et al. from Osaka University in Japan used a CNN-based neural network to calculate the first 32 Zernike coefficients using only one intensity image. However, due to the multiple solutions of multiple wavefronts corresponding to the same far field, directly using a neural network to restore the Zernike coefficients or wavefronts has a certain risk of overfitting.
[0004] The phase difference method is developed from the phase recovery algorithm. By simultaneously acquiring the focal plane and the defocused plane images determined by the defocus amount, the unique wavefront phase distribution is inverted according to the light intensity information or amplitude information of the two images. The algorithm mainly establishes the objective function through the maximum probability theory in the mathematical model, loads the object information and wavefront phase distribution information into the objective function, and selects the numerical optimization algorithm to solve the wavefront phase distribution and restore the original target. The algorithm eliminates the information of the observed object in the evaluation function, so it is not only suitable for point light sources, but also can solve the aberration of extended targets. However, the traditional phase difference involves complex algorithms, slow operation efficiency, and cannot meet the high real-time requirements. In addition, it generally requires multiple cameras to perform synchronous sampling on the in-focus and defocused planes, which has a complex structure and is difficult to ensure the same frequency. If the in-focus and defocused images at the same time can be obtained under the condition of one camera, it will greatly simplify the experimental equipment and avoid the problem of asynchronous sampling of multiple cameras. With the help of the powerful nonlinear fitting ability of the neural network, the accuracy and efficiency of wavefront restoration in point target and extended target application scenarios can be improved.
[0005] Therefore, the present invention proposes a wavefront phase restoration method for extended target imaging based on a defocused grating and a neural network. Summary of the invention
[0006] The technical problem to be solved by the present invention is: while ensuring one camera, the in-focus and positive and negative defocus far-field light spots of point targets or extended targets are collected at the same time, and the unique solution of the near-field wavefront phase can be inverted for different extended targets, and the accuracy and real-time performance of wavefront restoration are guaranteed. The traditional phase difference method needs to simultaneously obtain the defocus plane image determined by the focal plane and the defocus amount, and invert the wavefront phase distribution according to the light intensity information or amplitude information of the two images, but the algorithm involved in the traditional phase difference is relatively complex, the operation efficiency is slow, and it cannot meet the real-time higher requirements. The present invention uses a defocus grating, which can be presented on the camera respectively in-focus and positive and negative defocus far-field light spots, that is, one camera collects the required light spot in-focus information and defocus information. Then feature extraction is performed on the three images, the extended target information is eliminated, and only the aberration information is retained, so that the feature image has a one-to-one correspondence with the aberration and is independent of the extended target shape. In addition, combined with neural network technology, there is no need for iterative optimization, and a more real-time and efficient wavefront restoration method is studied.
[0007] The technical solution adopted by the present invention to solve the above problems is: a wavefront phase restoration method for extended target imaging based on a defocused grating and a neural network, a data set fully sampled in the sample space is selected for the neural network (U-Net) to fit the mapping relationship between the feature image and the near-field wavefront phase, and after the network training converges, the far-field spot image is input to obtain its corresponding wavefront aberration. This mapping solution process no longer requires iterative calculations, and the wavefront wave surface diagram is directly restored, reducing the calculation time. The specific implementation steps are as follows:
[0008] Step 1: Design a wavefront sensor based on a defocus grating, including the focal length of the short focal length lens, entrance pupil radius, defocus amount, lateral distance, number of phase steps, phase step width, and height parameter design;
[0009] Step 2: Verify whether the sensor designed in step 1 can simultaneously obtain non-overlapping, non-overflowing in-focus and positive and negative defocus far-field light intensity distributions on the focal plane of the short focal length lens, control the diffraction efficiency of positive and negative defocus to be consistent, and the spot radius should not be too small, otherwise it will be difficult for the neural network to extract features for a small number of pixels;
[0010] Step 3: If step 2 can be achieved, use the defocus grating designed in step 1 to introduce random aberrations that conform to the atmospheric transmission model to the ideal parallel light, obtain the far-field light intensity distribution of focus and positive and negative defocus, and then perform feature extraction, eliminate the extended target information, and retain the aberration information. If step 2 cannot be achieved, repeat step 1 to design the sensor again until step 2 is achieved;
[0011] Step 4: Record the characteristic image and near-field wavefront data, and use the characteristic image and the near-field wavefront corresponding to it as samples and labels respectively to produce an extended target fast phase inversion dataset based on defocused grating and neural network;
[0012] Step 5: Configure the deep learning environment and build a neural network;
[0013] Step 6: Select the first 90% of the samples in the data set as the training set for the network to learn the nonlinear mapping relationship between the feature image and the near-field wave surface; the remaining 10% of the data set is used as the validation set to adjust the network hyperparameters and verify the effectiveness of the algorithm. Finally, several sets of data with the same data distribution as the training set and the validation set are prepared as test sets for unbiased estimation to measure the accuracy and real-time performance of the method.
[0014] Step 7: Change different neural networks for testing, record the test set results, and select the network with the best performance.
[0015] Wherein, the defocus grating in step 1 is essentially an off-axis Fresnel zone plate, which has symmetrically distributed ±1-order diffraction optical axes and a pair of conjugate focal lengths on the ±1-order. The defocus grating is used in close contact with a short focal length lens, and the focal length of the short focal length lens should be much smaller than the focal length of the defocus grating. The defocus grating plays a role of splitting light, and the short focal length lens plays a role of focusing light.
[0016] In step 2, the defocus grating uses a quadratic displacement function to change the structure of the ordinary grating, and adds different degrees of defocus phase shift to each diffraction order. Under the paraxial approximation, such a defocus phase shift is equivalent to applying a phase shift of a thin convex lens and a thin concave lens respectively. Adjusting the off-axis amount of the defocus grating can correspond to the lateral offset of the three images on the camera, so that the camera only receives diffracted light of the 0th order and ±1st order.
[0017] Among them, in step 2, a binary phase-type Fresnel diffraction lens needs to be designed to meet the condition that the diffraction efficiency of the ±1-order diffraction light is equal, and the step width and height directly affect the diffraction efficiency of the 0th order and ±1st order.
[0018] Among them, in step 3, it is necessary to design a feature extraction method that can eliminate the extended target information and only retain the aberration information, so as to achieve rapid restoration of the wavefront aberration for different extended targets.
[0019] The number of samples in the data set in step 3 should be at least 20,000 groups, that is, sufficient sampling should be performed in the sample space.
[0020] The output of the neural network in step 5 is a wave surface image, which is an image-to-image task, and a network such as U-Net is preferred.
[0021] The division method of the training set, validation set and test set in step 5 can be appropriately changed according to actual needs and the size of the data set.
[0022] Among them, the 1000 sets of data in step 6 are used as test sets for unbiased estimation.
[0023] The advantages of the present invention compared with the prior art are:
[0024] (1) The present invention utilizes a defocus grating and only requires one camera to realize simultaneous acquisition of in-focus and positive and negative defocus images. It has a simple structure and compact equipment, and can overcome the multi-solution phenomenon of the traditional single-frame light intensity phase inversion algorithm.
[0025] (2) The feature extraction method proposed in the present invention is independent of the imaging target and can be applied to wavefront restoration in point target and arbitrary extended target imaging scenarios.
[0026] (3) The present invention utilizes the powerful nonlinear fitting ability of the neural network, does not require iteration, and directly outputs the near-field wavefront image from the characteristic image, eliminating the step of restoring the wavefront using the Zernike coefficients, and has strong real-time performance.
[0027] (4) In the U-Net network of the present invention, the traditional convolutional layer is replaced by a more efficient depth-separable convolutional layer, which reduces the inference time while ensuring the restoration accuracy. After testing, the time for inferring one frame of wavefront is less than 8ms. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] Figure 1 A schematic diagram of the working principle of a method for wavefront phase restoration of extended target imaging based on a defocused grating and a neural network according to the present invention;
[0029] Figure 2 It is a work flow chart of the present invention;
[0030] Figure 3 This is the optical path diagram of wavefront restoration based on phase-type defocused grating;
[0031] Figure 4 This is a schematic diagram of the far-field light intensity distribution after the ideal parallel light is imaged by the defocused grating;
[0032] Figure 5 Schematic diagram of the neural network architecture used in the present invention. DETAILED DESCRIPTION
[0033] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in combination with specific implementation cases and with reference to the accompanying drawings.
[0034] Figure 1 It is a schematic diagram of the working principle of the wavefront phase restoration method for extended target imaging based on a defocused grating and a neural network. Figure 2 It is the workflow diagram of the present invention, and the specific implementation process is:
[0035] Step 1: Design a wavefront sensor based on phase-type defocus grating modulation, use the off-axis Fresnel zone plate in close contact with the short focal length lens, and place the camera in the focal plane of the short focal length lens. The parameters of the phase-type defocus grating are as follows: the central wavelength is 520nm, the defocus grating defocus amount is 3λ, the entrance pupil diameter is 12mm, the long focal length defocus grating plays a splitting role, the focal length is 11.538m, the defocus grating off-axis amount is 173.077mm, the lateral distance between the in-focus, positive and negative defocus spots is 3mm, the short focal length lens plays a focusing role, the focal length is 200mm, and the defocus grating phase step height is 0.94π (the phase distribution of adjacent arc bands is 0.06π and π alternately). Figure 3 This is the optical path diagram of wavefront restoration based on phase-type defocused grating;
[0036] Step 2: Verify whether the sensor designed in step 1 can obtain positive and negative defocus and focal plane far-field light intensity distribution on the focal plane of the short focal length lens at the same time. Determine the center position of the three light spots according to the designed lateral distance, and extract 128×128 pixel negative defocus, in-focus, and positive defocus images with the center position as the center. Figure 4 This is a schematic diagram of the far-field light intensity distribution after the ideal parallel light is imaged by the defocused grating;
[0037] Step 3: If step 2 can be achieved, random aberrations that conform to the atmospheric transmission model are introduced into the ideal parallel light, and negative defocus, in-focus, and positive defocus images of the extended target are collected according to step 1, and feature extraction is performed. If step 2 cannot be achieved, step 1 is repeated to design the sensor again until step 2 is achieved;
[0038] Step 4: The feature map and its one-to-one corresponding near-field wavefront map are used as samples and labels in the data set respectively;
[0039] Step 5: Configure the deep learning environment and build a neural network. Figure 5 The schematic diagram of the U-Net architecture of the present invention is as follows. In this neural network structure, the input is the feature image of two channels, and the output is the near-field wavefront map. U-Net is mainly composed of three parts: the downsampling channel part, which is the same as the contraction channel of the traditional convolutional neural network; the upsampling channel part is built according to the expansion channel structure of the classic U-Net. The biggest difference from the traditional U-Net is that most of the convolutional layers do not use traditional convolutional layers, but use depth-separable convolutional layers, that is, in the convolution process, different channels are convolved channel by channel, and then the feature maps are fused through point-by-point convolution. Such an architectural design reduces the number of trainable parameters in the deep model, thereby improving the training efficiency. The network outputs a near-field wavefront map of size 128*128. U-Net selects the Adam function as the gradient descent function, and the initial learning rate is 10 -3 , the Batchsize is set to 64.
[0040] Step 6: Select the first 90% of the samples in the data set as the training set for the network to learn the nonlinear mapping relationship between the far-field spot and the near-field wavefront; the remaining 10% of the data set is used as the validation set to adjust the network hyperparameters and verify the effectiveness of the algorithm. Finally, another 1,000 sets of data with the same data distribution as the training set and validation set are prepared as test sets for unbiased estimation to measure the accuracy and real-time performance of the method.
[0041] After the training makes the network converge, it only needs to input a single-frame focal plane intensity image to the network, and the network can output the near-field wavefront information corresponding to the far-field spot. This process no longer involves iterative calculations, skips the Zernike coefficient wavefront restoration link, and is completed in one step, greatly improving the calculation speed. Simulation results show that the time required for the present invention to complete a wavefront restoration can be less than 8 milliseconds.
[0042] In step 1, the defocus amount W 20 It represents the defocusing ability of the grating, which is equal to the additional optical path difference of the ±1st order diffraction wavefront at the edge of the pupil. By definition, where f g is the focal length of the off-axis Fresnel zone plate, is the distance from the center of the defocused grating pupil to the center of the Fresnel zone plate, and R is the entrance pupil radius. The lateral distance σ between the ±1st order diffraction spot and the zeroth order spot on the detection plane is given by the focal length of the lens f L , grating focal length f g And the off-axis value x0 determines:
[0043]
[0044] Therefore, the central wavelength λ and the defocus amount W are determined 20 , entrance pupil radius R, lens focal length f L After the lateral distance σ is determined, the design parameters of the off-axis Fresnel zone plate can be determined intuitively and completely, and the focal length f of the off-axis Fresnel zone plate can be calculated in turn. g and off-axis amount x0, the present invention designs the defocus grating based on the above method.
[0045] In step 3, the intensity distribution of the image captured by the camera can be written as follows:
[0046] i=o*psf
[0047] Among them, i is the image captured by the camera, o is the detected target, and psf is the point spread function of the optical system. Perform Fourier transform on the above formula to obtain the spectrum information of the image:
[0048] I=O·OTF
[0049] Among them, I is the spectrum information of the image collected by the detector, O is the spectrum information of the detected target, and OTF is the optical transfer function of the optical system. Therefore, the spectrum I of the negative defocus, in-focus, and positive defocus images collected by the camera d- ,I f ,I d+ It can be written as follows:
[0050] I d- =O·OTF d-
[0051] I f =O·OTF f
[0052] I d+ =O·OTF d+
[0053] Among them, OTF d- is the optical transfer function of the optical system under negative defocus, OTF f is the optical transfer function of the optical system in the focus state, OTF d+ is the optical transfer function of the optical system under positive defocus condition.
[0054] By dividing the negative out-of-focus image spectrum with the in-focus image spectrum, we can get:
[0055]
[0056] After inverse Fourier transform, we get feature image 1:
[0057]
[0058] By dividing the spectrum of the defocused image by the spectrum of the in-focus image, we can get:
[0059]
[0060] After inverse Fourier transform, we get feature image 2:
[0061]
[0062] This feature extraction method eliminates the extended target information and retains the aberration information. Using feature image 1 and feature image 2 as input, a neural network can be used to invert the unique near-field wavefront.
[0063] The above description is only a specific implementation of the present invention, but the protection scope of the present invention is not limited thereto. Any person familiar with the technology can understand and think of any changes or substitutions within the technical scope disclosed by the present invention, which should be included in the scope of the present invention.
Claims
1. A method for wavefront phase restoration of extended target imaging based on defocused grating and neural network, characterized in that: This is accomplished by following these steps: Step 1: Design a wavefront sensor based on a defocus grating, including the focal length of the short focal length lens, entrance pupil radius, defocus amount, lateral distance, number of phase steps, phase step width, and height parameter design; Step 2: Verify whether the sensor designed in step 1 can simultaneously obtain non-overlapping, non-overflowing in-focus and positive and negative defocus far-field light intensity distributions on the focal plane of the short focal length lens, control the diffraction efficiency of positive and negative defocus to be consistent, and the spot radius should not be too small, otherwise it will be difficult for the neural network to extract features for a small number of pixels; Step 3: If step 2 can be achieved, use the defocus grating designed in step 1 to introduce random aberrations that conform to the atmospheric transmission model into the ideal parallel light, obtain the far-field light intensity distribution of focus and positive and negative defocus, perform feature extraction, eliminate the extended target information, and retain the aberration information; if step 2 cannot be achieved, repeat step 1 to design the sensor again until step 2 is achieved; Step 4: Record the characteristic image and near-field wavefront data, and use the characteristic image and the near-field wavefront corresponding to it as samples and labels to produce an extended target fast phase inversion dataset based on defocused grating and neural network; Step 5: Configure the deep learning environment and build a neural network; Step 6: Select the first 90% of the samples in the data set as the training set for the network to learn the nonlinear mapping relationship between the feature image and the near-field wave surface; the remaining 10% of the data set is used as the validation set to adjust the network hyperparameters and verify the effectiveness of the algorithm. Finally, a number of data with the same data distribution as the training set and validation set are prepared as the test set for unbiased estimation to measure the accuracy and real-time performance of the method. Step 7: Change different neural networks for testing, record the test set results, and select the network with the best performance; The defocused grating in step 1 is essentially an off-axis Fresnel zone plate, which has symmetrically distributed The first-order diffraction axis and There is a pair of conjugate focal lengths on the first level. The defocused grating is used in close contact with the short focal length lens. The focal length of the short focal length lens should be much smaller than the focal length of the defocused grating. The defocused grating plays the role of light splitting, and the short focal length lens plays the role of light focusing. In step 2, the defocus grating uses a quadratic displacement function to change the structure of the ordinary grating, and adds different degrees of defocus phase shift to each diffraction order. Under the paraxial approximation, such a defocus phase shift is equivalent to applying a phase shift of a thin convex lens and a thin concave lens respectively. Adjusting the off-axis amount of the defocus grating can correspond to the lateral offset of the three images on the camera, so that the camera only receives the 0th, 1st order diffracted light; In step 2, a binary phase-type Fresnel diffraction lens needs to be designed to meet the condition that the diffraction efficiency of ±1-order diffraction light is equal. The step width and height directly affect the 0th and 1st order diffraction lights. Diffraction efficiency of 1st order.
2. The method for wavefront phase restoration of extended target imaging based on defocused grating and neural network according to claim 1, characterized in that: In step 3, a feature extraction method that can eliminate the extended target information and only retain the aberration information needs to be designed, so that the wavefront aberration can be quickly restored for different extended targets.
3. The method for wavefront phase restoration of extended target imaging based on defocused grating and neural network according to claim 1, characterized in that: The number of samples in the data set in step 3 should be at least 20,000 groups, that is, sufficient sampling should be performed in the sample space.
4. The method for wavefront phase restoration of extended target imaging based on defocused grating and neural network according to claim 1, characterized in that: The output of the neural network in step 5 is a wave surface image, which is an image-to-image task, and a U-Net network is selected.
5. The method for wavefront phase restoration of extended target imaging based on defocused grating and neural network according to claim 1, characterized in that: The division method of the training set, validation set and test set in step 5 can be appropriately changed according to actual needs and the size of the data set.
6. The method for wavefront phase restoration of extended target imaging based on defocused grating and neural network according to claim 1, characterized in that: In step 6, the 1000 sets of data are used as test sets for unbiased estimation.
Citation Information
Patent Citations
Single-frame focal plane light intensity image deep learning phase difference method based on grating modulation
CN112179504A
Extended target super-resolution wavefront restoration method
CN115294422A
Transmission device and receiving device
JP2023079977A
