Design and optimization method of dynamic phase mask based on deep learning

Abstract

The invention provides a dynamic phase mask design and optimization method based on deep learning. The method comprises the following steps: step 1, obtaining a PSF set; step 2, acquiring a coded image; and step 3, carrying out synchronous optimization on the dynamic phase mask and the image reconstruction network. According to the method, the time-averaged dynamic phase mask is designed and optimized through deep learning end-to-end, the dynamic phase mask has higher expressive force than a traditional static mask in a computational imaging system, and the performance of depth estimation and extended depth-of-field imaging in computational imaging can be enhanced.

Description

Technical Field

[0001] The present invention belongs to the technical field of computational imaging, and in particular relates to a design and optimization method of a dynamic phase mask based on deep learning. Background Art

[0002] Computational imaging is a discipline that uses computational methods to enhance or extract image information. It combines technologies from the fields of optics, signal processing, and computer vision, and can break through the limitations of traditional optical imaging to achieve more efficient and flexible image processing and analysis. In the field of computational imaging, phase mask technology has attracted widespread attention because it can extend the depth of field (DOF) and improve the accuracy of depth estimation. Compared with the static phase mask used in traditional computational imaging engineering, the dynamic phase mask is considered to have stronger expressiveness. This is because the set of PSFs output by the dynamic phase mask is strictly non-convex, so the design space of the PSF generated by the dynamic phase mask is fundamentally more suitable for computational imaging engineering than the static mask. However, in the process of designing a dynamic phase mask, the phase accuracy of the modulation is limited due to the large number of quantization processes involved, which creates a demand for high-quality time-averaged dynamic phase masks.

[0003] A spatial light modulator (SLM) is an optical device that can modulate the spatial distribution characteristics of light waves. SLM controls the light field by changing the phase, amplitude or polarization of the incident light, and is therefore widely used in holographic display, optical information processing, adaptive optics, optical communications and other fields. In recent years, SLM based on microelectromechanical systems (MEMS) has achieved rapid development. This is an optical device that uses micromechanical structures to achieve light wave modulation. MEMS SLM adjusts the phase, amplitude or direction of light waves through micron-level mechanical movement to achieve precise control of the light field. It has the characteristics of high response speed, low power consumption and integration. Because it has very small inertia and a faster response speed than traditional liquid crystal SLM, it is very suitable for high-speed dynamic modulation. For the design of dynamic phase masks, the effective accuracy can be improved by averaging the intensity of multiple frames to overcome the impact of quantization.

[0004] Traditional optical depth estimation methods use sensors and optical devices to encode and recover depth information. This method takes advantage of the depth-dependent blur caused by aperture. These methods compare blurs of different ranges with the expected blur caused by an aperture focused at a fixed distance, retaining more high-frequency information about the scene through aperture encoding to obtain more depth information. Similar to depth estimation, static phase masks are widely used to produce invariant customized PSFs, thereby achieving better depth imaging effects. However, these traditional optically driven methods have been gradually replaced by deep learning neural networks that allow joint optimization of optical elements and neural reconstruction networks in terms of performance. Deep learning techniques can be used to jointly train optical parameters and image reconstruction network parameters by encoding images to retain additional information about the scene, and then using neural networks to reconstruct the image. This process uses a differentiable model for forward propagation of light, and uses backpropagation to allow the neural network to simultaneously participate in updating system parameters. The effectiveness of this method has been demonstrated in extended depth of field, depth estimation, and holography. Although these previous methods have successfully improved imaging performance, they generally focus on designing and optimizing simple optical structures. Summary of the invention

[0005] The purpose of the present invention is to provide a design and optimization method of a dynamic phase mask based on deep learning, aiming to design and optimize a time-averaged dynamic phase mask end-to-end through deep learning. This dynamic phase mask has stronger expressiveness in the computational imaging system than the traditional static mask, and can enhance the performance of depth estimation and extended depth of field imaging in computational imaging. The technical solution adopted is:

[0006] A design and optimization method of a dynamic phase mask based on deep learning, comprising the following steps:

[0007] Step 1, setting N initial phase masks to be optimized and generating their corresponding PSFs;

[0008] The specific steps include:

[0009] Step 1A, calling Zemax optical simulation software, inputting parameters of an initial phase mask and imaging lens structure, and obtaining the PSF of the optical system;

[0010] Step 1B, switching different initial phase mask inputs, repeating step 1A, and outputting the PSF set of the optical system under the influence of N phase masks for subsequent optimization;

[0011] Step 2: Input the initial image, use the generated series of PSFs to convolve with the image to obtain a depth-dependent convolution result, and average the images generated by each phase mask to obtain the final encoded image, which specifically includes the following steps:

[0012] Step 2A, the image processing unit receives the initial image C of the target, and performs a convolution operation on the initial image and each PSF to generate blurred images corresponding to different phase masks;

[0013] Step 2B, averaging the N blurred images to generate a coded image B;

[0014] Step 3: synchronously optimizing the dynamic phase mask and the image reconstruction network, specifically including the following steps:

[0015] Step 3A, the image reconstruction network receives the encoded image and outputs a network image W;

[0016] Step 3B: The image processing unit performs a loss comparison between the network image W and the initial image C in step 2A, and adjusts the training parameters in the loss function of the phase mask sequence and the image reconstruction network according to the comparison result.

[0017] Preferably, the specific steps of calculating the loss in step 3B include:

[0018] Based on the monocular depth estimation network, the predicted depth D of the image is calculated, and then the mean square error loss L is calculated. Depth ;

[0019]

[0020] Where N is the number of pixels;

[0021] -True depth.

[0022] Preferably, the specific steps of calculating the loss in step 3B include:

[0023] Use Attention U-Net to reconstruct the full focus image, output the reconstructed prediction image I', and then calculate the mean square error loss L AiF ;

[0024]

[0025] Where, N-number of pixels; AiF-all focus;

[0026] -True fully focused image.

[0027] Preferably, the loss function in step 3B is for:

[0028]

[0029] Where, I-initial image C;

[0030] - coded picture B;

[0031] - Network image W;

[0032] -Image simulation consistency loss;

[0033] - Loss of original image;

[0034] - Reconstruction losses;

[0035] d – depth of the image;

[0036] d' - depth of another image;

[0037] ω1-first hyperparameter; ω2-second hyperparameter;

[0038] Among them, in step 4B, the neural network weights are adjusted to reduce and

[0039] Preferably, the specific steps of phase switching in step 1 during actual application are:

[0040] In the actual imaging process, high-speed switching between the optimized N phase masks is achieved through the MEMS spatial light modulator.

[0041] Preferably, step 2B specifically includes the following steps:

[0042] The blurred image is cropped or resized, the intensity of each corresponding pixel is accumulated and the average is calculated, the calculated average intensity value matrix is ​​converted into an image format, and the final average encoded image is generated, which is used in subsequent end-to-end optimization.

[0043] Compared with the prior art, the advantages of the present invention are:

[0044] The present invention designs and optimizes a time-averaged dynamic phase mask through end-to-end.

[0045] The method uses high-speed phase modulation achieved by the SLM to switch between multiple phase masks during a single exposure, and since only a single frame is captured, the system uses less memory. Single-exposure imaging is also more light-efficient.

[0046] A single exposure allows capturing all the light in a scene at a fixed time interval. An end-to-end optimization system based on deep learning can simultaneously optimize the parameters of the dynamic phase mask and the image reconstruction system.

[0047] This dynamic phase mask has stronger expressiveness in computational imaging systems than traditional static masks, and can enhance the performance of depth estimation and extended depth of field imaging in computational imaging.

[0048] The present invention can effectively improve the performance of optical imaging systems including phase masks, and provides a new technical path for computational imaging tasks such as depth estimation and extended depth of field imaging. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 The figure is a schematic diagram of the overall process of the method of the present invention.

[0050] Figure 2 The figure is a specific flow chart of the method of the present invention.

[0051] Figure 3 Schematic diagram of the process of generating dynamic phase mask encoded images using SLM.

[0052] Figure 4 Schematic diagram of the end-to-end optimization process based on deep learning.

[0053] Figure 5 Schematic diagram for phase mask sequence optimization. DETAILED DESCRIPTION

[0054] The design and optimization method of the dynamic phase mask based on deep learning of the present invention will be described in more detail below in conjunction with a schematic diagram, wherein a preferred embodiment of the present invention is shown, and it should be understood that those skilled in the art can modify the present invention described herein while still achieving the beneficial effects of the present invention. Therefore, the following description should be understood as being widely known to those skilled in the art, and not as a limitation of the present invention.

[0055] like Figures 1 to 5 A design and optimization method for a dynamic phase mask based on deep learning can effectively enhance the performance of depth estimation and extended depth of field imaging in computational imaging engineering. Compared with the existing phase mask design method, the dynamic phase mask designed and optimized by the present invention has higher phase accuracy and better expressiveness, opening up a new direction for phase mask technology and computational imaging system design.

[0056] The specific steps include:

[0057] The MEMS spatial light modulator is a spatial light modulator (SLM) based on micro-electromechanical system (MEMS) technology, which can achieve high-response phase modulation, construct a series of time-averaged (the time of each switching is consistent) dynamic phase mask sequences and obtain their encoded images.

[0058] Step 1: Set N initial phase masks to be optimized and generate their corresponding PSFs.

[0059] Step 1A, calling Zemax optical simulation software, inputting parameters of an initial phase mask and imaging lens structure, and obtaining the PSF of the optical system;

[0060] Step 1B, switch different initial phase mask inputs, repeat step 1A, and output the PSF set of the optical system under the influence of N phase masks for subsequent optimization.

[0061] During the overall optimization process, the optimization system continuously adjusts the phase mask sequence according to the neural network feedback to ensure that the PSF set of the optical system under the influence of the optimized dynamic phase mask is suitable for the computational imaging task (such as depth of field extension). In the actual imaging process, the MEMS-based SLM is used to achieve high-response phase modulation, and switching between multiple phase masks is achieved during a single exposure, thereby obtaining a coded image output by multiple phase masks.

[0062] Since the set of PSFs output by multiple phase masks is strictly non-convex, the design space of point spread functions generated by dynamic SLM phase masks is fundamentally more expressive than that of static masks.

[0063] like Figure 3 As shown, the optical model images a static scene through multiple phase masks.

[0064] First, a series of dynamic phase masks are designed and phase modulation is performed using SLM. During a single exposure, the SLM switches at high speed between certain phase modes to obtain a coded image under the influence of the dynamic phase mask.

[0065] This process can be modeled as a total exposure time of 100 milliseconds, with the switching time for each exchange varying from 1 to 16 milliseconds, with the performance of the joint optimization system decreasing as the switching time between phase masks increases.

[0066] When the switching time is less than 25% of the total exposure, the noise can still be overcome by increasing the effective accuracy through intensity averaging over multiple frames.

[0067] Step 2: Get the encoded image.

[0068] Step 2A, the image processing unit receives the initial image C of the target, and performs a convolution operation on the initial image and each PSF to generate a blurred image corresponding to the phase mask;

[0069] Step 2B: N blurred images are averaged to generate a coded image B.

[0070] That is, the initial static scene is input for imaging, and a series of depth-dependent PSFs are generated using the designed time-averaged phase mask sequence. These PSFs are convolved with the depth-independent image to obtain depth-dependent convolution results.

[0071] The intensity of each corresponding pixel of these blurred images is accumulated and the average value is calculated. The calculated average intensity value matrix is ​​converted into image format to generate the final average encoded image, which is input into the downstream deep network for image processing. The image output by the network will be compared with the actual image for loss. The system then adjusts the parameters of the dynamic phase mask and the image reconstruction system according to the comparison result, thereby achieving coordinated optimization of the dynamic phase mask and the image reconstruction network, and finally obtaining a dynamic phase mask that is more suitable for computational imaging engineering.

[0072] Step 3: Synchronously optimize the phase mask and image reconstruction network.

[0073] This step involves the simultaneous optimization of the phase mask and the reconstruction algorithm (image reconstruction network).

[0074] In practical applications, the designed dynamic phase mask can be applied to computational imaging systems related to phase mask technology to obtain better extended depth of field imaging and depth estimation performance.

[0075] Step 3A, the image reconstruction network receives the encoded image and outputs a network image W;

[0076] Step 3B: The image processing unit performs a loss comparison between the network image W and the initial image C in step 2A, and then adjusts the training parameters in the loss function of the phase mask sequence and the image reconstruction network according to the comparison result.

[0077] During the end-to-end optimization process, the system will continuously optimize the phase mask sequence through deep learning.

[0078] The hybrid surface representing the phase mask can be represented by a function containing the radial distance of the optical axis, the radius of curvature, the aspheric coefficient, high-order polynomial coefficients, and an odd-order polynomial with phase mask characteristics. By continuously training and optimizing the relevant optical parameters, a dynamic phase mask that is more suitable for computational imaging systems such as extended depth of field imaging can be obtained.

[0079] The image reconstruction network is an important part of this optimization process.

[0080] NAFNet can be used as the image reconstruction network, which is a UNet-shaped network with excellent inter-block and intra-block networks, making it computationally efficient and easy to train.

[0081] In addition, NAFNet shows excellent performance in multiple image deblurring tasks, achieving a balance between performance and computational efficiency, making the network suitable for end-to-end optimization systems. The dynamic phase mask after system optimization has better performance than the traditional static mask in computational imaging tasks involving phase mask technology.

[0082] For the depth estimation task, you can use the pre-trained weights of the MiDaS Small architecture for monocular depth estimation, which is a well-known convolutional monocular depth estimation network designed to take in natural images and output relative depth maps.

[0083] The network is trained end-to-end using phase masks to predict D and the true depth from depth reconstruction. The mean square error (MSE) loss term is defined as:

[0084]

[0085] Where N is the number of pixels, this process allows to simultaneously optimize the phase template and fine-tune MiDaS to achieve the reconstruction of the average final encoded image.

[0086] For the extended depth of field task, the Attention U-Net can be used to reconstruct the full-focus image. The network is jointly optimized with the phase mask sequence to reconstruct the predicted image I and the true full-focus image. Define the MSE error loss term:

[0087]

[0088] Where N is the number of pixels.

[0089] like Figure 4 , during the forward image simulation (black arrow), the optimization system tracks the gradient of each parameter,

[0090] and backpropagates the error (red arrow) from the network output of the end-to-end optimization system,

[0091] In this process, the dynamic phase mask and image reconstruction network are jointly optimized.

[0092] During this optimization process, the system will continuously change the phase mask sequence through deep learning feedback, and the PSF of the optical system corresponding to the dynamic phase mask will also change continuously. The goal of optimizing the phase mask sequence is to make the final encoded image more suitable for computational imaging tasks such as extended depth of field or depth estimation.

[0093] NAFNet is used as the image reconstruction network, which is a UNet-shaped network with high computational efficiency and easy training, suitable for end-to-end optimization system.

[0094] At this stage, in order to design a dynamic phase mask that is more suitable for computational imaging systems such as extended depth of field, different depths can be set. The training parameters of the dynamic phase mask are optimized by pursuing the consistency of its image simulation.

[0095] The loss function of the system is designed as:

[0096]

[0097] In the formula, I, and Represent the real image, encoded image and reconstruction result respectively.

[0098] The hyperparameters ω1 and ω2 balance the different loss terms.

[0099] By optimizing the optical parameters of the dynamic phase mask and the parameters of the reconstruction network, the pursuit of different depths is maximized. image simulation consistency, while also striving for the highest simulation quality This enables joint optimization of the dynamic phase mask and the image reconstruction network.

[0100] The setting of the training depth can be reasonably selected near the focal length value of the different optical imaging systems that are initially set.

[0101] like Figure 5 As shown in the figure, each circle represents the visualization of a phase mask. Taking the case where there are three phase masks in the phase mask sequence as an example, a phase mask with a random initial starting point can be set for optimization (in the actual optimization process, a suitable optimization starting point can also be selected by yourself). During the optimization process, as the neural network feedback continuously changes the network parameters, the structure of the phase mask sequence is also adjusted accordingly. The two are optimized towards a common goal, and finally the optimized phase mask sequence constitutes a dynamic phase mask.

[0102] The above is only a preferred embodiment of the present invention and does not limit the present invention in any way. Any technician in the relevant technical field, without departing from the scope of the technical solution of the present invention, makes any form of equivalent replacement or modification to the technical solution and technical content disclosed in the present invention, which does not depart from the content of the technical solution of the present invention and still falls within the protection scope of the present invention.

Claims

1. A design and optimization method for a dynamic phase mask based on deep learning, characterized in that: The following steps are involved: Step 1, setting N initial phase masks to be optimized and generating their corresponding PSFs; The specific steps include: Step 1A, calling Zemax optical simulation software, inputting parameters of an initial phase mask and imaging lens structure, and obtaining the PSF of the optical system; Step 1B, switching different initial phase mask inputs, repeating step 1A, and outputting the PSF set of the optical system under the influence of N phase masks for subsequent optimization; Step 2: Input the initial image, use the generated series of PSFs to convolve with the image to obtain a depth-dependent convolution result, and average the images generated by each phase mask to obtain the final encoded image, which specifically includes the following steps: Step 2A, the image processing unit receives the initial image C of the target, and performs a convolution operation on the initial image and each PSF to generate blurred images corresponding to different phase masks; Step 2B, averaging the N blurred images to generate a coded image B; Step 3: synchronously optimizing the dynamic phase mask and the image reconstruction network, specifically including the following steps: Step 3A, the image reconstruction network receives the encoded image and outputs a network image W; Step 3B: The image processing unit performs a loss comparison between the network image W and the initial image C in step 2A, and adjusts the training parameters in the loss function of the phase mask sequence and the image reconstruction network according to the comparison result.

2. The method for designing and optimizing a dynamic phase mask based on deep learning according to claim 1, characterized in that: The specific steps for calculating the loss in step 3B include: Based on the monocular depth estimation network, the predicted depth D of the image is calculated, and then the mean square error loss L is calculated. Depth ; Where N is the number of pixels; -True depth.

3. The method for designing and optimizing a dynamic phase mask based on deep learning according to claim 1, characterized in that: The specific steps for calculating the loss in step 3B include: Use Attention U-Net to reconstruct the full focus image, output the reconstructed prediction image I', and then calculate the mean square error loss L AiF ; Where, N-number of pixels; AiF-all focus; -True fully focused image.

4. The method for designing and optimizing a dynamic phase mask based on deep learning according to claim 1, characterized in that: Loss function in step 3B for: Where, I-initial image C; - coded picture B; - Network image W; -Image simulation consistency loss; - Loss of original image; - Reconstruction losses; d – depth of the image; d' - depth of another image; ω1-first hyperparameter; ω2-second hyperparameter; Among them, in step 4B, the neural network weights are adjusted to reduce and 5. The method for designing and optimizing a dynamic phase mask based on deep learning according to claim 1, characterized in that: The specific steps of phase switching in step 1 in actual application are: In the actual imaging process, high-speed switching between the optimized N phase masks is achieved through the MEMS spatial light modulator.

6. The method for designing and optimizing a dynamic phase mask based on deep learning according to claim 1, characterized in that: Step 2B specifically includes the following steps: The blurred image is cropped or resized, the intensity of each corresponding pixel is accumulated and the average is calculated, the calculated average intensity value matrix is ​​converted into an image format, and the final average encoded image is generated, which is used in subsequent end-to-end optimization.

Images