A design method for an end-to-end refractive-diffractive hybrid imaging system based on ray tracing

Abstract

The present invention discloses a method for designing an end-to-end refractive-diffractive hybrid imaging system based on ray tracing. The method uses Newton's method to find the intersection of light rays and optical elements, utilizes Snell's law to implement ray tracing of refractive optical elements, and utilizes local grating equations to implement ray tracing of diffractive optical elements, thus achieving differentiable ray tracing of the entire optical path. The ray tracing results are used to calculate a simulated aberration image, and an image restoration neural network sensitive to local image texture is constructed. The input is the aberration image, and the output is an image with some aberrations removed. A loss function is constructed using the quality of the final output image as a judgment criterion, and end-to-end gradient descent optimization is performed on both optical parameters and neural network parameters to achieve the design of a collaborative imaging system for refractive-diffractive hybrid optical elements and deep learning. The present invention constructs a new ray tracing framework for the refractive-diffractive hybrid optical system, achieving end-to-end optimization, increasing the adaptability of the optical system to the optimization algorithm, and realizing high-quality imaging.

Description

Technical Field

[0001] The present invention relates to the fields of optical design and deep learning, and in particular to a design method for an end-to-end refractive-diffractive hybrid imaging system based on ray tracing. Background Art

[0002] End-to-end optical design has recently attracted considerable attention. This integrated approach simultaneously optimizes optical systems and neural networks via gradient backpropagation, aiming to identify the most efficient combination of optical components and back-end algorithms for a specific task. Within this framework, optical elements such as refractive lenses, diffractive optical elements, or metasurfaces are part of the overall imaging network. They collaborate with back-end processing networks to perform the imaging task and reconstruct the final output, which can include an image or other visual representation. This approach shows great promise in scientific imaging, computer vision, and the evolution of optical systems beyond traditional lens design methods.

[0003] Diffractive optical elements manipulate the phase of light by optimizing the design of phase surfaces. Compared to traditional lenses, they exhibit opposite chromatic and thermal aberration characteristics and possess powerful aberration correction capabilities. The diffraction grating, invented in the late 18th century, was the earliest diffractive optical element and is widely used in optical instruments. In 1820, Fresnel proposed the Fresnel lens, which was successfully developed in 1822. In the mid-1980s, the MIT Lincoln Laboratory in the United States pioneered the concept of "binary optics." Based on the diffraction theory of broadcasting, this technique creates a relief structure with two or more step depths etched onto the surface of a traditional optical element, creating a type of diffractive optical element that boasts pure phase, coaxial reproduction, and extremely high diffraction efficiency. In 1988, Swanson Veuknmp et al. exploited the dispersion properties of diffractive optical elements to correct for on-axis chromatic aberration and spherical aberration in a single lens, developing a multi-order phase lens. Since then, research on the application of diffractive optical elements in optical imaging has begun. Refractive-diffractive hybrid lenses combine the advantages of refractive and diffractive optical elements, showing great promise for the next generation of optical imaging systems, particularly in the field of infrared optics. However, current differentiable image formation models do not fully support the end-to-end design of more complex refractive-diffractive hybrid optical systems. Traditional wave optics models often oversimplify diffractive elements as thin phase plates, ignoring optical aberrations. Recent researchers have used a ray model for refractive lenses and a wave diffraction model for diffractive optical elements located at the end of the optical system. This model has three main problems. First, the conversion between rays and waves results in a loss of accuracy. Second, the diffractive elements in this model can only be machined on a flat substrate, and cannot track diffractive elements machined on an aspheric surface. Third, it only supports placing the diffractive element at the very end of the optical system, which limits its scope of application. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this paper proposes a ray-tracing-based design method for an end-to-end refractive-diffractive hybrid imaging system. This method employs a full ray-tracing model, integrates refractive and diffractive optical elements into a unified framework, and designs an aberration-correcting neural network tailored to the characteristics of aberrations. This method enables end-to-end collaborative design of high-precision refractive-diffractive hybrid optical imaging systems and is applicable to various types of refractive-diffractive hybrid optical systems.

[0005] The purpose of the present invention is achieved through the following technical solutions:

[0006] A method for designing an end-to-end refractive-diffractive hybrid imaging system based on ray tracing includes the following steps:

[0007] S1: Establish a virtual diffractive-refractive hybrid optical structure and the initial structure of the detector plane as a virtual optical system; and construct a data set consisting of real clear images;

[0008] S2: Establish a ray tracing model for a refractive-diffractive hybrid optical system. For refractive surfaces, use Snell's theorem to calculate the ray deflection direction. For diffractive surfaces, use the local grating equation to calculate the ray deflection direction.

[0009] S3: Setting a clear image in the data set as a virtual object plane at a preset object distance, sampling the object plane light to obtain an initial light, and propagating it into the virtual optical system. Using the ray tracing model established in S2, tracing the light from the incident point to the detector plane to obtain a simulated aberration-degraded image;

[0010] S4: Establish an image restoration neural network, the input of which is a simulated degraded image with aberrations, and the output is a picture with some aberrations removed; the image restoration neural network as a whole has a U-shaped structure, including a downsampling part and an upsampling part, and a full-field attention module connecting the downsampling part and the upsampling part; the downsampling part and the upsampling part each include a 3×3 convolution module and four full-field attention modules, and jump connections are performed between the corresponding full-field attention modules of the downsampling part and the upsampling part to promote the transmission of effective information; the full-field attention module includes two parts, the first part has five layers, which are LN layer, 1×1 convolution layer, 3×3 convolution layer, activation function ReLU and full-dimensional dynamic convolution in sequence; the second part has four layers, which are LN layer, 1×1 convolution layer, activation function ReLU and full-dimensional dynamic convolution layer in sequence; the input and output of the first part of the full-field attention module are added element-wise as the input of the second part; the input of the second part and the output of the second part are added element-wise as the output of the entire full-field attention module;

[0011] The full-dimensional dynamic convolution layer adopts a parallel strategy inside any convolution kernel to learn the attention value inside the convolution kernel from four dimensions: position, input channel, filter, and convolution kernel, thereby obtaining the full-dimensional convolution kernel attention value;

[0012] S5: Inputting the simulated aberration-containing image obtained in step S3 into the image restoration neural network, constructing a loss function, calculating the error between the image with some aberrations removed output by the image restoration neural network and the real clear image, and using back-propagation gradient descent to simultaneously optimize the parameters of the refractive-diffractive hybrid optical structure and the neural network parameters;

[0013] S6: Using all the real clear pictures in the dataset, repeat S3 to S6 until clear imaging is achieved, obtain the optimized refractive-diffractive hybrid optical structure and image restoration neural network, and complete the design of the end-to-end refractive-diffractive hybrid imaging system based on ray tracing.

[0014] Furthermore, the initial structure of the refractive-diffractive hybrid optical structure is two flat plates. Along the direction from incident to outgoing light, the first light-transmitting surface of the first flat plate is set as a diffractive surface, and the remaining light-transmitting surfaces of the two flat plates are set as even-refractive aspheric surfaces.

[0015] The expressions for all refractive and diffractive surfaces are:

[0016]

[0017] Where c is the curvature, k is the cone coefficient, α 2x is the high-order coefficient of the aspheric surface, r is the radial distance, x is the aspheric order, and X is the maximum aspheric order;

[0018] The additional phase introduced by the diffraction surface is:

[0019]

[0020] Among them, β y is the phase coefficient of the diffraction surface, R is the maximum radius of the diffraction element, y is the phase order, and Y is the maximum phase order.

[0021] Furthermore, in S2, the direction vector d of the outgoing light of the refractive surface t The vector form of the expression is

[0022]

[0023] Among them, n1 is the normal vector of the incident point of the refracting surface, d i is the direction vector of the incident light, n i is the refractive index of the material on the incident side, n tis the refractive index of the material on the output side;

[0024] The vector form of the outgoing light from the diffraction surface is expressed as:

[0025]

[0026] Where n2 is the normal vector of the incident point of the aspheric surface of the diffraction surface base, λ is the wavelength of the incident light, m ​​is the selected diffraction order, φ(r) is the additional phase introduced by the diffraction surface, and f is the direction vector of the local grating.

[0027] Furthermore, in S3, the Newton iteration method is used to calculate the intersection of the light and the virtual optical system to obtain the incident point; the light deflection angle is calculated for the refractive surface and the diffraction surface respectively, and the light is traced through the refractive surface and the diffraction surface to the detector plane; for the light on the detector plane, it is weighted according to the intensity of its virtual image plane sampling point to obtain a simulated aberration-containing degraded image P1.

[0028] Furthermore, the loss function in S5 is expressed as:

[0029] L=||P2(θ,n)-P0|| 2 +εL opt

[0030] Among them, the first term represents the gap between the output image of the image restoration neural network and the real image, P2 represents the picture output by the image restoration neural network, θ represents the parameters of the virtual optical system, n represents the parameters of the image restoration neural network; ε is a set proportional coefficient, L opt Represents the limitations of the virtual optical system itself.

[0031] The beneficial effects of the present invention are as follows:

[0032] The end-to-end refractive-diffractive hybrid imaging system design method proposed in the present invention constructs a unified ray tracing model for the refractive-diffractive hybrid optical system through Snell's law and a local grating equation model, and proposes an aberration recovery network that can enhance local texture features, thereby realizing the end-to-end optical design of any refractive-diffractive hybrid system; the refractive-diffractive hybrid imaging system obtained by the end-to-end design of the present invention can achieve high-precision and high-real-time imaging, and the design method can be adapted to the design of various forms of refractive-diffractive hybrid optical systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is a flow chart of a method for designing an end-to-end refractive-diffractive hybrid imaging system based on ray tracing in an embodiment of the present invention.

[0034] Figure 2 Graph showing the operating results of the ray tracing method in an embodiment of the present invention.

[0035] Figure 3 2 is a structural diagram of the image restoration neural network in an embodiment of the present invention.

[0036] Figure 4 These are the direct imaging results of the end-to-end design in an embodiment of the present invention, where (a) is a clear image without aberrations input to the system, (b) is the direct imaging result through the optical system, and (c) is the restoration result of the neural network. DETAILED DESCRIPTION

[0037] The present invention will be described in detail below with reference to the accompanying drawings and preferred embodiments, and the purpose and effects of the present invention will become more apparent. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0038] like Figure 1 As shown, the end-to-end refractive-diffractive hybrid imaging system design method based on ray tracing of this embodiment includes the following steps:

[0039] Step 1: Establish a virtual refractive-diffractive hybrid optical structure and the initial structure of the detector plane as a virtual optical system; and construct a data set consisting of real clear images.

[0040] As one implementation, to demonstrate the optimization capabilities of the present invention's method, the initial structure of a refractive-diffractive hybrid optical structure consists of two flat plates. Along the direction of incident to outgoing light, the first light-transmitting surface of the first plate is configured as a binary diffractive surface, while the remaining light-transmitting surfaces of both plates are configured as even-order refractive aspheric surfaces. The diffractive surfaces can be fabricated on any substrate. In this embodiment, the substrate is an even-order aspheric surface.

[0041] The expressions for diffractive base aspheric surfaces and refractive aspheric surfaces are both:

[0042]

[0043] Where c is the curvature, k is the cone coefficient, α 2x is the high-order coefficient of the aspheric surface, r is the radial distance, x is the aspheric order, and X is the maximum aspheric order.

[0044] The additional phase introduced by the diffraction surface is:

[0045]

[0046] Among them, β y is the phase coefficient of the diffraction surface, R is the maximum radius of the diffraction element, y is the phase order, and Y is the maximum phase order.

[0047] Step 2: Establish a ray tracing model for the refractive-diffractive hybrid optical system. For the refractive surface, use Snell's theorem to calculate the light deflection direction. For the diffractive surface, use the local grating equation to calculate the light deflection direction, thus forming a unified ray tracing model.

[0048] The direction vector d of the outgoing light from the refractive surface t The vector form of the expression is

[0049]

[0050] Among them, n1 is the normal vector of the incident point of the refracting surface, d i is the direction vector of the incident light, n i is the refractive index of the material on the incident side, n t is the refractive index of the material on the output side.

[0051] The vector form of the local grating equation of the diffraction surface is expressed as:

[0052]

[0053] Among them, n2 is the normal vector of the incident point of the aspheric surface of the diffraction surface base described in S1, λ is the wavelength of the incident light, m ​​is the selected diffraction order, φ(r) is the additional phase introduced by the diffraction surface described in S1, and f is the direction vector of the local grating.

[0054] Through vector calculation, the vector form of the outgoing light of the diffraction surface can be obtained as follows:

[0055]

[0056]

[0057] Here, n2 is the normal vector of the incident light on the aspheric surface of the diffraction surface base, λ is the wavelength of the incident light, m ​​is the selected diffraction order, φ(r) is the additional phase introduced by the diffraction surface, and f is the direction vector of the local grating. This allows the tracking of refractive and diffractive elements to be incorporated into a unified framework.

[0058] Step 3: Set a clear image in the data set as a virtual object plane at a preset object distance, sample the object plane light, obtain the initial light, and propagate it into the virtual optical system. Use the ray tracing model established in S2 to trace the light from the incident point to the detector plane to obtain a simulated aberration-degraded image.

[0059] In this embodiment, the object distance is set to 8000 meters. According to the size of the detector, the light is traced in reverse and intersected with the object surface to obtain the size of the virtual object surface. Then, for each pixel point of the virtual image surface, 64 light rays are sampled and forward propagated. Figure 2 As shown, the Newton iteration method is used to calculate the intersection of the light and the optical system to obtain the incident point. The light deflection angle is calculated for the refractive surface and the diffraction surface respectively. The light passes through the refractive surface and the diffraction surface and is traced to the detector plane. The light on the detector plane is weighted according to the intensity of its virtual image plane sampling point to obtain a simulated aberration-containing degraded image P1.

[0060] Step 4: Establish an image restoration neural network, such as Figure 3 As shown in the figure, the input of the image restoration neural network is a simulated aberration-degraded image, and the output is a picture with some aberrations removed; the image restoration neural network has a U-shaped structure as a whole, including a downsampling part and an upsampling part, as well as a full-field attention module FDA-Block connecting the downsampling part and the upsampling part; the downsampling part and the upsampling part each include a 3×3 convolution module and four full-field attention modules FDA-Block, and the corresponding full-field attention modules of the downsampling part and the upsampling part are skipped to promote the transmission of effective information; the full-field attention module consists of two parts, the first part has five layers, namely LN layer, 1×1 convolution layer, 3×3 convolution layer, activation function ReLU and full-dimensional dynamic convolution; the second part has four layers, namely LN layer, 1×1 convolution layer, activation function ReLU and full-dimensional dynamic convolution layer; the input and output of the first part of the full-field attention module FDA-Block are added element-wise and used as the input of the second part; the input of the second part and the output of the second part are added element-wise and used as the output of the entire full-field attention module.

[0061] In this embodiment, in the input and output of the network, based on the degradation characteristics of the aberration as the field of view changes, the field of view prior is introduced into the input. Specifically, the input one-channel blurred image is converted into a three-channel input: one channel blurred image, one channel X-direction field of view change, and one channel Y-direction field of view change. The input of the three channels is 640×512 in size, and the final output is a single-channel 640×512 clear image. This input can be used for regional learning of different fields of view, and different restoration measures can be taken to effectively improve the restoration quality of the image, especially the field of view near the edge. In addition, the residual connection of the overall network structure is also used, that is, the network learns not the output image, but the difference between the output image and the input image.

[0062] The full-dimensional dynamic convolution layer uses a parallel strategy within any convolution kernel to learn the internal attention value of the convolution kernel from four dimensions: position, input channel, filter, and convolution kernel, thereby obtaining the full-dimensional convolution kernel attention value. This can better integrate the input field of view prior and make the weights of coarse and fine textures different, strengthening local texture learning and enhancing attention.

[0063] Step 5: Input the simulated aberration-containing image obtained in step 3 into the image restoration neural network, construct a loss function, calculate the error between the image with some aberrations removed output by the image restoration neural network and the real clear image, and use backpropagation gradient descent to simultaneously optimize the parameters of the refractive-diffractive hybrid optical structure and the neural network parameters.

[0064] The output image of the neural network is recorded as P2, which is related to the optical system parameter θ and the neural network parameter n, so it can be expressed as P2(θ,n). The constructed loss function expression is as follows:

[0065] L=||P2(θ,n)-P0|| 2 +εL opt

[0066] The first term represents the gap between the neural network output image and the real image, ε is a set proportional coefficient, and L opt Represents the limitations of the optical system itself, such as aperture size, maximum lens thickness, minimum air gap, etc.

[0067] Step 6: Using all the real clear images in the dataset, repeat steps 3 to 6 until clear imaging is achieved. This results in the optimized refractive-diffractive hybrid optical structure and image restoration neural network, completing the design of an end-to-end refractive-diffractive hybrid imaging system based on ray tracing.

[0068] A specific embodiment of the method of the present invention is given below to illustrate the technical effects of the present invention.

[0069] In this embodiment, 1000 real-shot infrared data sets of 640*512 size are used, and the clear pictures in the data set are recorded as P0. In addition, a virtual lens is established, and the optimization starting point is a flat plate. Along the direction from the incident light to the outgoing light, the first light-transmitting surface of the first flat plate is a binary diffraction surface, and the remaining surfaces are spherical surfaces. The number of pixels on the detector plane is 640×512, and the pixel size is 15 microns. A virtual image plane is set at an object distance of 8000 meters, and sampling is performed, with 64 light rays sampled for each pixel. A unified ray tracing model for a refractive-diffractive hybrid optical system is established, and all incident light rays are traced to the detector plane to obtain a simulated picture with aberrations. Figure 3As shown, an image restoration neural network is constructed and trained. Within the full-field attention module (FDA-Block), the 1×1 convolution has a kernel size of 1, padding of 0, and a stride of 1; the 3×3 convolution has a kernel size of 3, padding of 1, and a stride of 1; and the full-dimensional dynamic convolution has a kernel size of 3, padding of 1, and a stride of 1. Within the full-field attention module (FDA-Block), the image size remains unchanged. With each downsampling pass, the number of channels in the image doubles, and with each upsampling pass, the number of channels is divided by 2. At the final 3×3 convolution, the number of channels in the image is reduced to 1, resulting in the output image.

[0070] A simulated three-channel 640×512 image containing aberrations and a field of view prior was input into the image restoration neural network. The network restored a single-channel 640×512 image. During training, a learning rate of 0.001 was set, and the back-propagation Adam algorithm was used for gradient descent. The parameters of the refractive-diffractive hybrid optical system and the neural network were simultaneously optimized. The optimization was iterated 100 times to obtain the final optimized result.

[0071] like Figure 4 As shown in the figure, it is the result of this embodiment, where (a) is a clear picture without aberration of the input system, (b) is the direct imaging result through the optical system, with a structural similarity of 0.65, and (c) is the restoration result of the neural network. Figure 4 The final imaging result has a structural similarity of over 0.9 compared to the original image, demonstrating that this method can achieve high-precision design of an end-to-end refractive-diffractive hybrid optical system. For a 640×512 infrared image, the imaging system designed using this method can achieve a single-image computation time of less than 30ms and can run at 30 frames per second, demonstrating its high real-time performance.

[0072] Those skilled in the art will understand that the foregoing descriptions are merely preferred embodiments of the invention and are not intended to limit the invention. Although the invention has been described in detail with reference to the foregoing examples, those skilled in the art will still be able to modify the technical solutions described in the foregoing examples or substitute equivalents for some of the technical features therein. Any modifications, equivalent substitutions, etc. made within the spirit and principles of the invention shall be included within the scope of protection of the invention.

Claims

1. A design method for an end-to-end refractive-diffractive hybrid imaging system based on ray tracing, characterized in that: The following steps are involved: S1: Establish a virtual refractive-diffractive hybrid optical structure and the initial structure of the detector plane as a virtual optical system; And build a dataset consisting of real clear pictures; S2: Establish a ray tracing model for a refractive-diffractive hybrid optical system. For refractive surfaces, use Snell's theorem to calculate the ray deflection direction. For diffractive surfaces, use the local grating equation to calculate the ray deflection direction. S3: Setting a clear image in the data set as a virtual object plane at a preset object distance, sampling the object plane light to obtain an initial light, and propagating it into the virtual optical system. Using the ray tracing model established in S2, tracing the light from the incident point to the detector plane to obtain a simulated aberration-degraded image; S4: Establish an image restoration neural network, the input of which is a simulated degraded image with aberrations, and the output is a picture with some aberrations removed; the image restoration neural network as a whole has a U-shaped structure, including a downsampling part and an upsampling part, and a full-field attention module connecting the downsampling part and the upsampling part; the downsampling part and the upsampling part each include a 3×3 convolution module and four full-field attention modules, and jump connections are performed between the corresponding full-field attention modules of the downsampling part and the upsampling part to promote the transmission of effective information; the full-field attention module includes two parts, the first part has five layers, which are LN layer, 1×1 convolution layer, 3×3 convolution layer, activation function ReLU and full-dimensional dynamic convolution in sequence; the second part has four layers, which are LN layer, 1×1 convolution layer, activation function ReLU and full-dimensional dynamic convolution layer in sequence; the input and output of the first part of the full-field attention module are added element-wise and used as the input of the second part; the input of the second part and the output of the second part are added element-wise and used as the output of the entire full-field attention module; The full-dimensional dynamic convolution layer adopts a parallel strategy inside any convolution kernel to learn the attention value inside the convolution kernel from four dimensions: position, input channel, filter, and convolution kernel, thereby obtaining the full-dimensional convolution kernel attention value; S5: Inputting the simulated aberration-containing image obtained in step S3 into the image restoration neural network, constructing a loss function, calculating the error between the image with some aberrations removed output by the image restoration neural network and the real clear image, and using back-propagation gradient descent to simultaneously optimize the parameters of the refractive-diffractive hybrid optical structure and the neural network parameters; S6: Using all the real clear images in the dataset, repeat S3 to S5 until clear imaging is achieved, obtain the optimized refractive-diffractive hybrid optical structure and image restoration neural network, and complete the design of the end-to-end refractive-diffractive hybrid imaging system based on ray tracing.

2. The method for designing an end-to-end refractive-diffractive hybrid imaging system based on ray tracing according to claim 1, characterized in that: The initial structure of the refractive-diffractive hybrid optical structure is two flat plates. Along the direction from incident to outgoing light, the first light-transmitting surface of the first flat plate is set as a diffractive surface, and the remaining light-transmitting surfaces of the two flat plates are set as even-refractive aspheric surfaces. The expressions for all refractive and diffractive surfaces are: ； in, is the curvature, is the cone coefficient, are the high-order coefficients of the aspheric surface, is the radial distance, , is the maximum aspheric order; The additional phase introduced by the diffraction surface is: ； in, is the phase coefficient of the diffraction surface, is the maximum radius of the diffraction element, Phase order, is the maximum phase order.

3. The method for designing an end-to-end refractive-diffractive hybrid imaging system based on ray tracing according to claim 1, wherein: In S2, the direction vector of the outgoing light from the refractive surface The vector form of the expression is ； in, is the normal vector of the incident point of the refracting surface, is the direction vector of the incident light, is the refractive index of the material on the incident side, is the refractive index of the material on the output side; The vector form of the outgoing light from the diffraction surface is expressed as: ； ； in, is the normal vector of the incident point of the aspheric surface of the diffraction surface base, is the wavelength of the incident light, is the selected diffraction order, is the additional phase introduced by the diffraction surface, is the direction vector of the local grating.

4. The method for designing an end-to-end refractive-diffractive hybrid imaging system based on ray tracing according to claim 1, wherein: In S3, the Newton iteration method is used to calculate the intersection of the light and the virtual optical system to obtain the incident point; the light deflection angle is calculated for the refractive surface and the diffraction surface respectively, and the light is traced through the refractive surface and the diffraction surface to the detector plane; for the light on the detector plane, the intensity of its virtual image plane sampling point is weighted to obtain a simulated aberration-containing degraded image .

5. The method for designing an end-to-end refractive-diffractive hybrid imaging system based on ray tracing according to claim 1, wherein: The expression of the loss function in S5 is: ； Among them, the first term represents the gap between the output image of the image restoration neural network and the real image, represents the picture output by the image restoration neural network, represents the parameters of the virtual optical system, Represents the parameters of the image restoration neural network; is a set proportional coefficient, Represents the limitations of the virtual optical system itself.

Images