Design method of multi-scale convolutional network and single-plane diffraction element optical number cooperation system

Abstract

The invention relates to the technical field of optical system design, in particular to a design method of a multi-scale convolutional network and single-plane diffraction element optical number cooperation system, and aims to solve the problems that an existing diffraction calculation imaging technology is poor in restoration real-time performance and low in restoration precision, the method comprises the following steps: step 1, designing a single-plane diffraction element; 2, constructing a multi-scale convolutional network image restoration module; and step 3, constructing an image restoration loss function. According to the invention, a high-quality color image is obtained while a unit piece system is extremely simplified. An MSE loss function considering full-view-field wavefront aberration and diffraction efficiency weight of RGB three channels is constructed, and the restoration effect of the design method is improved by 80% compared with the average PSNR of an image before restoration; compared with a traditional PSF deconvolution image restoration algorithm, the average PSNR is improved by 30%, and a new thought is provided for miniaturization development of an optical system in the future.

Description

Technical Field

[0001] The present invention relates to the technical field of optical system design, and particularly to a design method for a multi-scale convolutional network and single-plane diffractive element optical number collaborative system. Background Art

[0002] Traditional optical elements and systems usually use complex structures to achieve high-performance indicators. Diffractive computational imaging combines diffractive optical elements with computational imaging, achieving high performance while making the instrument thinner and lighter. Single-plane diffractive elements have advantages such as simple structure, small volume, and low cost, so they show great potential in miniaturized and lightweight optical systems.

[0003] In the Chinese Patent Publication No. "CN114647079 B", a "monolithic wide-band diffractive computational imaging method" is disclosed. This method performs achromatic aberration optimization design on traditional diffractive elements, designs a restoration algorithm according to the point spread function of the diffractive element after achromatic aberration optimization, and uses this algorithm to restore the imaging image of the achromatic diffractive lens. However, the design process of the achromatic diffractive lens in this method is complex, and the restoration time of the final PSF deconvolution algorithm is long, with poor restoration real-time performance, low restoration accuracy, and still blurred restoration images. Summary of the Invention

[0004] In order to solve the problems of poor restoration real-time performance and low restoration accuracy in existing diffractive computational imaging technologies, the present invention provides a design method for a multi-scale convolutional network and single-plane diffractive element optical number collaborative system, which can achieve better effects and higher efficiency than traditional image restoration methods.

[0005] To achieve the above object, the present invention adopts the following technical solutions:

[0006] A design method for a multi-scale convolutional network and single-plane diffractive element optical number collaborative system, the method comprising the following steps:

[0007] Step 1, design a single-plane diffractive element;

[0008] Step 2, construct a multi-scale convolutional network image restoration module;

[0009] Step 3, construct an image restoration loss function.

[0010] The single-plane diffractive element includes a Fresnel surface and a diffractive surface;

[0011] For the Fresnel surface, the surface formula of its annulus is:

[0012]

[0013] Where z is the sag height at each point of the aspheric surface, c is the curvature at the vertex of the aspheric surface, r is the radial coordinate of each point on the aspheric surface, k is the conic constant, A, B, and C are the aspheric coefficients; n is the number of annuli, and h is the sag height of the aspheric substrate.

[0014] For the diffraction surface, the diffraction phase equation is determined by the following formula:

[0015] ,

[0016] Where M is the diffraction order, B1 and B2 are the diffraction surface coefficients respectively, and ρ is the radial coordinate of the refractive-diffractive surface.

[0017] The multi-scale convolutional network image restoration module first inputs the blurred image Bi, extracts primary features through the input block, enters Encoder-1 and Encoder-2, and down-samples step by step to extract multi-scale features from low resolution to high resolution. After the lowest resolution branch extracts the global blur features, it restores the spatial resolution step by step through the bilinear up-sampling Decoder-1 and Decoder-2. Each level of the decoder combines the skip connection features of the corresponding scale for feature fusion and reconstruction, and finally generates the clear image Ii through the output block.

[0018] The two encoders in the scale convolutional network image restoration module have the same structure. Each level first contains a convolutional unit composed of a convolutional layer and an activation function, which is used to initially extract the local features of the current scale; then three residual blocks are connected. Each residual block consists of a first convolutional layer, an activation function, and a second convolutional layer, and the input and output are added through skip connections to achieve efficient feature expression and information retention. The encoder extracts deep blur features from high resolution to low resolution through a multi-level structure. In particular, the feature map of the lowest resolution layer helps to capture global context information.

[0019] The two decoders in the scale convolutional network image restoration module have the same structure. Each level first restores the structural details of the coarse-scale features passed from the previous level through three residual blocks. Each residual block also consists of a first convolutional layer, an activation function, and a second convolutional layer, and then uses a transposed convolutional layer and an activation function to upsample the feature map spatially, thereby increasing the resolution. Each decoding stage is also fused with the intermediate feature map of the encoder of the corresponding scale through skip connections to achieve the combination of local and global information and fully retain the multi-scale context features. Finally, through step-by-step decoding and fusion, a high-quality image with clear structure and rich details is restored.

[0020] The loss function of the multi-scale convolutional network image restoration method is determined by the following formula:

[0021] ,

[0022] Where is the scale weight, is the weight of each channel, and respectively represent the predicted value and the target value of channel c at scale s, is the weight of the VGG perceptual loss, and are respectively the predicted value and the target value of the perceptual loss of the 256-scale image;

[0023] The weight of each channel of the multi-scale convolutional network loss function described above is determined by the following formula:

[0024] ,

[0025] In the formula is the normalized diffraction efficiency coefficient of channel c, is the normalized wavefront aberration coefficient of each channel, is the normalized actual human eye color appearance perception sensitivity; where is the wavefront aberration coefficient of each channel, and the weight is distributed proportionally according to the size of the wavefront aberration; is the diffraction efficiency of each channel;

[0026] The normalized diffraction efficiency of each channel of the multi-scale convolutional network loss function described above is determined by the following formula:

[0027] ,

[0028] In the formula, the B channel with the lowest diffraction efficiency is used as a reference, and its normalized weight is 1, and the weights of other channels are calculated.

[0029] The beneficial technical effects of the present invention are as follows:

[0030] Through the collaborative design of a single-plane diffraction element and a deep learning multi-scale convolutional network unit, the present invention achieves a high-quality color image while realizing the extreme simplification of the unit component system. An MSE loss function considering the full-field wavefront aberration and diffraction efficiency weights of the RGB three channels is constructed. The restoration effect of the present invention has an average PSNR improvement of 80% compared to the image before restoration; compared with the traditional PSF deconvolution image restoration algorithm, the average PSNR is improved by 30%, which also provides a new idea for the future development of miniaturization of optical systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 is a schematic structural diagram of the optical number collaborative system of the multi-scale convolutional network and the single-plane diffraction element described in the present invention;

[0032] Figure 2 is a schematic structural diagram of the multi-scale convolutional network module described in the present invention;

[0033] Figure 3 Schematic diagram of the encoder structure of the present invention;

[0034] Figure 4 Schematic diagram of the decoder structure of the present invention;

[0035] Figure 5 Among them, (a), (b), and (c) are the full-field wavefront aberration diagrams of the RGB three channels respectively;

[0036] Figure 6 Diffraction efficiency curve diagrams of each order and wavelength of the single-plane diffractive objective of the present invention;

[0037] Figure 7 Comparison diagram before and after restoration of the present invention, (a) is the blurred image, and (b) is the clear image after restoration. Specific implementation manners

[0038] The present invention will be described in detail below in conjunction with the accompanying drawings and specific implementation manners.

[0039] As Figure 1 shown, the multi-scale convolutional network and the single-plane diffractive element optical number cooperation system include a single-plane diffractive objective 1, which uses a Fresnel surface on the side close to the object surface and a diffractive surface on the side close to the image surface; an image sensor 2, and the single-plane diffractive objective 1 forms an image on the image sensor 2; a multi-scale convolutional network image restoration module 3, which is electrically connected to the image sensor 2, and the high-speed image sensor 2 transmits image information to the multi-scale convolutional network image restoration module 3 for restoration, wherein the multi-scale convolutional neural network image restoration module 3 is pre-trained by a training set captured by the single-plane diffractive objective 1 and the image sensor 2.

[0040] Design method of the multi-scale convolutional network and the single-plane diffractive element optical number cooperation system, the method includes the following steps:

[0041] Step 1, design the single-plane diffractive element 1;

[0042] The single-plane diffractive element 1 includes a Fresnel surface and a diffractive surface.

[0043] For the Fresnel surface, the surface formula of its annulus is:

[0044]

[0045] In the formula, z is the sag height at each point of the aspheric surface, c is the curvature at the vertex of the aspheric surface, r is the radial coordinate of each point on the aspheric surface, k is the conic constant, A, B, and C are aspheric coefficients; n is the number of annuli, and h is the sag height of the aspheric surface base.

[0046] For the diffractive surface, the diffractive phase equation is determined by the following formula:

[0047] ,

[0048] where M is the diffraction order, B1 and B2 are the diffraction plane coefficients respectively, and ρ is the radial coordinate of the refraction and diffraction plane.

[0049] Step 2: Construct the multi-scale convolutional network image restoration module 3;

[0050] As Figure 2 shown, the multi-scale convolutional network image restoration module 3 first inputs the blurred image Bi, extracts primary features through the input block, enters Encoder 1 and Encoder 2, and successively downsamples to extract multi-scale features from low resolution to high resolution. After the lowest resolution branch extracts the global blurred features, it restores the spatial resolution step by step through the bilinear upsampling Decoder 1 and Decoder 2. Each level of the decoder combines the skip connection features of the corresponding scale for feature fusion and reconstruction, and finally generates the clear image Ii through the output block.

[0051] As Figure 3 shown, the two encoders in the multi-scale convolutional network image restoration module 3 have the same structure. Each level of it first contains a convolutional unit composed of a convolutional layer and an activation function, which is used to initially extract the local features of the current scale; then three residual blocks are connected. Each residual block is composed of Convolutional Layer 1, an activation function, and Convolutional Layer 2, and the input and output are added through skip connections to achieve efficient feature expression and information preservation. The encoder extracts deep blurred features from high resolution to low resolution through a multi-level structure. In particular, the feature map of the lowest resolution layer helps to capture global context information.

[0052] As Figure 4 shown, the two decoders in the multi-scale convolutional network image restoration module 3 have the same structure. Each level of it first restores the structural details of the coarse-scale features passed from the previous level through three residual blocks. Each residual block is also composed of Convolutional Layer 1, an activation function, and Convolutional Layer 2, and then an anti-convolutional layer and an activation function are used to upsample the feature map spatially, thereby improving the resolution. Each decoding stage is also fused with the intermediate feature map of the encoder of the corresponding scale through skip connections to achieve the combination of local and global information and fully retain the multi-scale context features. Finally, through successive decoding and fusion, a high-quality image with clear structure and rich details is restored.

[0053] Step 3: Construct the loss function of the image restoration method.

[0054] The loss function of the multi-scale convolutional network image restoration method is determined by the following formula:

[0055] ,

[0056] where is the scale weight, are the weights of each channel, and respectively represent the predicted value and the target value of channel c at scale s, is the VGG perceptual loss weight, and are respectively the predicted value and the target value of the perceptual loss of the 256-scale image.

[0057] The weights of each channel of the multi-scale convolutional network loss function described above are determined by the following formula:

[0058] ,

[0059] In the formula is the normalized diffraction efficiency coefficient of channel c, is the normalized wavefront aberration coefficient of each channel, is the normalized actual human eye color appearance perception sensitivity. Among them is the wavefront aberration coefficient of each channel, and the weights are distributed proportionally according to the size of the wavefront aberration; is the diffraction efficiency of each channel.

[0060] The normalized diffraction efficiency of each channel of the multi-scale convolutional network loss function described above is determined by the following formula:

[0061] ,

[0062] In the formula, the B channel with the lowest diffraction efficiency is used as a reference, and its normalized weight is 1, and the weights of other channels are calculated.

[0063] Example:

[0064] Step 1, design a single-plane diffractive element 1:

[0065] The single-plane diffractive optical element has a working wavelength range in the visible light band, a focal length of 50 mm, an F number of 5, and a full field of view angle of 10°. The relevant parameters of the front and back surfaces of the single-plane diffractive optical element are shown in Table 1 below:

[0066] Table 1

[0067] Surface Radius of curvature Thickness Material S1 14.272 1 E48R S2 Infinity 24 - S12 Infinity - -

[0068] In this example, the aspheric coefficients of each lens are shown in Table 2:

[0069] Table 2

[0070] Surface A B C S1 -4.3E-5 -6.56E-7 -1.70E-9

[0071] In this example, the diffraction coefficient of the diffraction surface S12 in the single-plane diffraction element 1 is B1 = -182.5.

[0072] Step 2: Construct a multi-scale convolutional network image restoration module 3:

[0073] By building a multi-scale convolutional network composed of an encoder, a decoder, and cross-scale skip connections, the structural design and initialization of the image restoration module 3 are completed. The structure of the multi-scale convolutional network image restoration module in the embodiment is as Figure 2 shown, and its specific structure is as follows:

[0074] The input blurred image Bi first extracts primary features through the input block and enters Encoder 1 and Encoder 2, successively downsampling to extract multi-scale features from low resolution to high resolution. After the lowest-resolution branch extracts the global blurred features, the spatial resolution is gradually restored through the bilinear upsampling Decoder 1 and Decoder 2. At each level of the decoder, the corresponding scale skip connection features are combined for feature fusion and reconstruction, and finally, the clear image Ii is generated by the output block.

[0075] As Figure 3 shown, the two encoders have the same structure. Each level of it first includes a convolutional unit composed of a convolutional layer and an activation function, which is used to initially extract the local features of the current scale; then three residual blocks are connected. Each residual block consists of a first convolutional layer, an activation function, and a second convolutional layer, and the input and output are added through skip connections to achieve efficient feature expression and information preservation. The encoder extracts deep blurred features from high resolution to low resolution through a multi-level structure, and in particular, the feature map of the lowest-resolution layer helps to capture global context information.

[0076] As Figure 4 shown, the two decoders have the same structure. Each level of it first restores the structural details of the coarse-scale features passed from the previous level through three residual blocks. Each residual block also consists of a first convolutional layer, an activation function, and a second convolutional layer, and then an anti-convolutional layer and an activation function are used to upsample the feature map spatially, thereby increasing the resolution. Each decoding stage is also fused with the intermediate feature maps of the corresponding scale encoder through skip connections to achieve the combination of local and global information and fully retain the multi-scale context features. Finally, through successive decoding and fusion, a high-quality image with clear structure and rich details is restored.

[0077] Step 3: Construct the loss function of the image restoration method:

[0078] The multi-scale convolutional network in the embodiment processes from low scale to high scale, and realizes image deblurring through multi-stage progressive optimization, where the weights of each scale are 0.4, 0.3, and 0.3 respectively.

[0079] The full-field wavefront aberration of the RGB channels of the single-plane diffractive element in the embodiment is as follows Figure 5 As shown in Figures 5(a), 5(b), and 5(c) which are the wavefront aberration diagrams of the RGB three channels respectively, the MSE loss weights of the wavefront aberration of the RGB channels are shown in Table 3:

[0080] Table 3

[0081] R G B Wavefront aberration 0.95λ 0.71λ 1.32λ MSE coefficient 0.29 0.23 0.48

[0082] The diffraction efficiency of the RGB channels of the single-plane diffractive element in this example is as follows Figure 4 As shown, the MSE loss weights of the diffraction efficiency of the RGB channels are shown in Table 4.

[0083] Table 4

[0084] R G B Diffraction efficiency 84.25% 90.75% 63.27% MSE coefficient 0.34 0.295 0.365

[0085] The MSE loss weights of the RGB channels integrating wavefront aberration and diffraction efficiency in this example are shown in Table 5:

[0086] Table 5

[0087] R G B MSE coefficient 0.3 0.45 0.25

[0088] As Figure 7 shown, Figure 7 (a) is the blurred image before restoration, Figure 7 (b) is the clear image after restoration. The clarity of the image restored by the multi-scale convolutional network is significantly improved.

[0089] The images restored by the multi-scale convolutional network are evaluated using two methods: PSNR and SSIM. The evaluation results are shown in Table 6.

[0090] Table 6

[0091] Evaluation method PSNR SSIM Blurred image 15.5136 0.6190 Restored image of traditional PSF 21.2779 0.7343 Multi-scale convolutional neural network 28.1382 0.8918

Claims

1. Design method of multi-scale convolutional network and single-plane diffractive element optical number collaboration system, characterized in that The method includes the following steps: Step 1: Design a single-plane diffraction element; Step 2: Construct a multi-scale convolutional network image restoration module; Step 3: Construct a loss function for the image restoration method.

2. The design method of the multi-scale convolution network and single-plane diffraction element optical number cooperation system according to claim 1, wherein The single-plane diffraction element includes a Fresnel surface and a diffraction surface; For the Fresnel surface, the surface formula of its annulus is: ， In the formula, z is the sagittal height at each point of the aspheric surface, c is the curvature at the vertex of the aspheric surface, r is the radial coordinate of each point on the aspheric surface, k is the conic constant, A, B, and C are the aspheric coefficients; n is the number of annuli, and h is the sagittal height of the aspheric base; For the diffraction surface, the diffraction phase equation is determined by the following formula: ， In the formula, M is the diffraction order, B1 and B2 are the diffraction surface coefficients respectively, and ρ is the radial coordinate of the refraction-diffraction surface.

3. The design method of the multi-scale convolutional network and single-plane diffraction element optical number collaboration system according to claim 1, characterized in that The multi-scale convolutional network image restoration module first inputs the blurred image Bi, extracts primary features through the input block, enters Encoder 1 and Encoder 2, and downsamples step by step to extract multi-scale features from low resolution to high resolution. After the lowest-resolution branch extracts the global blurred features, it restores the spatial resolution step by step through the bilinear upsampling Decoder 1 and Decoder 2. Each level of the decoder combines the skip connection features of the corresponding scale for feature fusion and reconstruction, and finally generates the clear image Ii through the output block.

4. The design method of the multi-scale convolutional network and single-plane diffraction element optical number cooperation system according to claim 3, characterized in that The two encoders in the multi-scale convolutional network image restoration module have the same structure. Each level of it first includes a convolutional unit composed of a convolutional layer and an activation function, which is used to initially extract the local features of the current scale; Subsequently, three residual blocks are connected. Each residual block is composed of a first convolutional layer, an activation function, and a second convolutional layer, and the input and output are added through a skip connection to achieve efficient expression and information retention of features; the encoder extracts deep blurred features from high resolution to low resolution through a multi-level structure. In particular, the feature map of the lowest-resolution layer helps to capture global context information.

5. The design method of the multi-scale convolution network and single-plane diffraction element light number cooperation system according to claim 3, characterized in that, The two decoders in the multi-scale convolutional network image restoration module have the same structure. Each level of it first restores the structural details of the coarse-scale features passed from the previous level through three residual blocks. Each residual block is also composed of a first convolutional layer, an activation function, and a second convolutional layer, and then uses a transposed convolutional layer and an activation function to upsample the feature map spatially, thereby improving the resolution; each decoding stage is also fused with the intermediate feature map of the encoder of the corresponding scale through a skip connection method to realize the combination of local and global information and fully retain the multi-scale context features; finally, through step-by-step decoding and fusion, a high-quality image with clear structure and rich details is restored.

6. The design method of the multi-scale convolutional network and single-plane diffraction element optical number collaborative system according to claim 1, characterized in that The loss function of the multi-scale convolutional network image restoration method is determined by the following formula: ， where is the scale weight, is the weight of each channel, and respectively represent the predicted value and the target value of channel c at scale s, is the VGG perceptual loss weight, and are respectively the predicted value and the target value of the 256-scale image perceptual loss; The weights of each channel of the multi-scale convolutional network loss function are determined by the following formula: ， In the formula is the normalized diffraction efficiency coefficient of channel c, is the normalized wavefront aberration coefficient of each channel, is the normalized actual human eye color appearance perception sensitivity; wherein is the wavefront aberration coefficient of each channel, and the weights are proportionally allocated according to the magnitude of the wavefront aberration; is the diffraction efficiency of each channel; The normalized diffraction efficiency of each channel of the multi-scale convolutional network loss function is determined by the following formula: ， In the formula, the B channel with the lowest diffraction efficiency is used as a reference, and its normalized weight is 1, and the weights of other channels are calculated.

Images