Design method of optical-digital collaborative system based on multi-scale convolutional network and single-plane diffraction element

Abstract

The design method of the optical-digital collaborative system of a multi-scale convolutional network and a single-plane diffraction element relates to the field of optical system design technology. In order to solve the problems of poor real-time restoration and low restoration accuracy of existing diffraction computational imaging technology, the method includes the following steps: Step 1, designing a single-plane diffraction element; Step 2, constructing a multi-scale convolutional network image restoration module; Step 3, constructing an image restoration loss function. The present invention achieves high-quality color images while achieving extreme simplification of the single-element system. An MSE loss function of the RGB three-channel considering the full-field wavefront aberration and diffraction efficiency weights is constructed. The restoration effect of this design method is improved by 80% compared with the average PSNR of the image before restoration; compared with the traditional PSF deconvolution image restoration algorithm, the average PSNR is improved by 30%, which also provides a new idea for the future miniaturization of optical systems.

Description

Technical Field

[0001] The present invention relates to the technical field of optical system design, and in particular to a design method for a multi-scale convolutional network and single-plane diffraction element optical-digital collaborative system. Background Art

[0002] Traditional optical components and systems often use complex structures to achieve high performance. Diffraction computational imaging combines diffractive optical elements with computational imaging to achieve lightweight and high-performance instruments. Single-plane diffractive elements have advantages such as simple structure, small size, and low cost, and therefore show great potential in miniaturized and lightweight optical systems.

[0003] Chinese patent publication number "CN114647079 B" discloses a "monolithic broadband diffraction computational imaging method." This method optimizes the achromatic design of a conventional diffraction element, designs a restoration algorithm based on the point spread function of the optimized diffraction element, and uses this algorithm to restore the image formed by the achromatic diffraction lens. However, this method involves a complex design process for the achromatic diffraction lens, and the final PSF deconvolution algorithm takes a long time to restore, resulting in poor real-time performance, low restoration accuracy, and blurred images. Summary of the Invention

[0004] In order to solve the problems of poor real-time restoration and low restoration accuracy in existing diffraction computational imaging technology, the present invention provides a design method for a multi-scale convolutional network and single-plane diffraction element optical-digital collaborative system, which can achieve better results and higher efficiency than traditional image restoration methods.

[0005] To achieve the above object, the present invention adopts the following technical solutions:

[0006] A method for designing a multi-scale convolutional network and a single-plane diffraction element optical-digital collaborative system, the method comprising the following steps:

[0007] Step 1: design a single-plane diffraction element;

[0008] Step 2: Construct a multi-scale convolutional network image restoration module;

[0009] Step three: construct the image restoration loss function.

[0010] The single-plane diffraction element includes a Fresnel surface and a diffraction surface;

[0011] The surface formula of the annular zone of the Fresnel surface is:

[0012]

[0013] Where z is the sagittal height at each point on the aspheric surface, c is the curvature at the vertex of the aspheric surface, r is the radial coordinate of each point on the aspheric surface, k is the conic constant, A, B, and C are the aspheric coefficients; n is the number of rings, and h is the sagittal height of the aspheric base;

[0014] The diffraction surface and the diffraction phase equation are determined by the following formula:

[0015] ,

[0016] Where M is the diffraction order, B1 and B2 are the diffraction surface coefficients, and ρ is the radial coordinate of the diffraction surface.

[0017] The multi-scale convolutional network image restoration module first inputs the blurred image Bi and extracts primary features through the input block, enters encoder 1 and encoder 2, downsamples step by step to extract multi-scale features from low resolution to high resolution, and after the lowest resolution branch extracts the global blurred features, the spatial resolution is restored step by step through bilinear upsampling decoder 1 and decoder 2. Each level of decoder combines the jump connection features of the corresponding scale to perform feature fusion and reconstruction, and finally generates a clear image Ii from the output block.

[0018] The two encoders in the scaled convolutional network image restoration module share the same structure. Each level first contains a convolutional unit consisting of a convolutional layer and an activation function, which is used to initially extract local features at the current scale. Three residual blocks are then connected. Each residual block consists of a convolutional layer 1, an activation function, and a convolutional layer 2. The input and output are added via skip connections, achieving efficient feature expression and information preservation. The encoder extracts deep fuzzy features from high to low resolution through a multi-level structure. In particular, the feature maps of the lowest resolution layer help capture global contextual information.

[0019] The two decoders in the scaled convolutional network image restoration module share the same structure. Each stage first restores the structural details of the coarse-scale features passed from the previous stage using three residual blocks. Each residual block also consists of a convolutional layer (1), an activation function, and a convolutional layer (2). A deconvolutional layer and activation function are then used to spatially upsample the feature map to improve resolution. Each decoding stage is also fused with the intermediate feature map of the corresponding scale encoder via skip connections, combining local and global information and fully preserving multi-scale contextual features. Ultimately, through progressive decoding and fusion, a high-quality image with clear structure and rich details is restored.

[0020] The loss function of the multi-scale convolutional network image restoration method is determined by the following formula:

[0021] ,

[0022] In the formula is the scale weight, is the weight of each channel, and They represent the predicted value and target value of channel c at scale s, is the VGG perceptual loss weight, and are the predicted and target values ​​of the 256-scale image perceptual loss;

[0023] The multi-scale convolutional network loss function channel weights Determined by the following formula:

[0024] ,

[0025] In the formula is the normalized diffraction efficiency coefficient of channel c, is the normalized wavefront aberration coefficient of each channel, is the normalized actual human eye color perception sensitivity; is the wavefront aberration coefficient of each channel, and the weight is allocated proportionally according to the size of the wavefront aberration; is the diffraction efficiency of each channel;

[0026] The multi-scale convolutional network loss function is the normalized diffraction efficiency of each channel Determined by the following formula:

[0027] ,

[0028] In the formula, the B channel with the lowest diffraction efficiency is taken as a reference and its normalized weight is 1, and the weights of other channels are calculated.

[0029] The beneficial technical effects of the present invention are as follows:

[0030] This invention achieves high-quality color images while minimizing the single-element system through the collaborative design of a single-plane diffraction element and a deep learning multi-scale convolutional network unit. A three-channel RGB MSE loss function, weighted by considering full-field wavefront aberrations and diffraction efficiency, was constructed. The restoration effect of this invention improved the average PSNR of the original image by 80% and the average PSNR by 30% compared to the traditional PSF deconvolution image restoration algorithm, providing new ideas for the future miniaturization of optical systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 Schematic diagram of the structure of the multi-scale convolutional network and single-plane diffraction element optical-digital collaborative system of the present invention;

[0032] Figure 2 This is a schematic diagram of the multi-scale convolutional network module structure of the present invention;

[0033] Figure 3 This is a schematic diagram of the encoder structure of the present invention;

[0034] Figure 4 This is a schematic diagram of the decoder structure of the present invention;

[0035] Figure 5 (a), (b) and (c) are the RGB three-channel full-field wavefront aberration maps respectively;

[0036] Figure 6 This is a graph showing the diffraction efficiency of the single-plane diffraction objective lens at various levels and wavelengths according to the present invention;

[0037] Figure 7 The comparison diagrams before and after restoration of the present invention are as follows: (a) is a blurred image, and (b) is a clear image after restoration. DETAILED DESCRIPTION

[0038] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0039] like Figure 1 As shown, a multi-scale convolutional network and a single-plane diffraction element optical digital collaborative system includes a single-plane diffraction objective lens 1, which uses a Fresnel surface on the side close to the object plane and a diffraction surface on the side close to the image plane; an image sensor 2, on which the single-plane diffraction objective lens 1 is imaged; a multi-scale convolutional network image restoration module 3, which is electrically connected to the image sensor 2, and the high-speed image sensor 2 transmits the image information to the multi-scale convolutional network image restoration module 3 for restoration, wherein the multi-scale convolutional neural network image restoration module 3 is pre-trained with a training set captured by the single-plane diffraction objective lens 1 and the image sensor 2.

[0040] A method for designing a multi-scale convolutional network and a single-plane diffraction element optical-digital collaborative system, the method comprising the following steps:

[0041] Step 1: design a single-plane diffraction element 1;

[0042] The single-plane diffraction element 1 includes a Fresnel surface and a diffraction surface.

[0043] The surface formula of the annular zone of the Fresnel surface is:

[0044]

[0045] Where z is the sagittal height at each point on the aspheric surface, c is the curvature at the vertex of the aspheric surface, r is the radial coordinate of each point on the aspheric surface, k is the conic constant, A, B, and C are the aspheric coefficients; n is the number of rings, and h is the sagittal height of the aspheric base.

[0046] The diffraction surface and the diffraction phase equation are determined by the following formula:

[0047] ,

[0048] Where M is the diffraction order, B1 and B2 are the diffraction surface coefficients, and ρ is the radial coordinate of the diffraction surface.

[0049] Step 2: construct a multi-scale convolutional network image restoration module 3;

[0050] like Figure 2 As shown, the multi-scale convolutional network image restoration module 3 first inputs the blurred image Bi and extracts primary features through the input block, enters the encoder 1 and the encoder 2, and downsamples step by step to extract multi-scale features from low resolution to high resolution. After the lowest resolution branch extracts the global blurred features, the spatial resolution is restored step by step through the bilinear upsampling decoder 1 and the decoder 2. Each level of the decoder combines the jump connection features of the corresponding scale to perform feature fusion and reconstruction, and finally generates a clear image Ii by the output block.

[0051] like Figure 3 As shown, the two encoders in the multi-scale convolutional network image restoration module 3 have the same structure. Each level first contains a convolution unit consisting of a convolution layer and an activation function, which is used to initially extract local features at the current scale. Three residual blocks are then connected. Each residual block consists of a convolution layer 1, an activation function, and a convolution layer 2. The input and output are added via skip connections to achieve efficient feature expression and information preservation. Through this multi-level structure, the encoder extracts deep fuzzy features from high to low resolution. In particular, the feature map of the lowest resolution layer helps capture global context information.

[0052] like Figure 4 As shown, the two decoders in the multi-scale convolutional network image restoration module 3 have the same structure. Each stage first restores the structural details of the coarse-scale features transmitted from the previous stage using three residual blocks. Each residual block also consists of a convolutional layer 1, an activation function, and a convolutional layer 2. A deconvolution layer and activation function are then used to spatially upsample the feature map to improve resolution. Each decoding stage is also fused with the intermediate feature map of the corresponding scale encoder via skip connections, combining local and global information and fully preserving multi-scale contextual features. Ultimately, through step-by-step decoding and fusion, a high-quality image with clear structure and rich details is restored.

[0053] Step three, construct the loss function of the image restoration method.

[0054] The loss function of the multi-scale convolutional network image restoration method is determined by the following formula:

[0055] ,

[0056] In the formula is the scale weight, is the weight of each channel, and They represent the predicted value and target value of channel c at scale s, is the VGG perceptual loss weight, and They are the predicted value and target value of 256-scale image perceptual loss respectively.

[0057] The multi-scale convolutional network loss function channel weights Determined by the following formula:

[0058] ,

[0059] In the formula is the normalized diffraction efficiency coefficient of channel c, is the normalized wavefront aberration coefficient of each channel, is the normalized actual human eye color perception sensitivity. is the wavefront aberration coefficient of each channel, and the weight is allocated proportionally according to the size of the wavefront aberration; is the diffraction efficiency of each channel.

[0060] The multi-scale convolutional network loss function is the normalized diffraction efficiency of each channel Determined by the following formula:

[0061] ,

[0062] In the formula, the B channel with the lowest diffraction efficiency is taken as a reference and its normalized weight is 1, and the weights of other channels are calculated.

[0063] Example:

[0064] Step 1: Design a single-plane diffraction element 1:

[0065] The single-plane diffractive optical element operates in the visible light band, has a focal length of 50mm, an F number of 5, and a full field of view of 10°. The relevant parameters of the front and rear surfaces of the single-plane diffractive optical element are shown in Table 1 below:

[0066] Table 1

[0067] surface Radius of curvature thickness Material S1 14.272 1 E48R S2 unlimited 24 - S12 unlimited - -

[0068] The aspheric coefficients of each lens in this example are shown in Table 2:

[0069] Table 2

[0070] surface A B C S1 -4.3E-5 -6.56E-7 -1.70E-9

[0071] In this example, the diffraction surface coefficient of the diffraction surface S12 in the single-plane diffraction element 1 is B1=-182.5.

[0072] Step 2: Construct a multi-scale convolutional network image restoration module 3:

[0073] By building a multi-scale convolutional network consisting of an encoder, a decoder and cross-scale jump connections, the structural design and initialization of the image restoration module 3 are completed. The multi-scale convolutional network image restoration module structure in the embodiment is as follows: Figure 2 As shown, its specific structure is as follows:

[0074] The input blurred image Bi first extracts primary features through the input block, enters encoder 1 and encoder 2, and downsamples step by step to extract multi-scale features from low resolution to high resolution. After the lowest resolution branch extracts the global blurred features, the spatial resolution is restored step by step through bilinear upsampling decoder 1 and decoder 2. Each level of decoder combines the jump connection features of the corresponding scale for feature fusion and reconstruction, and finally generates a clear image Ii from the output block.

[0075] like Figure 3 As shown in the figure, the two encoders have the same structure. Each level first contains a convolutional unit consisting of a convolutional layer and an activation function, which is used to initially extract local features at the current scale. Three residual blocks are then connected. Each residual block consists of a convolutional layer 1, an activation function, and a convolutional layer 2. The input and output are added via skip connections to achieve efficient feature expression and information preservation. The encoder extracts deep fuzzy features from high resolution to low resolution through a multi-level structure. In particular, the feature map of the lowest resolution layer helps capture global context information.

[0076] like Figure 4 As shown, the two decoders have the same architecture. Each stage first recovers structural details from the coarse-scale features passed from the previous stage using three residual blocks. Each residual block also consists of a convolutional layer (1), an activation function, and a convolutional layer (2). A deconvolutional layer and activation function are then used to spatially upsample the feature maps to increase resolution. Each decoding stage is also fused with the intermediate feature maps from the corresponding scale encoder via skip connections, combining local and global information and fully preserving multi-scale contextual features. Ultimately, through progressive decoding and fusion, a high-quality image with clear structure and rich details is restored.

[0077] Step 3: Construct the loss function of the image restoration method:

[0078] The multi-scale convolutional network in the embodiment processes step by step from low scale to high scale, and realizes image deblurring through multi-stage progressive optimization, wherein the weights of each scale are 0.4, 0.3, and 0.3 respectively.

[0079] The full-field wavefront aberration of the RGB channel of the single-plane diffraction element in the embodiment is as follows: Figure 5 As shown in Figure 5, 5(a), 5(b) and 5(c) are the RGB three-channel wavefront aberration images respectively, and the RGB channel wavefront aberration MSE loss weights are shown in Table 3:

[0080] Table 3

[0081] R G B Wavefront aberration 0.95λ 0.71λ 1.32λ MSE coefficient 0.29 0.23 0.48

[0082] In this example, the diffraction efficiency of the RGB channel of the single-plane diffraction element is as follows: Figure 4 As shown, the diffraction efficiency MSE loss weights of the RGB channels are shown in Table 4.

[0083] Table 4

[0084] R G B Diffraction efficiency 84.25% 90.75% 63.27% MSE coefficient 0.34 0.295 0.365

[0085] In this example, the MSE loss weights of the RGB channels for comprehensive wavefront aberration and diffraction efficiency are shown in Table 5:

[0086] Table 5

[0087] R G B MSE coefficient 0.3 0.45 0.25

[0088] like Figure 7 As shown, Figure 7 (a) is the blurred image before restoration, Figure 7 (b) is the clear image after restoration. The image clarity is significantly improved after restoration by the multi-scale convolutional network.

[0089] The images restored by the multi-scale convolutional network are evaluated using PSNR and SSIM. The evaluation results are shown in Table 6.

[0090] Table 6

[0091] Evaluation method PSNR SSIM Fuzzy Image 15.5136 0.6190 Traditional PSF restoration image 21.2779 0.7343 Multi-scale convolutional neural networks 28.1382 0.8918

Claims

1. A design method for a multi-scale convolutional network and single-plane diffraction element optical-digital collaborative system, characterized by: The method comprises the following steps: Step 1: design a single-plane diffraction element; Step 2: Construct a multi-scale convolutional network image restoration module; Step 3: Construct the loss function of the image restoration method; The single-plane diffraction element includes a Fresnel surface and a diffraction surface; The surface formula of the annular zone of the Fresnel surface is: Where z is the sagittal height at each point on the aspheric surface, c is the curvature at the vertex of the aspheric surface, r is the radial coordinate of each point on the aspheric surface, k is the conic constant, A, B, and C are the aspheric coefficients; n is the number of rings, and h is the sagittal height of the aspheric base; The diffraction surface and the diffraction phase equation are determined by the following formula: Φ=M(B1ρ 2 +B2p 4 ), Where M is the diffraction order, B1 and B2 are the diffraction surface coefficients, and ρ is the radial coordinate of the diffraction surface; The loss function of the multi-scale convolutional network image restoration method is determined by the following formula: Where α s is the scale weight, β c is the weight of each channel, and They represent the predicted value and target value of channel c at scale s, γ is the VGG perceptual loss weight, P 256 With T 256 are the predicted and target values ​​of the 256-scale image perceptual loss; The multi-scale convolutional network loss function has the channel weight β c Determined by the following formula: β c =(η cN ·W cN ·CPS cN ) N , Where η cN is the normalized diffraction efficiency coefficient of channel c, W cN is the normalized wavefront aberration coefficient of each channel, CPS cN is the normalized actual human eye color perception sensitivity; where W c is the wavefront aberration coefficient of each channel, and the weight is proportionally assigned according to the size of the wavefront aberration; η c is the diffraction efficiency of each channel; The multi-scale convolutional network loss function is the normalized diffraction efficiency η of each channel cN Determined by the following formula: In the formula, the B channel with the lowest diffraction efficiency is taken as a reference and its normalized weight is 1, and the weights of other channels are calculated.

2. The method for designing a multi-scale convolutional network and single-plane diffraction element optical-digital collaborative system according to claim 1, characterized in that: The multi-scale convolutional network image restoration module first inputs the blurred image Bi and extracts primary features through the input block, enters encoder 1 and encoder 2, downsamples step by step to extract multi-scale features from low resolution to high resolution, and after the lowest resolution branch extracts the global blurred features, the spatial resolution is restored step by step through bilinear upsampling decoder 1 and decoder 2. Each level of decoder combines the jump connection features of the corresponding scale to perform feature fusion and reconstruction, and finally generates a clear image Ii from the output block.

3. The method for designing a multi-scale convolutional network and single-plane diffraction element optical-digital collaborative system according to claim 2, characterized in that: The two encoders in the multi-scale convolutional network image restoration module have the same structure. Each level first contains a convolution unit composed of a convolution layer and an activation function, which is used to preliminarily extract local features of the current scale. Three residual blocks are then connected. Each residual block consists of a convolutional layer one, an activation function, and a convolutional layer two. The input and output are added through a jump connection to achieve efficient feature expression and information retention. The encoder extracts deep fuzzy features from high resolution to low resolution through a multi-level structure, especially the feature map of the lowest resolution layer helps to capture global context information.

4. The method for designing a multi-scale convolutional network and single-plane diffraction element optical-digital collaborative system according to claim 2, wherein: The two decoders in the multi-scale convolutional network image restoration module have the same structure. Each level first restores the structural details of the coarse-scale features transmitted from the previous level through three residual blocks. Each residual block is also composed of a convolutional layer 1, an activation function, and a convolutional layer 2. A deconvolution layer and an activation function are then used to spatially upsample the feature map to improve the resolution. Each decoding stage is also fused with the intermediate feature map of the corresponding scale encoder through a jump connection to achieve a combination of local and global information and fully preserve multi-scale contextual features. Finally, through step-by-step decoding and fusion, a high-quality image with clear structure and rich details is restored.

Images