Method for jointly optimizing metasurface-based color imaging system by using shunt mode and image reconstruction

Abstract

The invention provides a method for jointly optimizing a color imaging system based on a metasurface by using a shunt mode and image reconstruction. The method comprises the following steps of: establishing a micro mathematical model of an image sensor based on a shunt; establishing a deep neural network capable of simultaneously processing a plurality of image processing tasks to realize image reconstruction; the image model based on the splitter is realized as a shunt network; and the parameters of the shunt network and the reconstruction network are jointly trained to realize automatic optimization of the shunt mode. According to the invention, automatic optimization of the shunt mode can be realized, and the overall efficiency and accuracy of image sensing and reconstruction are improved.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing, and in particular to a method for jointly optimizing a color imaging system based on a metasurface using a branching mode and image reconstruction. Background Art

[0002] Challenging applications for high-sensitivity image sensors include high-resolution imaging, shooting in low-light environments, and ultra-high-speed imaging, where the amount of incident light received by each sensor pixel is extremely limited. Conventional color image sensors usually use color filters to distinguish colors, but the inherent drawback of this approach is that color filters attenuate the intensity of the incident light, resulting in its light sensitivity not meeting the requirements of demanding applications. To overcome this challenge, the latest technological progress is to replace color filters with metasurface-based color distributors. Metasurfaces are planar optical elements composed of subwavelength-sized structures. By carefully designing and arranging these structures, metasurfaces can control key properties of light waves, such as amplitude and phase, at specific frequencies with subwavelength spatial resolution. Therefore, metasurfaces can act as color separators, flexibly directing primary color light (red, green, and blue) at a certain location to specific sensor pixels at different locations. Image sensors using this metasurface-based distributor are expected to have light intensity sensitivity that is about three times higher than that of sensors using color filters, thus showing great potential in the above-mentioned application areas.

[0003] The design process of a metasurface-based splitter can be divided into three main steps. The first step is to determine the splitter pattern, which involves planning a set of splitters and their corresponding focus layouts for different colors. The second step is to calculate the phase profile based on the established focus. The third step is to optimize the subwavelength structure of the metasurface to approximate the required phase profile. To date, most research efforts have focused on the latter two steps. The arrangement of the subwavelength structure can be performed using a variety of optimization techniques, such as gradient descent, adjoint optimization, and deep learning methods. However, previous studies have not explored the first step, the design of the splitter pattern, in depth. Therefore, the design principles are not clear, and existing models lack sufficient analysis and evaluation. This issue cannot be ignored because the design of the splitter pattern is the basis for subsequent steps and it determines the limit of theoretical performance. Summary of the invention

[0004] In order to solve at least one problem of the prior art, the present disclosure provides a method for jointly optimizing a metasurface-based color imaging system using branching mode and image reconstruction.

[0005] The method disclosed herein comprises the following steps:

[0006] Establish a differentiable mathematical model of splitter-based image sensor;

[0007] Build deep neural networks that can handle multiple image processing tasks simultaneously to achieve image reconstruction;

[0008] Implementing the splitter-based image model as a splitter network;

[0009] The parameters of the branching network and the reconstruction network are jointly trained to achieve automatic optimization of the branching mode.

[0010] In some embodiments, the splitter distributes the color information of the input signal to specific sensor pixels. The construction of the differentiable mathematical model of the image sensor based on the splitter adopts a sampling operator, which can retain the value of a specific position and set the value of other positions to zero, thereby distributing the color information of a single position to designated sensor pixels at different positions.

[0011] In some embodiments, a differentiable mathematical model of a splitter-based image sensor is established, and the system imaging process can be modeled as follows:

[0012]

[0013] S represents the set split, g∈R H×W Indicates the original image detected, f∈R H×W×3 Represents color image detection, n∈R H ×W represents noise, ★ represents transposed convolution, ↓ s and ω s ∈R M×N×3 Represents the sampling operator and parameters of the transposed convolution s-th splitter.

[0014] In some embodiments, an equivalent form of the model is:

[0015]

[0016] Among them, S represents the color set in the original image, * represents the convolution operator, ↓ c and k c ∈R M×N×3 Represents the sampling operator and convolution kernel of the c-th color in the original image.

[0017] Although the two mathematical model forms mentioned above are equivalent in mathematical expressions, they each emphasize different physical processes. The first form focuses on how light information is transferred through each splitter. To reflect this, the model can be implemented in the form of a split network, with the aim of improving system performance by jointly optimizing the split pattern and image reconstruction method. In contrast, the second form focuses on describing the accumulation process of light information on each sensor pixel. This statement is more suitable for analyzing the quality of the original image captured by the detector sensor. It reveals some problems with the detected original image, such as ambiguity between spatial positions or color channels, incomplete sampling, and the image is affected by noise. Through this form, we can have a deeper understanding of the limitations and degradation phenomena that the sensor may encounter when capturing images.

[0018] In some embodiments, establishing a deep neural network comprises the following steps:

[0019] Perform color sampling on the original image, and import each color light into different channels. For the non-existent color information, use zero value to fill it.

[0020] After sampling, the sampled tensor will be processed through a layer of convolution, a CBAM attention module, ten residual blocks, and another layer of convolution.

[0021] A CBAM attention module is used to generate the final output image.

[0022] In addition, a long skip connection is included in the network structure, which connects the output of the first convolutional layer and the input of the last convolutional layer to enhance the learning ability and feature transfer of the network.

[0023] In some embodiments, the optimizer and loss function used to train the deep neural network are the Adam algorithm, and the mean square error (MSE) function is:

[0024]

[0025] T is the number of training images, ||·|| is the L2 norm, D(g t ; θ) represents the reconstruction network, and the parameter is Θ.

[0026] In some embodiments, the imaging model g=∑ s∈S (f↓ s ★ω s )+n is implemented as a branch network g=ε(.;ω), and the network is substituted into the loss function In g, and ω is set as the optimization variable, the loss function is modified as follows:

[0027] In some embodiments, the steps include:

[0028] Apply the softmax function T(·) on the auxiliary variable zs to obtain ωs, that is: ωs = T(zs). The final imaging model and the loss function for joint optimization are:

[0029]

[0030]

[0031] Initialize z using the branching mode and use the Adam optimizer to optimize this loss function to achieve automatic optimization of the branching mode.

[0032] The benefits of the present disclosure include: proposing two equivalent differentiable mathematical models of splitter-based image sensors. The first model aims to construct a split network to improve system performance by jointly optimizing the split mode and image reconstruction method. The second model reveals the noise interference, sampling incompleteness and spatial ambiguity of the detected original image. In addition, the present invention develops a deep neural network for image reconstruction and conducts a detailed evaluation of the five existing split modes. Based on these evaluation results, three split mode design principles are proposed. In addition, the present disclosure implements the proposed differentiable mathematical model through a split network and connects the network with the image reconstruction network. By jointly training the two networks, automatic optimization of the split mode is achieved, and the overall efficiency and accuracy of image sensing and reconstruction are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 An illustration of a mathematical model of a splitter-based image sensor according to some examples of the present disclosure is shown.

[0034] Figure 2 The structure diagram of the reconstruction network described in some examples of the present disclosure is illustrated.

[0035] Figure 3 Illustrations of some example splitter mode evaluation experiments of the present disclosure are illustrated.

[0036] Figure 4 Some exemplary joint optimization experiments of the present disclosure are illustrated.

[0037] Figure 5 Visual results of the splitter pattern evaluation experiment for a zoomed region of image kodim05 at a noise level ≤10 for the five splitter patterns.

[0038] Figure 6Visual results of the splitter mode evaluation experiment on a zoomed region of the image kodim24 at noise level 5 for the five splitter modes.

[0039] Figure 7 The visual effects of mode A and optimized modes of some examples of the present disclosure on the enlarged area of ​​image kodim05 are illustrated.

[0040] Figure 8 The visual effects of some exemplary modes D and optimized modes of the present disclosure on the enlarged area of ​​the image kodim24 are illustrated. DETAILED DESCRIPTION

[0041] The following embodiments of the technical solution of the present application are described in detail in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present application, and are therefore only used as examples, and cannot be used to limit the scope of protection of the present application.

[0042] In some examples of the present disclosure, a method for jointly optimizing a metasurface-based color imaging system using a branching pattern and image reconstruction includes the following steps:

[0043] S1. Establish a differentiable mathematical model of the image sensor based on the splitter;

[0044] S2. Build a deep neural network that can handle multiple image processing tasks simultaneously to achieve image reconstruction;

[0045] S3, implementing the splitter-based image model as a split network;

[0046] S4. Jointly train the parameters of the branch network and the reconstruction network to achieve automatic optimization of the branch mode.

[0047] In step S1 , a differentiable mathematical model of a splitter-based image sensor is established.

[0048] Figure 1 The mathematical model of the image sensor based on the splitter in some examples of the present disclosure is illustrated. Please refer to Figure 1 , the splitter-based system imaging model proposed in the present disclosure can be simulated in two equivalent forms.

[0049] The first form of the mathematical model is:

[0050]

[0051] The focus of the model is how light information is transferred through each splitter. To reflect this, the present disclosure adopts a split network to implement this model, the purpose of which is to improve system performance by jointly optimizing the split mode and image reconstruction method. The model uses a transposed convolution operator for modeling.

[0052] The second form of the mathematical model is:

[0053]

[0054] The second form of the mathematical model focuses on describing the accumulation process of light information on each sensor pixel. This description is more suitable for analyzing the quality of the original image captured by the detector sensor. It reveals some problems with the detected original image, such as ambiguity between spatial positions or color channels, incomplete sampling, and the image is affected by noise. Through this form, we can have a deeper understanding of the limitations and degradation phenomena that the sensor may encounter when capturing images. The second form of the mathematical model uses the convolution operator for modeling.

[0055] In step S2, a deep neural network is developed that can simultaneously process multiple image processing tasks to achieve image reconstruction.

[0056] Figure 2 The deep neural network developed in some examples of the present disclosure that can simultaneously process multiple image processing tasks to achieve image reconstruction is shown. Figure 2 As shown in the figure, the reconstruction network first samples the color of the input original image, assigns different colors to their respective channels, and fills the non-existent color information with zero values. After sampling, the sampled tensor will undergo the following processing flow: first pass through a convolutional layer, followed by a CBAM attention module, then ten residual blocks, then another convolutional layer, and finally pass through a CBAM attention module. In addition, the network structure also includes a long skip connection, which connects the output of the first convolutional layer and the input of the last convolutional layer to enhance the network's learning ability and feature transfer. The optimizer and loss function used to train the network are the Adam algorithm, and the mean square error (MSE) function is:

[0057]

[0058] Where T is the number of training images, ||·|| is the L2 norm, and D(g t ; θ) represents the reconstruction network, and the parameter is Θ.

[0059] In step S3, the splitter-based imaging model is implemented as a splitter network.

[0060] In some examples, the imaging model g=∑ s∈S (f↓ s ★ω s )+n is implemented as a branch network g=ε(.;ω), and the network is substituted into the loss function In g, and let ω be the optimization variable, so that the loss function is modified as follows:

[0061]

[0062] In step S4, the parameters of the branching network and the reconstruction network are jointly trained to achieve automatic optimization of the branching mode.

[0063] In some examples, a softmax function T(·) is applied to the auxiliary variable zs to obtain ωs, that is: ωs = T(zs), so the final imaging model and the loss function for joint optimization are:

[0064]

[0065]

[0066] Use the branch mode to initialize z and the Adam optimizer to optimize this loss function, thereby achieving automatic optimization of the branch mode.

[0067] In order to verify the effectiveness of the present invention, this embodiment uses the reconstruction network to evaluate five existing modes. The five modes are:

[0068] Mode A: using a microchromatic separator (MiCS), which uses a micro-plate-like structure with a high refractive index to separate colors to induce a spectrum-dependent deflection;

[0069] Mode B: Uses a full-color sorting metalens for high-sensitivity image sensors. It is polarization-insensitive and can sort and classify primary colors on high-density pixels;

[0070] Mode C: using the optical path design method based on dual-focus phase distribution;

[0071] Mode D: Uses full-color optical routing for image sensor pixel level and enhances the signal and polarization insensitivity of primary color bands through a multi-objective optimization approach;

[0072] Mode E: High-sensitivity color imaging using a pixel-level color separator based on a dielectric metasurface.

[0073] First, a degraded original image is generated for each demultiplexing mode, and then the Figure 2 The reconstruction network estimates the reference image from each original image and finally evaluates each restored image. The visual evaluation of the five modes is shown in Figure 5 , Figure 6 shown.

[0074] This disclosure establishes three split-mode design principles from these evaluations:

[0075] (1) Avoid using sub-pixel structure. Sub-pixel structure refers to the design adopted by mode B and mode C. The characteristic of this structure is that the incident light at a set of positions is regarded as a unified body, and the different colors of these lights are assigned to the corresponding color sub-pixels. Table 1 shows the average PSNR (dB) and SSIM of the five splitter modes on Kodak24 and McMaster when the noise level is 0, 5, 10, 15, and 20 in the embodiment of the present invention. Table 1 shows the average PSNR (dB) and SSIM of the five splitter modes on Kodak24 and McMaster when the noise level is 0, 5, 10, 15, and 20 in the embodiment of the present invention. As shown in Table 1, among all the compared modes, mode B and mode C perform the least well in terms of evaluation indicators. In addition, these two modes using sub-pixel structure often cause the appearance of false textures or produce severe blurring effects.

[0076] Table 1

[0077]

[0078] (2) Colors should be divided into areas as small as possible. In the modes that do not use a sub-pixel structure, blurring is almost unobservable in modes D and E, while some blurring occurs in mode A. The reason for this phenomenon is that the first two color separator modes divide colors into 1×2 or 1×3 pixel arrays at only a single location, while the latter divides colors into 3×3 pixel arrays. This result indicates that when designing the separator pattern, colors should be divided into areas as small as possible to reduce or eliminate potential blurring problems.

[0079] (3) The incident light should be completely separated. The color deviation phenomenon in mode D and mode E may be due to the fact that a large proportion of the incident light in these two splitter modes is not effectively split. In this case, in addition to the blurring effect between spatial positions, there is also a potential problem, namely the blurring effect between color channels. In order to more effectively separate color information during imaging, more beam splitters should be used to ensure that the incident light is completely separated. In this way, cross-interference between color channels can be reduced, thereby improving the color accuracy and clarity of the image.

[0080] In addition, in step 4 of this example, we successfully achieved automatic optimization of the splitting mode by jointly training the parameters of the splitting network and the reconstruction network. This method was applied to two existing splitter designs. In both cases, the optimized splitter mode showed better performance than the original mode. This result fully verified the effectiveness and practicality of the joint optimization method proposed in the present invention. Through this optimization process, we improved the design quality of the splitter and enhanced the performance of the entire imaging system.

[0081] Optimization of Mode A. Table 2 shows the average PSNR (dB) and SSIM of Mode A and its optimized modes on Kodak24 and McMaster when the noise level is 0, 5, 10, 15, and 20. As shown in Table 2, in all cases, the optimized mode is better than the original mode in terms of PSNR and SSIM. The improvement of PSNR can reach 0.78dB. In addition, Figure 7 As shown, the optimized pattern successfully solves the blurring problem of the original pattern.

[0082] Table 2

[0083]

[0084] Optimization of mode D. Table 3 shows the average PSNR (dB) and SSIM of mode D and its optimized mode on Kodak24 and McMaster when the noise level is 0, 5, 10, 15, and 20. As shown in Table 3, in all cases, the optimized mode is better than the original mode in terms of PSNR and SSIM. The improvement of PSNR can reach 0.75dB. In addition, Figure 8 As shown, the optimized pattern can correct the false color caused by the original pattern.

[0085] Table 3

[0086]

Claims

1. A method for jointly optimizing a metasurface-based color imaging system using a split-path mode and image reconstruction, comprising the following steps: Establish a differentiable mathematical model of splitter-based image sensor; Build deep neural networks that can handle multiple image processing tasks simultaneously to achieve image reconstruction; Implementing the splitter-based image model as a splitter network; The parameters of the branching network and the reconstruction network are jointly trained to achieve automatic optimization of the branching mode.

2. The method for jointly optimizing a color imaging system based on a metasurface using a branching mode and image reconstruction according to claim 1, characterized in that: The splitter distributes the color information of the input signal to specific sensor pixels.

3. The method for jointly optimizing a color imaging system based on a metasurface using a branching mode and image reconstruction according to claim 1, characterized in that: A differentiable mathematical model of the image sensor based on the splitter is established, and the system imaging process can be modeled as follows: S represents the set split, g∈R H×W Indicates the original image detected, f∈R H×W×3 Represents color image detection, n∈R H×W represents noise, ★ represents transposed convolution, ↓ s and ω s ∈R M×N×3 Represents the sampling operator and parameters of the transposed convolution s-th splitter.

4. The method for jointly optimizing a color imaging system based on a metasurface using a branching mode and image reconstruction according to claim 3, characterized in that: The equivalent form of the model is: Among them, S represents the color set in the original image, * represents the convolution operator, ↓ c and k c ∈R M×N×3 Represents the sampling operator and convolution kernel of the c-th color in the original image.

5. The method for jointly optimizing a color imaging system based on a metasurface using a branching mode and image reconstruction according to claim 1, characterized in that: Building a deep neural network involves the following steps: Perform color sampling on the original image, and import each color light into different channels. For the non-existent color information, use zero value to fill it. After sampling, the sampled tensor will be processed through a layer of convolution, a CBAM attention module, ten residual blocks, and another layer of convolution. A CBAM attention module is used to generate the final output image.

6. The method for jointly optimizing a color imaging system based on a metasurface using a branching mode and image reconstruction according to claim 5, characterized in that: The optimizer and loss function used to train the deep neural network are the Adam algorithm, and the mean square error (MSE) function is: T is the number of training images, ||·|| is the L2 norm, D(g t ; θ) represents the reconstruction network, and the parameter is Θ.

7. The method for jointly optimizing a color imaging system based on a metasurface using a branching mode and image reconstruction according to claim 1, characterized in that: The imaging model g = ∑ s∈S (f↓ s ★ω s )+n is implemented as a branch network g=ε(.;ω), and the network is substituted into the loss function In g, and ω is set as the optimization variable, the loss function is modified as follows:

8. The method for jointly optimizing a color imaging system based on a metasurface using a branching mode and image reconstruction according to claim 1, characterized in that: The following steps are involved: Apply the softmax function T(·) on the auxiliary variable zs to obtain ωs, that is: ωs = T(zs). The final imaging model and the loss function for joint optimization are: Initialize z using the branching mode and use the Adam optimizer to optimize this loss function to achieve automatic optimization of the branching mode.

Images