Self-encoding and decoding spectrum reconstruction method based on neural network

CN118913445AInactive Publication Date: 2024-11-08HARBIN INST OF TECH
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
CN202411137991.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-19
Publication Date
2024-11-08
Estimated Expiration
Not applicable · inactive patent

Smart Images

  • Figure CN118913445A_ABST
    Figure CN118913445A_ABST
Patent Text Reader

Abstract

The invention discloses a self-encoding and decoding spectrum reconstruction method based on a neural network, and the method comprises the following steps: 1, carrying out the pre-training of a constraint encoding neural network, so as to learn the relation between a super-structure lens parameter and a spectrum response, and providing a weight constrained by a structure parameter for the encoding neural network; 2, the random spectrum coding neural network is used for carrying out random coding on the input spectrum and providing input for decoding and reconstructing the neural network; and step 3, the spectrum reconstruction neural network is used for decoding the input encoded spectrum and reconstructing the spectrum with high precision. According to the method, the optimal coding filter parameter can be found through pre-training, and the spectrum can be reconstructed with high precision only by using a few filters, so that the self-coding and decoding algorithm has higher precision in the spectrum reconstruction process, accords with a super-structure lens production process, and reduces the use cost.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to a spectrum reconstruction method, in particular to a self-encoding and decoding spectrum reconstruction method based on a neural network. Background Art

[0002] Spectral reconstruction algorithms are a class of algorithms used to infer a complete spectrum from partially known spectral data. These algorithms usually use mathematical and statistical methods, as well as prior knowledge of spectral data, to infer and reconstruct the spectrum.

[0003] The interpolation method is one of the simplest and most direct spectral reconstruction algorithms. It infers the values ​​of unknown data points by linear or nonlinear interpolation between known data points. Common interpolation methods include linear interpolation, polynomial interpolation (such as Lagrange interpolation, Newton interpolation), spline interpolation, etc. The above methods are usually suitable for cases where data points are dense, but may not perform well when the data is sparse or noisy. The compressed sensing method uses the sparsity or low rank of the signal to reconstruct the signal. In spectral reconstruction, the compressed sensing method measures the spectrum in a low-dimensional space and then reconstructs the complete spectrum using a sparse representation or a low-rank matrix. Common compressed sensing methods include sparse representation-based methods (such as sparse representation reconstruction, compressed sensing reconstruction) and low-rank matrix-based methods (such as matrix completion). There are also optimization-based spectral reconstruction methods, which transform the spectral reconstruction problem into an optimization problem by defining an optimization problem, and use a numerical optimization algorithm to solve the problem. Common optimization methods include gradient-based methods, model-based methods, evolutionary algorithms, etc. These methods usually require the definition of a suitable loss function or regularization term to balance data fitting and model complexity. The spectral reconstruction method based on machine learning uses a machine learning model to learn the mapping relationship from partial spectral data to the complete spectrum. It usually requires a large amount of known spectral data to train the model, and can use various machine learning models, such as neural networks, support vector machines, random forests, etc. After the training is completed, the model can be used to reconstruct new partial spectral data.

[0004] Metasurface is a type of artificial two-dimensional structure with a subwavelength structure that can achieve flexible control of the phase and frequency of the light field. At the same time, metasurface has the advantages of miniaturization and integration, which provides a new idea for solving the color filtering and integration problems of image sensors. Summary of the invention

[0005] In order to improve the accuracy of spectrum reconstruction, the present invention provides a self-encoding and decoding spectrum reconstruction method based on a neural network. This method can find the optimal encoding filter parameters through pre-training, and can reconstruct the spectrum with high accuracy using only a few filters, so that the self-encoding and decoding algorithm has higher accuracy in the spectrum reconstruction process, and complies with the production process of the meta-lens, reducing the cost of use.

[0006] The objective of the present invention is achieved through the following technical solutions:

[0007] A self-encoding and decoding spectrum reconstruction method based on a neural network comprises the following steps:

[0008] Step 1: pre-training a constrained encoding neural network to learn the relationship between the meta-lens parameters and the spectral response, and providing weights constrained by the structural parameters for the encoding neural network;

[0009] Step 2: A random spectrum encoding neural network is used to randomly encode the input spectrum to provide input for the decoding and reconstruction neural network;

[0010] Step 3: Spectral reconstruction neural network is used to decode the input encoded spectrum and reconstruct the spectrum with high precision.

[0011] Compared with the prior art, the present invention has the following advantages:

[0012] 1. In the process of training the neural network, the present invention optimizes the parameters of the filter (coded meta-lens) with the best performance, ensuring the use of the optimal structure, reducing the filtering channels and achieving higher accuracy.

[0013] 2. While reconstructing the spectrum, the present invention designs the parameters of the optimal meta-lens to reduce the complexity of the system to alleviate the manufacturing difficulty. The meta-lens processing technology is compatible with the mature complementary metal semiconductor sensor preparation technology and can be processed and assembled on the same production line. Therefore, this method has a strong application background. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is the overall flow chart of the self-encoding and decoding spectral reconstruction algorithm based on neural network.

[0015] Figure 2 It is the result of reconstructing the spectral information of the incident light using a neural network.

[0016] Figure 3 It is a pseudo-color image of the spectrum reconstruction of the ordinary camera photo of the real scene and the spectrum image obtained by the spectral camera.

[0017] Figure 4 It is the calculated spectral reconstruction pseudo-color image of the real scene and the calculated spectral response curve reconstruction result of the characteristic pixel points in the picture. DETAILED DESCRIPTION

[0018] The technical solution of the present invention is further described below in conjunction with the accompanying drawings, but is not limited thereto. Any modification or equivalent replacement of the technical solution of the present invention without departing from the spirit and scope of the technical solution of the present invention should be included in the protection scope of the present invention.

[0019] The present invention provides a self-encoding and decoding spectrum reconstruction method based on a neural network, the method includes a pre-trained constraint encoding neural network, a random spectrum encoding neural network and a spectrum decoding and reconstruction neural network, so as to realize the function of spectrum reconstruction. Figure 1 As shown, the specific steps are as follows:

[0020] Step 1: Pre-train the constrained encoding neural network to learn the relationship between the meta-lens parameters and the spectral response, and provide the encoding neural network with weights constrained by the structural parameters. The specific steps are as follows:

[0021] The structural parameters of the meta-lens are D = [d 1 ,d 2 ,d 3 ,…,d n ] T As the neural network training input, the spectral response W is the output. After training, W = fm(D), and each column wi in the matrix W = [w i1 ,w i2 ,…,w im ] T , and the spectral response is normalized, which is convenient for training in random spectral encoding neural networks to produce convergence results, where: d 1 ,d 2 ,d 3 ,…,d n are the specific parameters of the meta-lens unit structure, such as length, width, height, spacing or rotation angle, etc.; fm function is the forward spectral response prediction function; w i1 ,w i2 ,…,w im is the spectral transmittance of each wavelength of the filter channel obtained; i is the number of filter channels.

[0022] In this step, the constraints of the pre-trained neural network include: adding a regularization constraint term L(D) to the neural network during the training process, ensuring that the structural parameters are within the manufacturing process range during training, and reducing production costs.

[0023] In this step, the network training process can directly obtain the required metalens parameters at the end, which is convenient for aligning the production process and verifying the feasibility.

[0024] In this step, the purpose of training is to optimize the structural parameter D and the decoding neural network parameter θ to satisfy the equation given the number of filter channels i:

[0025]

[0026] Step 2: Random spectral encoding neural network is used to randomly encode the input spectrum to provide input for the decoding and reconstruction neural network. The specific steps are as follows:

[0027] A series of actual discrete spectra S = [s 1 ,s 2 ,…,s m ] T As training input, the spectrum is encoded with the spectral response W of the pre-trained neural network as the weight, and the encoded spectrum Y = W S is passed as output to the spectral decoding neural network to provide input for the decoding and reconstruction neural network, where: 1 ,s 2 ,…,s m is the spectral intensity distribution of each wavelength.

[0028] Step 3: Spectral reconstruction neural network is used to decode the input encoded spectrum and reconstruct the spectrum with high precision. The specific steps are as follows:

[0029] The encoded spectrum Y is taken as input and the original spectrum S is taken as output. After training, S=fn(Y), where the fn function is the forward spectrum decoding prediction function. Finally, the encoded spectrum is decoded and the spectrum is reconstructed.

[0030] In the present invention, the pre-trained constraint coding neural network in step 1 is a DNN, including an input layer, a hidden layer, and an output layer; the random spectrum coding neural network in step 2 is a single hidden layer neural network, specifically a fully connected layer, with a weight of W and a bias of b=0, and the forward coding process of the neural network is consistent with the compressed sensing algorithm, where the spectrum S is obtained through i filtering channels to obtain the encoded spectrum Y; the spectrum reconstruction neural network in step 3 is a DNN. The specific parameters of the above three-layer neural network include activation function, number of hidden layers, and number of neurons, all of which need to be optimized according to specific data to be confirmed.

[0031] Figure 2 The figure shows the reconstruction result of the incident light spectrum information calculated using a neural network. The result is calculated using a spectrum modulation unit composed of four metasurface unit structures. Some random input spectra are encoded by the metasurface and decoded by the neural network algorithm. The reconstructed spectral response curve is compared with the spectral response curve carried by the incident light. In order to quantify the overlap between them, the concept of fidelity is introduced, as shown in Figure 1. Figure 2The reconstruction results shown all have a large spectral reconstruction fidelity, which can prove that the spectral modulation unit and neural network algorithm designed in the present invention can effectively realize spectral reconstruction. I is the original spectrum, I' is the reconstructed spectrum, and the fidelity F(I,I') is defined as:

[0032]

[0033] Where n represents the number of spectral data samples.

[0034] Figure 3 Shown is a spectrally reconstructed pseudo-color image of a real scene photographed by an ordinary camera and a spectral image obtained by a spectral camera. The result is still calculated using a spectral modulation unit composed of four metasurface unit structures. A spectral camera is used to take pictures and decoded through neural network calculations. The reconstructed spatial spectral data set is converted into a pseudo-color spectral image and compared with the real picture taken by an ordinary camera. From the results, it can be seen that the pseudo-color picture obtained after reconstruction restores the texture in the picture well, including non-visible details hidden in the same color or shadows in ordinary photos.

[0035] Figure 4 The figure shows the calculated spectral reconstruction pseudo-color image of the real scene and the calculated spectral response curve reconstruction result of the characteristic pixel points in the picture. Selecting two characteristic pixel points, red and blue, to draw the spectral response curve, it can be seen that the reconstructed curve has a high degree of overlap with the original curve. The spectral response of the red pixel points is concentrated in the range of 650-750nm, and the spectral response of the blue pixel points is concentrated in the range of 450-500nm. The spectral response corresponds to the color, demonstrating the spectral reconstruction imaging effect of the spectral encoder and decoder of the present invention.

Claims

1. A self-encoding and decoding spectrum reconstruction method based on a neural network, characterized in that The method comprises the following steps: Step 1: pre-training a constrained encoding neural network to learn the relationship between the meta-lens parameters and the spectral response, and providing weights constrained by the structural parameters for the encoding neural network; Step 2: A random spectrum encoding neural network is used to randomly encode the input spectrum to provide input for the decoding and reconstruction neural network; Step 3: Spectral reconstruction neural network is used to decode the input encoded spectrum and reconstruct the spectrum with high precision.

2. The self-encoding and decoding spectrum reconstruction method based on a neural network according to claim 1 is characterized in that The specific steps of step 1 are as follows: using the structural parameters D of the meta-lens = [d1, d2, d3, ..., d n ] T As the neural network training input, the spectral response W is the output. After training, W = fm(D), and each column wi in the matrix W = [w i1 ,w i2 ,…,w im ] T , and the spectral response is normalized, where: d1, d2, d3, …, d n are the specific parameters of the meta-lens unit structure; fm function is the forward spectral response prediction function; w i1 ,w i2 ,…,w im is the spectral transmittance of each wavelength of the filter channel obtained; i is the number of filter channels.

3. The self-encoding and decoding spectrum reconstruction method based on neural network according to claim 2 is characterized in that During the training process, a regularization constraint term L(D) is added to the neural network.

4. The self-encoding and decoding spectrum reconstruction method based on neural network according to claim 2 is characterized in that The structural parameters D and the decoding neural network parameters θ are optimized to satisfy the equation 5. The self-encoding and decoding spectrum reconstruction method based on neural network according to claim 1 is characterized in that The specific steps of step 2 are as follows: a series of actual discrete spectra S = [s1, s2, ..., s m ] T As input, the spectrum is encoded with the spectral response W of the pre-trained neural network as the weight, and the encoded spectrum Y = W S is passed to the spectral decoding neural network as output to provide input for the decoding and reconstruction neural network, where: s1, s2, …, s m is the spectral intensity distribution of each wavelength.

6. The self-encoding and decoding spectrum reconstruction method based on a neural network according to claim 1 or 5, characterized in that The encoding neural network is a fully connected layer with a weight of W and a bias of b=0. The forward encoding process of the neural network is consistent with the compressed sensing algorithm, and the spectrum S is obtained by passing the i filtering channels to obtain the encoded spectrum Y.

7. The self-encoding and decoding spectrum reconstruction method based on neural network according to claim 1 is characterized in that The specific steps of step three are as follows: take the encoded spectrum Y as input and the original spectrum S as output, after training, S=fn(Y), the fn function is the forward spectrum decoding prediction function, and finally decode the encoded spectrum to reconstruct the spectrum.

Citation Information

Patent Citations

  • Image reconstruction and coding method in spectral tomography

    CN112270725A

  • Spectrum reconstruction method based on arrayed multi-peak-valley broadband optical filter and micro spectrometer

    CN114910166A

  • Calculation type spectrometer based on dispersion super lens and working method

    CN116878657A

  • Calculation type visible-near infrared spectrum imaging chip based on metasurface

    CN117490843A

  • Construction method of wide-spectrum integrated snapshot spectrometer based on metasurface

    CN117911555A