Method for realizing spectrum reconstruction based on spectrum physical calibration

Abstract

The invention discloses a method for realizing spectrum reconstruction based on spectrum physical calibration, and belongs to the technical field of infrared spectrum. The method comprises the following steps of S1, data acquisition, S2, spectrum-electricity joint data preprocessing, S3, spectrum reconstruction neural network construction, S4, model training, S5, multi-objective optimization and S6, model testing. In the constructed spectrum reconstruction neural network, the neural network is optimized by combining the spectrum physical calibration error and the multi-objective optimization algorithm, and the reconstruction precision and efficiency of the spectrum are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of infrared spectroscopy, and particularly relates to a method for spectral reconstruction based on spectral physical calibration, which is applicable to the precise calibration of devices such as spectrometers and photodetectors. Background Art

[0002] Near-infrared spectroscopy technology developed in the 1950s and has made remarkable progress with the rapid development of computer technology. Its analysis speed is extremely fast, and it can complete the determination of multiple performance indicators in a short time, greatly improving work efficiency. At the same time, the entire analysis process does not require complex pretreatment of samples and will not damage the samples, truly realizing non-destructive testing. In addition, a major highlight of this technology is that it can simultaneously measure multiple components in the same mode, greatly simplifying the measurement operation process, reducing labor and time costs, and being widely used in fields such as material analysis, environmental monitoring, and biomedicine. In order to obtain better spectral data, physical calibration of spectral instruments and design of spectral reconstruction algorithms are indispensable.

[0003] The existing physical calibration of spectral devices mainly uses commercial standard spectrometers and lasers for point-by-point scanning to generate a responsivity function / matrix. And traditional spectral reconstruction is basically based on the compressed sensing theory to recover the original spectral signal from a small number of measurement values. Its reconstruction result depends to a large extent on the design of the measurement matrix and the performance of the adopted reconstruction algorithm. However, when the signal sparsity is not obvious, it will seriously affect the reconstruction accuracy. With the continuous development of deep learning, spectral reconstruction algorithms combined with deep learning have gradually emerged. Spectral reconstruction using deep learning is achieved by leveraging the powerful non-linear fitting ability of deep neural networks. The deep learning model can automatically learn complex features and mapping relationships from a large amount of spectral data without making too many assumptions about the characteristics such as the sparsity of the signal in advance. And most deep learning-based spectral reconstruction algorithms usually use data such as the known photocurrent sequence Eij, the wavelength corresponding responsivity Aap, and the device bias V as inputs and the light intensity value as the output.

[0004] The accuracy of traditional deep learning-based spectral reconstruction is not only limited by the data volume and model characteristics, but also by the physical errors in device calibration. And to reduce physical errors, more rigorous physical calibration and better devices are required, which is costly. Summary of the Invention

[0005] In order to improve the accuracy of spectral reconstruction and reduce the calibration cost, the present invention proposes a method for spectral reconstruction based on spectral physical calibration. In the spectral reconstruction neural network constructed by the present invention, the neural network is optimized by combining spectral physical calibration errors and multi-objective optimization algorithms, improving the accuracy and efficiency of spectral reconstruction.

[0006] The technical solution adopted by the present invention is as follows:

[0007] A method for spectral reconstruction based on spectral physical calibration, comprising the following steps:

[0008] S1. Data acquisition:

[0009] Collect and record the photocurrent sequence Eij, device bias voltage V, responsivity A, and the manually detected light intensity value I at different wavelengths, and generate an original data set.

[0010] S2. Spectral-electrical joint data preprocessing:

[0011] Perform logarithmic normalization processing and label data normalization processing on each item of data in the original data set to obtain the input data set of the neural network; among them, logarithmic normalization processing can solve the problem of small-signal quantization noise, and label data normalization processing can map the original spectral density data to the [0, 100] interval through linear transformation.

[0012] S3. Construction of spectral reconstruction neural network:

[0013] Construct a spectral reconstruction neural network with physical constraint embedding to achieve high-fidelity reconstruction of spectral features. The spectral reconstruction neural network includes a cascaded input layer, a residual block group, and an output layer;

[0014] The input data of the input layer is the photocurrent sequence Eij, device bias voltage V, responsivity A, and the manually detected light intensity value I;

[0015] The residual block group includes at least one dense residual block and at least one identity residual block, and the two are arranged alternately;

[0016] The dense residual block is composed of a stack of fully connected layers, batch normalization, and ReLU activation functions to achieve non-linear feature mapping;

[0017] The identity residual block is used to retain the original feature path and avoid gradient disappearance;

[0018] The output layer adopts a 2048-dimensional fully connected layer to perform high-resolution reconstruction on the data output by the residual block group and output the predicted light intensity value corresponding to the wavelength.

[0019] S4. Model training:

[0020] Divide the input data set obtained in step S2 into a training set and a test set; use the mean squared error (MSE) as the loss function, and use the training set to train the spectral reconstruction neural network. Optimize the model parameters of the spectral reconstruction neural network through the backpropagation algorithm to obtain a preliminarily trained spectral reconstruction neural network. In the initial stage of model training, only MSE is used as the loss function, which can minimize the difference between the predicted value and the true value and quickly fit the data.

[0021] S5. Multi-objective optimization:

[0022] Introduce a multi-objective optimization algorithm into the spectral reconstruction neural network after preliminary training is completed. By minimizing the difference between the model predicted value and the actual value, further optimize the parameters of the spectral reconstruction neural network and the sparsity and complexity of the model to obtain a trained spectral reconstruction neural network.

[0023] S6. Model testing:

[0024] Use the test set to test and verify the spectral reconstruction neural network. When the predicted light intensity value output by the spectral reconstruction neural network meets the error requirements, the spectral reconstruction neural network can be used for spectral reconstruction based on spectral physical calibration.

[0025] Preferably, in step S5, the objective function of the multi-objective optimization is:

[0026] ,

[0027] ,

[0028] ,

[0029] ,

[0030] where is the light intensity prediction error, is the light intensity prediction value of the spectral reconstruction neural network, is the light intensity value of the i-th channel detected manually, is the total number of channels;

[0031] Adopt L1 regularization as the sparsity regularization term. By minimizing L RE , force the predicted light intensity value of the spectral reconstruction neural network to be as sparse as possible, so as to better fit the physical characteristics of the true spectrum. At the same time, it can suppress non-physical solutions caused by noise (such as abnormal peaks / troughs), thereby preventing overfitting and improving the generalization ability of the spectral reconstruction neural network. represents the k-th weight in the spectral reconstruction neural network, and K represents the number of weights;

[0032] is the spectral physical calibration error, represents the responsivity of the i-th channel related to the wavelength ; represents the j-th light intensity value in the i-th channel related to the wavelength ; , respectively represent the lower wavelength limit and the upper wavelength limit of the laser; The spectral physical calibration results ( ) detected manually in the early stage are used in the spectral reconstruction neural network to measure the difference between the theoretical value of photocurrent (inverse spectral reconstruction value ) and the target actual value (the true value of the photocurrent output by the device ), making the spectral reconstruction neural network more representative and robust.

[0033] Preferably, in the objective function, the weight coefficients satisfy and .

[0034] Preferably, during the training process of the spectral reconstruction neural network, the weight coefficients are optimized according to the test set error and the Pareto solution distribution density .

[0035] Preferably, the optimization method of the weight coefficient is as follows:

[0036] S51. Initialize the population:

[0037] In the initialization stage, the Das-Dennis method of uniform sampling is used to generate 100 groups of weight coefficient combinations , avoiding the search bias caused by traditional fixed initial weights (such as [0.6, 0.2, 0.2]).

[0038] S52. Generate the Pareto solution set and evaluate the density:

[0039] Define , for each group of weight coefficient combinations , optimize the parameters of the spectral reconstruction neural network through backpropagation, and calculate three objective values: ;

[0040] Identify the Pareto front through fast non-dominated sorting. After calculating the normalized crowding distance of the Pareto solutions, arrange them in ascending order for each objective dimension, and then calculate the sum of the distances between adjacent Pareto solutions and normalize it to [0, 1]; and mark the Pareto solutions in the sparse region (the sparse region is selected as the top 20% of the crowding distance rankings).

[0041] S53. Density-driven dynamic adjustment of weights:

[0042] The weight adjustment formula is as follows:

[0043] ,

[0044] where represents the normalized crowding distance of the current p-th solution, , represent the population average crowding distance and standard deviation respectively, represents the adjustment factor;

[0045] For the solutions in the dense region: , reduce the weight λ1 of the dominant objective; for the solutions in the sparse region: , increase the weights λ2 and λ3.

[0046] S54. Genetic evolution and frontier update:

[0047] After updating the weight coefficients, perform simulated binary crossover and polynomial mutation operations to generate the offspring population; merge the historical solutions and offspring solutions through the environmental selection mechanism, and retain the individuals with a high non-dominated level and sparse distribution.

[0048] S55. Termination condition and optimal solution selection:

[0049] The termination condition is set to continuously monitor the convergence index of the Pareto frontier during the iteration process, such as the hypervolume growth rate;

[0050] The optimal solution is the Knee point density-performance balance optimal solution selected by comprehensively evaluating L DL 、L RE 、L AE on the training set.

[0051] The beneficial effects of the present invention are as follows:

[0052] 1. Improve the spectral resolution of the device: Through the spectral reconstruction method based on deep learning, the responsivity and photocurrent characteristics corresponding to more optical wavelength values are obtained, and it is no longer limited by the optical wavelength interval selected in the physical calibration process, thereby greatly improving the spectral resolution of the device.

[0053] 2. Improve the accuracy of the instrument: The spectral reconstruction neural network autonomously learns the non-linear mapping relationship between the photocurrent sequence and the light intensity value based on the physical calibration of the previous manual detection to perform spectral reconstruction, improving the accuracy of the instrument.

[0054] 3. Enhance the generalization ability of the spectral reconstruction neural network: A multi-objective optimization algorithm is introduced into the spectral reconstruction neural network to further optimize the neural network parameters, improving the generalization ability and robustness of the neural network.

[0055] 4. Reduce the calibration cost: It reduces the complexity and cost of manual calibration and improves the calibration efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 is a flowchart of the present invention;

[0057] Figure 2 is a weight optimization method;

[0058] Figure 3 is a comparative diagram of the relative error distribution between the method of the present invention and the traditional method;

[0059] Figure 4 is a comparative diagram of the training loss decline between the method of the present invention and the traditional method;

[0060] Figure 5 is a comparative diagram of the spectral reconstruction results between the method of the present invention and the traditional method in the wavelength range of 500nm - 900nm. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0061] The technical solution of the present invention will be further described below in conjunction with the drawings and specific embodiments.

[0062] This embodiment provides a method for spectral reconstruction based on spectral physical calibration, as Figure 1 shown, including the following steps:

[0063] S1. Data acquisition:

[0064] Collect and record the photocurrent sequence Eij, device bias voltage V, responsivity A, and the manually detected light intensity value I at different wavelengths (the wavelength range is 500nm - 900nm, the acquisition step size is 5nm, and there are 80 groups in total) to generate the original data set.

[0065] S2. Spectral - electrical joint data pre - processing:

[0066] Perform logarithmic normalization processing and label data normalization processing on each item of data in the original data set to obtain the input data set of the neural network; among them, logarithmic normalization processing can solve the problem of small - signal quantization noise, and label data normalization processing can map the original spectral density data to the interval [0, 100] through linear transformation.

[0067] S3. Spectral reconstruction neural network construction:

[0068] Construct a spectral reconstruction neural network with physical constraint embedding to achieve high - fidelity reconstruction of spectral features. The spectral reconstruction neural network includes a cascaded input layer, a residual block group, and an output layer.

[0069] The input data of the input layer are the photocurrent sequence Eij, the device bias voltage V, the responsivity A, and the light intensity value I detected manually.

[0070] The residual block group includes 6 dense residual blocks and 3 identity residual blocks, and one identity residual block is connected after every two dense residual blocks, arranged alternately in sequence.

[0071] The dense residual block is composed of a stack of fully connected layers, batch normalization, and ReLU activation functions to achieve non-linear feature mapping.

[0072] The identity residual block is used to retain the original feature path to avoid gradient disappearance.

[0073] The output layer adopts a 2048-dimensional fully connected layer to perform high-resolution reconstruction on the data output by the residual block group and output the predicted light intensity value corresponding to the wavelength.

[0074] S4. Model training:

[0075] The input data set obtained in step S2 is divided into a training set and a test set in a ratio of 4:1.

[0076] The mean square error (MSE) is used as the loss function, and the training set is used to train the spectral reconstruction neural network. The model parameters of the spectral reconstruction neural network are optimized through the backpropagation algorithm to obtain a preliminarily trained spectral reconstruction neural network. Only using MSE as the loss function in the initial stage of model training can minimize the difference between the predicted value and the true value and quickly fit the data.

[0077] S5. Multi-objective optimization:

[0078] In the spectral reconstruction neural network after preliminary training, a multi-objective optimization algorithm is introduced. By minimizing the difference between the model predicted value and the actual value, the parameters of the spectral reconstruction neural network and the sparsity and complexity of the model are further optimized to obtain a trained spectral reconstruction neural network.

[0079] Specifically, the objective function of the multi-objective optimization is:

[0080] ,

[0081] ,

[0082] ,

[0083] ,

[0084] where is the light intensity prediction error, is the predicted light intensity value of the spectral reconstruction neural network, is the light intensity value of the i-th channel detected manually, is the total number of channels; in this embodiment, n takes 18.

[0085] Adopt L1 regularization as the sparsity regularization term, and by minimizing L RE , force the predicted light intensity value of the spectral reconstruction neural network to be as sparse as possible, so as to better fit the physical characteristics of the real spectrum. At the same time, it can suppress the non-physical solutions caused by noise (such as abnormal peaks / troughs), and thus prevent overfitting and improve the generalization ability of the spectral reconstruction neural network. represents the k-th weight in the spectral reconstruction neural network, and K represents the number of weights.

[0086] is the spectral physical calibration error, represents the responsivity of the i-th channel related to the wavelength , represents the j-th light intensity value in the i-th channel related to the wavelength , , respectively represent the lower wavelength limit and the upper wavelength limit of the laser; use the previously manually detected spectral physical calibration results ( ) for the spectral reconstruction neural network to measure the difference between the theoretical value of the photocurrent (the inverse spectral reconstruction value ) and the target actual value (the true value of the photocurrent output by the device ), so that the spectral reconstruction neural network is more representative and robust.

[0087] As Figure 2 shown, the optimization method of the weight coefficient is as follows:

[0088] S51. Initialize the population:

[0089] In the initialization stage, use the uniform sampling Das-Dennis method to generate 100 groups of weight coefficient combinations , covering the weight space that satisfies , and avoid the search bias caused by traditional fixed initial weights (such as [0.6, 0.2, 0.2]).

[0090] S52. Generate the Pareto solution set and density evaluation:

[0091] Define , for each group of weight coefficient combinations , optimize the parameters of the spectral reconstruction neural network through backpropagation, and calculate the three objective values: .

[0092] Identify the Pareto front by fast non-dominated sorting. After calculating the normalized crowding distance of the Pareto solutions, sort them in ascending order for each objective dimension, and then calculate the sum of the distances between adjacent Pareto solutions and normalize it to [0, 1]; and mark the Pareto solutions corresponding to the sparse regions with the top 20% of the crowding distance rankings.

[0093] S53. Density-driven weight dynamic adjustment:

[0094] The weight adjustment formula is:

[0095] (i = 1, 2, 3),

[0096] where, represents the normalized crowding distance of the current p-th solution, and represent the average crowding distance and standard deviation of the population respectively, = 0.1 represents the adjustment factor.

[0097] For the solutions in the dense region: , reduce the weight λ1 of the dominant objective; for the solutions in the sparse region: , increase the weights λ2 and λ3.

[0098] S54. Genetic evolution and frontier update:

[0099] After updating the weight coefficients, perform simulated binary crossover and polynomial mutation operations to generate the offspring population. Combine the historical solutions and the offspring solutions through the environmental selection mechanism, and retain the individuals with a higher non-dominated level and a sparser distribution. In this embodiment, the crossover probability in the simulated binary crossover is set to 0.85, and the mutation probability is set to 0.15.

[0100] S55. Termination condition and optimal solution selection:

[0101] The termination condition is set to that the hypervolume change rate of the Pareto front is < 1% for 10 consecutive generations during the iteration process.

[0102] The optimal solution is the Knee point density-performance balance optimal solution selected by comprehensively evaluating L DL 、L RE 、L AE on the training set. The finally obtained optimal weight coefficients are: . This combination significantly reduces the number of abnormal peaks (the reduction amplitude reaches 58%) and effectively suppresses model overfitting.

[0103] S6. Model testing:

[0104] The spectral reconstruction neural network is tested and verified using a test set. When the predicted light intensity value output by the spectral reconstruction neural network meets the error requirements, the spectral reconstruction neural network can be used to achieve spectral reconstruction based on spectral physical calibration.

[0105] Table 1: Performance comparison table

[0106]

[0107] Table 1 shows the performance comparison between the method of this embodiment and the traditional method (deep neural network method without physical modeling). It can be seen that the method of the present invention has significant advantages over the traditional method in multiple key performance indicators, specifically including: the average relative error is reduced by 60%, the error in the key wavelength band is reduced by 62.5%, the number of abnormal wave peaks is reduced by 58%, and the spectral resolution of the instrument is increased by 21.52%. The above results verify the effectiveness and superiority of the present invention in improving the spectral reconstruction accuracy and physical consistency.

[0108] Figure 3 This is a comparison chart of the relative error distribution between the method of the present invention and the traditional method. It can be seen that the error distribution of the spectral signal reconstructed by the present invention is more concentrated, the overall error amplitude is smaller, and the fluctuation range is significantly narrowed, indicating that the model has stronger stability and anti-noise ability during the prediction process and has better generalization ability.

[0109] Figure 4 This is a comparison chart of the training loss decline between the method of the present invention and the traditional method. Under the same number of training rounds, the spectral reconstruction network proposed by the present invention has a faster convergence speed and a lower final loss value, verifying the effective guiding role of the multi-objective optimization mechanism based on Pareto density regulation in updating network parameters during the training stage, which helps the model improve the prediction accuracy while keeping the complexity controllable.

[0110] Figure 5 This is a comparison chart of the spectral reconstruction results between the method of the present invention and the traditional method in the wavelength range of 500nm - 900nm. It can be observed that at the key wavelength band positions such as absorption peaks and emission peaks, the reconstructed curve of the method of the present invention has a higher degree of coincidence with the true spectrum, and the error is less than 1.5%. This result shows that the present invention can more accurately restore the physical characteristics in the spectral structure and improve the high-fidelity spectral reconstruction ability.

[0111] In summary, the present invention realizes high-precision and low-error spectral reconstruction on the basis of retaining the physical consistency of the photoelectric response by introducing spectral physical calibration information and multi-objective optimization strategies. The relevant experimental data and graphical results all show that this method has excellent stability, robustness and engineering applicability, and has broad application prospects.

Claims

1. A method for spectrum reconstruction based on spectrum physical calibration, characterized in that: The steps include: S1. Data collection: Collect and record the photocurrent sequence Eij, device bias voltage V, responsivity A and manually detected light intensity value I at different wavelengths to generate the original data set; S2. Spectroscopic-electrical joint data preprocessing: Perform logarithmic normalization and label data normalization on each item in the original data set to obtain the input data set of the neural network; S3. Spectral reconstruction neural network construction: Construct a spectral reconstruction neural network with physical constraint embedding to achieve high-fidelity reconstruction of spectral features. The spectral reconstruction neural network includes a cascaded input layer, a residual block group, and an output layer. The input data of the input layer are the photocurrent sequence Eij, the device bias voltage V, the responsivity A and the manually detected light intensity value I; The residual block group includes at least one dense residual block and at least one identical residual block, and the two are arranged alternately; The dense residual block is composed of a fully connected layer, batch normalization, and a ReLU activation function stack to achieve feature nonlinear mapping; The constant residual block is used to retain the original feature path and avoid gradient disappearance; The output layer uses a 2048-dimensional fully connected layer to reconstruct the data output by the residual block group with high resolution and output the predicted light intensity value of the corresponding wavelength; S4. Model training: The input data set obtained in step S2 is divided into a training set and a test set; the mean square error is used as a loss function, the spectral reconstruction neural network is trained using the training set, and the model parameters of the spectral reconstruction neural network are optimized by a back propagation algorithm to obtain a spectral reconstruction neural network that has been preliminarily trained; S5. Multi-objective optimization: After the initial training, a multi-objective optimization algorithm is introduced into the spectral reconstruction neural network. By minimizing the difference between the model prediction value and the actual value, the spectral reconstruction neural network parameters and the sparsity and complexity of the model are further optimized to obtain a trained spectral reconstruction neural network. S6. Model testing: The test set is used to test and verify the spectral reconstruction neural network. When the predicted light intensity value output by the spectral reconstruction neural network meets the error requirement, the spectral reconstruction neural network can be used to realize spectral reconstruction based on spectral physical calibration.

2. The method for realizing spectrum reconstruction based on spectrum physical calibration according to claim 1, characterized in that: In step S5, the objective function of the multi-objective optimization is for: ， ， ， ， in, is the light intensity prediction error, Light intensity prediction value for spectral reconstruction neural network, is the light intensity value of the ith channel detected manually, is the total number of channels; L1 regularization is used as the sparsity regularization term. represents the kth weight in the spectral reconstruction neural network, and K represents the number of weights; is the spectral physical calibration error, Representation and wavelength The responsivity of the associated ith channel, Representation and wavelength The jth light intensity value in the i-th channel is related to, , They represent the lower and upper wavelength limits of the laser respectively.

3. The method for realizing spectrum reconstruction based on spectrum physical calibration according to claim 2, characterized in that: In the objective function, the weight coefficient satisfies and .

4. The method for realizing spectrum reconstruction based on spectrum physical calibration according to claim 3, characterized in that: During the training process of the spectral reconstruction neural network, the weight coefficient is optimized according to the test set error and the Pareto solution distribution density. .

5. The method for realizing spectrum reconstruction based on spectrum physical calibration according to claim 4, characterized in that: The weight coefficient The optimization method is as follows: S51. Initialize the population: In the initialization phase, the uniformly sampled Das-Dennis method is used to generate 100 sets of weight coefficient combinations [ ]; S52. Pareto solution set generation and density evaluation: definition , for each group of weight coefficient combinations [ ], optimize the spectral reconstruction neural network parameters by back propagation, and calculate the three objective values: ; The Pareto front is identified by fast non-dominated sorting. After calculating the normalized crowding distance of the Pareto solution, each target dimension is sorted in ascending order. Then the sum of the distances between adjacent Pareto solutions is calculated and normalized to [0,1]; and the Pareto solutions in the sparse area are marked. S53. Density-driven weight dynamic adjustment: The weight adjustment formula is: ， in, represents the normalized crowding distance of the current p-th solution, , They represent the average crowding distance and standard deviation of the population, represents the regulating factor; Solution for dense regions: , reduce the dominant target weight ; Solution for sparse regions: , increase the weight and ; S54.Genetic evolution and frontier updates: After updating the weight coefficients, simulated binary crossover and polynomial mutation operations are performed to generate the offspring population; historical solutions and offspring solutions are merged through the environmental selection mechanism to retain individuals with high non-dominated levels and sparse distribution; S55. Termination conditions and optimal solution selection: The termination condition is set as the convergence index of the Pareto frontier that is continuously monitored during the iteration process; the optimal solution is to comprehensively evaluate L on the training set DL , L RE , L AE , the optimal solution for the selected Knee point density-performance balance.

Images