A method for spectrum reconstruction based on spectrum physical calibration

Abstract

This invention discloses a method for spectral reconstruction based on spectral physical calibration, belonging to the field of infrared spectroscopy. The method comprises the following steps: S1. Data acquisition, S2. Spectral-electrical combined data preprocessing, S3. Construction of a spectral reconstruction neural network, S4. Model training, S5. Multi-objective optimization, and S6. Model testing. In the constructed spectral reconstruction neural network, the invention optimizes the neural network by combining spectral physical calibration errors with a multi-objective optimization algorithm, thereby improving the accuracy and efficiency of spectral reconstruction.

Description

Technical Field

[0001] The present invention belongs to the field of infrared spectroscopy technology, and in particular relates to a method for realizing spectrum reconstruction based on spectrum physical calibration, which is suitable for accurate calibration of devices such as spectrometers and photoelectric detectors. Background Art

[0002] Near-infrared spectroscopy (NIR) technology was developed in the 1950s and has made significant progress with the rapid development of computer technology. Its analysis speed is extremely fast, enabling the determination of multiple performance indicators in a short period of time, greatly improving work efficiency. Furthermore, the entire analysis process does not require complex sample pretreatment or damage, truly achieving non-destructive testing. Furthermore, a major highlight of this technology is its ability to simultaneously measure multiple components in the same mode, greatly simplifying the measurement process and reducing labor and time costs. It is widely used in materials analysis, environmental monitoring, biomedicine, and other fields. However, obtaining better spectral data requires the physical calibration of the spectroscopic instrument and the design of spectral reconstruction algorithms.

[0003] Existing physical calibration of spectral devices primarily utilizes commercial standard spectrometers and lasers for point-by-point scanning to generate a responsivity function / matrix. Traditional spectral reconstruction, based on compressed sensing theory, recovers the original spectral signal from a small number of measured values. The resulting reconstruction results rely heavily on the design of the measurement matrix and the performance of the reconstruction algorithm employed. However, signal sparsity can significantly impact reconstruction accuracy. With the continuous advancement of deep learning, spectral reconstruction algorithms incorporating deep learning have gradually emerged. Spectral reconstruction using deep learning leverages the powerful nonlinear fitting capabilities of deep neural networks. Deep learning models can automatically learn complex features and mapping relationships from large amounts of spectral data, eliminating the need for prior assumptions about signal sparsity and other characteristics. Most deep learning-based spectral reconstruction algorithms typically use known data such as the photocurrent sequence Eij, the wavelength-corresponding responsivity Aap, and the device bias voltage V as input, and output light intensity values.

[0004] The accuracy of traditional deep learning-based spectral reconstruction is limited not only by the amount of data and model characteristics, but also by the physical errors in device calibration. Reducing physical errors requires more rigorous physical calibration and the use of better devices, which is costly. Summary of the Invention

[0005] To improve the accuracy of spectral reconstruction and reduce calibration costs, this paper proposes a method for spectral reconstruction based on spectral physical calibration. This method combines spectral physical calibration errors with a multi-objective optimization algorithm to optimize the neural network, improving both the accuracy and efficiency of spectral reconstruction.

[0006] The technical solution adopted in the present invention is as follows:

[0007] A method for realizing spectrum reconstruction based on spectrum physical calibration includes the following steps:

[0008] S1. Data Collection:

[0009] The photocurrent sequence Eij, device bias voltage V, responsivity A and manually detected light intensity value I at different wavelengths are collected and recorded to generate the original data set.

[0010] S2. Spectroscopic-electrical joint data preprocessing:

[0011] Each data item in the original data set is logarithmically normalized and labeled data normalized to obtain the input data set of the neural network; among them, the logarithmic normalization processing can solve the problem of small signal quantization noise, and the label data normalization processing can map the original spectral density data to the [0,100] interval through linear transformation.

[0012] S3. Spectral reconstruction neural network construction:

[0013] Construct a spectral reconstruction neural network with physical constraint embedding to achieve high-fidelity reconstruction of spectral features. The spectral reconstruction neural network consists of a cascaded input layer, a residual block group, and an output layer.

[0014] The input data of the input layer are the photocurrent sequence Eij, the device bias voltage V, the responsivity A, and the manually detected light intensity value I is the true value label;

[0015] The residual block group includes at least one dense residual block and at least one identical residual block, and the two are interleaved;

[0016] The dense residual block is constructed by stacking a fully connected layer, batch normalization, and ReLU activation function to achieve nonlinear mapping of features;

[0017] The identical residual block is used to preserve the original feature path and avoid gradient disappearance;

[0018] The output layer uses a 2048-dimensional fully connected layer to reconstruct the data output by the residual block group at high resolution and output the predicted light intensity value of the corresponding wavelength.

[0019] S4. Model training:

[0020] The input dataset obtained in step S2 is divided into a training set and a test set. The spectral reconstruction neural network is trained on the training set using the mean squared error (MSE) as the loss function. The model parameters of the spectral reconstruction neural network are optimized using the backpropagation algorithm to obtain a preliminarily trained spectral reconstruction neural network. Using only the MSE as the loss function in the initial stage of model training minimizes the difference between the predicted and true values, allowing for rapid data fitting.

[0021] S5. Multi-objective optimization:

[0022] After the initial training is completed, a multi-objective optimization algorithm is introduced into the spectral reconstruction neural network. By minimizing the difference between the model predicted value and the actual value, the spectral reconstruction neural network parameters and the sparsity and complexity of the model are further optimized to obtain a trained spectral reconstruction neural network.

[0023] S6. Model testing:

[0024] The test set is used to test and verify the spectral reconstruction neural network. When the predicted light intensity value output by the spectral reconstruction neural network meets the error requirement, the spectral reconstruction neural network can be used to realize spectral reconstruction based on spectral physical calibration.

[0025] Preferably, in step S5, the objective function L of the multi-objective optimization is all for:

[0026] L all =λ1L DL +λ2L RE +λ3L AE ,

[0027]

[0028] Among them, L DL is the light intensity prediction error, is the light intensity prediction value of the spectral reconstruction neural network, I i is the light intensity value of the manually detected channel i, and n is the total number of channels;

[0029] L RE L1 regularization is used as the sparsity regularization term, by minimizing L RE , forcing the spectral reconstruction neural network to predict the light intensity value As sparse as possible, so as to better fit the physical characteristics of the real spectrum. At the same time, it can suppress the non-physical solution caused by noise (such as abnormal peaks / troughs), thereby preventing overfitting and improving the generalization ability of the spectral reconstruction neural network. k represents the kth weight in the spectral reconstruction neural network, and K represents the number of weights;

[0030] LAE is the spectral physical calibration error, A i (λ) represents the responsivity of the i-th channel related to the wavelength λ, I ij (λ) represents the jth light intensity value in the i-th channel associated with the wavelength λ, λ0, λ q Respectively represent the lower and upper wavelength limits of the laser; the spectral physical calibration results (A i , I ij ) is used for spectral reconstruction neural network to measure the theoretical value of photocurrent (inverse spectral reconstruction value ) and the target actual value (the actual value of the photocurrent output by the device E ij ), making the spectral reconstruction neural network more representative and robust.

[0031] Preferably, in the objective function, the weight coefficients satisfy λ1, λ2, λ3≥0, and λ1+λ2+λ3=1.

[0032] Preferably, during the training process of the spectral reconstruction neural network, the weight coefficients λ1, λ2, and λ3 are optimized according to the test set error and the Pareto solution distribution density.

[0033] Preferably, the weight coefficients λ1, λ2, and λ3 are optimized as follows:

[0034] S51. Initialize the population:

[0035] In the initialization phase, the uniformly sampled Das-Dennis method is used to generate 100 sets of weight coefficient combinations [λ1,λ2,λ3] to avoid the search bias caused by traditional fixed initial weights (such as [0.6,0.2,0.2]).

[0036] S52. Pareto solution generation and density assessment:

[0037] Define f1 = L DL ,f2=L RE ,f3=L AE , for each set of weight coefficient combinations [λ1,λ2,λ3], optimize the spectrum reconstruction neural network parameters through back propagation and calculate the three target values: f1,f2,f3;

[0038] The Pareto front is identified through fast non-dominated sorting. After calculating the normalized crowding distance of the Pareto solutions, each target dimension is sorted in ascending order. The sum of the distances between adjacent Pareto solutions is calculated and normalized to [0, 1]. The Pareto solutions in sparse areas are marked (the sparse areas are selected as the top 20% in crowding distance).

[0039] S53. Density-driven weight dynamic adjustment:

[0040] The weight adjustment formula is:

[0041]

[0042] Among them, D p represents the normalized crowding distance of the current p-th solution, σ D They represent the average crowding distance and standard deviation of the population, respectively, and γ = 0.1 represents the adjustment factor;

[0043] Solution for dense regions: Reduce the dominant target weight λ1; for sparse area solution: Increase the weights λ2 and λ3.

[0044] S54.Genetic Evolution and Frontier Updates:

[0045] After updating the weight coefficients, simulated binary crossover and polynomial mutation operations are performed to generate the offspring population; historical solutions and offspring solutions are merged through the environmental selection mechanism to retain individuals with high non-dominated levels and sparse distribution.

[0046] S55. Termination conditions and optimal solution selection:

[0047] The termination condition is set as the convergence indicator of the Pareto front that is continuously monitored during the iteration process, such as the hypervolume growth rate;

[0048] The optimal solution is to comprehensively evaluate L on the training set DL 、L RE 、L AE , the optimal solution of Knee point density-performance balance is selected.

[0049] The beneficial effects of the present invention are as follows:

[0050] 1. Improve device spectral resolution: Through a deep learning-based spectral reconstruction method, the corresponding responsivity and photocurrent characteristics at more optical wavelength values ​​are obtained, no longer limited by the optical wavelength interval selected during the physical calibration process, thereby greatly improving the device spectral resolution.

[0051] 2. Improve instrument accuracy: The spectrum reconstruction neural network autonomously learns the nonlinear mapping relationship between the photocurrent sequence and the light intensity value based on the physical calibration of the previous manual detection to reconstruct the spectrum, thereby improving the instrument accuracy.

[0052] 3. Enhance the generalization ability of the spectral reconstruction neural network: A multi-objective optimization algorithm is introduced into the spectral reconstruction neural network to further optimize the neural network parameters and improve the generalization ability and robustness of the neural network.

[0053] 4. Reduce calibration costs: Reduce the complexity and cost of manual calibration and improve calibration efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 This is a patent flow chart;

[0055] Figure 2 is the weight optimization method;

[0056] Figure 3 A comparison diagram of relative error distribution between the method of the present invention and the traditional method;

[0057] Figure 4 This is a comparison chart of the reduction in training loss between the method of the present invention and the traditional method;

[0058] Figure 5 This is a comparison chart of the spectrum reconstruction results of the method of the present invention and the traditional method in the 500nm-900nm band. DETAILED DESCRIPTION

[0059] The technical solution of the present invention is further described below with reference to the accompanying drawings and specific embodiments.

[0060] This embodiment provides a method for realizing spectrum reconstruction based on spectrum physical calibration, such as Figure 1 As shown, the following steps are included:

[0061] S1. Data Collection:

[0062] The photocurrent sequence Eij, device bias voltage V, responsivity A and manually detected light intensity value I at different wavelengths (wavelength range: 500nm-900nm, acquisition step size: 5nm, a total of 80 groups) were collected and recorded to generate the original data set.

[0063] S2. Spectroscopic-electrical joint data preprocessing:

[0064] Each data item in the original data set is logarithmically normalized and labeled data normalized to obtain the input data set of the neural network; among them, the logarithmic normalization processing can solve the problem of small signal quantization noise, and the label data normalization processing can map the original spectral density data to the [0,100] interval through linear transformation.

[0065] S3. Spectral reconstruction neural network construction:

[0066] A spectral reconstruction neural network with physical constraint embedding is constructed to achieve high-fidelity reconstruction of spectral features. The spectral reconstruction neural network consists of a cascaded input layer, a residual block group, and an output layer.

[0067] The input data of the input layer are the photocurrent sequence Eij, the device bias voltage V, the responsivity A, and the manually detected light intensity value I is the true value label.

[0068] The residual block group includes 6 dense residual blocks and 3 identical residual blocks, and two dense residual blocks are followed by one identical residual block, which are interleaved in sequence.

[0069] The dense residual block is constructed by stacking a fully connected layer, batch normalization, and a ReLU activation function to achieve nonlinear mapping of features.

[0070] The identical residual block is used to preserve the original feature path and avoid gradient disappearance.

[0071] The output layer uses a 2048-dimensional fully connected layer to reconstruct the data output by the residual block group at high resolution and output the predicted light intensity value of the corresponding wavelength.

[0072] S4. Model training:

[0073] The input dataset obtained in step S2 is divided into a training set and a test set in a ratio of 4:1.

[0074] The spectral reconstruction neural network was trained using the training set using the mean squared error (MSE) as the loss function. The model parameters of the spectral reconstruction neural network were optimized using the backpropagation algorithm to obtain a preliminarily trained spectral reconstruction neural network. Using only the MSE as the loss function in the initial stage of model training minimizes the difference between the predicted and true values, allowing for rapid data fitting.

[0075] S5. Multi-objective optimization:

[0076] After the initial training is completed, a multi-objective optimization algorithm is introduced into the spectral reconstruction neural network. By minimizing the difference between the model predicted value and the actual value, the spectral reconstruction neural network parameters and the sparsity and complexity of the model are further optimized to obtain a trained spectral reconstruction neural network.

[0077] Specifically, the objective function L of the multi-objective optimization is all for:

[0078] L all =λ1L DL +λ2L RE +λ3L AE ,

[0079]

[0080] Among them, L DL is the light intensity prediction error, is the light intensity prediction value of the spectral reconstruction neural network, I i is the light intensity value of the manually detected i-th channel, and n is the total number of channels; in this embodiment, n is 18.

[0081] L REL1 regularization is used as the sparsity regularization term, by minimizing L RE , forcing the spectral reconstruction neural network to predict the light intensity value As sparse as possible, so as to better fit the physical characteristics of the real spectrum. At the same time, it can suppress the non-physical solution caused by noise (such as abnormal peaks / troughs), thereby preventing overfitting and improving the generalization ability of the spectral reconstruction neural network. k represents the kth weight in the spectral reconstruction neural network, and K represents the number of weights.

[0082] L AE is the spectral physical calibration error, A i (λ) represents the responsivity of the i-th channel related to the wavelength λ, I ij (λ) represents the jth light intensity value in the i-th channel associated with the wavelength λ, λ0, λ q Respectively represent the lower and upper wavelength limits of the laser; the spectral physical calibration results (A i , I ij ) is used for spectral reconstruction neural network to measure the theoretical value of photocurrent (inverse spectral reconstruction value ) and the target actual value (the actual value of the photocurrent output by the device E ij ), making the spectral reconstruction neural network more representative and robust.

[0083] like Figure 2 As shown, the optimization method of the weight coefficients λ1, λ2, and λ3 is as follows:

[0084] S51. Initialize the population:

[0085] In the initialization phase, the uniformly sampled Das-Dennis method is used to generate 100 sets of weight coefficient combinations [λ1,λ2,λ3], covering the weight space satisfying λ1+λ2+λ3=1, avoiding the search bias caused by traditional fixed initial weights (such as [0.6,0.2,0.2]).

[0086] S52. Pareto solution generation and density assessment:

[0087] Define f1 = L DL ,f2=L RE ,f3=L AE , for each set of weight coefficient combinations [λ1,λ2,λ3], the spectral reconstruction neural network parameters are optimized by back propagation, and the three target values ​​are calculated: f1,f2,f3.

[0088] The Pareto front is identified through fast non-dominated sorting. After calculating the normalized crowding distance of the Pareto solutions, each target dimension is sorted in ascending order. The sum of the distances between adjacent Pareto solutions is calculated and normalized to [0, 1]. The Pareto solutions corresponding to the top 20% of sparse regions in crowding distance are marked.

[0089] S53. Density-driven weight dynamic adjustment:

[0090] The weight adjustment formula is:

[0091]

[0092] Among them, D p represents the normalized crowding distance of the current p-th solution, σ D They represent the average crowding distance and standard deviation of the population respectively, and γ = 0.1 represents the adjustment factor.

[0093] Solution for dense regions: Reduce the dominant target weight λ1; for sparse area solution: Increase the weights λ2 and λ3.

[0094] S54.Genetic Evolution and Frontier Updates:

[0095] After updating the weight coefficients, simulated binary crossover and polynomial mutation operations are performed to generate the offspring population. The environmental selection mechanism merges historical solutions with offspring solutions, retaining individuals with high non-dominated hierarchies and sparse distributions. In this example, the crossover probability in the simulated binary crossover is set to 0.85, and the mutation probability is set to 0.15.

[0096] S55. Termination conditions and optimal solution selection:

[0097] The termination condition is set as the change rate of the Pareto front hypervolume is less than 1% for 10 consecutive generations during the iteration process.

[0098] The optimal solution is to comprehensively evaluate L on the training set DL 、L RE 、L AE The optimal solution for knee point density-performance balance was selected, resulting in optimal weight coefficients of λ1 = 0.42, λ2 = 0.31, and λ3 = 0.27. This combination significantly reduced the number of abnormal peaks by 58%, effectively suppressing model overfitting.

[0099] S6. Model testing:

[0100] The test set is used to test and verify the spectral reconstruction neural network. When the predicted light intensity value output by the spectral reconstruction neural network meets the error requirement, the spectral reconstruction neural network can be used to realize spectral reconstruction based on spectral physical calibration.

[0101] Table 1: Performance comparison table

[0102] index Traditional methods Patented method Improvement Average relative error (%) 5.73 2.33 ↓59.33％ Key band error (%) 4.21 1.71 ↓59.38％ Number of abnormal peaks 50 21 ↓58.00％ Spectral resolution improvement (%) 0 21.52 ↑21.52％

[0103] Table 1 compares the performance of the method of this embodiment with a traditional method (a deep neural network method without physical modeling). As can be seen, the method of the present invention exhibits significant advantages over the traditional method in several key performance indicators, including a 60% reduction in average relative error, a 62.5% reduction in key band error, a 58% reduction in the number of abnormal peaks, and a 21.52% improvement in instrument spectral resolution. These results demonstrate the effectiveness and superiority of the present invention in improving spectral reconstruction accuracy and physical consistency.

[0104] Figure 3 This is a comparison diagram of the relative error distribution between the method of the present invention and the traditional method; it can be seen that the error distribution of the spectral signal reconstructed by the present invention is more concentrated, the overall error amplitude is smaller, and the fluctuation range is significantly narrowed, indicating that the model has stronger stability and noise resistance in the prediction process and has better generalization ability.

[0105] Figure 4 This is a comparison chart of the reduction in training loss between the method of the present invention and the traditional method. Under the same training rounds, the spectral reconstruction network proposed in the present invention has a faster convergence speed and a lower final loss value, which verifies the effective guiding role of the multi-objective optimization mechanism based on Pareto density control on the update of network parameters during the training phase, which helps the model improve prediction accuracy while keeping the complexity controllable.

[0106] Figure 5 This figure compares the spectral reconstruction results of the inventive method and traditional methods in the 500nm–900nm band. It can be observed that at key wavelengths such as absorption and emission peaks, the reconstructed curves of the inventive method match the true spectrum more closely, with an error of less than 1.5%. This result demonstrates that the inventive method can more accurately restore the physical features of the spectral structure, improving the ability to reconstruct high-fidelity spectra.

[0107] In summary, this method, by incorporating spectral physical calibration information and a multi-objective optimization strategy, achieves high-precision, low-error spectral reconstruction while preserving the physical consistency of the photoelectric response. The relevant experimental data and graphical results demonstrate that this method possesses excellent stability, robustness, and engineering applicability, and has broad application prospects.

Claims

1. A method for spectrum reconstruction based on spectrum physical calibration, characterized in that: The steps include: S1. Data Collection: Collect and record the photocurrent sequence Eij, device bias voltage V, responsivity A, and manually detected light intensity value I at different wavelengths to generate the original data set; S2. Spectroscopic-electrical joint data preprocessing: Perform logarithmic normalization and label data normalization on each data item in the original data set to obtain the input data set of the neural network; S3. Spectral reconstruction neural network construction: Construct a spectral reconstruction neural network with physical constraint embedding to achieve high-fidelity reconstruction of spectral features. The spectral reconstruction neural network consists of a cascaded input layer, a residual block group, and an output layer. The input data of the input layer are the photocurrent sequence Eij, the device bias voltage V, the responsivity A, and the manually detected light intensity value I is the true value label; The residual block group includes at least one dense residual block and at least one identical residual block, and the two are interleaved; The dense residual block is constructed by stacking a fully connected layer, batch normalization, and ReLU activation function to achieve nonlinear mapping of features; The identical residual block is used to preserve the original feature path and avoid gradient disappearance; The output layer uses a 2048-dimensional fully connected layer to reconstruct the data output by the residual block group at high resolution and output the predicted light intensity value of the corresponding wavelength; S4. Model training: The input data set obtained in step S2 is divided into a training set and a test set; the spectrum reconstruction neural network is trained using the training set using the mean square error as the loss function, and the model parameters of the spectrum reconstruction neural network are optimized by the back propagation algorithm to obtain a spectrum reconstruction neural network that has been preliminarily trained; S5. Multi-objective optimization: After the initial training, a multi-objective optimization algorithm is introduced into the spectral reconstruction neural network. By minimizing the difference between the model prediction value and the actual value, the spectral reconstruction neural network parameters and the sparsity and complexity of the model are further optimized to obtain a trained spectral reconstruction neural network. The objective function L of the multi-objective optimization algorithm all for: L all =λ1L DL +λ2L RE +λ3L AE , Among them, L DL is the light intensity prediction error, is the light intensity prediction value of the spectral reconstruction neural network, I i is the light intensity value of the manually detected channel i, and n is the total number of channels; L RE L1 regularization is used as the sparsity regularization term, w k represents the kth weight in the spectral reconstruction neural network, and K represents the number of weights; L AE is the spectral physical calibration error, A i (λ) represents the responsivity of the i-th channel related to the wavelength λ, I ij (λ) represents the jth light intensity value in the i-th channel associated with the wavelength λ, λ0, λ q Respectively represent the lower and upper wavelength limits of the laser; S6. Model testing: The test set is used to test and verify the spectral reconstruction neural network. When the predicted light intensity value output by the spectral reconstruction neural network meets the error requirement, the spectral reconstruction neural network can be used to realize spectral reconstruction based on spectral physical calibration.

2. The method for realizing spectrum reconstruction based on spectrum physical calibration according to claim 1, characterized in that: In the objective function, the weight coefficients satisfy λ1, λ2, λ3 ≥ 0, and λ1 + λ2 + λ3 = 1.

3. The method for realizing spectrum reconstruction based on spectrum physical calibration according to claim 2, characterized in that: During the training process of the spectral reconstruction neural network, the weight coefficients λ1, λ2, and λ3 are optimized according to the test set error and the Pareto solution distribution density.

4. The method for realizing spectrum reconstruction based on spectrum physical calibration according to claim 3, characterized in that: The optimization method of the weight coefficients λ1, λ2, and λ3 is as follows: S51. Initialize the population: In the initialization phase, the uniformly sampled Das-Dennis method is used to generate 100 sets of weight coefficient combinations [λ1,λ2,λ3]; S52. Pareto solution generation and density assessment: Define f1 = L DL ,f2=L RE ,f3=L AE , for each set of weight coefficient combinations [λ1,λ2,λ3], optimize the spectrum reconstruction neural network parameters through back propagation and calculate the three target values: f1,f2,f3; Identify the Pareto frontier through fast non-dominated sorting, calculate the normalized crowding distance of the Pareto solutions, sort each target dimension in ascending order, calculate the sum of the distances between adjacent Pareto solutions and normalize it to [0,1]; and mark the Pareto solutions in the sparse area; S53. Density-driven weight dynamic adjustment: The weight adjustment formula is: Among them, D p represents the normalized crowding distance of the current p-th solution, σ D They represent the average crowding distance and standard deviation of the population, respectively, and γ = 0.1 represents the adjustment factor; Solution for dense regions: Reduce the dominant target weight λ1; for sparse area solution: Increase weights λ2 and λ3; S54.Genetic Evolution and Frontier Updates: After updating the weight coefficients, simulated binary crossover and polynomial mutation operations are performed to generate the offspring population; historical solutions and offspring solutions are merged through the environmental selection mechanism to retain individuals with high non-dominated levels and sparse distribution; S55. Termination conditions and optimal solution selection: The termination condition is set to continuously monitor the convergence index of the Pareto front during the iteration process; the optimal solution is to comprehensively evaluate L on the training set DL 、L RE 、L AE , the optimal solution of Knee point density-performance balance is selected.

Images