Method for screening optimal model parameters in a spectral transmission model and application thereof
The optimal parameters of the spectral transfer model were determined by singular value decomposition and correlation coefficient calculation, which solved the accuracy problem of the spectral correction model across different spectroscopic instruments, realized the effective sharing and stability of the spectral correction model, and reduced the sample quantity requirement.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHINA TOBACCO HUNAN IND CORP
- Filing Date
- 2024-12-12
- Publication Date
- 2026-06-12
Smart Images
Figure CN122201483A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a method for screening spectral model parameters, specifically a method for screening optimal model parameters in a spectral transfer model and its application, belonging to the field of spectral analysis technology. Background Technology
[0002] When using spectroscopic instruments for in-situ, real-time, and rapid analysis of complex chemical and biological systems, it is typically necessary to measure the spectral data of a batch of calibration samples and then establish a calibration model between the spectral data of the calibration samples and the concentration of the analyte or other chemical and physical properties in the calibration samples. Once established, the calibration model can be used to predict the concentration of the analyte or other chemical and physical properties in the test sample from the spectral data of the test sample. However, the validity of the calibration model's predictions rests on the assumption that the spectra of the test sample and the calibration sample were measured using the same spectroscopic instrument under the same optical parameters. If these assumptions are not met, the accuracy of the calibration model's predictions is difficult to guarantee. In practical applications, when a spectral calibration model established on one spectrometer is applied to other spectrometers of the same type, it is difficult to guarantee the validity of the spectral calibration model's predictions. Furthermore, the aging of spectroscopic instrument components and the replacement of instrument components can severely affect the accuracy of the spectral calibration model's predictions. In such cases, a "spectral calibration model transfer method" can be used to maintain the accuracy of the spectral calibration model's predictions, thereby avoiding the need to expend significant manpower and resources to rebuild the spectral calibration model.
[0003] Existing methods for transferring spectral calibration models can be broadly categorized into three types: calibration model parameter update methods, prediction result correction methods, and spectral normalization methods. Calibration model parameter update methods and prediction result correction methods require spectral data of standard samples on the current spectrometer, along with the content data of the target component in these standard samples, to establish a transfer model. Spectral normalization methods, however, only require spectral data of standard samples measured on two spectrometers to establish a transfer model. Therefore, spectral normalization methods have a wider range of applications and are more favored by users. Commonly used spectral normalization methods include direct normalization, segmented direct normalization, spectral spatial transformation, Prokopius analysis, and canonical correlation analysis. Among these, spectral spatial transformation is currently one of the best-performing and most widely used spectral normalization methods.
[0004] Existing methods for determining the optimal model parameters of spectral spatial transfer models divide the standard sample set into a modeling sample set and a validation sample set. The modeling sample set is used to construct the spectral spatial transfer model, while the validation sample set is used to determine the optimal model parameter values. However, these methods are only suitable for situations with a large number of standard samples. When the number of standard samples used to construct the transfer model is small, making it difficult to create a validation sample set, the optimal model parameters of the spectral spatial transfer model become difficult to determine accurately. In practical applications, obtaining and storing a large number of standard samples for extended periods poses significant challenges for users of spectral spatial transfer methods, severely limiting the wider application of existing methods for determining the optimal model parameters. Therefore, it is essential to develop a simple and practical method for determining the optimal model parameters of spectral spatial transfer models. Summary of the Invention
[0005] To address the problems existing in the prior art, the first objective of this invention is to provide a method for selecting the optimal model parameters in a spectral transfer model. This method can determine the optimal model parameters of a spectral spatial transfer model without requiring a validation sample set, thereby establishing an effective spectral spatial transfer model, enabling the sharing of spectral correction models among similar instruments, and ensuring long-term effectiveness even when the spectrometer undergoes changes, such as natural aging, maintenance, or component replacement.
[0006] The second objective of this invention is to provide an application of a method for screening optimal model parameters in a spectral transfer model. The method provided by this invention can quickly determine the optimal model parameters of a spectral spatial transfer model even with a small number of standard samples or when it is difficult to divide a validation sample set. Testing has shown that the spectral spatial transfer model constructed using the optimal model parameters obtained by the screening method provided by this invention has the smallest sum of squared residuals between the standardized spectral data of the test samples and the actually measured spectral data, effectively solving the problem of the large number of samples required in the determination of optimal model parameters in existing technologies.
[0007] To achieve the above technical objectives, this invention provides a method for selecting optimal model parameters in a spectral transfer model, comprising:
[0008] Step S1: Obtain spectral standard samples, detect the standard samples with a spectrometer, and obtain the main spectral data matrix X1 and the secondary spectral data matrix X2;
[0009] Step S2: Combine matrix X1 and matrix X2 into X comb = [X1, X2], then perform singular value decomposition on it, and select a submatrix of its singular value decomposition matrix to obtain the reconstructed spectral vector V1. *and V2 * ;
[0010] Step S3: Perform singular value decomposition on X1 and X2 respectively, and obtain X comb With the left singular value vector matrix X1, X comb With the left singular value vector matrix X2, V1 * With the left singular value vector matrix X1 and V2 * The absolute value of the correlation coefficient between X2 and the left singular value vector matrix;
[0011] Step S4: Set a threshold for the absolute value of the correlation coefficient and determine the optimal model parameters in the spectral transfer model through an iterative method.
[0012] As a preferred embodiment, the main spectral data matrix X1 and the slave spectral data matrix X2 are obtained from the main spectrometer and the slave spectrometer respectively detecting spectral standard samples, or from the same spectrometer detecting spectral standard samples at different times.
[0013] As a preferred embodiment, the i-th row of the main spectral data matrix X1 is the spectral data of the i-th standard sample measured on the main spectrometer.
[0014] As a preferred embodiment, the i-th row of the spectral data matrix X2 represents the spectral data of the i-th standard sample measured from the spectrometer.
[0015] As a preferred embodiment, the reconstructed spectral vector V1 * and V2 * The acquisition process is as follows:
[0016] Step S2-1, for X comb The singular value decomposition of a matrix is performed as follows:
[0017] Formula 1:
[0018] Step S2-2, according to X comb Given the left singular value vector matrix and the singular value diagonal matrix of the matrix, calculate V1. * and V2 * The process is as follows:
[0019] Formula 2:
[0020] Formula 3:
[0021] As a preferred embodiment, the singular value decomposition process of X1 and X2 is as follows:
[0022] Formula 4:
[0023] Formula 5:
[0024] As a preferred embodiment, the calculation process for the absolute value of the correlation coefficient is as follows:
[0025] Formula 6:
[0026] Formula 7:
[0027] Formula 8:
[0028] Formula 9:
[0029] In equations 6 to 9: cov(x,y) is the covariance between two vectors x and y; σ(x) and σ(y) represent the standard deviations of vectors x and y, respectively.
[0030] As a preferred embodiment, the process for determining the optimal model parameters is as follows: when Coeff1(i), Coeff2(i), Coeff3(i), and Coeff4(i) are all greater than a set threshold, but at least one of Coeff1(i+1), Coeff2(i+1), Coeff3(i+1), and Coeff4(i+1) is less than the set threshold, then the optimal model parameter value is i.
[0031] This invention also provides an application of a method for selecting optimal model parameters in a spectral transfer model, used to construct a transfer model using the spectral spatial transformation method.
[0032] As a preferred embodiment, the process of constructing the transfer model using the spectral spatial transformation method is as follows: determining the value of the optimal model parameter c, and calculating V1. * and V2 * Then, a transfer model is constructed, and its calculation process is as follows:
[0033] Formula 10:
[0034] Formula 11:
[0035] In Equations 10 and 11: x test x represents the spectral data of the sample to be tested measured on a spectrometer. trans This is the corresponding spectrum on the standardized main spectrometer.
[0036] Compared with the prior art, the beneficial technical effects of the technical solution provided by the present invention are as follows:
[0037] 1) The screening method provided by this invention can determine the optimal model parameters of the spectral spatial transfer model without verifying the sample set, thereby establishing an effective spectral spatial transfer model, realizing the sharing of spectral correction models among similar instruments, and ensuring long-term effectiveness in the event of changes in the spectral instrument, such as natural aging, maintenance, or replacement of parts.
[0038] 2) The technical solution provided by this invention can quickly determine the optimal model parameters of the spectral spatial transfer model even when there are a small number of standard samples or when it is difficult to divide the verification sample set. After testing, the spectral spatial transfer model constructed by the optimal model parameters obtained by the screening method provided by this invention has the smallest sum of squared residuals between the standardized spectral data of the test samples and the actual measured spectral data, which effectively solves the problem of the large number of samples required in the process of determining the optimal model parameters of the spectral spatial transfer model in the prior art. Attached Figure Description
[0039] Figure 1 A flowchart of the spectral normalization model provided for a specific embodiment of the present invention;
[0040] Figure 2 The images shown are near-infrared spectra of the same tablet sample measured on two Foss NIR Systems near-infrared spectrometers in Example 1 of this invention.
[0041] Figure 3 The near-infrared spectrum of a certain tablet sample measured on the main spectrometer in Embodiment 1 of the present invention and the spectrum obtained by normalizing the near-infrared spectrum of the same sample measured from the spectrometer to the main spectrometer using the SST model with 6 principal components;
[0042] Figure 4 The near-infrared spectra of a certain almond protein sugar sample measured on two NIR Systems 6500 spectrometers in Embodiment 2 of the present invention are obtained after SNV preprocessing and the spectrum obtained after normalizing the spectrum measured on the main spectrometer to the spectrum obtained on the secondary spectrometer using an SST model with two principal components.
[0043] Figure 5 The near-infrared spectrum of a corn silage sample measured on two online near-infrared spectrometers in Embodiment 2 of the present invention and the spectrum obtained after normalizing the spectrum measured on the main spectrometer to the spectrum obtained on the secondary spectrometer using an SST model with two principal components are described. Detailed Implementation
[0044] To facilitate understanding of the present invention, a more comprehensive and detailed description of the invention will be provided below in conjunction with the accompanying drawings and preferred embodiments. It should be noted that the described embodiments are merely some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.
[0045] This invention provides a method for selecting optimal model parameters in a spectral transfer model, the specific process of which is as follows:
[0046] 1) Use two spectrometers, one as the master spectrometer and the other as the slave spectrometer, to measure the spectral data matrices X1 and X2 of the standard sample, respectively. The i-th row of X1 and X2 represents the spectral data of the i-th standard sample measured on the master spectrometer and the slave spectrometer, respectively.
[0047] 2) Combine X1 and X2 to form X comb = [X1, X2], and perform singular value decomposition on it:
[0048] 3) Calculation
[0049] 4) Perform singular value decomposition on X1 and X2 respectively:
[0050] 5) Calculate U comb Between (:,i) and U1(:,i), U comb Between (:,i) and U2(:,i), Between V1(:,i), The absolute values of the correlation coefficients between Coeff1(i), Coeff2(i), Coeff3(i), and Coeff4(i) with V2(i, :, i) (i = 1, 2, ..., m; m is the number of standard samples);
[0051] 6) If Coeff1(i), Coeff2(i), Coeff3(i), and Coeff4(i) are all greater than a certain threshold (e.g., 0.8), but at least one of Coeff1(i+1), Coeff2(i+1), Coeff3(i+1), and Coeff4(i+1) is less than the threshold, then the optimal model parameter value of the spectral spatial transfer model is i.
[0052] Example 1
[0053] The near-infrared spectral data of the tablet samples used in this embodiment are publicly available near-infrared spectral data, sourced from http: / / www.idrc-chambersburg.org / shootout2002.html. This spectral data consists of 1310 absorption spectra from 655 tablets measured on two FossNIR Systems near-infrared spectrometers, one called the master spectrometer and the other the slave spectrometer. The spectral wavelength range is 600–1898 nm, with a wavelength interval of 2 nm. The content of the active ingredient in each tablet was determined by high-performance liquid chromatography (HPLC). After removing 13 suspected abnormal samples, the remaining 642 samples were divided into a calibration set (151 samples), a validation set (451 samples), and a test set (40 samples). The spectral data of these samples in the 600–1638 nm range were selected for subsequent data analysis.
[0054] The optimal model parameters for the spectral spatial transfer model were determined using the screening method described above.
[0055] 1) The near-infrared spectra of all tablet samples were preprocessed using SNV (Standard Normal Variation);
[0056] 2) Randomly select 15 samples from the validation set as standard samples for constructing the Spectral Spatial Transformation (SST) transfer model, calculate Coeff1(i), Coeff2(i), Coeff3(i) and Coeff4(i) (i = 1, 2, ..., 15), and determine the optimal model parameter values of the SST model accordingly;
[0057] 3) The Partial Least Squares Regression (PLSR) method was used to establish a PLSR1 calibration model between the spectral data of the calibration sample measured on the main spectrometer and the content of active components in the calibration sample. The number of latent variables used in the PLSR1 calibration model was determined by examining the root mean square error of prediction (RMSEP) of the calibration model on the validation set.
[0058] 4) The PLSR1 calibration model was combined with the SST model with different principal component numbers and applied to the quantitative analysis of spectral data measured from the spectrometer to verify the performance of the optimal SST model constructed using this invention.
[0059] like Figure 2As shown, there are significant differences in the near-infrared spectra of the same tablet sample measured on two Foss NIR Systems near-infrared spectrometers in the wavelength range of 600–800 nm. Therefore, a transfer model is needed to eliminate the differences in spectral response caused by the different spectrometers. Table 1 lists the Coeff1(i), Coeff2(i), Coeff3(i), and Coeff4(i) (i = 1, 2, …, 15) calculated based on the spectral data of 15 standard tablet samples used to construct the spectral spatial transfer model. From the results in Table 1, it can be seen that when 1 ≤ i ≤ 6, Coeff1(i), Coeff2(i), Coeff3(i), and Coeff4(i) are all greater than the threshold of 0.8, but Coeff1(7), Coeff2(7), Coeff3(7), and Coeff4(7) are all significantly less than this threshold. Therefore, the optimal model parameter value of the SST model can be determined to be 6. To verify that the optimal SST model constructed using this invention does indeed possess good model transfer performance, this experiment combined the PLSR1 calibration model with SST models using different principal component numbers for quantitative analysis of spectral data measured on the secondary spectrometer (Table 2). As shown in Table 2, the quantitative analysis results of the PLSR1 calibration model combined with the SST model using 6 principal components on the spectral data of the calibration, validation, and test samples measured on the secondary spectrometer are closest to the quantitative analysis results of the PLSR1 calibration model on the spectral data of the corresponding samples measured on the primary spectrometer. Moreover, the SST model using 6 principal components obtains the spectral data X obtained after standardizing the sample spectral data measured on the secondary spectrometer to the primary spectrometer. trans Compared with the spectral data X actually measured on the main spectrometer actual The sum of squared spectral residuals (SSD) is relatively small, and the differences in spectral response caused by different spectrometers are essentially eliminated. Figure 3 When the number of principal components used in the SST model increases from 6 to 7, X trans With X actual The sum of squared spectral residuals (SSD) increased by approximately 37.4%, indicating that the optimal model parameter for the SST model should be 6. These experimental results verify that the present invention can quickly and accurately determine the optimal model parameter values for the SST model.
[0060] Table 1
[0061]
[0062] Table 2
[0063]
[0064] Note 1: SSD = sum{diag[(X actual -X trans )×(X actual -X trans ) T ]}, where X trans X is the spectral data matrix obtained after normalizing the sample spectral data matrix measured from the spectrometer to the main spectrometer using the SST model. actual Let diag(X) be the spectral data matrix of the same sample measured on the main spectrometer, and let sum(x) be the column vector consisting of the diagonal elements of the square matrix X.
[0065] Example 2
[0066] The near-infrared spectral data of the almond protein sugar samples used in this embodiment are publicly available near-infrared spectral data from the internet, sourced from https: / / ucphchemometrics.com / datasets. This spectral data consists of 64 reflectance spectra from 32 corn samples measured on two NIR Systems 6500 near-infrared spectrometers, one as the main spectrometer and the other as the spectrometer. The wavelength range of the spectra is 850–2048 nm, with a wavelength interval of 2 nm. The moisture content of each sample was determined using conventional chemical detection methods. The 32 spectra measured on each spectrometer were divided into a calibration set (16 spectra) and a validation set (16 spectra).
[0067] The main steps of this experiment are as follows:
[0068] 1) Near-infrared spectra of all almond protein sugar samples were preprocessed using SNV (Surface Infrared Variation) technology;
[0069] 2) Randomly select 7 samples from the validation set as standard samples for constructing the SST model, calculate Coeff1(i), Coeff2(i), Coeff3(i) and Coeff4(i) (i = 1, 2, ..., 7), and determine the optimal model parameter values of the SST model accordingly;
[0070] 3) Establish a PLSR1 calibration model between the spectral data of the calibration sample measured on the main spectrometer using PLSR and the moisture content of the calibration sample. Determine the number of latent variables used in the PLSR1 calibration model by examining the root mean square prediction error (RMSEP) of the calibration model on the validation set.
[0071] 4) The PLSR1 model was combined with SST models using different principal component numbers and applied to the quantitative analysis of spectral data measured from a spectrometer to verify the performance of the optimal SST model constructed using this invention.
[0072] Table 3 lists the calculated Coeff1(i), Coeff2(i), Coeff3(i), and Coeff4(i) (i = 1, 2, ..., 7) based on the spectral data of seven almond protein sugar standard samples used to construct the spectral spatial transfer model. From the results in Table 3, it can be seen that when 1 ≤ i ≤ 2, Coeff1(i), Coeff2(i), Coeff3(i), and Coeff4(i) are all greater than the threshold of 0.8, but Coeff2(3), Coeff3(3), and Coeff4(3) are less than this threshold. Therefore, the optimal model parameter value for the SST model can be determined to be 2. This experiment will combine the PLSR1 established on the main spectrometer with the SST model using different principal component numbers for quantitative analysis of spectral data measured on the secondary spectrometer (Table 4). As shown in Table 4, the quantitative analysis results of the spectral data of the calibration and validation samples measured on the spectrometer using the PLSR1 calibration model combined with the SST model employing two principal components are very close to the quantitative analysis results of the spectral data of the corresponding samples measured on the main spectrometer using the PLSR1 calibration model. The differences in spectral response caused by different spectrometers have been largely eliminated. Figure 4 Furthermore, the SST model using two principal components will normalize the sample spectral data measured from the spectrometer to the main spectrometer, resulting in the spectral data X. trans Compared with the spectral data X actually measured on the main spectrometer actual The minimum sum of squared spectral residuals (SSD) further proves that the optimal model parameter values of the SST model determined by this invention are correct.
[0073] Table 3
[0074]
[0075] Table 4
[0076]
[0077] Example 3
[0078] The near-infrared spectral data of the corn silage samples used in this embodiment consisted of 28 reflectance spectra from 14 corn silage samples measured on two self-made online near-infrared spectrometers: the main spectrometer ('OnlineNirs1') and the secondary spectrometer ('OnlineNirs2'). The wavelength range of the spectra was 902–1680 nm, with a wavelength interval of 2 nm. The 14 spectra measured on each instrument were divided into a calibration set (7 spectra) and a validation set (7 spectra).
[0079] The main steps of this experiment are as follows:
[0080] 1) Randomly select 7 samples from the validation set as standard samples for constructing the SST model, calculate Coeff1(i), Coeff2(i), Coeff3(i) and Coeff4(i) (i = 1, 2, ..., 7), and determine the optimal model parameter values of the SST model accordingly;
[0081] 2) SST models with different principal component numbers were applied to near-infrared spectral data of maize silage samples measured on 'OnlineNirs2' to obtain standardized spectral data X. trans Calculate X trans Near-infrared spectral data X of the same sample actually measured on 'OnlineNirs1' actual The sum of squared spectral residuals (SSD) was used to verify the performance of the optimal SST model constructed using this invention.
[0082] Table 5 lists the calculated Coeff1(i), Coeff2(i), Coeff3(i), and Coeff4(i) (i = 1, 2, ..., 7) based on the spectral data of the seven standard maize silage samples used to construct the SST model. From the results in Table 5, it can be seen that when 1 ≤ i ≤ 2, Coeff1(i), Coeff2(i), Coeff3(i), and Coeff4(i) are all greater than the threshold of 0.8, but Coeff2(3), Coeff3(3), and Coeff4(3) are less than this threshold. Therefore, the optimal model parameter value for the SST model can be determined to be 2. Table 6 shows that the SST model using two principal components will normalize the sample spectral data measured on 'OnlineNirs2' to 'OnlineNirs1' to obtain the spectral data X. trans Compared with the spectral data actually measured on 'OnlineNirs1' X actual The sum of squared spectral residuals (SSD) is minimized, and the differences in spectral response caused by different spectrometers are essentially eliminated. Figure 5 The result indicates that the optimal model parameter value of the SST model should be 2. This experimental result further proves that the present invention can quickly and accurately determine the optimal model parameter value of the SST model.
[0083] Table 5
[0084]
[0085] Table 6
[0086]
[0087]
Claims
1. A method for selecting optimal model parameters in a spectral transfer model, characterized in that, include: Step S1: Obtain spectral standard samples, detect the standard samples with a spectrometer, and obtain the main spectral data matrix X1 and the secondary spectral data matrix X2; Step S2: Combine matrix X1 and matrix X2 into X comb = [X1, X2], then perform singular value decomposition on it, and select a submatrix of its singular value decomposition matrix to obtain the reconstructed spectral vector V1. * and V2 * ; Step S3: Perform singular value decomposition on X1 and X2 respectively, and obtain X comb With the left singular value vector matrix X1, X comb With the left singular value vector matrix X2, V1 * With the left singular value vector matrix X1 and V2 * The absolute value of the correlation coefficient between X2 and the left singular value vector matrix; Step S4: Set a threshold for the absolute value of the correlation coefficient and determine the optimal model parameters in the spectral transfer model through an iterative method.
2. The method for selecting optimal model parameters in a spectral transfer model according to claim 1, characterized in that: The main spectral data matrix X1 and the slave spectral data matrix X2 are obtained from the main spectrometer and the slave spectrometer respectively detecting spectral standard samples, or from the same spectrometer detecting spectral standard samples at different times.
3. The method for selecting optimal model parameters in a spectral transfer model according to claim 1, characterized in that: The i-th row of the main spectral data matrix X1 is the spectral data of the i-th standard sample measured on the main spectral instrument; The i-th row of the spectral data matrix X2 represents the spectral data of the i-th standard sample measured from the spectrometer.
4. The method for selecting optimal model parameters in a spectral transfer model according to claim 1, characterized in that: The reconstructed spectral vector V1 * and V2 * The acquisition process is as follows: Step S2-1, for X comb The singular value decomposition of a matrix is performed as follows: Formula 1: Step S2-2, according to X comb Given the left singular value vector matrix and the singular value diagonal matrix of the matrix, calculate V1. * and V2 * The process is as follows: Formula 2: Formula 3:
5. The method for selecting optimal model parameters in a spectral transfer model according to claim 1, characterized in that: The singular value decomposition process of X1 and X2 is as follows: Formula 4: Formula 5:
6. The method for selecting optimal model parameters in a spectral transfer model according to claim 1, characterized in that: The calculation process for the absolute value of the correlation coefficient is as follows: Formula 6: Formula 7: Formula 8: Formula 9: In equations 6 to 9: cov(x,y) is the covariance between two vectors x and y; σ(x) and σ(y) represent the standard deviations of vectors x and y, respectively.
7. The method for selecting optimal model parameters in a spectral transfer model according to claim 1, characterized in that: The process of determining the optimal model parameters is as follows: when Coeff1(i), Coeff2(i), Coeff3(i) and Coeff4(i) are all greater than the set threshold, but at least one of Coeff1(i+1), Coeff2(i+1), Coeff3(i+1) and Coeff4(i+1) is less than the set threshold, then the optimal model parameter value is i.
8. The application of the method for selecting optimal model parameters in a spectral transfer model according to any one of claims 1 to 7, characterized in that: Used to construct a transfer model for spectral spatial transformation.
9. The application of the method for selecting optimal model parameters in a spectral transfer model according to claim 8, characterized in that: The process of constructing the transfer model using the spectral space transformation method is as follows: determine the value of the optimal model parameter c, and calculate V1. * and V2 * Then, a transfer model is constructed, and its calculation process is as follows: Formula 10: Formula 11: In Equations 10 and 11: x test x represents the spectral data of the sample to be tested measured on a spectrometer. trans This is the corresponding spectrum on the standardized main spectrometer.