Multi-index collaborative prediction method based on spectral data

Abstract

The present invention relates to near infrared spectroscopy analysis technology. In order to improve the stability and prediction accuracy of near infrared spectroscopy data in the analysis and prediction process, a multi-index collaborative prediction method based on spectral data is provided, including: 1. determining a preprocessing method and a parameter corresponding to the preprocessing method; 2. determining the main component setting interval when the partial least squares method is used to model spectral data; 3. performing partial least squares modeling according to step 2 and step 1 to obtain a combination model; 4. selecting at least 3 combination model screening indicators; 5. fully arranging the combination model screening indicators; 6. using the combination model screening indicators after full arrangement to screen the combination models respectively; 7. using the screened combination models to predict spectral data to obtain prediction values; 8. performing mean processing on multiple prediction values ​​to obtain a final prediction value. The above method improves the stability and prediction accuracy in the analysis and prediction process.

Description

Technical Field

[0001] The invention relates to the technical field of near-infrared spectral analysis, and in particular to a multi-index collaborative prediction method based on spectral data. Background Art

[0002] In recent years, near-infrared spectroscopy analysis technology has developed rapidly and has been applied in many fields such as chemical industry, pharmaceutical industry, military industry, food, etc. Near-infrared spectroscopy technology belongs to molecular spectroscopy technology, which can show the composition and properties of substances at the molecular level. It has achieved very high benefits in terms of both economic and social impact and has great development potential.

[0003] However, currently most material composition and property information detection is mainly carried out using large laboratory near-infrared spectroscopic instruments. Although these methods are quantitatively accurate and highly sensitive, the equipment required is bulky and expensive, the sample preparation time is long and the sample preparation method is strict. The detection equipment and sample preparation require professional operation, the detection environment is fixed, and the analysis time is long. They are not suitable for on-site detection and are not easy to promote and use.

[0004] With the development of portable near-infrared spectroscopy technology, the mainstream large-scale near-infrared spectrometers in the market are moving towards small size, low price and portable direction. At present, the portable near-infrared spectroscopy analysis method mainly uses a single indicator for modeling and prediction, and the commonly used ones are MSE (mean square error) and MAE (mean absolute error). The optimal values ​​of these indicators are used to screen the best model for component prediction of unknown samples. Due to the singleness of its indicators, the prediction results are often more random. At the same time, since the portable near-infrared spectrometer is easily affected by the light source, detector, usage method, environmental conditions, etc., it is easy to cause the collected spectral data to have poor stability, low precision, and easy deviation of spectral data, which will further reduce the reliability of the single indicator and the prediction accuracy. Summary of the invention

[0005] In order to improve the stability and prediction accuracy of portable near-infrared spectral data during analysis and prediction, the present invention provides a multi-index collaborative prediction method based on spectral data.

[0006] The technical solution adopted by the present invention to solve the above problems is:

[0007] The multi-index collaborative prediction method based on spectral data includes:

[0008] Step 1, determining the pretreatment method and the parameters corresponding to the pretreatment method;

[0009] Step 2, determining the principal component setting interval when using the partial least squares method to model spectral data;

[0010] Step 3, according to the main component setting interval determined in step 2, the spectral data preprocessed according to step 1 are respectively modeled by partial least squares method to obtain a combined model;

[0011] Step 4: Select at least 3 combination model screening indicators;

[0012] Step 5: Arrange all the screening indicators of the combined model;

[0013] Step 6, using the fully arranged combination model screening index to screen the combination models respectively;

[0014] Step 7, respectively using the combined models selected in step 6 to predict the spectral data to obtain predicted values;

[0015] Step 8: Perform averaging on the multiple predicted values ​​obtained in step 7 to obtain the final predicted value.

[0016] Furthermore, in step 4, three combined model screening indicators are selected, namely: root mean square error, model determination coefficient, and compromise accuracy.

[0017] Furthermore, the root mean square error is calculated as follows: Where RMESCV represents the root mean square error, Z is the number of spectral data in the training set, yi represents the calibration value of the i-th sample, Represents the predicted value of the i-th sample.

[0018] Furthermore, the calculation method of the model determination coefficient is: Where R 2 represents the model determination coefficient, Z is the number of spectral data in the training set, and y i,act is the calibration value of the i-th sample, y i,pre is the predicted value of the i-th sample, y i,ave is the average of all calibration values ​​of Z samples.

[0019] Furthermore, the compromise accuracy is calculated as follows: T=100%*S / Z, where T is the compromise accuracy, S is the number of correctly predicted samples, and Z is the number of spectral data in the training set.

[0020] Compared with the prior art, the present invention has the following beneficial effects: first, M spectral data with known calibration values ​​and N spectral data with unknown calibration values ​​to be analyzed are imported, and three indicators, namely, root mean square error, model determination coefficient, and compromise accuracy, are introduced for collaborative prediction. The three indicators are respectively set as the first, second, and third indicators to predict the N spectral data with unknown calibration values ​​to be analyzed, and 6N component prediction values ​​are obtained. Finally, the 6N prediction values ​​are processed by single spectrum averaging to obtain N single prediction values, which are the final component prediction values ​​of the N spectral data with unknown calibration values ​​to be analyzed. By introducing multiple indicators for collaborative prediction, the correlation of the influence of a single indicator on the fluctuation of the original data is reduced, and the reliability and prediction accuracy of the spectral analysis are further improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 Flow chart of the multi-index collaborative prediction method based on spectral data. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0023] like Figure 1 As shown, the multi-index collaborative prediction method based on spectral data is characterized by comprising:

[0024] Step 1, determining the pretreatment method and the parameters corresponding to the pretreatment method;

[0025] Step 2, determining the principal component setting interval when using the partial least squares method to model spectral data;

[0026] Step 3, according to the main component setting interval determined in step 2, the spectral data preprocessed according to step 1 are respectively modeled by partial least squares method to obtain a combined model;

[0027] Step 4: Select at least 3 combination model screening indicators;

[0028] Step 5: Arrange all the screening indicators of the combined model;

[0029] Step 6, using the fully arranged combination model screening index to screen the combination models respectively;

[0030] Step 7, respectively using the combined models selected in step 6 to predict the spectral data to obtain predicted values;

[0031] Step 8: Perform averaging on the multiple predicted values ​​obtained in step 7 to obtain the final predicted value.

[0032] Specifically, in step 4, three combination model screening indicators are selected, namely: root mean square error, model determination coefficient, and compromise accuracy.

[0033] The root mean square error is calculated as: Where RMESCV represents the root mean square error, Z is the number of spectral data in the training set, yi represents the calibration value of the i-th sample, Represents the predicted value of the i-th sample.

[0034] The determination coefficient of the model is calculated as follows: Where R 2 represents the model determination coefficient, Z is the number of spectral data in the training set, and y i,act is the calibration value of the i-th sample, y i,pre is the predicted value of the i-th sample, y i,ave is the average of all calibration values ​​of Z samples.

[0035] The compromise accuracy is calculated as follows: T=100%*S / Z, where T is the compromise accuracy, S is the number of correctly predicted samples, and Z is the number of spectral data in the training set.

[0036] Example

[0037] In this embodiment, M spectral data with known calibration values ​​are imported as known quantities for spectral modeling, 0.8M spectral data are randomly selected as the training set, the remaining 0.2M spectral data are used as the compromise validation set, and N spectral data with unknown calibration values ​​to be analyzed are used as the prediction set.

[0038] Step 1, determine the preprocessing method and the parameters corresponding to the preprocessing method. This embodiment selects two preprocessing methods: Gaussian smoothing and detrending are performed on 0.8M training set spectral data, where the Gaussian smoothing window width is set in the interval [1, 15] and is an odd number, and the detrending order is set in the interval [1, 4]. The type, number and parameters of the preprocessing method can be limited according to actual needs.

[0039] Step 2, determine that the principal component setting interval for spectral data modeling using the partial least squares method is [1, 15];

[0040] Step 3, according to the main component setting interval determined in step 2, PLS (partial least squares) modeling is performed on the 0.8M spectral data of the training set after preprocessing to obtain a combination model, and there are a total of 8×4×15=480 combination models;

[0041] Step 4, select at least 3 combination model screening indicators. In this embodiment, 3 combination model screening indicators are selected, namely root mean square error, model determination coefficient and compromise accuracy. Other types and quantities of combination model screening indicators can also be selected according to actual needs. At the same time, in order to further improve the prediction accuracy, more combination models are retained for model screening. When the three indicators are used as the first indicators for model screening, a larger threshold range can be set.

[0042] The root mean square error (RMESCV) is calculated by cross-validation generation using the leave-one-out method, and the expression is as follows:

[0043]

[0044] Each spectral data corresponds to a sample. There are 0.8M samples in the training set. Each time, a sample X is taken from it. i (i-th sample, i=1, 2, 3...0.8M), use the remaining (0.8M-1) spectral data to build a model, and use the built model to predict the sample X that was taken out i of y i Value (calibration value), get the predicted value of the sample Then, the root mean square error of all the predicted values ​​is calculated to obtain the RMSECV value. Since the leave-one-out method traverses all samples in the training set, the accuracy of the RMSECV indicator for model screening is higher and more stable than the traditional MSE and MAE indicators.

[0045] Model determination coefficient R 2 The calculation method is as follows: The model determination coefficient represents the percentage of the variation of the dependent variable that can be explained by the controlled independent variable. The higher the degree of explanation of the independent variable on the dependent variable, the higher the percentage of the variation caused by the independent variable in the total variation, that is, the higher the model determination coefficient, the better the model effect, and the better the spectral prediction effect. For the spectral modeling of 0.8M spectral data in the training set, the specific calculation formula of the model determination coefficient is as follows:

[0046]

[0047] Among them, y i,act is the calibration value of the i-th sample, y i,pre is the predicted value of the i-th sample within the model, y i,ave It is the average of all calibration values ​​of 0.8M samples.

[0048] The compromise accuracy is calculated as follows: spectral modeling is performed on 0.8M samples in the training set, and the spectral model generated by the combination of the preprocessing method under different parameters and the modeling method under different modeling parameters is used to predict the compromise verification set. If the deviation between the predicted value and the calibrated value is within the tolerance range, the prediction is judged to be correct. The total number of samples with correct predictions is S, and the compromise accuracy T is: T = 100% * S / (0.8M).

[0049] The three indicators of root mean square error, compromise accuracy and model determination coefficient characterize the performance of each model under different combinations of preprocessing methods and modeling methods from different dimensions. The three indicators are used as three-dimensional independent variables, and the accuracy of the prediction set is used as the dependent variable for collaborative prediction. Finally, the best model is screened according to the different priorities of the three indicators, and then the best model is used to predict unknown samples. Through this method, compared with the single minimum root mean square error indicator, the maximum compromise accuracy indicator or the maximum model determination coefficient indicator, this method can effectively avoid model overfitting and reduce the impact of unreasonable division of the compromise validation set, and finally achieve the purpose of improving the prediction accuracy of portable near-infrared equipment.

[0050] Step 5: Arrange all the screening indicators of the combined model;

[0051] Step 6, using the fully arranged combination model screening index to screen the combination models respectively;

[0052] Step 7, respectively using the combined models selected in step 6 to predict the spectral data to obtain predicted values;

[0053] A total of 6 prediction values ​​can be obtained by fully arranging the screening indicators of the three combined models. The specific process is as follows:

[0054] 1. Taking the root mean square error as the first indicator, the model determination coefficient as the second indicator, and the compromise accuracy as the third indicator, the specific steps of modeling and prediction are:

[0055] (1) Root mean square error screening: Through leave-one-out cross-validation, the RMSECV values ​​corresponding to 480 combination models were obtained, and the minimum RMSECV value G was selected as the benchmark value. The model combination with RMSECV value less than 1.2G was retained, and the combination model with RMSECV value greater than 1.2G was eliminated;

[0056] (2) Model determination coefficient screening: Through leave-one-out cross-validation, obtain the R corresponding to the combined model retained in step (1). 2 Value, choose R 2 The maximum value B is taken as the reference value, and R is retained. 2 The combined model with a value greater than 0.9B is eliminated. 2Combined models with values ​​less than 0.9B;

[0057] (3) Compromise accuracy screening: Use the combined model retained in step (2) to predict 0.2M spectral data of the compromise validation set, calculate the prediction accuracy of each combined model for the compromise validation set, and select the combined model with the highest prediction accuracy as the final model;

[0058] Spectral model prediction: The final model selected in step (3) is used to predict the results of N spectral data with unknown calibration values ​​to be analyzed to obtain the predicted value N1.

[0059] 2. Taking the root mean square error as the first indicator, the compromise accuracy as the second indicator, and the model determination coefficient as the third indicator, the specific steps of modeling and prediction are:

[0060] (1) Root mean square error screening: Through leave-one-out cross-validation, the RMSECV values ​​corresponding to 480 combination models were obtained, and the minimum RMSECV value G was selected as the benchmark value. The model combination with RMSECV value less than 1.2G was retained, and the combination model with RMSECV value greater than 1.2G was eliminated;

[0061] (2) Compromise accuracy screening: Use the combined models retained in step (1) to predict 0.2M compromise validation set spectral data, calculate the prediction accuracy of each combined model for the compromise validation set, select the highest prediction accuracy value C as the benchmark value, retain the combined model with a prediction accuracy higher than 0.9C, and eliminate the combined model with an accuracy lower than 0.9C;

[0062] (3) Model determination coefficient screening: Through leave-one-out cross-validation, obtain the R corresponding to the combined model retained in step (2). 2 value, select the maximum R 2 The combined model of the values ​​is the final model;

[0063] Spectral model prediction: The final model selected in step (3) is used to predict the results of N spectral data with unknown calibration values ​​to be analyzed to obtain the predicted value N2.

[0064] 3. Taking the model determination coefficient as the first indicator, the root mean square error as the second indicator, and the compromise accuracy as the third indicator, the specific steps of modeling and prediction are:

[0065] (1) Model determination coefficient screening: Through leave-one-out cross-validation, the R values ​​corresponding to 480 combination models were obtained. 2 Value, choose R 2 The maximum value B is taken as the reference value, and R is retained. 2 The combined model with a value greater than 0.8B is eliminated. 2 Combined model with a value less than 0.8B;

[0066] (2) Root mean square error screening: Through leave-one-out cross-validation, obtain the RMSECV value corresponding to the combined model retained in step (1), select the minimum RMSECV value G as the benchmark value, retain the model combination with RMSECV value less than 1.1G, and eliminate the combined model with RMSECV value greater than 1.1G;

[0067] (3) Compromise accuracy screening: Use the combined model retained in step (2) to predict 0.2M spectral data of the compromise validation set, calculate the prediction accuracy of each combined model for the compromise validation set, and select the combined model with the highest prediction accuracy as the final model;

[0068] Spectral model prediction: The final model selected in step (3) is used to predict the results of N spectral data with unknown calibration values ​​to be analyzed to obtain the predicted value N3.

[0069] 4. Taking the model determination coefficient as the first indicator, the compromise accuracy as the second indicator, and the root mean square error as the third indicator, the specific steps of modeling and prediction are:

[0070] (1) Model determination coefficient screening: Through leave-one-out cross-validation, the R values ​​corresponding to 480 combination models were obtained. 2 Value, choose R 2 The maximum value B is taken as the reference value, and R is retained. 2 The combined model with a value greater than 0.8B is eliminated. 2 Combined model with a value less than 0.8B;

[0071] (2) Compromise accuracy screening: Use the combined models retained in step (1) to predict 0.2M compromise validation set spectral data, calculate the prediction accuracy of each combined model for the compromise validation set, select the highest prediction accuracy value C as the benchmark value, retain the combined model with a prediction accuracy higher than 0.9C, and eliminate the combined model with an accuracy lower than 0.9C;

[0072] (3) Root mean square error screening: Through leave-one-out cross-validation, obtain the RMSECV value corresponding to the combined model retained in step (2), and select the combined model with the minimum RMSECV value as the final model;

[0073] Spectral model prediction: The final model selected in step (3) is used to predict the results of N spectral data with unknown calibration values ​​to be analyzed to obtain the predicted value N4.

[0074] 5. The compromise accuracy is used as the first indicator, the root mean square error is used as the second indicator, and the model determination coefficient is used as the third indicator. The specific steps of modeling and prediction are:

[0075] (1) Compromise accuracy screening: 480 combination models were used to predict 0.2M compromise validation set spectral data, and the prediction accuracy of each combination model for the compromise validation set was calculated. The combination model with the highest prediction accuracy C was selected as the benchmark value, and the combination model with a prediction accuracy higher than 0.8C was retained, and the combination model with an accuracy lower than 0.8C was eliminated;

[0076] (2) Root mean square error screening: Through leave-one-out cross-validation, obtain the RMSECV value corresponding to the combined model retained in step (1), select the minimum RMSECV value G as the benchmark value, retain the model combination with RMSECV value less than 1.1G, and eliminate the combined model with RMSECV value greater than 1.1G;

[0077] (3) Model determination coefficient screening: Through leave-one-out cross-validation, obtain the R corresponding to the combined model retained in step (2). 2 value, select the maximum R 2 The combined model of the values ​​is the final model;

[0078] Spectral model prediction: The final model selected in step (3) is used to predict the results of N spectral data with unknown calibration values ​​to be analyzed to obtain the predicted value N5.

[0079] 6. Taking the compromise accuracy as the first indicator, the model determination coefficient as the second indicator, and the root mean square error as the third indicator, the specific steps of modeling and prediction are:

[0080] (1) Compromise accuracy screening: 480 combination models were used to predict 0.2M compromise validation set spectral data, and the prediction accuracy of each combination model for the compromise validation set was calculated. The combination model with the highest prediction accuracy C was selected as the benchmark value, and the combination model with a prediction accuracy higher than 0.8C was retained, and the combination model with an accuracy lower than 0.8C was eliminated;

[0081] (2) Model determination coefficient screening: Through leave-one-out cross-validation, obtain the R corresponding to the combined model retained in step (1). 2 Value, choose R 2 The maximum value B is taken as the reference value, and R is retained. 2 The combined model with a value greater than 0.9B is eliminated. 2 Combined models with values ​​less than 0.9B;

[0082] (3) Root mean square error screening: Through leave-one-out cross-validation, obtain the RMSECV value corresponding to the combined model retained in step (2), and select the combined model with the minimum RMSECV value as the final model;

[0083] Spectral model prediction: The final model selected in step (3) is used to predict the results of N spectral data with unknown calibration values ​​to be analyzed, and the predicted value N6 is obtained;

[0084] Step 8: N pieces of spectral data with unknown calibration values ​​to be analyzed correspond to N component prediction values. The prediction values ​​N1, N2, N3, N4, N5, and N6 obtained in the above steps respectively contain N values. Set the prediction value N1 = (z1, z2, ... z N ), N2=(c1,c2,……c N ), N3=(v1,v2,……v N ), N4=(r1,r2,……r N ), N5=(e1,e2,……e N ), N6=o1,o2,……o N ), the obtained multiple prediction values ​​are averaged to obtain a single prediction value N7: The single prediction value N7 is the final component prediction value of the spectral data with N unknown calibration values ​​to be analyzed.

Claims

1. A multi-index collaborative prediction method based on spectral data, characterized in that: include: Step 1, determining the pretreatment method and the parameters corresponding to the pretreatment method; Step 2, determining the principal component setting interval when using the partial least squares method to model spectral data; Step 3, according to the main component setting interval determined in step 2, the spectral data preprocessed according to step 1 are respectively modeled by partial least squares method to obtain a combined model; Step 4: Select at least 3 combination model screening indicators; Step 5: Arrange all the screening indicators of the combined model; Step 6, using the fully arranged combination model screening index to screen the combination models respectively; Step 7, respectively using the combined models selected in step 6 to predict the spectral data to obtain predicted values; Step 8: Perform averaging on the multiple predicted values ​​obtained in step 7 to obtain the final predicted value.

2. The multi-index collaborative prediction method based on spectral data according to claim 1 is characterized in that: In step 4, three combined model screening indicators are selected, namely: root mean square error, model determination coefficient, and compromise accuracy.

3. The multi-index collaborative prediction method based on spectral data according to claim 2 is characterized in that: The root mean square error is calculated as: Where RMESCV represents the root mean square error, Z is the number of spectral data in the training set, and y i represents the calibration value of the i-th sample, Represents the predicted value of the i-th sample.

4. The multi-index collaborative prediction method based on spectral data according to claim 2 is characterized in that: The determination coefficient of the model is calculated as follows: Where R 2 represents the model determination coefficient, Z is the number of spectral data in the training set, and y i,act is the calibration value of the i-th sample, y i,pre is the predicted value of the i-th sample, y i,ave is the average of all calibration values ​​of Z samples.

5. The multi-index collaborative prediction method based on spectral data according to claim 2 is characterized in that: The compromise accuracy is calculated as follows: T=100%*S / Z, where T is the compromise accuracy, S is the number of correctly predicted samples, and Z is the number of spectral data in the training set.

Images