Online modeling method for near-infrared spectroscopy based on partial least squares method

Abstract

The present invention relates to the field of near-infrared spectroscopy detection technology. To improve the efficiency of near-infrared spectroscopy online modeling, a near-infrared spectroscopy online modeling method based on partial least squares is provided, comprising: step 1, inputting original spectral data; step 2, preprocessing the original spectral data; step 3, modeling using optimized partial least squares method; step 4, setting model evaluation indicators; and step 5, screening models based on the model evaluation indicators and saving the optimal model. The above method can improve the efficiency of near-infrared spectroscopy online modeling.

Description

Technical Field

[0001] The present invention relates to the technical field of near-infrared spectrum detection, in particular to a near-infrared spectrum online modeling method based on partial least squares method. Background Art

[0002] Near-infrared spectroscopy equipment primarily consists of traditional large-scale spectral analysis systems and portable spectrometers. With the development of microelectromechanical (MEMS) technology, portable spectrometers have been widely used in various fields, including food, brewing, medicine, and coal, in recent years. Compared to traditional large-scale spectral analysis systems, portable spectrometers offer advantages such as simple structure, low cost, portability, and real-time results, making them a popular product in the spectroscopy field. However, they also have certain limitations, such as stability, operational efficiency, and accuracy, which still lag behind traditional large-scale laboratory equipment. Continuous optimization is needed in terms of hardware and algorithms.

[0003] Currently, there are two sampling methods for portable spectrometers. One is centralized sampling and uploading to the cloud, and then offline modeling, analysis, and prediction are performed after the data is downloaded from the cloud. The other is online prediction, which is divided into offline model import prediction and online data modeling prediction. Currently, due to the impact of algorithm modeling operation efficiency, most products use offline modeling, importing the selected optimal model into the system to predict unknown samples; and in order to improve the operating efficiency of the system, online modeling prediction often uses compressed search parameter intervals to reduce the amount of calculation to achieve the effect of online modeling prediction. However, the compression of the search parameter interval can easily reduce the prediction accuracy. How to improve the efficiency of algorithm modeling operations without compressing the search parameter interval has become one of the directions of portable spectrometer algorithm optimization. Summary of the Invention

[0004] In order to improve the efficiency of near-infrared spectrum online modeling, the present invention provides a near-infrared spectrum online modeling method based on partial least squares method.

[0005] The technical solution adopted by the present invention to solve the above problems is:

[0006] The online modeling method of near infrared spectroscopy based on partial least squares method includes:

[0007] Step 1: Input the original spectrum data;

[0008] Step 2: preprocessing the original spectral data;

[0009] Step 3: Use partial least squares method to build a model;

[0010] Step 4: Set model evaluation indicators;

[0011] Step 5: Screen the model based on the model evaluation index and save the optimal model;

[0012] The step 3 is specifically as follows:

[0013] Step 31: Set the search interval for the main component of modeling;

[0014] Step 32: extract the maximum number of principal components in the search parameter interval;

[0015] Step 33: Centralize the preprocessed data to obtain the set X v ;

[0016] Step 34: Substitute the data obtained in step 33 into the partial least squares method to obtain X under the largest principal component. r 、Y l The final fitting slope is X r ×Y l , the intercept is Y mean A linear model where X r is the rotation load coefficient matrix, Y l is the linear load coefficient matrix;

[0017] Step 35: Repeat step 34 to calculate the linear model under all principal components in the search parameter interval;

[0018] Step 36: Calculate the spectral prediction value y corresponding to the linear model p , y corresponding to all linear models p Composition set Y p .

[0019] Furthermore, the specific steps of preprocessing the original spectral data in step 2 are:

[0020] Step 21: Select a preprocessing model, wherein the preprocessing model includes a preprocessing method and a corresponding parameter range;

[0021] Step 22: Use a preprocessing model to preprocess the original spectral data.

[0022] Furthermore, the preprocessing method includes: Gaussian smoothing, Gaussian derivative, SG smoothing method, detrending and standardization.

[0023] Furthermore, in step 36, the spectrum prediction value y corresponding to the linear model p The calculation method of y is: p =X v ×(X r ×Y l )+Y mean .

[0024] Furthermore, the model evaluation index in step 4 adopts mean square error or mean absolute error.

[0025] Compared with the prior art, the present invention has the following advantages: by extracting the maximum number of principal components, the X under the maximum principal component is calculated only once in the early operation. r 、Y l value, and then calculate the predicted value (y p ) when introducing a loop and gradually extracting the X corresponding to the principal component r , Y l value, and then output the predicted value set Y under all principal components at one time p , then for Y p The matrix calculates the mean square error or mean absolute error at once, and the principal component corresponding to the minimum mean square error or mean absolute error is used as the optimal parameter of the model. This method can directly reduce the number of model operations from N×M to N (N is the number of preprocessing methods, and M is the number of principal components), greatly improving the model's operational efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is a flow chart of the online modeling method for near-infrared spectroscopy based on partial least squares method;

[0027] Figure 2 This is a flow chart of modeling and prediction in the embodiment. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0029] In the traditional modeling method, assuming that there are N combinations of preprocessing schemes, and the principal component interval in the PLS modeling algorithm is set to n_component = [1, 2, ..., 15], then there will be N × 15 combinations, which will greatly increase the load of the model operation.

[0030] The inventors have found that the PLS algorithm uses an iterative approach to find residuals. The load matrix and score matrix under the maximum number of principal components (max_component) already contain 1-(max_component-1) effective information. Therefore, the present invention optimizes the algorithm and only calculates the X under the maximum principal component once in the first half of the operation. r 、Y l value, and then calculate y p When the value is introduced, the loop is gradually extracted to obtain the X under the corresponding principal component. r 、Y lvalue, and then output all y values of 1-max_component at once p The algorithm then calculates the mean square error (MSE) or mean absolute error (MAE) for the Yp matrix. The result corresponding to the smallest MSE or MAE is used as the prediction result under the optimal model. This solution reduces the number of model operations from N × 15 to N, greatly improving the model's operational efficiency.

[0031] like Figure 1 As shown, the near-infrared spectroscopy online modeling method based on partial least squares method includes:

[0032] Step 1: Input the original spectral data; perform spectral detection on the target object using a portable spectrometer and upload the spectral data to the online modeling platform; select a data set from the platform database as required as the input data for model training, i.e., the training set;

[0033] Step 2: preprocessing the original spectral data; including selecting a preprocessing model, wherein the preprocessing model includes a preprocessing method and a corresponding parameter range; and then preprocessing the original spectral data using the preprocessing model; the preprocessing methods include: Gaussian smoothing, Gaussian derivative, SG smoothing method, detrending, standardization, etc.;

[0034] Step 3: Modeling using the partial least squares method after loop optimization; specifically:

[0035] Step 31: Set the search interval for the main component of modeling;

[0036] Step 32: extract the maximum number of principal components in the search parameter interval;

[0037] Step 33: Centralize the preprocessed data to obtain the set X v , centering means subtracting the column mean from the data of each column;

[0038] Step 34: Substitute the data obtained in step 33 into the partial least squares method to obtain X under the largest principal component. r 、Y l The final fitting slope is X r ×Y l , the intercept is Y mean A linear model where X r is the rotation load coefficient matrix, Y l is the linear load coefficient matrix;

[0039] Step 35: Repeat step 34 to calculate the linear model under all principal components in the search parameter interval;

[0040] Step 36: According to formula y p =Xv ×(X r ×Y l )+Y mean , calculate the spectral prediction value y corresponding to the linear model p , y corresponding to all linear models p Composition set Y p .

[0041] Step 4: Set the model evaluation index, which can be evaluated using mean square error or mean absolute error;

[0042] Step 5: Screen the model based on the model evaluation indicators and save the optimal model.

[0043] Example

[0044] like Figure 2 As shown, take the data collected by the near-infrared spectrometer in the 1350nm-1650nm band as an example:

[0045] a. Importing the original spectrum: In this embodiment, the original spectrum data is a 340×51 matrix data, ie, containing 340 spectral data, each spectral data band number is 50, and the calibration value data is 1;

[0046] b. Preprocessing the original spectral data: This example uses 10 preprocessing methods, including Gaussian smoothing, Gaussian derivative, and SG smoothing. Each preprocessing method provides a wide range of parameter settings, totaling 2,000 preprocessing combinations. This wide range of preprocessing combinations is intended to increase the robustness of the online modeling model and thereby improve the prediction accuracy of the portable spectrometer.

[0047] c. Modeling using partial least squares: Set the search interval of the modeling principal component to [1, 2, ..., 15], and extract the maximum value of 15 in this interval as the maximum number of principal components of this algorithm. Decompose and calculate the pre-processed data in step b, and find the X corresponding to the principal component of 15. r , Y l Matrix, the slope is X r ×Y l , the intercept is Y mean Linear model; loop extracting X corresponding to principal components 1-14 r , Y l The predicted values under all principal components are finally calculated. Using the above method, only one model calculation under the largest principal component is required. Combined with step b, a total of 2000 model calculations are required.

[0048] d. Model evaluation index setting: For regression algorithms, evaluation indicators generally use MSE (mean square error) and MAE (mean absolute error). For this data modeling optimal model screening, the selected indicator is MSE;

[0049] e. Model screening: Using the model prediction values obtained in step b under different preprocessing combinations and different PLS principal component combinations, combined with the input calibration values, calculate the MSE matrix and select the model with the minimum MSE value as the optimal model for this modeling;

[0050] f. Save the optimal model: Through the calculation in step e, an optimal model can be finally locked, and the corresponding preprocessing combination and parameters, the principal component of the optimal PLS, and the minimum MSE value of the optimal model can be obtained and saved simultaneously;

[0051] g. Input of data to be predicted: Use a portable spectrometer to sample the spectrum of unknown samples in preparation for predictive analysis of the corresponding indicators of the target object;

[0052] h. Model prediction: The data obtained in step g is used as input, and the optimal model is selected as the relationship mapping between the spectral data and the calibration value. It can achieve real-time sampling, prediction, output, display and cloud storage of each item.

[0053] This example verifies that the original model screening process takes 18 days, but after optimization, the entire process only takes 30 hours, greatly improving the modeling efficiency.

Claims

1. Near infrared spectroscopy online modeling method based on partial least squares method, including: Step 1: Input the original spectrum data; Step 2: preprocessing the original spectral data; Step 3: Use partial least squares method to build a model; Step 4: Set model evaluation indicators; Step 5: Screen the model based on the model evaluation index and save the optimal model; It is characterized in that the step 3 specifically comprises: Step 31: Set the search interval for the main component of modeling; Step 32: extract the maximum number of principal components in the search parameter interval; Step 33: Centralize the preprocessed data to obtain the set X v ; Step 34: Substitute the data obtained in step 33 into the partial least squares method to obtain X under the largest principal component. r 、Y l value, and then fitting to get the slope as X r ×Y l , the intercept is Y mean A linear model where X r is the rotation load coefficient matrix, Y l is the linear load coefficient matrix; Step 35: Repeat step 34 to calculate the linear model under all principal components in the search parameter interval; Step 36: Calculate the spectral prediction value y corresponding to the linear model p , y corresponding to all linear models p Composition set Y p .

2. The near-infrared spectroscopy online modeling method based on partial least squares method according to claim 1, wherein The specific steps of preprocessing the original spectral data in step 2 are: Step 21: Select a preprocessing model, wherein the preprocessing model includes a preprocessing method and a corresponding parameter range; Step 22: Use a preprocessing model to preprocess the original spectral data.

3. The near-infrared spectroscopy online modeling method based on partial least squares method according to claim 2, characterized in that, The preprocessing methods include: Gaussian smoothing, Gaussian derivative, SG smoothing, detrending and standardization.

4. The near-infrared spectroscopy online modeling method based on partial least squares method according to claim 1, wherein In step 36, the spectrum prediction value y corresponding to the linear model p The calculation method of y is: p =X v ×(X r ×Y l )+Y mean .

5. The near-infrared spectroscopy online modeling method based on partial least squares method according to any one of claims 1 to 4, characterized in that: The model evaluation index in step 4 is the mean square error or the mean absolute error.

Images