Robust spectrum quantitative model screening method and system

Abstract

The invention provides a robust spectrum quantitative model screening method and system. The method comprises the following steps: collecting sample spectrums under multiple groups of different measurement conditions; setting different optimization conditions; constructing a model according to the set optimization condition, and calculating corresponding R2 and RMSECV; and evaluating the robustness of the model by using PrR2cv and PrRMSECV, and screening to obtain the most robust model. The technical problem that the prediction performance of the model is reduced due to the change of environmental conditions is solved.

Description

Technical Field

[0001] The present invention relates to the field of spectral analysis, and in particular to a robust spectral quantitative model screening method and system. Background Art

[0002] Spectral quantitative analysis technologies such as near-infrared and hyperspectral belong to secondary analysis technologies. Usually, it is necessary to build a relationship model between spectral data and reference values ​​obtained by a set of standard methods before it can be used for prediction. However, the prediction results of quantitative models are easily affected by changes in detection conditions, which is a long-standing problem of modern spectral technologies such as near-infrared spectroscopy and hyperspectral technology. When the sample state, environmental conditions, and instrument state change, the test results often deviate and may even invalidate the model.

[0003] The existing invention patent application document "LIBS spectroscopy quantitative gold mine method based on ultraviolet band and model stacking strategy" with publication number CN118193977A, the existing method includes: S1, using a LIBS spectrometer with an ultraviolet band to collect spectral data and establish a data set; S3, spectral data preprocessing and spectral feature extraction; S4, base learner and meta learner model selection and establishment; S5, using a model stacking strategy to establish a quantitative inversion model; S6, result verification.

[0004] The existing invention patent application document "A method for determining soil organic matter based on an optimization model" with publication number CN111488926A, the existing method includes: collecting soil samples; determining the organic matter content of soil samples by chemical methods; collecting soil laser induced breakdown spectrum information after tableting, and collecting it multiple times continuously; preprocessing the spectrum, including baseline correction, noise reduction, averaging, and standardization; collecting laser induced breakdown spectrum information of the sample to be tested, and collecting it multiple times continuously, and performing spectrum preprocessing; calculating the mixed vector value of the spectrum of the sample to be tested and the spectrum of the calibration set sample, and sorting them from small to large; selecting the optimal sub-model according to the prediction parameters of the model for each sub-calibration set and the leave-one-out cross-validation method; using the optimal sub-model to predict the organic matter content of the soil sample to be tested.

[0005] In the existing public document "Quantitative Analysis of Iron Ore Slurry by Laser Induced Breakdown Spectroscopy Based on Partial Least Squares with Mutual Information Feature Screening", the partial least squares algorithm (PLS) is used to analyze and model the LIBS full spectrum data, in which the determination coefficient R° of the iron element reaches 0.826 and the root mean square error (RMSE) is 2.86%. The laser induced breakdown spectroscopy analysis technology is applied to the rapid detection of iron grade in iron ore tailings, and a modeling method combining mutual information feature screening with partial least squares is proposed. A method based on mutual information feature screening partial least squares is proposed to perform feature screening. When performing feature screening, this method considers the correlation between each column feature of the spectrum and the grade value concentration, including linear and nonlinear relationships. However, in the above two existing patents and public documents, the sample set used for modeling is only collected and processed under uniform collection conditions, and is not processed under a variety of different collection conditions; in addition, the indicators for evaluating the model effect are only the correlation coefficient R or the determination coefficient R 2 Or traditional indicators such as root mean square error (RMSE), these parameters are calculated based on the predicted values ​​and reference values ​​of the same set of models, and do not fully consider the impact of environmental differences on measurements. In order to overcome the impact of different environments on measurements, it is still necessary to design an experimental scheme that can test and evaluate the prediction performance of the model under multiple detection conditions and a corresponding new evaluation and screening index. This screening index is calculated based on the predicted values ​​and reference values ​​of the model under multiple sets of prediction conditions, rather than just one set.

[0006] In summary, the existing technology has the technical problem that the model prediction performance decreases due to changes in environmental conditions. Summary of the invention

[0007] The technical problem to be solved by the present invention is: how to solve the technical problem in the prior art that the model prediction performance decreases due to changes in environmental conditions.

[0008] The present invention adopts the following technical solutions to solve the above technical problems: The robust spectral quantitative model screening method comprises:

[0009] S1. Under multiple groups of different measurement conditions, collect n groups of sample spectra to be measured and obtain sample reference values;

[0010] S2. Setting difference optimization conditions;

[0011] S3. According to the difference optimization conditions, for the mn group spectra and sample reference values, a spectrum-reference value relationship model is constructed, and the cross-validation determination coefficient R corresponding to each spectrum-reference value relationship model is calculated. 2 cv and the cross-validation root mean square error R MSECV ;

[0012] S4. Using the cross-validation determination coefficient R2 cv , cross validation root mean square error R MSECV The robustness of the model of the relationship between the spectrum and the reference value was evaluated, and suitable robust models were screened accordingly.

[0013] The present invention performs reasonable optimization in the modeling stage, screens models with high robustness, obtains models that are less affected by the measurement environment, can avoid re-modeling, and ensures the applicability and accuracy of the model during prediction.

[0014] In a more specific technical solution, in S2, m types of pre-processing and band range combinations are designed to process the above n groups of sample spectra to be measured to obtain mn groups of spectra.

[0015] In a more specific technical solution, in S2, the qth optimized spectrum under the pth acquisition environment is recorded as X (p,q) .

[0016] In a more specific technical solution, in S2, the optimized processing characteristic bands of the m-group sample spectra to be tested use the same bands as the n-group sample spectra to be tested.

[0017] The present invention adopts a quantitative multi-condition measurement method (QAMCM) to obtain a model with robust optimization parameters, thereby avoiding the problem of degradation of model prediction performance due to changes in environmental conditions.

[0018] In a more specific technical solution, in S3, for the mn groups of spectra and the corresponding sample reference values ​​Y, a preset quantitative modeling algorithm is used to construct a relationship model between the spectra of the mn groups and the reference values.

[0019] In a more specific technical solution, in S3, the preset quantitative modeling algorithms include: partial least squares method, artificial neural network and support vector machine.

[0020] In a more specific technical solution, in S3, under the pth acquisition environment, the qth optimization process is performed, and the cross-validation determination coefficient R 2 cv , cross validation root mean square error R MSECV , respectively RMSECV (p,q) .

[0021] The present invention can be used to screen out a partial least squares model with robust optimization parameters, effectively solving the problem of decreased model prediction performance due to changes in environmental conditions. Therefore, the screened model parameters can maintain good prediction performance even when the acquisition conditions change.

[0022] In a more specific technical solution, in S4, according to the cross-validation determination coefficient R 2 cv , cross validation root mean square error R MSECV The model of the relationship between the spectrum and the reference value is evaluated. For the qth group of optimization conditions, the following logic is used to process the cross-validation determination coefficient R 2 cv , cross validation root mean square error R MSECV :

[0023] .

[0025] In a more specific technical solution, in S4, according to the cross-validation determination coefficient R 2 cv , cross validation root mean square error R MSECV , processing to obtain the cross-validation determination coefficient parameter of m groups and the cross-validation root mean square error parameter Pr for m groups RMSECV , specifically by setting applicable conditions; under applicable conditions, a applicable robust model is constructed.

[0026] In a more specific technical solution, the robust spectral quantitative model screening system includes:

[0027] The spectrum and reference value acquisition module is used to collect n groups of sample spectra to be measured under multiple groups of different measurement conditions and obtain sample reference values;

[0028] Condition setting module, used to set difference optimization conditions;

[0029] The model index obtaining module is used to construct the spectrum and reference value relationship model for mn groups of spectra and sample reference values ​​according to the difference optimization conditions, and calculate the cross-validation determination coefficient R corresponding to each spectrum and reference value relationship model 2 cv and cross validation root mean square error R MSECV , the model index obtaining module is connected with the condition setting module and the spectrum and reference value obtaining module;

[0030] Model screening module to use the cross-validation determination coefficient R 2 cv , cross validation root mean square error R MSECV The robustness of the spectrum-reference value relationship model is evaluated, and a suitable robust model is screened based on it. The model screening module is connected to the model indicator obtaining module.

[0031] Compared with the prior art, the present invention has the following advantages:

[0032] The present invention performs reasonable optimization in the modeling stage, screens models with high robustness, obtains models that are less affected by the measurement environment, can avoid re-modeling, and ensures the applicability and accuracy of the model during prediction.

[0033] The present invention adopts a quantitative multi-condition measurement method (QAMCM) to obtain a model with robust optimization parameters, thereby avoiding the problem of degradation of model prediction performance due to changes in environmental conditions.

[0034] The present invention can be used to screen out a partial least squares model with robust optimization parameters, effectively solving the problem of decreased model prediction performance due to changes in environmental conditions. Therefore, the screened model parameters can maintain good prediction performance even when the acquisition conditions change.

[0035] The present invention solves the technical problem in the prior art that the model prediction performance decreases due to changes in environmental conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 Schematic diagram of the basic steps of the robust spectral quantitative model screening method of Example 1 of the present invention. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in combination with the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0038] Example 1

[0039] like Figure 1 As shown, the robust spectral quantitative model screening method provided by the present invention includes the following basic steps:

[0040] S1. Collect sample spectra under multiple sets of different measurement conditions;

[0041] In this embodiment, under two or more different instrument or environmental conditions, assuming that there are n types of acquisition environments, which are respectively recorded as environment 1, environment 2, ... environment n; the spectrum of the target sample is collected and the reference value of each sample is obtained, and finally n sets of spectra can be obtained, whose spectrum matrices are respectively recorded as X1 and X2, and the corresponding reference values ​​of the components to be measured are the same, which are recorded as Y;

[0042] S2, setting different optimization conditions;

[0043] In this embodiment, m combinations of pre-processing and band ranges are designed to process the above n groups of spectra to be measured, and obtain mn groups of spectra, wherein the spectrum after the qth (q∈[1,m] and p∈N*) optimization process under the pth acquisition environment (p∈[1,n] and q∈N*) is recorded as X (p,q) ;

[0044] In this embodiment, the characteristic bands used in the m groups of optimization processing are the bands common to the spectra collected under the n environments;

[0045] In this embodiment, it can be combined with various band screening methods, including but not limited to: interval interval partial least squares method. These band screening methods only need to meet the condition that the band range of each screening of the spectra collected under these n environments is the same, and can be used in combination with the method of the present invention.

[0046] S3, build a model according to the set optimization conditions and calculate the corresponding R2 and RMSECV;

[0047] In this embodiment, for the above mn groups of spectra and the corresponding reference values ​​Y of the components to be measured, various quantitative modeling algorithms, such as partial least squares method, are used to construct a relationship model between the spectra and the reference values, and a total of mn groups of models are obtained. The cross-validation determination coefficient R of each model is calculated. 2 cv and cross validation root mean square error R MSECV ;

[0048] In this embodiment, the qth optimization process under the pth acquisition environment, R 2 cv and R MSECV Denoted as and RMSECV (p,q) ;

[0049] In this embodiment, the method of constructing a relationship model between the spectrum and the reference value includes but is not limited to: partial least squares method for modeling, and other modeling algorithms such as artificial neural network, support vector machine, etc. for modeling; in this embodiment, partial least squares method can be replaced by the other aforementioned modeling algorithms.

[0050] S4. Using the cross-validation determination coefficient R 2 cv and cross validation root mean square error R MSECV Evaluate the robustness of the model and select the most robust model.

[0051] In this embodiment, using or Pr RMSECV Two indicators are used to evaluate the optimization performance of the model. For the qth group of optimization conditions, and Pr RMSECV The formulas are:

[0052]

[0053] According to the above formula, a total of m groups can be obtained and m group Pr RMSECV ,by Maximum or Pr RMSECV The optimization condition at the lowest value is the optimal condition, and all models built under this condition are considered to be the most robust and accurate models.

[0054] Example 2

[0055] The present invention takes the near infrared detection and hyperspectral detection of the vitamin C content in pepper as an example, and provides a method for screening a robust spectral quantitative model for the vitamin C content in pepper. In the quantitative model multi-condition measurement method (QAMCM), 70 pepper fruits are used as test samples, and these samples come from 11 pepper varieties, and each sample is composed of multiple fruits of the same variety. The spectra of these pepper samples are collected under two different instrument conditions and the reference value of each sample is obtained.

[0056] The first acquisition condition includes: using a MPA Fourier transform spectrometer to collect spectra. Each pepper fruit is placed in the center of the acquisition window, and the spectrum is collected in a diffuse reflection mode. The spectrum collection range is 11988-3996cm -1 , with a resolution of 7.7cm -1 Since each pepper sample is composed of multiple fruits of the same variety, after the pepper fruits belonging to the same sample are collected, they are averaged into a spectrum as the near-infrared spectrum of the sample.

[0057] The second acquisition condition includes: using a Specim FX17 hyperspectral camera to collect spectral data of peppers in the near infrared (NIR) region (900-1700nm). The camera has a spectral resolution of 8nm, a spectral pixel of 3.5nm, and is equipped with a mobile platform for line scanning sampling.

[0058] In this embodiment, in order to reduce the influence of unstable light source and dark current of sensor during experimental acquisition, black and white correction is used to improve the quality of experimental data. The black and white correction formula is as follows:

[0059]

[0060] Among them, I is the sample hyperspectral image after black and white correction, I raw It represents the original hyperspectral image of the sample collected by placing the sample on the sampling platform; I whiteIt represents the white spectrum image of a polytetrafluoroethylene white board with a reflectivity close to 100% collected under complete lighting conditions; dark Represents a dark spectrum image acquired in complete darkness.

[0061] In this embodiment, the region of interest (ROI) algorithm is used to segment the pepper image from the background image. In the pepper fruit hyperspectral image segmentation operation, I is linearly scaled from the interval [0,1] to the interval [0,255] to correspond to the pixel value interval. Since HSI contains 224 channels, each channel records the spectrum of the corresponding band, and then the average pixel value of the spectrum in each channel is calculated, and the i-th channel is W i ; Since the pepper hyperspectral image contains two parts: pepper image and black background image, the pixel value of the black background must be smaller than the pixel mean W i +C. C is a certain threshold, which is determined by adjusting the parameters. i The pixels of +C are extracted to achieve the segmentation of the pepper hyperspectral image and the background image.

[0062] In this embodiment, the segmented pepper hyperspectral images are averaged to obtain a spectrum, which is the average hyperspectral curve of the current pepper fruit. Afterwards, as in the step described in acquisition condition 1, since each pepper sample is composed of multiple fruits of the same variety, after the hyperspectral curves of each pepper fruit belonging to the same sample are calculated, they are further averaged into a spectrum as the hyperspectral data of the sample for subsequent modeling analysis.

[0063] According to the Chinese national standard “GB5009.86-2016-kw National Food Safety Standard Determination of Ascorbic Acid in Food”, the vitamin C content of each pepper sample was chemically determined to obtain 70 vitamin C reference values, which were sequentially constructed into a 70×1 vector, recorded as Y.

[0064] In this embodiment, 12 experimental groups with different pretreatments and band ranges are designed, and two groups of spectra to be measured under acquisition conditions 1 and acquisition conditions 2 are processed to obtain a total of 24 groups of spectra. Since the range of the pepper hyperspectral spectrum is narrow and the range of the near-infrared spectrum is wide, the range of the pepper hyperspectral spectrum, i.e., 861-1746nm, is uniformly used in experimental groups 1, 2, and 3 for modeling, and three different pretreatments are set: no pretreatment, vector normalization (snv), and first-order derivative; in experimental groups 4-9, two typical groups of bands are set: 1000-1450nm and 1400-1740nm for modeling, and the three pretreatments are also combined for analysis; in experimental groups 10-12, further selection is made within the range of 1000-1450nm, and 1300-1230, 1190-1150, and 1112-1040nm are used for modeling, and these bands are also combined with the three pretreatments.

[0065] For the above 24 groups of spectra, including 12 groups of spectra under condition 1 and 12 groups of spectra under condition 2, and the corresponding reference value Y of the component to be measured, the relationship model between the pepper spectrum and the vitamin C reference value was constructed using partial least squares method, and a total of 24 groups of models were obtained. The cross-validation determination coefficient (R 2 cv ) and cross-validation root mean square error (RMSECV). The cross-validation results of the near-infrared model and the hyperspectral model of the vitamin C content of peppers under various optimized treatment conditions are shown in Table 1:

[0066] Table 1 Cross-validation results of near-infrared model and hyperspectral model for vitamin C content in peppers under various optimized treatment conditions

[0067]

[0068]

[0069] From the results in Table 1 above, we can see that different optimization conditions have a huge impact on the model. For the pepper hyperspectral, the processing model of experimental group 10 has the highest accuracy, while for the pepper near-infrared spectrum, the processing model of experimental group 11 has the highest accuracy. Since the optimal processing conditions are different, choosing which optimization condition is the most robust modeling is the next step.

[0070] In this embodiment, for the aforementioned p-th experimental group, record and are the cross-validation determination coefficients of the partial least squares models under acquisition conditions 1 and 2 respectively; denoted by RMSECV (1,p) and RMSECV (2,p)The cross-validation root mean square error of the partial least squares model under collection condition 1 and collection condition 2 is respectively obtained; or Pr RMSECV Two indicators are used to evaluate the optimization performance of the model: Maximum or Pr RMSECV The lowest model is the optimal model, which is considered to be the most robust and accurate model among the optimized models. and Pr RMSECV The formulas are:

[0071]

[0072] Pr RMSECV =RMSECV (1,p) ×RMSECV (2,p)

[0073] The screening results of the most robust pepper vitamin C spectral model under each optimization condition are shown in Table 2 below:

[0074] Table 2 Screening results of the most robust pepper vitamin C spectral model under various optimization conditions

[0075]

[0076] In Table 2 above, each experimental group and Pr RMSECV It is based on the R of the pepper hyperspectral model and near infrared spectroscopy model with the same number in Table 1 above. 2 cv and R MSECV As shown in Table 2 above, according to this step, the problem of inconsistent optimal model parameters in the two measurement environments in the previous step is solved, and the comprehensive optimal parameters are experimental group 11 (1300-1230, 1190-1150, 1112-1040nm, SNV pretreatment). The highest among all experimental groups, Pr RMSECV It is the lowest among all experimental groups, so the near-infrared model and hyperspectral model of pepper vitamin C constructed under this condition are considered to be the most robust models. Through the QAMCM method consisting of the above series of steps, the most robust modeling parameters are established, and the model constructed based on this parameter can show better prediction performance and anti-environmental interference ability than other models as a whole.

[0077] In summary, the present invention performs reasonable optimization in the modeling stage, screens models with high robustness, obtains models that are less affected by the measurement environment, can avoid re-modeling, and ensures the applicability and accuracy of the model during prediction.

[0078] The present invention adopts a quantitative multi-condition measurement method (QAMCM) to obtain a model with robust optimization parameters, thereby avoiding the problem of degradation of model prediction performance due to changes in environmental conditions.

[0079] The present invention can be used to screen out a partial least squares model with robust optimization parameters, effectively solving the problem of decreased model prediction performance due to changes in environmental conditions. Therefore, the screened model parameters can maintain good prediction performance even when the acquisition conditions change.

[0080] The present invention solves the technical problem in the prior art that the model prediction performance decreases due to changes in environmental conditions.

[0081] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A robust spectral quantitative model screening method, characterized in that: The method comprises: S1. Under multiple groups of different measurement conditions, collect n groups of sample spectra to be measured and obtain sample reference values; S2. Setting difference optimization conditions; S3, constructing a spectrum-reference value relationship model for the mn group spectrum and the sample reference value according to the difference optimization condition, and calculating the cross-validation determination coefficient R corresponding to each spectrum-reference value relationship model 2 cv and the cross-validation root mean square error R MSECV ; S4, using the cross-validation determination coefficient R 2 cv , the cross validation root mean square error R MSECV The robustness of the model of the relationship between the spectrum and the reference value is evaluated, and a suitable robust model is screened accordingly.

2. The robust spectral quantitative model screening method according to claim 1, characterized in that: In the above S2, m types of pre-processing and wavelength range combinations are designed to process the above n groups of sample spectra to be measured, and obtain mn groups of spectra.

3. The robust spectral quantitative model screening method according to claim 1, characterized in that: In S2, the qth optimized spectrum under the pth acquisition environment is recorded as X (p,q) .

4. The robust spectral quantitative model screening method according to claim 1, characterized in that: In S2, the optimized characteristic bands of the m groups of sample spectra to be tested use a common band with the n groups of sample spectra to be tested.

5. The robust spectral quantitative model screening method according to claim 1, characterized in that: In S3, a preset quantitative modeling algorithm is used to construct a relationship model between the spectrum and the reference value of the mn group of spectra and the corresponding sample reference value Y.

6. The robust spectral quantitative model screening method according to claim 1, characterized in that: The S3, the preset quantitative modeling algorithm includes: partial least squares method, artificial neural network and support vector machine.

7. The robust spectral quantitative model screening method according to claim 1, characterized in that: In S3, under the pth collection environment, the qth optimization process is performed to set the cross-validation determination coefficient R 2 cv , the cross validation root mean square error R MSECV , respectively denoted as R 2 cv(p,q) RMSECV (p,q) .

8. The robust spectral quantitative model screening method according to claim 1, characterized in that: In S4, according to the cross-validation determination coefficient R 2 cv , the cross validation root mean square error R MSECV Evaluate the spectrum-reference value relationship model, wherein the optimization conditions of the qth group are processed using the following logic to obtain the cross-validation determination coefficient R 2 cv , the cross validation root mean square error R MSECV : 。 9. The robust spectral quantitative model screening method according to claim 1, characterized in that: In S4, according to the cross-validation determination coefficient R 2 cv , the cross validation root mean square error R MSECV , processing to obtain the cross-validation determination coefficient parameter of m groups and the cross-validation root mean square error parameter Pr for m groups RMSECV , specifically by setting applicable conditions; under the applicable conditions, the applicable robust model is constructed.

10. A robust spectral quantitative model screening system, characterized in that: The system comprises: The spectrum and reference value acquisition module is used to collect n groups of sample spectra to be measured under multiple groups of different measurement conditions and obtain sample reference values; Condition setting module, used to set difference optimization conditions; The model index obtaining module is used to construct a spectrum-reference value relationship model for the mn group spectrum and the sample reference value according to the difference optimization condition, and calculate the cross-validation determination coefficient R corresponding to each spectrum-reference value relationship model. 2 cv and the cross-validation root mean square error R MSECV , the model index obtaining module is connected with the condition setting module and the spectrum and reference value obtaining module; Model screening module, used to use the cross-validation determination coefficient R 2 cv , the cross validation root mean square error R MSECV The robustness of the spectrum-reference value relationship model is evaluated, and a suitable robust model is screened accordingly. The model screening module is connected to the model index obtaining module.

Images