Multi-weight vector optimization based near infrared spectroscopy data variable selection method and system
By employing a multi-weighted vector optimization method, the problem of noise interference in near-infrared spectral data variable selection is solved, improving modeling accuracy and detection capability, simplifying the variable selection process, and enhancing the model's predictive performance.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HEFEI INSTITUTE OF PHYSICAL SCIENCE CHINESE ACADEMY OF SCIENCES
- Filing Date
- 2022-10-26
- Publication Date
- 2026-07-03
AI Technical Summary
In the selection of variables for near-infrared spectral data, existing technologies often rely on the weighted superposition of multiple weight vectors, which is susceptible to noise interference, leading to a decline in prediction performance and making it difficult to improve modeling accuracy while reducing the dimensionality of variables.
A multi-weight vector optimization method is adopted. A subset of data is generated by uniform probability random sampling, a sub-model is generated by partial least squares algorithm, and the optimal model subset is selected by cross-validation error sorting. The minimum cross-validation error of the weight vector is calculated by step search method, and the optimal weight vector and variable subset are selected. The iteration continues until the number of variables is 1 to reduce noise interference.
It improves the accuracy of near-infrared spectral modeling and detection, simplifies the variable selection process, enhances the predictive power and stability of the model, and eliminates the need for preprocessing spectral data.
Smart Images
Figure CN115630506B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of chemometrics variable selection in near-infrared spectroscopy data processing, and specifically to a method for variable selection in high-dimensional data. Background Technology
[0002] With advancements in spectral analysis and detection technologies and instruments, spectral techniques have been widely applied in numerous fields, including petroleum refining, agriculture, medicine, chemistry, and environmental protection. In the qualitative and quantitative analysis of spectral models, extracting feature information from complex high-dimensional spectral data, reducing the impact of noise and extraneous variables on the analysis results, and improving the model's predictive power are crucial for establishing robust predictive models. Multivariate calibration algorithms, such as principal component regression and partial least squares regression (PLSR) for quantitative analysis, and partial least squares discriminant analysis (PLS-DA) for qualitative analysis, are commonly used to address this issue by extracting latent information from spectral datasets. However, many studies have found that selecting spectral data variables after model construction can significantly improve the model's predictive and interpretative capabilities. Furthermore, this method can be used to develop low-cost, multi-channel spectral detection instruments. The spectral variable space is typically large (NP-hard) compared to the number of modeling samples. Selecting the most efficient wavelength combinations from a high-dimensional variable space to build a spectral prediction model remains challenging.
[0003] The present invention, disclosed in CN106950193A, relates to a method for near-infrared spectroscopy variable selection based on self-weighted variable combination cluster analysis. The specific implementation process is as follows: First, the variable space is randomly sampled using the binary matrix sampling method (BMS). Second, the contribution value of each spectral variable is obtained by weighting two information vectors (IVs): frequency of occurrence (Fre) and partial least squares regression coefficient (Reg), thus considering the influence of Fre and Reg IVs on the importance of the variable. Finally, variables with small contribution values are removed using the exponential decay function (EDF) to achieve feature variable selection. Compared with existing technologies, this method has the advantages of being fast and repeatable, and improves the prediction accuracy and stability of the model. However, this method uses two information vector weighting methods, resulting in a fixed-size variable set of 14 variables after iteration. Then, a full permutation calculation is needed to find the minimum value of all results. The final prediction result is 0.0248, which is not good. This method uses a weighted approach, superimposing two weight vectors with certain weights. However, the quality of the weight vectors themselves depends on the samples. For example, when there is a lot of noise, Reg will be significantly affected by the noise. Superimposing multiple weight vectors will inevitably reduce the performance of the prediction results. Summary of the Invention
[0004] The technical problem to be solved by this invention is how to select the optimal values of multiple weight vectors in order to reduce the dimensionality of variables while reducing the interference of noise on variable selection, thereby improving the accuracy of near-infrared spectral modeling and detection.
[0005] The present invention solves the above-mentioned technical problems through the following technical means:
[0006] A near-infrared spectral data variable selection method based on multi-weighted vector optimization includes the following steps:
[0007] S1: Randomly sample the sample variable space based on uniform probability to generate multiple subset datasets;
[0008] S2: Generate sub-models for each subset of data based on the partial least squares algorithm; sort all sub-models according to the cross-validation error, and select a certain proportion of the model subsets from them in ascending order as the optimal model subset;
[0009] S3: Select weight vectors, calculate weight vector values based on the optimal model subset, normalize, amplify and smooth the weight vectors, use the step search method to calculate the minimum cross-validation error of each weight vector and its corresponding variable subset; select the weight vector with the minimum cross-validation error as the optimal weight vector, and record the corresponding variable subset;
[0010] S4: Use the optimal weight vector as the variable selection probability to further sample the variable space and generate a new set of sub-models;
[0011] S5: Repeat steps S2 to S4 until the number of variables in the subset selected from the variable space is 1, then stop the iteration; select the subset with the lowest cross-validation error during the iteration process as the optimal variable set.
[0012] Furthermore, the weight vector in step S3 includes the regression coefficient weight vector, the variable projection importance weight vector, the selection ratio weight vector, the multivariate correlation weight vector, the inverse of the residual variance vector, and the frequency weight vector.
[0013] Furthermore, the weight vector is defined as follows:
[0014] (1) Regression coefficient weight vector
[0015] Regression coefficients measure the dependence of y on X; the absolute value of the regression coefficients is used as an estimate of the importance of the variable.
[0016] (2) Variable projection importance weight vector
[0017] The variable projection importance weight vector accumulates by weighting each PLS component to reflect the importance of each variable; it is defined as follows:
[0018]
[0019] Among them SS a It is the weighted sum of squares of the covariances between X and y; A represents the number of the largest latent variables; the subscript i represents the i-th variable;
[0020] (3) Select the weight vector
[0021] Selectivity ratio SR is expressed as the explained variance SS i,explained and residual variance SS i,residual The ratio;
[0022]
[0023] Where t TP p is obtained by projecting X rows onto the normalized regression coefficients; TPi This is obtained by projecting column X onto the top; It is the residual after X projection; the subscript i indicates the i-th variable;
[0024] (4) Multivariate correlation weight vector
[0025] The multivariate correlation weight vector can minimize the influence of irrelevant X structures and highlight the variables most relevant to the response; it is expressed as:
[0026]
[0027] in This reflects the relevant changes in the predicted response projected back into the original X variable space via PLS; B is the residual variance, and B is the regression coefficient. It is a predicted value. It is the residual after X-projection;
[0028] (5) Reciprocal of the residual variance vector
[0029] (6) Frequency weight vector
[0030] The frequency weight vector is obtained by calculating the number of times the variables appear in the optimal model subset, reflecting the frequency of occurrence of the variables in the optimal model subset.
[0031] Corresponding to the above method, the present invention also provides a near-infrared spectral data variable selection system based on multi-weighted vector optimization, comprising:
[0032] The subset generation module is used to randomly sample the sample variable space based on uniform probability to generate multiple subsets.
[0033] The optimal model subset selection module is used to generate sub-models for each subset of data based on the partial least squares algorithm; it sorts all the sub-models according to the cross-validation error and selects a certain proportion of the model subsets from them in ascending order as the optimal model subset;
[0034] The weight vector selection module is used to calculate weight vector values based on the optimal model subset, normalize, amplify, and smooth the weight vectors, calculate the minimum cross-validation error of each weight vector and its corresponding variable subset using a step search method, select the weight vector with the minimum cross-validation error as the optimal weight vector, and record the corresponding variable subset.
[0035] The sub-model generation module is used to further sample the variable space using the optimal weight vector as the variable selection probability to generate a new set of sub-models.
[0036] The iterative module repeats steps S2 to S4 until the number of variables in the subset selected from the variable space is 1, at which point the iteration stops; the subset with the lowest cross-validation error during the iteration process is selected as the optimal variable set.
[0037] Furthermore, the weight vector selection module includes a regression coefficient weight vector, a variable projection importance weight vector, a selection ratio weight vector, a multivariate correlation weight vector, the inverse of the residual variance vector, and a frequency weight vector.
[0038] Furthermore, the weight vector is defined as follows:
[0039] (1) Regression coefficient weight vector
[0040] Regression coefficients measure the dependence of y on X; the absolute value of the regression coefficients is used as an estimate of the importance of the variable.
[0041] (2) Variable projection importance weight vector
[0042] The variable projection importance weight vector accumulates by weighting each PLS component to reflect the importance of each variable; it is defined as follows:
[0043]
[0044] Among them SS a It is the weighted sum of squares of the covariances between X and y; A represents the number of the largest latent variables; the subscript i represents the i-th variable;
[0045] (3) Select the weight vector
[0046] Selectivity ratio SR is expressed as the explained variance SS i,explained and residual variance SS i,residual The ratio;
[0047]
[0048] Where t TP p is obtained by projecting X rows onto the normalized regression coefficients; TPi This is obtained by projecting column X onto the top; It is the residual after X projection; the subscript i indicates the i-th variable;
[0049] (4) Multivariate correlation weight vector
[0050] The multivariate correlation weight vector can minimize the influence of irrelevant X structures and highlight the variables most relevant to the response; it is expressed as:
[0051]
[0052] in This reflects the relevant changes in the predicted response projected back into the original X variable space via PLS; B is the residual variance, and B is the regression coefficient. It is a predicted value. It is the residual after X-projection;
[0053] (5) Reciprocal of the residual variance vector
[0054] (6) Frequency weight vector
[0055] The frequency weight vector is obtained by calculating the number of times the variables appear in the optimal model subset, reflecting the frequency of occurrence of the variables in the optimal model subset.
[0056] Corresponding to the above method, the present invention also provides a near-infrared spectral data variable selection device based on multi-weight vector optimization, including at least one processor and at least one memory communicatively connected to the processor, wherein: the memory stores program instructions executable by the processor, and the processor can execute the above method by calling the program instructions.
[0057] Corresponding to the above method, the present invention also provides a computer-readable storage medium, characterized in that the computer-readable storage medium stores computer instructions that cause the computer to perform the above method.
[0058] The advantages of this invention are:
[0059] This invention employs a step-by-step search strategy to find the optimal values of multiple weight vectors. Compared to a single weight vector, this approach combines the advantages of multiple weight vectors, enabling the selection of the most important variables. This invention also proposes a weight vector smoothing strategy to reduce interference from noisy variables and improve the effectiveness of variable selection. No preprocessing is required for the input near-infrared spectral data.
[0060] The model built based on the variables selected in this invention has high modeling and predictive capabilities. Because the step search strategy evaluates each weight vector individually and then compares the RMSECV values of multiple weight vectors, it offers strong flexibility and can easily be extended to include more weight vectors. Attached Figure Description
[0061] Figure 1 This is a flowchart of the near-infrared spectral data variable selection method based on multi-weight vector optimization in an embodiment of the present invention;
[0062] Figure 2 In this embodiment of the invention, corn spectral data was selected, with corn protein content as the chemical value, and the variable selection statistical results were obtained from 100 repeated calculations.
[0063] Figure 3 The statistical results of variable selection are obtained by selecting corn spectral data and using corn oil content as the chemical value in the embodiments of the present invention, and performing 100 repeated calculations. Detailed Implementation
[0064] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0065] The following embodiments use near-infrared spectral data of maize as an example to describe the present invention in detail. A method for selecting near-infrared spectral data variables based on multi-weighted vector optimization includes the following steps:
[0066] S1: Randomly sample the sample variable space based on uniform probability to generate multiple subset datasets.
[0067] The corn spectral sample contains 960 variables. The probability weights of all variables in the sample are set to an equal value of 1 / 960, resulting in 1000 subset datasets.
[0068] S2: Generate sub-models for each subset of data based on the partial least squares algorithm. Sort all sub-models according to the cross-validation error, and select a certain proportion of the sub-models from them in ascending order as the optimal sub-model set.
[0069] From 1000 datasets containing different subsets of variables, a sub-model was generated for each subset using the partial least squares algorithm, resulting in 1000 sub-models. These sub-models were then sorted in ascending order based on their cross-validation error. The top 10% of the sorted models were selected as the optimal model subset.
[0070] S3: Select a weight vector, calculate the weight vector value based on the optimal model subset, normalize, amplify and smooth the weight vector, and use the step search method to find the optimal weight vector and the corresponding optimal variable subset in the optimal model subset.
[0071] Six weight vectors were selected to process the maize spectral dataset, including the regression coefficient weight vector, variable projection importance weight vector, selection ratio weight vector, multivariate correlation weight vector, inverse of residual variance vector, and frequency weight vector. These weight vectors are defined as follows:
[0072] (1) Regression coefficients measure the dependence of y on X. The absolute value of the regression coefficient can be used as an estimate of the importance of the variable.
[0073] (2) Variable projection importance weight vector
[0074] The variable projection importance weight vector accumulates to reflect the importance of each variable by weighting each PLS component. It is defined as follows:
[0075]
[0076] Among them SS a It is the weighted sum of squares of the covariances between X and y. A represents the number of the largest latent variables. The subscript i indicates the i-th variable.
[0077] (3) Select the weight vector
[0078] The selectivity ratio (SR) can be expressed as the explained variance SS. i,explained and residual variance SS i,residual The ratio of .
[0079]
[0080] Where t TP This is obtained by projecting the X-row onto the normalized regression coefficients. TPi This is obtained by projecting column X onto the top. It is the residual after projection of X. The subscript i indicates the i-th variable.
[0081] (4) Multivariate correlation weight vector
[0082] The multivariate correlation weight vector can minimize the influence of irrelevant X structures and highlight the variables most relevant to the response. It can be represented as:
[0083]
[0084] in This reflects the relevant changes in the predicted response projected back into the original X variable space via PLS. B is the residual variance. B is the regression coefficient. It is a predicted value. It is the residual after X-projection.
[0085] (5) Reciprocal of the residual variance vector
[0086] (6) Frequency weight vector
[0087] The frequency weight vector is obtained by calculating the frequency of occurrence of variables in the optimal model subset. This vector reflects the frequency of occurrence of variables in the optimal model subset.
[0088] The minimum cross-validation error and corresponding variable subset of each of the six weight vectors are calculated using a step search method. The weight vector with the minimum cross-validation error is selected as the optimal weight vector, and its corresponding variable subset is recorded.
[0089] S4: Use the optimal weights as the variable selection probability to further sample the variable space and generate a new set of sub-models.
[0090] S5: Repeat steps two through four until the number of variables in the variable subset is 1, then stop the iteration. Select the subset with the lowest cross-validation error during the iteration process as the optimal variable set.
[0091] Variable selection was repeated 100 times for maize spectra, yielding statistical results and modeling prediction results for these 100 instances. The statistical results are shown in Tables 1 and 2. Table 1 shows the mean and standard deviation of modeling error, cross-validation error, and prediction error when maize protein content is selected as the chemical value y. Table 2 shows the mean and standard deviation of modeling error, cross-validation error, and prediction error when maize oil content is selected as the chemical value y. Tables 1 and 2 show that using the multi-weighted vector optimization variable selection method significantly improves the modeling and detection capabilities of maize spectral data. For ease of comparison, the tables also include results obtained without variable selection, i.e., using only partial least squares modeling and prediction.
[0092] Table 1. Results of corn protein data
[0093]
[0094] Table 2. Data Results on Corn Oil Content
[0095] method Number of variables ± standard deviation Modeling error ± standard deviation Cross-validation error ± standard deviation Prediction error ± standard deviation Partial Least Squares 700 0.049 0.064 0.0727 Multi-weighted vector optimization 61.87±8.59 0.010±0.001 0.014±0.001 0.015±0.001
[0096] Figure 2 and Figure 3 Statistical results of 100 repeated wavelength selections were presented when the protein content and oil content of corn were selected as chemical values.
[0097] Through the above technical solutions, the near-infrared spectral data variable selection method based on multi-weight vector optimization provided by the present invention does not require spectral preprocessing, is simple and fast, and can effectively select near-infrared spectral variables, thereby improving modeling and prediction accuracy.
[0098] Corresponding to the above method, the present invention also provides a near-infrared spectral data variable selection device based on multi-weight vector optimization, including at least one processor and at least one memory communicatively connected to the processor, wherein: the memory stores program instructions executable by the processor, and the processor can execute the above method by calling the program instructions.
[0099] Corresponding to the above method, the present invention also provides a computer-readable storage medium, characterized in that the computer-readable storage medium stores computer instructions that cause the computer to perform the above method.
[0100] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for selecting near-infrared spectral data variables based on multi-weighted vector optimization, characterized in that, Includes the following steps: S1: Randomly sample the sample variable space based on uniform probability to generate multiple subset datasets; S2: Generate sub-models for each subset of data based on the partial least squares algorithm; sort all sub-models according to the cross-validation error, and select a certain proportion of the model subsets from them in ascending order as the optimal model subset; S3: Select weight vectors, calculate weight vector values based on the optimal model subset, normalize, amplify and smooth the weight vectors, use the step search method to calculate the minimum cross-validation error of each weight vector and its corresponding variable subset; select the weight vector with the minimum cross-validation error as the optimal weight vector, and record the corresponding variable subset; S4: Use the optimal weight vector as the variable selection probability to further sample the variable space and generate a new set of sub-models; S5: Repeat steps S2 to S4 until the number of variables in the variable subset is 1, then stop the iteration; select the subset with the lowest cross-validation error during the iteration process as the optimal variable set; The weight vectors in S3 include the regression coefficient weight vector, the variable projection importance weight vector, the selection ratio weight vector, the multivariate correlation weight vector, the inverse of the residual variance vector, and the frequency weight vector; The weight vector is defined as: (1) Regression coefficient weight vector Regression coefficients measure the dependence of y on X; The absolute value of the regression coefficient is used as an estimate of the importance of the variable; (2) Variable projection importance weight vector The variable projection importance weight vector accumulates by weighting each PLS component to reflect the importance of each variable; Its definition is (1) in It is the weighted sum of squares of the covariances between X and y; A represents the number of the largest latent variables; the subscript i represents the i-th variable; (3) Select the ratio weight vector Selectivity ratio (SR) is expressed as the explained variance. and residual variance The ratio; (2) in This is obtained by projecting the X-row onto the normalized regression coefficients; This is obtained by projecting column X onto the top; It is the residual after X projection; the subscript i indicates the i-th variable; (4) Multivariate correlation weight vector The multivariate correlation weight vector can minimize the influence of irrelevant X structures and highlight the variables most relevant to the response; it is expressed as: (3) in This reflects the relevant changes in the predicted response projected back into the original X variable space via PLS; It is the residual variance. It is a predicted value. It is the residual after X-projection; (5) Reciprocal of the residual variance vector (6) Frequency weight vector The frequency weight vector is obtained by calculating the number of times the variables appear in the optimal model subset, reflecting the frequency of occurrence of the variables in the optimal model subset.
2. A near-infrared spectral data variable selection system based on multi-weighted vector optimization, characterized in that, include: The subset generation module is used to randomly sample the sample variable space based on uniform probability to generate multiple subsets. The optimal model subset selection module is used to generate sub-models for each subset of data based on the partial least squares algorithm; it sorts all the sub-models according to the cross-validation error and selects a certain proportion of the model subsets from them in ascending order as the optimal model subset; The weight vector selection module is used to calculate weight vector values based on the optimal model subset, normalize, amplify, and smooth the weight vectors, calculate the minimum cross-validation error of each weight vector and its corresponding variable subset using a step search method, select the weight vector with the minimum cross-validation error as the optimal weight vector, and record the corresponding variable subset. The sub-model generation module is used to further sample the variable space using the optimal weight vector as the variable selection probability to generate a new set of sub-models. The iteration module repeatedly executes the optimal model subset selection module, the weight vector selection module, and the sub-model generation module until the number of variables in the variable subset is 1, at which point the iteration stops. Select the subset with the lowest cross-validation error during the iteration process as the optimal variable set; The weight vector selection module includes the regression coefficient weight vector, variable projection importance weight vector, selection ratio weight vector, multivariate correlation weight vector, reciprocal of residual variance vector, and frequency weight vector. The weight vector is defined as: (1) Regression coefficient weight vector Regression coefficients measure the dependence of y on X; The absolute value of the regression coefficient is used as an estimate of the importance of the variable; (2) Variable projection importance weight vector The variable projection importance weight vector accumulates by weighting each PLS component to reflect the importance of each variable; Its definition is (1) in It is the weighted sum of squares of the covariances between X and y; A represents the number of the largest latent variables; the subscript i represents the i-th variable; (3) Select the ratio weight vector Selectivity ratio (SR) is expressed as the explained variance. and residual variance The ratio; (2) in This is obtained by projecting the X-row onto the normalized regression coefficients; This is obtained by projecting column X onto the top; It is the residual after X projection; the subscript i indicates the i-th variable; (4) Multivariate correlation weight vector The multivariate correlation weight vector can minimize the influence of irrelevant X structures and highlight the variables most relevant to the response; it is expressed as: (3) in This reflects the relevant changes in the predicted response projected back into the original X variable space via PLS; It is the residual variance. It is a predicted value. It is the residual after X-projection; (5) Reciprocal of the residual variance vector (6) Frequency weight vector The frequency weight vector is obtained by calculating the number of times the variables appear in the optimal model subset, reflecting the frequency of occurrence of the variables in the optimal model subset.
3. A near-infrared spectral data variable selection device based on multi-weighted vector optimization, characterized in that, The method includes at least one processor and at least one memory communicatively connected to the processor, wherein the memory stores program instructions executable by the processor, and the processor can execute the method as described in claim 1 by invoking the program instructions.
4. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause the computer to perform the method as described in claim 1.