Neural network model variable selection method and device based on SHAP value

Abstract

The invention provides a neural network model variable selection method based on an SHAP value, and belongs to the field of spectrum data processing chemometrics variable selection, and the method comprises the steps: S1, carrying out the standardization processing of a sample spectrum, and obtaining a spectrum data set; s2, selecting spectral data to train a neural network model based on the current variable set; s3, calculating an SHAP value of each variable in the current variable set based on K-Fold cross validation to obtain a variable importance vector; s4, sequencing the variable importance vectors, and removing unimportant variables based on an e exponential function to obtain a new variable set; s5, judging that the set number of iterations is reached, if so, selecting the variable set with the minimum cross validation error as the optimal variable set, and if not, returning to S2, and inputting a new variable set into the neural network model for continuous execution; the invention further provides a neural network model variable selection device based on the SHAP value, input variable importance evaluation and variable selection are achieved, and spectrum modeling and detection precision is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of spectral data processing chemometrics variable selection, specifically to a variable selection method for high-dimensional data, and particularly to a variable selection method for a neural network model based on SHAP values. Background Art

[0002] Spectral technology has been widely applied in various fields due to its rapid detection. An important goal of analyzing spectral data is to develop robust, easily interpretable, and highly accurate qualitative and quantitative models. Multivariate calibration models, including multiple linear regression, partial least squares, and artificial neural networks (ANN), are commonly used methods for spectral data analysis. In the modeling process, variable selection plays a crucial role. Selecting appropriate spectral data variables before model construction can significantly improve the prediction ability and interpretability of the model. Although many variable selection algorithms have been developed based on the partial least squares algorithm, such as the weight vector smoothing strategy proposed in the Chinese invention patent application "Near-infrared spectral data variable selection method and system based on multi-weight vector optimization" with the publication number CN115630506A, since the partial least squares algorithm is difficult to directly process the nonlinear part in spectral data, for spectral data with nonlinearity, the variable selection algorithm based on partial least squares has low accuracy. As a nonlinear modeling method, neural networks have natural modeling ability for nonlinear data, but how to evaluate the importance of neural network input variables and select variables to improve the spectral modeling accuracy and detection accuracy has not been solved by the prior art. Summary of the Invention

[0003] The technical problem to be solved by the present invention is how to evaluate the importance of neural network input variables and select variables to improve the spectral modeling accuracy and detection accuracy.

[0004] The present invention solves the above technical problem through the following technical solutions: A variable selection method for a neural network model based on SHAP values, the method comprising:

[0005] S1. Standardize the sample spectra to obtain a spectral data set;

[0006] S2. Select spectral data based on the current variable set to train a neural network model;

[0007] S3. Calculate the SHAP value of each variable in the current variable set based on K-Fold cross-validation to obtain a variable importance vector, and record the variable set and its cross-validation error;

[0008] S4. Sort the variable importance vector, and remove unimportant variables based on the e exponential function to obtain a new variable set;

[0009] S5. Determine whether the set number of iterations is reached. If yes, select the variable set with the smallest cross-validation error during the iteration as the optimal variable set. If not, return to S2, input the new variable set into the neural network model and continue the execution.

[0010] The present invention uses a neural network model to perform variable selection on spectral data, uses a cyclic iteration method to screen variables, realizes the evaluation of the importance of neural network input variables by calculating the SHAP value of features, realizes removing some unimportant variables each time through the e-exponential function to obtain a new variable set, and then evaluates the model based on the new variable set. During each iteration, the input variables will be reduced, and the input variables are selected based on cluster analysis to reduce the variable dimension while reducing the interference of noise and useless variables on modeling, and improve the spectral modeling accuracy and detection accuracy.

[0011] Preferably, in S2, the neural network model includes an input layer, a hidden layer, and an output layer. The number of neurons in the input layer is the same as the number of input variables. The hidden layer includes one or more convolutional layers, pooling layers, and fully connected layers, and different layers are connected through activation functions.

[0012] Preferably, in S2, the K-Fold cross-validation is used to determine the neural network parameter as the L2 regularization parameter. Specifically, the interval of L2 is set to select 6 points on the logarithmic scale between 1×10 -6 and 0.1. The model is trained respectively using the K-Fold cross-validation method, and the L2 regularization parameter corresponding to the minimum loss function is selected as the parameter of the neural network model.

[0013] Preferably, the calculation formula of the SHAP value in S3 is:

[0014]

[0015] where φ i is the contribution of feature i in the variable set, N is the variable set, S is the feature subset of all features in the variable set N that do not include feature i, and v is the value function.

[0016] Preferably, the calculation method of the cross-validation error in S3 is: divide the variable set into K parts, take out one part as the test set each time, and the remaining K-1 parts as the training set, repeat K times, calculate the root mean square error of each test set, and finally average the results of K times as the cross-validation error RMSECV:

[0017]

[0018] where RMSE k is the root mean square error of the test set corresponding to the kth cross-validation.

[0019] Preferably, the calculation formula of the e-exponential function in S4 is:

[0020]

[0021] Among them, p is the total number of variables, M is the total number of iterations, and r i is the ratio of the number of remaining variables after the i-th iteration to the total number of variables.

[0022] The present invention also provides a neural network model variable selection device based on SHAP values. The device includes:

[0023] A data preprocessing module for performing normalization processing on the sample spectrum to obtain a spectral data set;

[0024] A model training module for training a neural network model based on the current variable set by selecting spectral data;

[0025] A SHAP value calculation module for calculating the SHAP value of each variable in the current variable set based on K-Fold cross-validation, obtaining a variable importance vector, and recording the variable set and its cross-validation error;

[0026] A variable removal module for sorting the variable importance vector and removing unimportant variables based on the exponential function to obtain a new variable set;

[0027] An iterative calculation module for determining whether the set number of iterations is reached. If so, selecting the variable set with the smallest cross-validation error during the iteration as the optimal variable set. If not, inputting the new variable set into the neural network model to continue execution.

[0028] Preferably, the neural network model in the model training module includes an input layer, a hidden layer, and an output layer. The number of neurons in the input layer is the same as the number of input variables. The hidden layer includes one or more convolutional layers, pooling layers, and fully connected layers, and different layers are connected through activation functions.

[0029] Preferably, in the model training module, K-Fold cross-validation is used to determine the neural network parameter as the L2 regularization parameter. Specifically, the interval of L2 is set to select 6 points between 1×10 -6 to 0.1 on a logarithmic scale, and the K-Fold cross-validation method is used to train the model respectively. The L2 regularization parameter corresponding to the minimum loss function is selected as the parameter of the neural network model.

[0030] Preferably, the calculation formula of the SHAP value in the SHAP value calculation module is:

[0031]

[0032] Among them, φ iis the contribution of feature i in the variable set, N is the variable set, S is the subset of all features in the variable set N that do not include feature i, and v is the value function.

[0033] Preferably, the calculation method of the cross-validation error in the SHAP value calculation module is as follows: divide the variable set into K parts, take out one part as the test set each time, and the remaining K - 1 parts as the training set. Repeat K times, calculate the root mean square error of each test set, and finally average the results of K times as the cross-validation error RMSECV:

[0034]

[0035] where RMSE k is the root mean square error of the test set corresponding to the k-th cross-validation.

[0036] Preferably, the calculation formula of the e-exponential function in the variable removal module is:

[0037]

[0038] where p is the total number of variables, M is the total number of iterations, and r i is the ratio of the number of remaining variables after the i-th iteration to the total number of variables.

[0039] The advantages provided by the present invention are as follows:

[0040] (1) The present invention uses a neural network model to perform variable selection on spectral data, uses a cyclic iteration method to screen variables, realizes the evaluation of the importance of neural network input variables by calculating the SHAP values of features, and realizes removing some unimportant variables each time through the e-exponential function to obtain a new variable set. Then, based on the new variable set, the model is evaluated. During each iteration, the input variables will be reduced, and the input variables are selected based on cluster analysis to reduce the variable dimension while reducing the interference of noise and useless variables on modeling, and improve the spectral modeling accuracy and detection accuracy.

[0041] (2) The present invention regards the neural network as a black box system, and variable selection can be applied to any neural network model, including neural networks with self-attention mechanisms and Transform architectures, without changing the internal structure of the model, and can even be applied to other modeling algorithms such as support vector regression and principal component regression algorithms.

[0042] (3) The data preprocessing process of the present invention only needs to perform standardization processing on the spectral data, and the preprocessing is simple, fast. Description of the Drawings

[0043] Figure 1 is the flowchart of the method for variable selection of a neural network model based on SHAP values provided by an embodiment of the present invention;

[0044] Figure 2 This is a schematic diagram of the variable selection statistical results calculated 10 times repeatedly for the present invention using ANN1 as the neural network model, selecting corn spectral data, and using the water content of corn as the chemical value;

[0045] Figure 3 This is a schematic diagram of the variable selection statistical results calculated 10 times repeatedly for the present invention using ANN2 as the neural network model, selecting corn spectral data, and using the water content of corn as the chemical value;

[0046] Figure 4 This is a schematic diagram of the variable selection statistical results calculated 10 times repeatedly for the present invention using CNN1 as the neural network model, selecting corn spectral data, and using the water content of corn as the chemical value;

[0047] Figure 5 This is a schematic diagram of the variable selection statistical results calculated 10 times repeatedly for the present invention using CNN2 as the neural network model, selecting corn spectral data, and using the water content of corn as the chemical value. Specific implementation manners

[0048] In order to make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the following combines specific embodiments and refers to the accompanying drawings to clearly and completely describe the technical solutions of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present invention.

[0049] As Figure 1 shown, this embodiment provides a method for variable selection of a neural network model based on SHAP values, and the method includes the following steps:

[0050] S1. Standardize the sample spectra to obtain a spectral data set. The present invention uses a sample segmentation algorithm to divide the spectral data set into a training set and a test set. The sample segmentation algorithm can adopt the KS algorithm. The data preprocessing process only needs to standardize the spectral data, and the preprocessing is simple, fast and quick.

[0051] S2. Select spectral data based on the current variable set to train a neural network model.

[0052] The neural network model of the present invention can be a conventional artificial neural network or a convolutional neural network. The neural network model includes an input layer, a hidden layer, and an output layer. The number of neurons in the input layer is the same as the number of input variables. The hidden layer includes one or more convolutional layers, pooling layers, and fully connected layers. Different layers are connected by activation functions. The training of the neural network adopts the Adam algorithm. When constructing the neural network model, K-Fold cross-validation is used to determine the neural network parameter as the L2 regularization parameter. Specifically, the range of L2 is set to be between 1×10 -6 and 0.1, and 6 points are selected on the logarithmic scale. The model is trained using the K-Fold cross-validation method respectively, and the L2 regularization parameter corresponding to the minimum loss function is selected as the parameter of the neural network model.

[0053] In the present invention, the neural network is regarded as a black box system. Variable selection can apply any neural network model, including neural networks with self-attention mechanisms and Transformer architectures, without changing the internal structure of the model. It can even be applied to other modeling algorithms such as support vector regression and principal component regression.

[0054] S3. Calculate the SHAP value of each variable in the current variable set based on K-Fold cross-validation, obtain the variable importance vector, and record the variable set and its cross-validation error. Among them, the calculation formula of the SHAP value is:

[0055]

[0056] Among them, φ i is the contribution of feature i in the variable set, N is the variable set, S is the feature subset of all features in the variable set N that do not include feature i, and v is the value function.

[0057] The calculation method of the cross-validation error is as follows: divide the variable set into K parts, take out one part as the test set each time, and the remaining K - 1 parts as the training set. Repeat K times, calculate the root mean square error of each test set, and finally average the results of K times as the cross-validation error RMSECV:

[0058]

[0059] Among them, RMSE k is the root mean square error of the test set corresponding to the kth cross-validation.

[0060] S4. Sort the variable importance vector. Specifically, sort the variable importance vector from largest to smallest; remove unimportant variables based on the exponential function to obtain a new variable set. Among them, the calculation formula of the exponential function is:

[0061]

[0062] Where p is the total number of variables, M is the total number of iterations, and r i is the ratio of the number of remaining variables after the i-th iteration to the total number of variables.

[0063] S5. Determine whether the set number of iterations is reached. If so, select the variable subset with the smallest cross-validation error during the iteration as the optimal variable set. If not, return to S2, input the new variable set into the neural network model, and continue the execution.

[0064] The present invention uses a neural network model to perform variable selection on spectral data, uses a cyclic iteration method to screen variables, realizes the evaluation of the importance of neural network input variables by calculating the SHAP values of features, realizes removing some unimportant variables each time through the e-exponential function to obtain a new variable set, and then evaluates the model based on the new variable set. During each iteration, the input variables will be reduced, and the input variables are selected based on cluster analysis to reduce the variable dimension while reducing the interference of noise and useless variables on modeling, and improve the spectral modeling accuracy and detection accuracy.

[0065] The present invention also provides a device for variable selection of a neural network model based on SHAP values, which is characterized in that the device includes:

[0066] A data preprocessing module for performing standardization processing on the sample spectrum to obtain a spectral data set.

[0067] A model training module for training a neural network model based on the current variable set for the spectral data. The neural network model includes an input layer, a hidden layer, and an output layer. The number of neurons in the input layer is the same as the number of input variables. The hidden layer includes one or more convolutional layers, pooling layers, and fully connected layers, and different layers are connected through an activation function. The neural network model is constructed by using K-Fold cross-validation to determine the neural network parameter as the L2 regularization parameter. Specifically, the interval of L2 is set to select 6 points between 1×10 -6 and 0.1 on a logarithmic scale, and the model is trained respectively using the K-Fold cross-validation method, and the L2 regularization parameter corresponding to the minimum loss function is selected as the parameter of the neural network model.

[0068] A SHAP value calculation module for calculating the SHAP value of each variable in the current variable set based on K-Fold cross-validation to obtain a variable importance vector, and recording the variable set and its cross-validation error. The calculation formula of the SHAP value is:

[0069]

[0070] Where φ i is the contribution of feature i in the variable set, N is the variable set, S is the feature subset of all features in the variable set N that do not include feature i, and v is the value function.

[0071] The cross-validation error is calculated as follows: divide the variable set into K parts. Each time, take out one part as the test set, and the remaining K - 1 parts as the training set. Repeat this K times. Calculate the root mean square error of each test set, and finally average the results of K times as the cross-validation error RMSECV:

[0072]

[0073] Among them, RMSE k is the root mean square error of the test set corresponding to the k-th cross-validation.

[0074] The variable removal module is used to sort the variable importance vectors, remove unimportant variables based on the e-exponential function, and obtain a new variable set; the calculation formula of the e-exponential function is:

[0075]

[0076] Among them, p is the total number of variables, M is the total number of iterations, and r i is the ratio of the number of remaining variables to the total number of variables after the i-th iteration.

[0077] The iterative calculation module is used to determine whether the set number of iterations is reached. If so, select the variable set with the smallest cross-validation error during the iteration as the optimal variable set. If not, input the new variable set into the neural network model and continue to execute.

[0078] Case analysis

[0079] The present invention will be introduced in detail taking the near-infrared spectral data of corn as an example:

[0080] The corn spectral data set contains 80 samples, and each sample has 700 variables. Using the KS algorithm, the samples are divided into a training set and a test set. Among them, there are 60 samples in the training set and 20 samples in the test set.

[0081] S1. Standardize the sample spectra and set the number of iterations Iter to 20.

[0082] S2. Establish a neural network model and determine the neural network parameters based on K-Fold cross-validation; the present invention establishes 4 neural networks, and the specific parameters of the neural networks are shown in Table 1:

[0083] Table 1 Specific parameters of the neural network

[0084] ANN1 ANN2 CNN1 CNN2 Input 700 700 700 700 Activation function Identity(x) Elu(x) Identity(x) Elu(x) Convolution layer parameters - - 5 5 Activation function Identity() Elu() Identity() Elu() Fully connected layer 25 25 25 25 Activation function Identity(x) Elu(x) Identity(x) Elu(x) Fully connected layer 5 5 5 5 Output 1 1 1 1

[0085] After 5-Fold cross-validation, the L2 regularization parameter is set to 1×10 -5 .

[0086] S3. Based on the variable subset in the current variable space, calculate the SHAP values based on K-Fold cross-validation, evaluate the variable importance, and obtain the variable importance vector; record the current variable subset and the cross-validation error.

[0087] S4. Sort the variable importance vector from largest to smallest, and remove unimportant variables based on the exponential decay function.

[0088] S5. Repeat steps S2 to S5 until the iteration number reaches 20 and stop the iteration; select the subset with the lowest cross-validation error during the iteration as the optimal variable set.

[0089] Table 2 Results of Maize Water Data

[0090]

[0091] Figures 2 to 5 The statistical results of 10 repeated wavelength selections are given when the maize moisture content is selected as the chemical value.

[0092] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for variable selection of a neural network model based on SHAP values, characterized in that: The method includes: S1. Performing normalization processing on the sample spectrum to obtain a spectral data set; S2. Selecting spectral data based on the current variable set to train a neural network model; S3. Calculating the SHAP value of each variable in the current variable set based on K-Fold cross-validation to obtain a variable importance vector, and recording the variable set and its cross-validation error; S4. Sorting the variable importance vector, and removing unimportant variables based on the exponential function to obtain a new variable set; S5. Judging whether the set number of iterations is reached. If so, selecting the variable set with the smallest cross-validation error during the iteration as the optimal variable set. If not, returning to S2 and inputting the new variable set into the neural network model to continue the execution.

2. The method for selecting neural network model variables based on SHAP values according to claim 1, characterized in that: In S2, the neural network model includes an input layer, a hidden layer, and an output layer. The number of neurons in the input layer is the same as the number of input variables. The hidden layer includes one or more convolutional layers, pooling layers, and fully connected layers, and different layers are connected through activation functions.

3. The method for selecting neural network model variables based on SHAP values according to claim 1, characterized in that: In S2, the K-Fold cross-validation is used to determine the neural network parameter as the L2 regularization parameter. Specifically, the interval of L2 is set to be selected from 1×10 -6 to 0.1, and 6 points are selected on the logarithmic scale. The K-Fold cross-validation method is used to train the model respectively, and the L2 regularization parameter corresponding to the minimum loss function is selected as the parameter of the neural network model.

4. The method for selecting neural network model variables based on SHAP values according to claim 1, characterized in that: The calculation formula of the SHAP value in S3 is: where φ i is the contribution of feature i of the variable set, N is the variable set, S is the subset of all features in variable set N that do not include feature i, and v is the value function.

5. The method for selecting neural network model variables based on SHAP values according to claim 1, wherein: The calculation method of the cross-validation error in S3 is: dividing the variable set into K parts, taking out one part as the test set each time, and the remaining K - 1 parts as the training set, repeating K times, calculating the root mean square error of each test set, and finally averaging the results of K times as the cross-validation error RMSECV: Among them, RMSE k is the root mean square error of the corresponding test set for the k-th cross-validation.

6. The method for selecting neural network model variables based on SHAP values according to claim 1, wherein: The calculation formula of the exponential function in S4 is: where p is the total number of variables, M is the total number of iterations, and r i is the ratio of the number of remaining variables after the i-th iteration to the total number of variables.

7. A neural network model variable selection device based on SHAP values, characterized in that: The device includes: A data preprocessing module for performing normalization processing on the sample spectrum to obtain a spectral data set; A model training module for selecting spectral data based on the current variable set to train a neural network model; A SHAP value calculation module for calculating the SHAP value of each variable in the current variable set based on K-Fold cross-validation to obtain a variable importance vector, and recording the variable set and its cross-validation error; A variable removal module for sorting the variable importance vector and removing unimportant variables based on the exponential function to obtain a new variable set; An iterative calculation module for judging whether the set number of iterations is reached. If so, selecting the variable set with the smallest cross-validation error during the iteration as the optimal variable set. If not, inputting the new variable set into the neural network model to continue the execution.

8. The apparatus for selecting neural network model variables based on SHAP values according to claim 7, wherein: In the model training module, the neural network model includes an input layer, a hidden layer, and an output layer. The number of neurons in the input layer is the same as the number of input variables. The hidden layer includes one or more convolutional layers, pooling layers, and fully connected layers, and different layers are connected through activation functions.

9. The apparatus for selecting neural network model variables based on SHAP values according to claim 7, wherein: In the model training module, K-Fold cross-validation is used to determine the neural network parameter as the L2 regularization parameter. Specifically, the range of L2 is set to be between 1×10 -6 and 0.1, and 6 points are selected on the logarithmic scale. The model is trained using the K-Fold cross-validation method, and the L2 regularization parameter corresponding to the minimum loss function is selected as the parameter of the neural network model.

10. The neural network model variable selection device based on SHAP values according to claim 7, characterized in that: The calculation formula of the SHAP value in the SHAP value calculation module is: where φ i is the contribution of feature i in the variable subset, N is the variable subset, S is the subset of all features in the variable subset N that do not include feature i, and v is the value function.

Images