Method for evaluating extrapolation capability of industrial software model based on algorithm model

Abstract

The invention discloses a method for evaluating the extrapolation capability of an industrial software model based on an algorithm model. The method mainly solves the problem that traditional evaluation indexes such as the accuracy rate, the recall rate and the precision are insufficient in the aspect of evaluating the extrapolation capability of the model and in the aspect of evaluating the universality between different algorithm models. The method comprises the following steps: S1, obtaining a to-be-evaluated model ModelEva, an original data set D belonging to Rn * m and a target Y belonging to Rn, wherein n represents the total data amount and m represents the total feature amount; s2, performing preliminary evaluation on the model by using the accuracy rate, the recall rate and the precision rate; and S3, deeply evaluating the extrapolation capability of the industrial software model by using a method based on an algorithm model. Through the above design, a novel universal evaluation index (EA) is introduced, the defects of traditional evaluation indexes such as accuracy, recall rate and precision in the aspect of evaluating the extrapolation capability of the model are effectively overcome, and the method is suitable for evaluating various algorithm models, and has the advantages of being high in adaptability and high in practicability. Comprising but not limited to linear regression, random forest regression, a convolutional neural network, a graph neural network, Bayesian optimization and a deep neural network. The method not only can evaluate the performance of the model on a training or verification data set, but also can deeply evaluate the expression ability of the model in unknown data or new fields, namely the extrapolation ability of the model.

Description

Technical Field

[0001] The present invention relates to the field of machine learning, and in particular to a method for evaluating the extrapolation capability of an industrial software model based on an algorithm model. Background Art

[0002] In the field of machine learning, model evaluation usually relies on accuracy, recall, precision, F1 score, and e ) and other traditional indicators. These indicators are very useful in the model development and verification stages, but they usually only evaluate the performance of the model on the training or verification dataset, which is not enough to evaluate the model's extrapolation ability on new domains or unseen data. Therefore, it is necessary to develop a new general evaluation method that specifically evaluates the model's extrapolation ability. Summary of the invention

[0003] The purpose of the present invention is to provide a method for evaluating the extrapolation capability of industrial software models based on an algorithm model, mainly to solve the deficiencies of traditional evaluation indicators such as accuracy, recall rate, precision rate, etc. in evaluating the extrapolation capability of models and evaluating the universality between different algorithm models.

[0004] To achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0005] A method for evaluating the extrapolation capability of an industrial software model based on an algorithm model comprises the following steps:

[0006] S1, get the model to be evaluated Eva , original dataset D∈R n×m , target Y∈R n , where n represents the total amount of data and m represents the total amount of features;

[0007] S2, use accuracy, recall, and precision to conduct a preliminary evaluation of the model;

[0008] S3, using an algorithm-based approach to conduct an in-depth assessment of the extrapolation capability of industrial software models, including the following steps:

[0009] S31, data partitioning module: based on Z- sco r e Get the interval range to divide the data and specify a data anchor point x anchor To obtain D l , D r and the corresponding target Y l , and target Y r , Among them, D lis the left interval data, D r Represents right interval data;

[0010] S32, algorithm module: based on algorithm model Model base , including but not limited to linear regression, random forest regression, convolutional neural network, graph neural network, Bayesian optimization, deep neural network, to train data in different intervals; after training, use the model to perform reverse inference to generate a test data set D′;

[0011] S33, Model to be evaluated test: Using the model to be evaluated Model Eva Test the generated test data set D′ and obtain the target Y′;

[0012] S34, extrapolation capability calculation module of the model to be evaluated: based on steps S31 to S33, the extrapolation capability EA of the model to be evaluated is calculated.

[0013] Furthermore, in the present invention, the algorithm model to be evaluated includes but is not limited to linear regression, random forest regression, convolutional neural network, graph neural network, Bayesian optimization, and deep neural network.

[0014] Furthermore, in the present invention, the model extrapolation capability includes the prediction stability of the model to be evaluated in the new extrapolation space, the consistency of the prediction results, and the adaptation speed to new data.

[0015] Furthermore, the present invention also includes using cross-validation to evaluate the generalization ability of the model to be evaluated; the specific method includes:

[0016] i) 20 / 80 split: the dataset is divided into 80% training set and 20% test set;

[0017] ii) k-folds cross validation: divide the dataset into k subsets of equal size, take turns as test sets and the rest as training sets;

[0018] iii) Leave-One-Out Cross Validation (LOO): Leave one sample each time as the test set and the rest as the training set.

[0019] Furthermore, in step S31, the specific steps of the data partitioning module to implement data partitioning are:

[0020] S311, Assumption: For target data y∈R n , now we need to calculate the i-th row y of the target data y i to divide;

[0021] S312, specify the anchor point x of the target data y anchor, calculate the mean μ and standard deviation σ of the target Y. The specific formula is:

[0022]

[0023] S313, calculate target y i The Z-score value is as follows:

[0024]

[0025] S314, interval data identification: set the left threshold threshold left, identify all Z-score values ​​between the left threshold and x anchor The data between is the left interval data D l and target y l ; Set the right threshold threshold_right to identify all Z-score values ​​between x anchor The data between the right threshold and the right interval data D r and target y r .

[0026] Furthermore, in step S32, the specific steps of generating the test data set D′ are:

[0027] S321, based on algorithm model base , including but not limited to linear regression, random forest regression, convolutional neural network, graph neural network, Bayesian optimization, deep neural network, for different interval data D l and D r Train and get the model Model l 、Model r ;

[0028] S322, cross-inference based on the trained model, that is, based on the trained model Model l and r Reversely generate new data D′ r ;Model based on training r and l Reversely generate new data D′ l ;

[0029] S323, synthesize the test data set D′, that is: D′=D′ r +D′ l .

[0030] Furthermore, in step S34, the specific method for calculating the extrapolation ability EA of the model to be evaluated is:

[0031] S341, data extrapolation scale calculation: For any data xi , calculate its extrapolation scale The specific formula is:

[0032]

[0033] S342, setting of extrapolation range r: providing an extrapolation scale range r, for example: [0, 5], to obtain the extrapolation capability of the model to be evaluated;

[0034] S343, data real precision RP calculation: According to the target y and target y′, the real precision of each data under the extrapolation scale is calculated. The specific formula is:

[0035]

[0036] S344, extrapolation capability evaluation of the model to be evaluated: for the extrapolation range r set in step S342, calculate the capability EA of the model to be evaluated within the extrapolation range. r , the specific formula is:

[0037]

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] (1) The present invention introduces a new evaluation index (EA) to effectively make up for the shortcomings of traditional evaluation indicators such as accuracy, recall, precision, etc. in evaluating the extrapolation ability of the model. This method can not only evaluate the performance of the model on the training or validation data set, but more importantly, it can deeply evaluate the performance of the model on unknown data or new fields, that is, the extrapolation ability of the model.

[0040] (2) The present invention has the ability to dynamically adjust the evaluation interval: it can dynamically adjust the evaluation to adapt to changes in different models and data sets, thereby improving the accuracy, reliability and flexibility of the evaluation.

[0041] (3) The present invention is general and universal, and can be used to evaluate a variety of algorithm models, including but not limited to linear regression, random forest regression, convolutional neural network, graph neural network, Bayesian optimization, and deep neural network.

[0042] (4) The present invention provides a new perspective and tool for the evaluation of industrial software models, which helps to improve the predictive stability and consistency of the model in practical applications and enhance the credibility of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a structural schematic diagram of the present invention. DETAILED DESCRIPTION

[0044] The present invention is further described below in conjunction with the accompanying drawings and embodiments. The embodiments of the present invention include but are not limited to the following embodiments.

[0045] Example

[0046] like Figure 1 As shown, the present invention discloses a method for evaluating the extrapolation capability of industrial software models based on an algorithm model. First, a data acquisition module is used to obtain the model Model to be evaluated. Eva , original dataset D∈R n×m , target Y∈R n , where n represents the total amount of data and m represents the total amount of features; the model is preliminarily evaluated using the industrial software model preliminary evaluation module; the evaluation indicators include accuracy, recall, precision, generalization ability of the model, data distribution change test and evaluation model interpretability. y ): The most intuitive evaluation indicator, which measures the proportion of samples predicted correctly by the model to be evaluated to the total samples; Recall: It measures the proportion of samples predicted as positive examples by the model to be evaluated to the actual positive examples; Precision: It measures the proportion of samples predicted as positive examples by the model to be evaluated to the actual positive examples.

[0047] The generalization ability of the model to be evaluated is cross-validated. Common methods include:

[0048] i) 20 / 80 split: the dataset is divided into 80% training set and 20% test set;

[0049] ii) k-folds cross validation: divide the dataset into k subsets of equal size, take turns as test sets and the rest as training sets;

[0050] iii) Leave-One-Out Cross Validation (LOO): Leave one sample each time as the test set and the rest as the training set.

[0051] Data distribution change test:

[0052] i) Sensitivity: evaluates the sensitivity of the model output to small changes in input features;

[0053] ii) Tolerance e ): Evaluate the degree of change in the output results when the input features of the model change within a certain range.

[0054] Interpretability Evaluation:

[0055] i) Feature Importance: Evaluate the influence of each feature on the model prediction results;

[0056] ii) t-SNE, UMAP, PHATE, etc.: Dimensionality reduction techniques used to visualize the distribution of high-dimensional data;

[0057] iii) Kolmogrov-Amold KAN(Symbolic Regression): It finds the mathematical expression of the data through symbolic regression.

[0058] The industrial software model extrapolation capability assessment module uses an algorithm model-based method to conduct an in-depth assessment of the industrial software model extrapolation capability, wherein the model extrapolation capability includes the prediction stability of the model to be evaluated in the new extrapolation space, the consistency of the prediction results, and the adaptation speed to new data. The algorithm model includes but is not limited to linear regression, random forest regression, convolutional neural network, graph neural network, Bayesian optimization, and deep neural network. Specifically, the following steps are included:

[0059] Data partitioning module: Based on Z- sco r e Get the interval range to divide the data and specify a data anchor point x anchor To obtain D l , D r and the corresponding target Y l , and target Y r , Among them, D l is the left interval data, D r Represents the right interval data. The specific steps are:

[0060] Assumptions: For the target data y∈R n , now we need to calculate the i-th row y of the target data y i to divide;

[0061] Specify the anchor point x of the target data y anchor , calculate the mean μ and standard deviation σ of the target Y. The specific formula is:

[0062]

[0063] Calculate target y i The Z-score value is as follows:

[0064]

[0065] Interval data recognition: Set the left threshold threshold left to identify all Z- sco r e Value between left threshold and xanchor The data between is the left interval data D l and target y l ; Set the right threshold threshold_right to identify all Z-score values ​​between x anchor The data between the right threshold and the right interval data D r and target y r .

[0066] Algorithm module: based on algorithm model base , including but not limited to linear regression, random forest regression, convolutional neural network, graph neural network, Bayesian optimization, deep neural network, to train data in different intervals; after training, use the model to perform reverse inference to generate the test data set D′; the specific steps are:

[0067] Based on algorithm model base For different interval data D l and D r Train and get the model Model l 、Model r ;

[0068] Cross-inversion based on the trained model, that is: based on the trained model Model l and r Reversely generate new data D′ r ;Model based on training r and l Reversely generate new data D′ l ;

[0069] Synthetic test data set D′, that is: D′=D′ r +D′ l .

[0070] Model testing to be evaluated: Using the model to be evaluated Eva , including but not limited to linear regression, random forest regression, convolutional neural network, graph neural network, Bayesian optimization, deep neural network, test the generated test data set D′ to obtain the target Y′;

[0071] The module for calculating the extrapolation capability of the model to be evaluated is used to calculate the extrapolation capability EA of the model to be evaluated based on the above steps.

[0072] The specific method is:

[0073] Data extrapolation scale calculation: For any data x i , calculate its extrapolation scale The specific formula is:

[0074]

[0075] Extrapolation range r setting: Give an extrapolation scale range r, for example: [0, 5], to obtain the extrapolation ability of the model to be evaluated;

[0076] Data real precision RP calculation: According to the target y and target y′, the real precision of each data under the extrapolation scale is calculated The specific formula is:

[0077]

[0078] Extrapolation capability evaluation of the model to be evaluated: For the set extrapolation range r, calculate the capability EA of the model to be evaluated within the extrapolation range. r , the specific formula is:

[0079]

[0080] The advantages of the present invention are:

[0081] a) Model self-assessment capability: It provides a more advanced model self-assessment capability than the preliminary assessment module, which can achieve better performance and more reliable results in practical applications;

[0082] b) Higher adaptability: The extrapolated capability evaluation indicators can adapt to different application scenarios and needs, providing more personalized evaluation results;

[0083] c) Stronger prediction stability: The extrapolation ability evaluation index can better evaluate the stability of the model when facing new field data and improve the reliability of the model;

[0084] d) Better consistency assessment: The extrapolation capability assessment index can assess whether the prediction results of the model on new data are consistent with the prediction results on known data, thus enhancing the credibility of the model;

[0085] e) Excellent versatility: Extrapolation capability evaluation indicators can be used to judge a variety of algorithm models, including but not limited to linear regression, random forest regression, convolutional neural network, graph neural network, Bayesian optimization, and deep neural network. This versatility enables the evaluation indicators to provide fair and objective performance evaluation across different algorithms.

[0086] Through the above design, the present invention effectively makes up for the deficiencies of traditional evaluation indicators such as accuracy, recall, precision, etc. in evaluating the extrapolation ability of models by introducing a new evaluation index (EA), and is suitable for evaluating a variety of algorithm models, including but not limited to linear regression, random forest regression, convolutional neural network, graph neural network, Bayesian optimization, and deep neural network. This method can not only evaluate the performance of the model on the training or verification data set, but more importantly, it can deeply evaluate the performance of the model on unknown data or new fields, that is, the extrapolation ability of the model.

[0087] The above embodiment is only one of the preferred implementation modes of the present invention and should not be used to limit the protection scope of the present invention. Any changes or modifications that are made to the main design concept and spirit of the present invention and have no substantive significance, and the technical problems they solve are still consistent with the present invention, should be included in the protection scope of the present invention.

Claims

1. A method for evaluating the extrapolation capability of an industrial software model based on an algorithm model, characterized in that: The following steps are involved: S1, get the model to be evaluated Eva , original dataset D∈R n×m , target Y∈R n , where n represents the total amount of data and m represents the total amount of features; S2, use accuracy, recall, and precision to conduct a preliminary evaluation of the model; S3, using an algorithmic model-based approach to conduct an in-depth assessment of the extrapolation capability of industrial software models, including the following steps: S31, data partitioning module: based on the Z-score, obtain the interval range to perform data partitioning and specify a data anchor point x anchor To obtain D l , D r and the corresponding goals and goals Among them, D l is the left interval data, D r Represents right interval data; S32, algorithm module: based on algorithm model base , including but not limited to linear regression, random forest regression, convolutional neural network, graph neural network, Bayesian optimization, deep neural network, to train data in different intervals; after training, use the model to perform reverse inference to generate a test data set D'; S33, Model to be evaluated test: Using the model to be evaluated Model Eva Test the generated test data set D' and obtain the target Y'; S34, extrapolation capability calculation module of the model to be evaluated: based on steps S31 to S33, the extrapolation capability EA of the model to be evaluated is calculated.

2. The method for evaluating the extrapolation capability of industrial software models based on an algorithm model according to claim 1, characterized in that: The algorithm models to be evaluated include but are not limited to linear regression, random forest regression, convolutional neural network, graph neural network, Bayesian optimization, and deep neural network.

3. The method for evaluating the extrapolation capability of industrial software models based on an algorithm model according to claim 1, characterized in that: The model extrapolation capability includes the prediction stability of the model to be evaluated in the new extrapolation space, the consistency of the prediction results, and the speed of adapting to new data.

4. The method for evaluating the extrapolation capability of industrial software models based on an algorithm model according to claim 3 is characterized in that: It also includes the use of cross-validation to evaluate the generalization ability of the model to be evaluated; the specific methods include: i) 20 / 80 split: the dataset is divided into 80% training set and 20% test set; ii) k-folds cross validation: divide the dataset into k subsets of equal size, take turns as test sets and the rest as training sets; iii) Leave-One-Out Cross Validation (LOO): Leave one sample each time as the test set and the rest as the training set.

5. The method for evaluating the extrapolation capability of industrial software models based on an algorithm model according to claim 4 is characterized in that: In step S31, the specific steps of the data partitioning module to implement data partitioning are: S311, Assumption: For target data y∈R n , now we need to calculate the i-th row y of the target data y i to divide; S312, specify the anchor point x of the target data y anchor , calculate the mean μ and standard deviation σ of the target Y. The specific formula is: S313, calculate target y i The Z-score value is as follows: S314, interval data identification: set the left threshold threshold_left, identify all Z-score values ​​between the left threshold and x anchor The data between is the left interval data D l and target y l ; Set the right threshold threshold_right to identify all Z-score values ​​between x anchor The data between the right threshold and the right interval data D r and target y r .

6. The method for evaluating the extrapolation capability of industrial software models based on an algorithm model according to claim 5 is characterized in that: In step S32, the specific steps of generating the test data set D' are: S321, based on algorithm model base , including but not limited to linear regression, random forest regression, convolutional neural network, graph neural network, Bayesian optimization, deep neural network, for different interval data D l and D r Train and get the model Model l 、Model r ; S322, cross-inversion based on the trained model, that is, based on the trained model Model l and r Reverse to generate new data D' r ;Model based on training r and l Reverse to generate new data D' l ; S323, synthesize the test data set D', that is: D' = D' r +D' l .

7. The method for evaluating the extrapolation capability of industrial software models based on an algorithm model according to claim 6 is characterized in that: In step S34, the specific method for calculating the extrapolation ability EA of the model to be evaluated is: S341, data extrapolation scale calculation: For any data x i , calculate its extrapolation scale The specific formula is: S342, setting of extrapolation range r: providing an extrapolation scale range r, for example: [0, 5], to obtain the extrapolation capability of the model to be evaluated; S343, data real precision RP calculation: According to the target y and target y', the real precision of each data under the extrapolation scale is calculated. The specific formula is: S344, extrapolation capability evaluation of the model to be evaluated: for the extrapolation range r set in step S342, calculate the capability EA of the model to be evaluated within the extrapolation range. r , the specific formula is:

Images