Dynamic AI index system index prediction fitting method and device based on machine learning

Abstract

The invention discloses a dynamic AI index system index prediction fitting method and device based on machine learning. According to the method, through parallel training of multiple machine learning and deep learning models such as linear regression, polynomial regression, SVM, XGBoost, GBRT, DNN and Transformer, the relation between the index value of the index system and the comprehensive index value can be captured more comprehensively, and therefore the prediction accuracy is improved. The invention innovatively proposes a triple condition ranking screening method based on RMSE, R2 and time cost, can effectively screen out an optimal prediction model, and ensures the accuracy and reliability of a prediction result. The system can automatically detect updating of an index system or an actual data set and automatically execute a new round of model training, testing and prediction, and timeliness and accuracy of a prediction result are ensured. When the index system or the actual data set is updated each time, each model is retrained to ensure that the model can adapt to a new data environment and avoid outdated time.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence and indicator system index technology, and specifically relates to a dynamic AI indicator system index prediction fitting method and device based on machine learning. Background Art

[0002] The application of AI machine learning algorithms in the indicator evaluation system is becoming more and more extensive. The core lies in the effective selection of various indicators through algorithms to ensure the scientificity, accuracy and practicality of the evaluation results. When selecting indicators, AI machine learning algorithms follow the following principles: the principle of relevance, ensuring that the selected indicators are closely related to the evaluation objectives; the principle of independence, avoiding high correlation between indicators, resulting in information overlap; the principle of operability, the indicators should be easy to quantify and easy to process by the algorithm; the principle of comprehensiveness, ensuring that the indicators can fully reflect all aspects of the evaluation object. Machine learning algorithms such as decision trees, random forests, support vector machines, etc. have significant advantages in indicator selection. These algorithms can automatically filter out indicators that have a greater impact on the evaluation results through training data, thereby building an efficient evaluation model.

[0003] Although the research on AI machine learning algorithms has been widely used in the selection of various indicators in the indicator evaluation system, there are still deficiencies or gaps in whether there is a fitable relationship between each evaluation factor and the quantitative value of the target label evaluation to learn its rules for prediction. There is no method and device for establishing an AI simulator based on the existing evaluation indicator values ​​and evaluation index values ​​to explore the relationship between linear, nonlinear, spatial or multi-layer network machine learning and deep learning. Summary of the invention

[0004] The purpose of the present invention is to provide a dynamic AI indicator system index prediction fitting method and device based on machine learning in order to solve the above-mentioned problems.

[0005] The technical solution adopted by the present invention is as follows: a dynamic AI indicator system index prediction fitting method based on machine learning, the method comprising the following steps:

[0006] S1: Obtain the historical or real-time record data of each indicator value and comprehensive index of a certain level of the indicator system, with each indicator name and comprehensive index as the row field name of the data table, and the column data as the records of each indicator value entered multiple times for evaluating each thing or a single thing and the comprehensive index value calculated according to the indicator system calculation formula;

[0007] S2: Understand and prepare all rows of data for each record of each indicator value and comprehensive index value, and check whether the obtained indicator value data is in the probability distribution between 0 and 1 or between -1 and +1. If not, the system needs to automatically standardize or normalize each indicator value data to the probability distribution between 0 and 1 or between -1 and +1;

[0008] S3: Data preprocessing, the first step in creating a machine learning model. Performing data preprocessing includes data cleaning, data integration, data transformation, and data reduction methods. The steps include obtaining data sets, importing libraries, importing data sets, finding missing data, and encoding categorical data.

[0009] S4: Select each index after data preprocessing as feature Xi, the index value as its feature value, and take the comprehensive index value as the label value of the target variable Y; perform feature construction, feature extraction and feature selection of machine learning feature engineering;

[0010] S5: Construct machine learning prediction models, and construct machine learning and deep learning models such as linear and polynomial regression, support vector machine, extreme gradient boosting, gradient boosted regression tree, deep neural network and Transformer in parallel to train comprehensive index prediction; each time the comprehensive index of the indicator system is calculated for the target object, that is, when the indicator system or the actual data set is updated, each algorithm model will be automatically and dynamically retrained;

[0011] S6: Perform support vector machine, extreme gradient boosting and gradient boosting regression tree training for each model, define the root mean square error as the loss function, and the algorithm automatically applies cross-validation and grid search to optimize the parameters and hyperparameters of each model, and automatically select the best parameter combination to make the prediction accuracy of each model the highest;

[0012] S7: Test the optimal parameter models of linear and polynomial regression, support vector machine, extreme gradient boosting, gradient boosted regression tree, deep neural network and Transformer, and automatically calculate their root mean square error, determination coefficient and time cost on the test data set as accuracy evaluation indicators to evaluate the prediction performance of the model;

[0013] S8: Based on the prediction performance evaluation index values ​​of three different dimensions, namely, root mean square error, determination coefficient and time cost, the self-created triple condition ranking screening method is used to screen and select the optimal prediction model for each prediction model. Finally, the model with the smallest root mean square error in the last layer of conditional sorting is selected as the comprehensive index AI prediction model of this indicator system based on the existing data set and subsequent predictions are performed to output the comprehensive index prediction value of the selected model;

[0014] S9: After that, the AI ​​algorithm fitting prediction method and device of the comprehensive index of this indicator system are completed. When the indicator system or the actual data set is automatically checked for updates, it returns to the first step to perform a new round of comprehensive index AI algorithm fitting prediction.

[0015] In a preferred embodiment, in step S2, data standardization includes the following steps:

[0016] Data standardization is to scale the data so that it falls into a small specific range. Since the measurement units of various indicators in the credit index system are different, in order to be able to participate in the evaluation calculation, the indicators need to be normalized and their values ​​mapped to a certain numerical range through function transformation.

[0017] There are many methods for data standardization. Based on general experience, the present invention uses the min-max standardization method. After the above standardization process, the original data are converted into dimensionless index evaluation values, that is, each index value is at the same level, and comprehensive evaluation and analysis can be performed.

[0018] The min-max normalization method is to perform a linear transformation on the original data. Let minA and maxA be the minimum and maximum values ​​of attribute A respectively, and map an original value x of A to a value x' in the interval [0,1] through min-max normalization. The formula is:

[0019] newdata= / ;

[0020] y=(x-MinValue) / (MaxValue-MinValue)

[0021] Note: x and y are the values ​​before and after the conversion, respectively. MaxValue and MinValue are the maximum and minimum values ​​of the samples, respectively.

[0022] In a preferred embodiment, in step S2, data normalization includes the following steps:

[0023] First of all, the basic measurement units must be consistent. Neural networks are trained (probability calculation) and predicted based on the statistical probability of samples in events, and normalization is a statistical probability distribution between 0 and 1. SVM classifies and simulates by linearly dividing distances after dimensionality reduction, so spatiotemporal dimensionality reduction normalization is a statistical coordinate distribution that is unified between -1 and +1.

[0024] The normalization method is:

[0025] Linear function conversion, the expression is as follows:

[0026] y=(x-MinValue) / (MaxValue-MinValue)

[0027] Note: x and y are the values ​​before and after the conversion, respectively. MaxValue and MinValue are the maximum and minimum values ​​of the samples, respectively.

[0028] In a preferred embodiment, in step S3, data cleaning includes: mainly completing missing values ​​in the data, eliminating noise data, identifying or deleting outliers and resolving inconsistencies. The main goals achieved are: standardizing data formats, cleaning abnormal data, correcting errors, and removing duplicate data. Data cleaning includes: 1. Deleting redundant and duplicate data; 2. Repairing structural errors; 3. Managing unnecessary outliers; 4. Processing missing data, etc.

[0029] Data integration includes: integrating data from multiple data sources and storing them in a unified manner.

[0030] Data transformation converts data into a form suitable for data mining through smooth aggregation, data generalization, and normalization.

[0031] Data reduction reduces or simplifies a data set, which not only maintains the integrity of the original data but also makes the results after data reduction the same or almost the same as the results before reduction.

[0032] In view of the standardization and characteristics of the indicator system data set, missing values ​​in the records are filled with 0 according to the indicator recording habits; redundant duplicate data are deleted, including duplicate row records with exactly the same fields, and column data with the same value including 0 for each evaluation item of a certain indicator; then confirm whether the indicator values ​​are standardized or normalized during indicator design due to different measurement units. If there are indicators that have not undergone standardized or normalized data transformation, a unified probability distribution between 0 and 1 or a coordinate distribution between -1 and +1 is performed on all indicators. It is not allowed that some are 0-1 and some are -1-+1; the indicator system data is relatively standardized, and outliers have practical significance and are not deleted.

[0033] In a preferred embodiment, in step S4, feature extraction uses statistical methods to measure the relationship between a single feature and a response variable.

[0034] The Pearson correlation coefficient is used to measure the linear relationship between variables. The result is [-1,1]. -1 indicates a complete negative correlation, +1 indicates a positive correlation, and 0 indicates a non-linear correlation, but it does not mean that there is no other relationship. An obvious flaw of the Pearson coefficient is that it only measures linear relationships. If the relationship is nonlinear, even if the two variables have a one-to-one relationship, the Pearson correlation will be close to 0. The formula for the Pearson coefficient is the sample covariance divided by the product of the standard deviation of X and the standard deviation of Y. The Pearson correlation coefficient between two random variables can be directly calculated in sklearn. For example, the correlation coefficient between X and X*X calculated below is approximately 0;

[0035] The maximum information coefficient in the feature extraction process is obtained based on the mutual information. The mutual information formula is as follows. The maximum information coefficient MIC can not only measure the linear relationship between variables like the Pearson coefficient, but also measure the nonlinear relationship between variables. Although MIC can measure the nonlinear relationship between variables, when the relationship between variables is close to linear correlation, the Pearson correlation coefficient is still irreplaceable. First, the Pearson correlation coefficient is fast to calculate, which is very important when processing large-scale data. Second, the value range of the Pearson correlation coefficient is [-1, 1], while the MIC and distance correlation coefficient are both [0, 1]. This feature enables the Pearson correlation coefficient to represent richer relationships. The sign indicates the positive or negative relationship, and the absolute value can indicate the intensity.

[0036] In a preferred embodiment, in step S4, feature selection uses a linear regression model. Suppose we have a data set, and its true model should be Y=X1+X2. When we observe, we find that Y'=X1+X2+e, where e is noise. If there is a linear relationship between X1 and X2, for example, X1 is approximately equal to X2, then due to the existence of noise e, the model we learn may not be Y=X1+X2, but may be Y=2X1, or Y=-X1+3X2. By adding regularization terms to the model, feature selection can also be achieved. L1 regularization results in a relatively sparse model. Controlling the penalty term coefficient alpha will force the coefficients corresponding to those weak features to become 0. This feature makes L1 regularization a good feature selection method. L2 regularization will make the values ​​of the coefficients average. For correlated features, this means that they can obtain more similar corresponding coefficients.

[0037] In a preferred embodiment, the step S5 specifically includes:

[0038] Model building and training:

[0039] After data preprocessing is completed, select the appropriate machine learning algorithm for model design, development and training. This patent designs algorithm models such as linear regression, polynomial regression, gradient boosting regression tree, extreme gradient boosting, deep neural network and Transformer. At the same time, cross-validation, grid search and other methods are used to optimize and adjust the parameters and hyperparameters of each machine learning algorithm model, and select the parameter combination that can obtain the highest prediction result accuracy to improve the accuracy of the regression model.

[0040] Define the loss function:

[0041] Loss function, also known as cost function, is often used to represent or measure the error generated by the decision function. Loss function is one of the key factors affecting model performance. A deep understanding of various existing loss functions is the premise and basis for selecting and constructing loss functions. The loss function is used to evaluate the degree of difference between the model prediction results and the actual results. Commonly used loss functions can be selected, such as mean square error and root mean square error in the second chapter of concepts and theories. By minimizing the loss function, the model parameters are adjusted to make the model's prediction results more accurate. This paper selects RMSE as the loss function in each prediction algorithm program for development and design.

[0042]

[0043] In a preferred embodiment, step S7 specifically includes using a test set to evaluate the performance indicators of the model. This patent selects mainstream accuracy evaluation indicators such as mean square error, root mean square error, determination coefficient, mean absolute percentage error, and innovatively adds the time cost evaluation indicator of model training test prediction completion to evaluate the predictive performance of the model. At the same time, cross-validation, grid search and other methods are used to optimize the parameters and hyperparameters of the model to ensure the selection of the best parameter combination to improve the performance and accuracy of the regression model.

[0044] In a preferred embodiment, in step S8, a concise and efficient sorting and screening method, namely a triple conditional ranking and screening method, is used to select the optimal AI prediction model based on prediction performance evaluation indicators of three different dimensions, namely root mean square error, determination coefficient and time cost, which specifically includes the following steps:

[0045] a. Direct sorting: sort the data in optimal descending order according to the weight of each indicator.

[0046] b. Case ranking, a data analysis method used to sort data to determine the position of each observation in a sequence.

[0047] A method integrating direct sorting and case ranking was adopted to rank the three prediction performance evaluation indicators of root mean square error, determination coefficient and time cost, respectively, to rank the performance of linear and polynomial regression, support vector machine, extreme gradient boosting, gradient enhanced regression tree, deep neural network and Transformer model, with the best one ranked first.

[0048] Then the next step is to innovatively establish a triple condition ranking screening method. For the three evaluation factors X, Y, and Z, they are sorted and grouped according to X, Y, and Z at the same time, that is, triple sorting; because the selection condition sorting and screening is first sorted and screened according to one factor X, the result data after X sorting and screening is sorted and screened for Y, and the result data after Y sorting and screening is sorted for indicator Z and the final screened optimal result will be selected.

[0049] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0050] 1. In the present invention, through the parallel training of multiple machine learning and deep learning models such as linear regression, polynomial regression, SVM, XGBoost, GBRT, DNN and Transformer, the relationship between the index value of the index system and the comprehensive index value can be more comprehensively captured, thereby improving the accuracy of the prediction. The innovative triple conditional ranking screening method based on RMSE, R2 and time cost is proposed, which can effectively screen out the optimal prediction model and ensure the accuracy and reliability of the prediction results.

[0051] 2. In the present invention, the system will automatically detect the update of the indicator system or the actual data set, and automatically perform a new round of model training, testing and prediction to ensure the timeliness and accuracy of the prediction results. Each time the indicator system or the actual data set is updated, each model will be retrained to ensure that the model can adapt to the new data environment and avoid being outdated. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a technical route step diagram of the present invention;

[0053] Figure 2 It is a system block diagram of the present invention;

[0054] Figure 3 It is the mainstream index diagram for evaluating the regression prediction performance of the model in the present invention;

[0055] Figure 4 This is a general diagram of the dynamic AI indicator system index comprehensive prediction method and device in the present invention. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0057] Reference Figure 1-4 ,

[0058] Example:

[0059] Dynamic AI index system index prediction fitting method based on machine learning

[0060] The specific steps are as follows:

[0061] 1. Obtain the historical or real-time recorded data of each indicator value and comprehensive index of a certain level in the indicator system. The indicator name and comprehensive index are used as the row and field names of the data table, and the column data are the records of the indicator values ​​entered for evaluating various things or a single thing multiple times and the comprehensive index values ​​calculated according to the indicator system calculation formula.

[0062] 2. Since the measurement units of each indicator of the indicator system may be different when the indicator system is established and designed, each indicator will be standardized or normalized when participating in the calculation of the comprehensive index, that is, a probability distribution between 0 and 1 or a coordinate distribution between -1 and +1. Therefore, it is necessary to understand and prepare the data of all rows of data of each indicator value and comprehensive index value, and check whether the obtained indicator value data is in the probability distribution between 0 and 1 or between -1 and +1. If not, the system needs to automatically standardize or normalize each indicator value data to a probability distribution between 0 and 1 or between -1 and +1.

[0063] Data Standardization:

[0064] Data standardization is to scale the data so that it falls into a small specific range. Since the measurement units of various indicators in the credit index system are different, in order to be able to participate in the evaluation calculation, the indicators need to be normalized and their values ​​mapped to a certain numerical range through function transformation.

[0065] There are many methods for data standardization. Based on general experience, the present invention uses the min-max standardization method. After the above standardization process, the original data are converted into dimensionless index evaluation values, that is, each index value is at the same level, and comprehensive evaluation and analysis can be performed.

[0066] The min-max normalization method is to perform a linear transformation on the original data. Let minA and maxA be the minimum and maximum values ​​of attribute A respectively, and map an original value x of A to a value x' in the interval [0,1] through min-max normalization. The formula is:

[0067] New data = /

[0068] y=(x-MinValue) / (MaxValue-MinValue)

[0069] Note: x and y are the values ​​before and after the conversion, respectively. MaxValue and MinValue are the maximum and minimum values ​​of the samples, respectively.

[0070] (2) Data normalization

[0071] Normalization is a way to simplify calculations, that is, to transform dimensioned expressions into dimensionless expressions and become scalars. The specific role of normalization is to summarize and unify the statistical distribution of samples. Normalization between 0 and 1 is a statistical probability distribution, and normalization between -1 and +1 is a statistical coordinate distribution. Normalization means unity, unification and unification. Whether for modeling or calculation, the basic measurement unit must be the same. Neural networks are trained (probability calculation) and predicted based on the statistical probability of samples in events. Normalization is a statistical probability distribution that is unified between 0 and 1; SVM is based on linear division distance after dimensionality reduction for classification and simulation. Therefore, spatiotemporal dimensionality reduction normalization is a statistical coordinate distribution that is unified between -1 and +1.

[0072] There are mainly the following normalization methods, and this patent adopts the first one.

[0073] a. Linear function conversion, the expression is as follows:

[0074] y=(x-MinValue) / (MaxValue-MinValue)

[0075] Note: x and y are the values ​​before and after the conversion, respectively. MaxValue and MinValue are the maximum and minimum values ​​of the samples, respectively.

[0076] b. Logarithmic function conversion, the expression is as follows:

[0077] y=log10(x)

[0078] Description: Logarithmic function conversion with base 10.

[0079] c. Inverse cotangent function conversion, the expression is as follows:

[0080] y=atan(x)*2 / PI

[0081] 3. Data preprocessing, the first step in creating a machine learning model. Performing data preprocessing includes data cleaning, data integration, data transformation, and data reduction methods. The steps include obtaining data sets, importing libraries, importing data sets, finding missing data, and encoding categorical data.

[0082] Data cleaning: mainly to complete missing values ​​in the data, eliminate noise data, identify or delete outliers and resolve inconsistencies. The main goals are to standardize data formats, clean abnormal data, correct errors, and remove duplicate data. Data cleaning includes: 1. Deleting redundant and duplicate data; 2. Fixing structural errors; 3. Managing unnecessary outliers; 4. Processing missing data, etc.

[0083] Data integration: Integrate data from multiple data sources and store them in a unified manner.

[0084] Data transformation: Convert data into a form suitable for data mining through smooth aggregation, data generalization, normalization, etc.

[0085] Data reduction: Reduce or simplify a data set, not only maintaining the integrity of the original data but also making the results after data reduction the same or almost the same as before reduction.

[0086] In view of the standardization and characteristics of the indicator system data set, the data cleaning involved in this patent first handles data missing, fills missing values ​​in the record with 0 according to the indicator record habit; deletes redundant duplicate data, including duplicate row records with exactly the same fields, and column data with the same value including 0 for each evaluation item record value of a certain indicator; then confirms whether each indicator value has been standardized or normalized during indicator design due to different measurement units. If there is an indicator that has not undergone standardized or normalized data transformation, a unified probability distribution between 0 and 1 or a coordinate distribution between -1 and +1 is performed on all indicators. It is not allowed that some are 0-1 and some are -1-+1; the indicator system data is relatively standardized, and outliers have practical significance and are not deleted.

[0087] 4. Select each indicator after data preprocessing as the feature Xi, the indicator value as its feature value, and take the comprehensive index value as the label value of the target variable Y; perform feature construction, feature extraction and feature construction of machine learning feature engineering.

[0088] Feature engineering refers to the process of converting raw data into training data for a model. Its purpose is to obtain better training data features so that the machine learning model approaches this upper limit. Feature engineering can improve the performance of the model, and sometimes even achieve good results on simple models. Feature engineering plays a very important role in machine learning and is generally considered to include three parts: feature construction, feature extraction, and feature selection.

[0089] Feature extraction obtains a set of features with obvious physical or statistical significance through feature transformation. Feature selection is to eliminate irrelevant or redundant features, reduce the number of effective features, reduce the time of model training, and improve the accuracy and performance of the model. The following common methods are designed and selected based on general knowledge and experience.

[0090] Use statistical methods to measure the relationship between a single feature and the response variable.

[0091] The Pearson correlation coefficient measures the linear relationship between variables. The result is [-1,1]. -1 indicates a complete negative correlation, +1 indicates a positive correlation, and 0 indicates a non-linear correlation, but it does not mean that there is no other relationship. An obvious flaw of the Pearson coefficient is that it only measures linear relationships. If the relationship is nonlinear, even if the two variables have a one-to-one relationship, the Pearson correlation will be close to 0. The formula of the Pearson coefficient is the sample covariance divided by the product of the standard deviation of X and the standard deviation of Y. In sklearn, the Pearson correlation coefficient between two random variables can be directly calculated. For example, the correlation coefficient between X and X*X calculated below is approximately 0.

[0092] Maximum information coefficient. The maximum information coefficient is obtained based on mutual information. The following is the mutual information formula. The maximum information coefficient MIC can not only measure the linear relationship between variables like the Pearson coefficient, but also measure the nonlinear relationship between variables. Although MIC can measure the nonlinear relationship between variables, when the relationship between variables is close to linear correlation, the Pearson correlation coefficient is still irreplaceable. First, the Pearson correlation coefficient is fast to calculate, which is very important when processing large-scale data. Second, the value range of the Pearson correlation coefficient is [-1, 1], while the MIC and distance correlation coefficient are both [0, 1]. This feature enables the Pearson correlation coefficient to represent richer relationships. The sign indicates the positive or negative relationship, and the absolute value can indicate the intensity.

[0093] Feature selection based on machine learning algorithms

[0094] Linear Models and Regularization

[0095] When there is a linear relationship between the features and the response variable, and the features are relatively independent. You can try to use a linear regression model for feature selection, because the more important the feature, the larger the corresponding coefficient in the model, and the more irrelevant the feature is to the output variable, the closer the corresponding coefficient is to 0. In many actual data, there are often multiple interrelated features. At this time, the model becomes unstable, and a slight change in the data may lead to a huge change in the model, which makes the prediction of the model difficult. This phenomenon is also called multicollinearity. For example, suppose we have a data set, and its true model should be Y = X1 + X2. When we observe, we find that Y' = X1 + X2 + e, where e is noise. If there is a linear relationship between X1 and X2, for example, X1 is approximately equal to X2, then due to the existence of noise e, the model we learn may not be Y = X1 + X2, but may be Y = 2X1, or Y = -X1 + 3X2. By adding regularization terms to the model, feature selection can also be achieved. L1 regularization produces a relatively sparse model. Controlling the penalty coefficient alpha will force the coefficients corresponding to weak features to become 0. This feature makes L1 regularization a good feature selection method. L2 regularization will make the values ​​of the coefficients average. For related features, this means that they can obtain more similar corresponding coefficients.

[0096] Random Forest Model

[0097] Random forests are composed of multiple decision trees. Each node in the decision tree is a condition about a certain feature. Impurity can be used to determine the optimal feature for partitioning the data set. For classification problems, Gini impurity or information gain is usually used. For regression problems, variance or least squares fitting is usually used. When training a decision tree, it is possible to calculate how much each feature reduces the impurity of the tree. For a decision tree forest, it is possible to calculate how much each feature reduces the impurity on average, and use the average reduced impurity as the value of feature selection.

[0098] Another common feature selection method is to directly measure the impact of each feature on the model accuracy. The main idea is to shuffle the order of the eigenvalues ​​of each feature and measure the impact of the change in order on the model accuracy. Obviously, for unimportant variables, shuffling the order will not have much impact on the model accuracy, but for important variables, shuffling the order will reduce the model accuracy.

[0099] Feature selection is to select features with high correlation with the target variable from the original data set to reduce data dimensions and computational complexity while ensuring that important information in the data is retained. Because there is an irremovable computational relationship between each indicator and index after data preprocessing, features with high correlation with the target variable are not selected for the indicator system to reduce the number of features invested in machine learning training.

[0100] 5. Machine learning prediction model construction, which includes parallel construction of linear and polynomial regression, support vector machine, extreme gradient boosting, gradient enhanced regression tree, deep neural network, Transformer and other machine learning and deep learning models for comprehensive index prediction training; each time the comprehensive index of the indicator system is calculated for the target object, that is, when the indicator system or the actual data set is updated, each algorithm model will be automatically and dynamically retrained.

[0101] The second step of the AI ​​machine learning prediction model is model design and training, which is also the core step of machine learning. By expressing the relationship between input data and expected output as a mathematical model and using optimization algorithms to adjust the parameters of the model, the model can accurately predict new data.

[0102] Model building and training

[0103] After data preprocessing is completed, select the appropriate machine learning algorithm for model design, development and training. This patent designs algorithm models such as linear regression, polynomial regression, gradient boosting regression tree, extreme gradient boosting, deep neural network and Transformer. At the same time, cross-validation, grid search and other methods are used to optimize and adjust the parameters and hyperparameters of each machine learning algorithm model, and select the parameter combination that can obtain the highest prediction result accuracy to improve the accuracy of the regression model.

[0104] Defining the loss function

[0105] Loss function, also known as cost function, is often used to represent or measure the error generated by the decision function. Loss function is one of the key factors affecting model performance. A deep understanding of various existing loss functions is the premise and basis for selecting and constructing loss functions. The loss function is used to evaluate the degree of difference between the model prediction results and the actual results. Commonly used loss functions can be selected, such as mean square error and root mean square error in the second chapter of concepts and theories. By minimizing the loss function, the model parameters are adjusted to make the model's prediction results more accurate. This paper selects RMSE as the loss function in each prediction algorithm program for development and design.

[0106]

[0107] 6. When training the support vector machine, extreme gradient boosting and gradient boosted regression tree models, the root mean square error is defined as the loss function. The algorithm automatically applies cross-validation and grid search to optimize the parameters and hyperparameters of each model, and automatically selects the best parameter combination to make the prediction accuracy of each model the highest.

[0108] 7. Test the optimal parameter models of linear and polynomial regression, support vector machine, extreme gradient boosting, gradient boosted regression tree, deep neural network and Transformer, and automatically calculate the root mean square error, determination coefficient and time cost on the test data set as accuracy evaluation indicators to evaluate the prediction performance of the model. Evaluate and optimize each machine learning prediction model, including using the test set to evaluate the performance indicators of the model. This patent selects mainstream accuracy evaluation indicators such as mean square error, root mean square error, determination coefficient, mean absolute percentage error, and innovatively adds the time cost evaluation indicator of model training test prediction to evaluate the prediction performance of the model. At the same time, cross-validation, grid search and other methods are used to optimize the parameters and hyperparameters of the model to ensure the selection of the best parameter combination to improve the performance and accuracy of the regression model.

[0109] By designing and building machine learning models such as linear regression, support vector machine, extreme gradient boosting, gradient enhanced regression tree, deep neural network and Transformer, the index value of the evaluation index is predicted, and the performance and accuracy of each algorithm model are compared and analyzed. The mean square error, root mean square error, determination coefficient, mean absolute percentage error of the prediction results of each algorithm model on the test set, as well as the innovative indicators CPU and GPU execution time cost are compared. The results are shown in the following table.

[0110] Table 1 Comparison and ranking of prediction performance of various models for index prediction based on index system data

[0111]

[0112]

[0113] The root mean square error RMSE of the linear regression and polynomial regression models is the minimum value of 0, and the R-square is the maximum value of 1, with the best accuracy and fit; the second is the deep neural network prediction model, whose root mean square error RMSE is at least 0.4 higher than the remaining other machine learning algorithm models; the RMSE of extreme gradient boosting and gradient boosted regression trees can also reach an error within the range of 0.3 when the training set increases, and the R-square can also reach a maximum value of 1. The prediction results of the random forest model based on the experimental data in this experiment are relatively poor in accuracy and fit in both experiments compared with the prediction results of other machine learning models, so the dynamic AI indicator system index prediction comprehensive model of multiple machine learning models established by this patent does not include the random forest prediction model. The dynamic AI indicator system index comprehensive prediction method and device of this patent ultimately includes a variety of machine learning models including linear and polynomial regression, support vector machine, extreme gradient boosting, gradient boosted regression tree, deep neural network and Transformer, as shown in the figure below.

[0114] 8. Based on the above steps, a comprehensive prediction model of the dynamic AI indicator system index is established. Linear and polynomial regression, support vector machine, extreme gradient boosting, gradient enhanced regression tree, deep neural network and Transformer models are selected for automatic parallel training and testing based on the obtained indicator system data. The prediction performance is compared and the best one is selected as the selected model and the predicted comprehensive index value is output.

[0115] Based on the practical experience of AI model training and prediction, it is known that if the AI ​​prediction accuracy is high but the time cost is too long, it will not be actually selected for deployment and application; no matter how good the determination coefficient of the AI ​​prediction model is, but the root mean square error is very low, it will not be selected. In view of this, this patent innovatively proposes a concise and efficient sorting and screening method based on the prediction performance evaluation indicators of three different dimensions, namely the root mean square error, determination coefficient and time cost, to select the optimal AI prediction model, namely the triple condition ranking screening method:

[0116] a. Direct sorting: sort the data in optimal descending order according to the weight of each indicator.

[0117] b. Case ranking, a data analysis method used to sort data to determine the position of each observation in a sequence.

[0118] A method integrating direct sorting and case ranking was adopted to rank the three prediction performance evaluation indicators of root mean square error, determination coefficient and time cost, respectively, to rank the performance of linear and polynomial regression, support vector machine, extreme gradient boosting, gradient enhanced regression tree, deep neural network and Transformer model, with the best one ranked first.

[0119] Then the next step is to innovatively establish a triple condition ranking screening method. For the three evaluation factors X, Y, and Z, they are sorted and grouped according to X, Y, and Z at the same time, that is, triple sorting; because the selection condition sorting and screening is first sorted and screened according to one factor X, the result data after X sorting and screening is sorted and screened for Y, and the result data after Y sorting and screening is sorted for indicator Z and the final screened optimal result will be selected.

[0120] In practical application, three selected prediction performance evaluation indicators are selected: root mean square error, determination coefficient and time cost. In view of the fact that the time cost is too long and the determination coefficient is good but the root mean square error is very low, such AI prediction models will not be actually selected. First, the execution time cost sorting results are selected to filter and delete the worst model; the next step is to sort and filter the determination coefficient of each model after deleting the longest execution time model, and delete the model with the smallest R2 value. If there is no smallest R2 model and there are more than 1 model, the model will not be deleted in this step; the next step is also the last step to sort the root mean square error of each model after screening and exclude, and select the model with the smallest root mean square error as the final selected model. Then output the comprehensive index prediction value of the selected model.

[0121] According to the accuracy and ranking comparison results of linear and polynomial regression, support vector machine, extreme gradient boosting, gradient enhanced regression tree, deep neural network and Transformer models for index prediction based on the evaluation index system data in Table 1, a triple conditional ranking screening method is used for analysis and evaluation. First, the execution time performance of each model is sorted, and the model with the longest execution time is deleted; the next step is to sort and screen the determination coefficient of each model after deleting the model with the longest execution time, and delete the model with the smallest R2 value. If there is no smallest R2 model and there are more than 1 model, the model will not be deleted in this step; the last step is to sort the root mean square error of each model after screening and exclusion, and select the model with the smallest root mean square error as the final selected model, which is the linear and polynomial regression model based on the results in Table 1. Then, the index prediction value of the linear and polynomial regression model for the index data to be predicted is output as the comprehensive index prediction result of the method and device.

[0122] 9. At this point, the AI ​​algorithm fitting prediction method and device for the comprehensive index of this indicator system are completed. When the indicator system or actual data set is automatically checked for updates, return to the first step to perform a new round of comprehensive index AI algorithm fitting prediction.

[0123] From the above, we can know that:

[0124] In the present invention, through the parallel training of multiple machine learning and deep learning models such as linear regression, polynomial regression, SVM, XGBoost, GBRT, DNN and Transformer, the relationship between the index value of the index system and the comprehensive index value can be more comprehensively captured, thereby improving the accuracy of the prediction. The innovative triple conditional ranking screening method based on RMSE, R2 and time cost is proposed to effectively screen out the optimal prediction model and ensure the accuracy and reliability of the prediction results.

[0125] In the present invention, the system will automatically detect the update of the indicator system or the actual data set, and automatically perform a new round of model training, testing and prediction to ensure the timeliness and accuracy of the prediction results. Each time the indicator system or the actual data set is updated, each model will be retrained to ensure that the model can adapt to the new data environment and avoid being outdated.

[0126] The present invention has the advantages of parallel training of multiple models:

[0127] Model diversity: Different machine learning models have different characteristics. For example, the linear regression model is suitable for linear relationships, the DNN model is suitable for nonlinear relationships, and the SVM model is suitable for feature space division. By training multiple models in parallel, the relationship between the indicator value of the indicator system and the comprehensive index value can be captured more comprehensively, thereby improving the accuracy of the prediction.

[0128] Complementarity: Different models may be sensitive to different data features. By training multiple models in parallel, the complementarity of different models can be exploited to further improve the accuracy of predictions.

[0129] Robustness: When the prediction performance of some models degrades due to data noise or outliers, other models may still be able to maintain high prediction performance, thereby improving the robustness of the entire system.

[0130] Advantages of triple condition ranking screening method:

[0131] Comprehensive consideration of multiple indicators: This method simultaneously considers the three indicators of RMSE, R2 and time cost, which can more comprehensively evaluate the prediction performance of the model and avoid focusing on a single indicator while ignoring other important factors.

[0132] Highlight key indicators: This method first screens according to time cost, eliminates models with too long execution time, and ensures the practicality of the model; secondly, it screens according to R2, eliminates models with too low a degree of fit, and ensures the reliability of the model; finally, it screens according to RMSE, selects the model with the smallest prediction error, and ensures the accuracy of the model.

[0133] Simplify the selection process: This method can effectively simplify the model selection process through the triple condition ranking screening method, avoiding the tediousness and subjectivity of manual model selection.

[0134] Dynamic update is achieved in the present invention:

[0135] Automatic detection of data updates:

[0136] Real-time: The system can detect updates to the indicator system or actual data set in real time, and immediately execute a new round of model training, testing, and prediction to ensure the timeliness of the prediction results.

[0137] Reduce manual intervention: Automatic detection of data updates can reduce manual intervention and improve work efficiency.

[0138] Adapt to changes: Updates to the indicator system or actual data set may reflect changes in the external environment. Automatically detecting data updates and retraining the model can ensure that the model can adapt to the new data environment and avoid becoming outdated.

[0139] Model retraining:

[0140] Maintain model accuracy: Over time, the metric system or the actual data set may change, and retraining the model can ensure that the model's predictive performance is not affected.

[0141] Improve model generalization ability: Retraining the model can help the model learn new data features, improve the model's generalization ability, and enable it to better adapt to new data environments.

[0142] Avoid overfitting: Overfitting may occur during model training. Retraining the model can help the model generalize better and avoid overfitting.

[0143] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "comprise" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0144] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A dynamic AI indicator system index prediction fitting method based on machine learning, characterized by: The method comprises the following steps: S1: Obtain the historical or real-time record data of each indicator value and comprehensive index of a certain level of the indicator system, with each indicator name and comprehensive index as the row field name of the data table, and the column data as the records of each indicator value entered multiple times for evaluating each thing or a single thing and the comprehensive index value calculated according to the indicator system calculation formula; S2: Understand and prepare all rows of data for each record of each indicator value and comprehensive index value, and check whether the obtained indicator value data is in the probability distribution between 0 and 1 or between -1 and +1. If not, the system needs to automatically standardize or normalize each indicator value data to the probability distribution between 0 and 1 or between -1 and +1; S3: Data preprocessing, the first step in creating a machine learning model; performing data preprocessing includes data cleaning, data integration, data transformation, and data reduction methods. The steps include obtaining a dataset, importing a library, importing a dataset, finding missing data, and encoding categorical data; S4: Select each index after data preprocessing as feature Xi, the index value as its feature value, and take the comprehensive index value as the label value of the target variable Y; perform feature construction, feature extraction and feature selection of machine learning feature engineering; S5: Construct machine learning prediction models, and respectively construct linear and polynomial regression, support vector machine, extreme gradient boosting, gradient boosted regression tree, deep neural network and Transformer machine learning and deep learning models in parallel to train comprehensive index prediction; each time the comprehensive index of the indicator system is calculated for the target object, that is, when the indicator system or the actual data set is updated, each algorithm model will be automatically and dynamically retrained; S6: Perform support vector machine, extreme gradient boosting and gradient boosting regression tree training for each model, define the root mean square error as the loss function, and the algorithm automatically applies cross-validation and grid search to optimize the parameters and hyperparameters of each model, and automatically select the best parameter combination to make the prediction accuracy of each model the highest; S7: Test the optimal parameter models of linear and polynomial regression, support vector machine, extreme gradient boosting, gradient boosted regression tree, deep neural network and Transformer, and automatically calculate their root mean square error, determination coefficient and time cost on the test data set as accuracy evaluation indicators to evaluate the prediction performance of the model; S8: Based on the prediction performance evaluation index values ​​of three different dimensions, namely, root mean square error, determination coefficient and time cost, the self-created triple condition ranking screening method is used to screen and select the optimal prediction model for each prediction model. Finally, the model with the smallest root mean square error in the last layer of conditional sorting is selected as the comprehensive index AI prediction model of this indicator system based on the existing data set and subsequent predictions are performed to output the comprehensive index prediction value of the selected model; S9: After that, the AI ​​algorithm fitting prediction method and device for the comprehensive index of this indicator system are completed; when the indicator system or the actual data set is automatically checked for updates, it returns to the first step to perform a new round of comprehensive index AI algorithm fitting prediction.

2. The dynamic AI indicator system index prediction fitting method based on machine learning according to claim 1, characterized in that: In step S2, data standardization includes the following steps: Data standardization is to scale the data so that it falls into a small specific range. Since the measurement units of various indicators in the credit index system are different, in order to be able to participate in the evaluation calculation, the indicators need to be normalized and their values ​​mapped to a certain numerical range through function transformation. There are many methods for data standardization. Based on general experience, the present invention uses the min-max standardization method. After the above standardization process, the original data are converted into dimensionless index evaluation values, that is, the values ​​of each index are at the same level, and comprehensive evaluation and analysis can be performed. The min-max standardization method is to perform a linear transformation on the original data; let minA and maxA be the minimum and maximum values ​​of attribute A respectively, and map an original value x of A to a value x' in the interval [0,1] through min-max standardization. The formula is: newdata= / ; y=(x-MinValue) / (MaxValue-MinValue) Note: x and y are the values ​​before and after the conversion, respectively. MaxValue and MinValue are the maximum and minimum values ​​of the samples, respectively.

3. The dynamic AI indicator system index prediction fitting method based on machine learning according to claim 1, characterized in that: In step S2, data normalization includes the following steps: First, the basic measurement units must be consistent. Neural networks are trained (probability calculation) and predicted based on the statistical probability of samples in events. Normalization is a statistical probability distribution between 0 and 1. SVM is based on linear division of distances after dimensionality reduction for classification and simulation. Therefore, spatiotemporal dimensionality reduction normalization is a statistical coordinate distribution between -1 and +1. The normalization method is: Linear function conversion, the expression is as follows: y=(x-MinValue) / (MaxValue-MinValue) Note: x and y are the values ​​before and after the conversion, respectively. MaxValue and MinValue are the maximum and minimum values ​​of the samples, respectively.

4. The dynamic AI indicator system index prediction fitting method based on machine learning according to claim 1, characterized in that: In step S3, data cleaning includes: mainly completing missing values ​​in the data, eliminating noise data, identifying or deleting outliers and resolving inconsistencies; the main goals achieved are: standardizing data formats, cleaning abnormal data, correcting errors, and removing duplicate data; data cleaning includes:

1. deleting redundant and duplicate data; 2. fixing structural errors; 3. managing unnecessary outliers; 4. processing missing data; Data integration includes: integrating data from multiple data sources and storing them uniformly; Data transformation converts data into a form suitable for data mining through smooth aggregation, data generalization, and normalization; Data reduction reduces or simplifies a data set, not only maintaining the integrity of the original data but also making the results after data reduction the same or almost the same as before; In view of the standardization and characteristics of the indicator system data set, missing values ​​in the records are filled with 0 according to the indicator recording habits; redundant duplicate data are deleted, including duplicate row records with exactly the same fields, and column data with the same value including 0 for each evaluation item of a certain indicator; then confirm whether the indicator values ​​are standardized or normalized during indicator design due to different measurement units. If there are indicators that have not undergone standardized or normalized data transformation, a unified probability distribution between 0 and 1 or a coordinate distribution between -1 and +1 is performed on all indicators. It is not allowed that some are 0-1 and some are -1-+1; the indicator system data is relatively standardized, and outliers have practical significance and are not deleted.

5. The dynamic AI indicator system index prediction fitting method based on machine learning according to claim 1, characterized in that: In step S4, feature extraction uses statistical methods to measure the relationship between a single feature and the response variable. The Pearson correlation coefficient is used to measure the linear relationship between variables. The result is [-1,1]. -1 indicates a complete negative correlation, +1 indicates a positive correlation, and 0 indicates a non-linear correlation, but it does not mean that there is no other relationship. The obvious defect of the Pearson coefficient is that it only measures linear relationships. If the relationship is nonlinear, even if the two variables have a one-to-one relationship, the Pearson relationship will be close to 0. The formula of the Pearson coefficient is the sample covariance divided by the product of the standard deviation of X and the standard deviation of Y. The Pearson correlation coefficient between two random variables can be directly calculated in sklearn. For example, the correlation coefficient between X and X*X calculated below is approximately 0. The maximum information coefficient in the feature extraction process is obtained based on the mutual information. The following is the mutual information formula; the maximum information coefficient MIC can not only measure the linear relationship between variables like the Pearson coefficient, but also measure the nonlinear relationship between variables; Although MIC can measure the nonlinear relationship between variables, when the relationship between variables is close to linear correlation, Pearson correlation coefficient is still irreplaceable; First, the Pearson correlation coefficient is fast to calculate, which is very important when dealing with large-scale data; second, the value range of the Pearson correlation coefficient is [-1, 1], while the MIC and distance correlation coefficients are both [0, 1]; This feature enables the Pearson correlation coefficient to represent richer relationships, with the sign indicating the positive or negative relationship and the absolute value indicating the intensity.

6. The dynamic AI indicator system index prediction fitting method based on machine learning according to claim 1, characterized in that: In step S4, feature selection uses a linear regression model. Suppose we have a data set, and its true model should be Y=X1+X2. When we observe, we find that Y'=X1+X2+e, where e is noise. If there is a linear relationship between X1 and X2, for example, X1 is approximately equal to X2, then due to the existence of noise e, the model we learn may not be Y=X1+X2, but may be Y=2X1, or Y=-X1+3X2. By adding regularization terms to the model, feature selection can also be achieved. L1 regularization results in a relatively sparse model. Controlling the penalty term coefficient alpha will force the coefficients corresponding to weak features to become 0. This feature makes L1 regularization a good feature selection method. L2 regularization will make the values ​​of the coefficients average. For related features, this means that they can obtain more similar corresponding coefficients.

7. The dynamic AI indicator system index prediction fitting method based on machine learning according to claim 1, characterized in that: The step S5 specifically includes: Model building and training: After data preprocessing is completed, select the appropriate machine learning algorithm for model design, development and training; this patent designs linear regression, polynomial regression, gradient boosting regression tree, extreme gradient boosting, deep neural network and Transformer algorithm models; at the same time, cross-validation and grid search methods are used to optimize and adjust the parameters and hyperparameters of each machine learning algorithm model, and select the parameter combination that can obtain the highest prediction result accuracy to improve the accuracy of the regression model; Define the loss function: Loss function, also known as cost function, is often used to represent or measure the error generated by decision function; loss function is one of the key factors affecting model performance; in-depth understanding of various existing loss functions is the premise and basis for selecting and constructing loss functions; loss function is used to evaluate the difference between the model prediction results and the actual results. For commonly used loss functions, see the mean square error and root mean square error in the second chapter concepts and theories. By minimizing the loss function, the model parameters are adjusted to make the model prediction results more accurate; this paper selects RMSE as the loss function in each prediction algorithm program for development and design; 8. The dynamic AI indicator system index prediction fitting method based on machine learning according to claim 1, characterized in that: In the step S7, specifically, the test set is used to evaluate the performance indicators of the model. This patent selects the mainstream accuracy evaluation indicators of mean square error, root mean square error, determination coefficient, and mean absolute percentage error, and innovatively adds the time cost evaluation indicator of model training test prediction completion to evaluate the prediction performance of the model. At the same time, cross-validation and grid search methods are used to optimize the parameters and hyperparameters of the model to ensure the selection of the best parameter combination to improve the performance and accuracy of the regression model.

9. The dynamic AI indicator system index prediction fitting method based on machine learning according to claim 1, characterized in that: In step S8, a concise and efficient ranking and screening method, namely a triple condition ranking and screening method, is used to select the optimal AI prediction model using prediction performance evaluation indicators based on three different dimensions: root mean square error, determination coefficient, and time cost. The method specifically includes the following steps: a. Direct sorting: sort the data in optimal descending order according to the weight of each indicator; b. Case ranking, a data analysis method used to sort data to determine the position of each observation in a sequence; The method of integrating direct sorting and case ranking was adopted to rank the performance of linear and polynomial regression, support vector machine, extreme gradient boosting, gradient boosted regression tree, deep neural network and Transformer model based on the three prediction performance evaluation indicators of root mean square error, determination coefficient and time cost, and the best one was ranked first. Then the next step is to innovatively establish a triple condition ranking screening method; for the three evaluation factors X, Y, and Z, they are sorted and grouped according to X, Y, and Z at the same time, that is, triple sorting; due to the selection condition sorting and screening, it is first sorted and screened according to one factor X, and the result data after X sorting and screening is sorted and screened for Y, and the result data after Y sorting and screening is sorted for indicator Z and the final screened optimal result will be selected.

10. A dynamic AI indicator system comprehensive index prediction and fitting device based on multiple machine learning, the device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, it implements the process of the dynamic AI indicator system comprehensive index prediction and fitting method based on multiple machine learning as described in any one of claims 1 to 9.

Images