An artificial intelligence decision optimization method and system based on big data drive

Abstract

The present invention relates to the field of artificial intelligence data processing and decision optimization technology, specifically, an artificial intelligence decision optimization method and system driven by big data, including obtaining a sample data set; converting the category feature into a category feature standard value according to the attribute gain and correlation of the category feature, and converting the continuous feature into a continuous feature standard value according to the standard scaling and correlation of the continuous feature; constructing a prediction model of the target variable and the category feature standard value and the continuous feature standard value; calculating the optimal value of the constraint condition according to the Lagrange multiplier method to achieve artificial intelligence decision optimization. The prediction model puts the category feature and the continuous feature into the same prediction model, thereby satisfying the integrity of the prediction model; at the same time, the mutual influence between the attribute features is taken into account, thereby improving the accuracy of the prediction model.

Description

Technical Field

[0001] The present invention relates to the technical field of artificial intelligence data processing and decision optimization, and specifically to an artificial intelligence decision optimization method and system driven by big data. Background Art

[0002] With the rapid development of IoT devices and Internet platforms, prediction models driven by big data are becoming more and more perfect, and there are more and more ways to optimize artificial intelligence decisions based on limited conditions and target variables. However, there are still some shortcomings in the existing prediction models. First, when processing data, it is often difficult for existing prediction models to include both categorical features and continuous features as independent variable factors into the model at the same time. The categorical features are usually converted into numerical forms using One-Hot Encoding or Label Encoding, while the continuous features directly use the original numerical values. This processing method easily leads to the difficulty of categorical features and continuous features to work together in the same model, affecting the collaborative performance of the respective variable factors in the prediction model. Second, in the actual data processing process, there may be a correlation between categorical features and continuous features, and this correlation may have an adverse effect on the training process of the model. In view of the above problems, the present invention proposes a new method for processing categorical features and continuous features, which converts categorical features and continuous features into standard values ​​together so that they can be used simultaneously in the same prediction model to improve the collaborative performance of the prediction model. At the same time, the degree of correlation between features is considered when performing feature standardization, thereby improving the accuracy of the prediction model. Summary of the invention

[0003] (1) Technical issues to be resolved

[0004] The purpose of the present invention is to provide an artificial intelligence decision optimization method and system based on big data drive, so as to solve the problem of converting categorical features and continuous features into standard values, so as to use them simultaneously in the same prediction model to improve the prediction ability of the model. At the same time, when performing feature standardization, the degree of correlation between features is considered, thereby improving the accuracy of the prediction model.

[0005] (2) Technical solution

[0006] To achieve the above objectives, on the one hand, the present invention provides an artificial intelligence decision optimization method based on big data drive, the method comprising:

[0007] S1. Obtain a sample data set and divide the sample data set into a training set and a validation set, wherein the sample data set includes attribute features, attribute values, and a target variable, wherein the attribute features include category features and continuous features;

[0008] S2. Processing the data according to a preprocessing method, wherein the preprocessing method includes removing the upper threshold of the sample data set; calculating the attribute gain and correlation of the category feature and converting them into the category feature standard value according to a preset conversion method, calculating the standard scaling and correlation of the continuous feature and converting them into the continuous feature standard value according to a preset conversion method;

[0009] S3, construct a prediction model for the target variable and the standard value of the categorical feature and the standard value of the continuous feature, bring the training set into the prediction model to train the prediction model parameters, and bring the validation set into the prediction model to optimize the prediction model parameters;

[0010] S4, dividing the attribute features into certain features and uncertain features, setting a value range for the uncertain features as a constraint condition according to the preset condition classification method and setting the target variable as a fixed target;

[0011] S5. Input the constraints and fixed targets into the prediction model, and calculate the optimal value of the uncertain feature according to the Lagrange multiplier method; obtain a new sample data set for the next preset period and optimize the prediction model parameters according to the new sample data set, and recalculate the new optimal value of the uncertain feature by inputting new attribute features to achieve artificial intelligence decision optimization.

[0012] Furthermore, the method of calculating the attribute gain and correlation of the category feature and converting them into the category feature standard value according to the preset conversion method, and calculating the standard scaling and correlation of the continuous feature and converting them into the continuous feature standard value according to the preset conversion method includes:

[0013] Calculate the attribute gain and correlation of the category feature, calculate the standard scaling and correlation of the continuous feature; convert the category feature into the category feature standard value according to the first preset ratio, and convert the continuous feature into the continuous feature standard value according to the second preset ratio; convert the category feature standard value The calculation formula is:

[0014] ;

[0015] in is the category feature, is the attribute gain of the category feature, is the correlation degree of the category feature, , are the weight coefficients of category feature attribute gain and category feature association, is the adjustment parameter for the standard value of the category feature;

[0016] Continuous characteristic standard value The calculation formula is:

[0017] ;

[0018] in is a continuous feature, is the standard scaling of continuous features, is the correlation degree of continuous features, , are the weight coefficients of the continuous feature standard scaling and the continuous feature association, is the adjustment parameter of the standard value of the continuous feature.

[0019] Furthermore, the method of calculating the attribute gain and correlation of the category feature, calculating the standard scaling and correlation of the continuous feature; converting the category feature into the category feature standard value according to the first preset ratio, and converting the continuous feature into the continuous feature standard value according to the second preset ratio includes:

[0020] Divide the target variable into target categories according to the preset classification method, obtain the total number of entries in the sample data set and the number of entries corresponding to each target category, and the information entropy of the sample data set The calculation formula is:

[0021] ;

[0022] in is a sample data set, is the information entropy of the sample data set, For the target categories, is the total number of target categories, For the The probability of the target category;

[0023] The total number of entries corresponding to the category feature and the number of different values ​​of the target category under the category feature are counted respectively, and the probability of the number of different values ​​of the category feature accounting for the total number of entries corresponding to the category feature is calculated. The conditional entropy of different values ​​of the category feature is obtained respectively, and the conditional entropy of different values ​​of the category feature is summed to obtain the total conditional entropy of the category feature. The total conditional entropy of the category feature The calculation formula is:

[0024] ;

[0025] in is the first Category features, is a category feature The number of categories, The first Value categories, Indicates the value is The number of entries corresponding to the value category, Indicates the total number of entries corresponding to the category feature, Indicates The value category is equal to The probability of

[0026] The information entropy of the sample data set Subtract the total conditional entropy of the categorical features Get category feature attribute gain, category feature attribute gain The calculation formula is:

[0027] ;

[0028] According to the normal distribution characteristics of continuous features, the mean and standard deviation of continuous features are calculated. According to the characteristics of the standard deviation standardization method that reflects the degree of data deviation from the mean, the standard deviation standardization method is negative and then added with a preset constant to obtain the standard scaling value of the continuous feature. The calculation formula is:

[0029] ;

[0030] in is the original value of the continuous feature, is the standard scaling value, is the mean value of the continuous feature, is the standard deviation of the continuous feature, is a preset constant.

[0031] Furthermore, the method of calculating the attribute gain and correlation of the category feature, calculating the standard scaling and correlation of the continuous feature; converting the category feature into the category feature standard value according to the first preset ratio, and converting the continuous feature into the continuous feature standard value according to the second preset ratio includes:

[0032] The continuous features are divided into stage features according to the preset classification method, and the attribute gain of the stage feature is calculated. The maximum value of the attribute gain of the stage feature and the attribute gain of the category feature is used as the benchmark feature. The other stage features and category features are used as reference features. A contingency table between the reference features and the benchmark features is established and the chi-square value is calculated. The chi-square value The calculation formula is:

[0033] ;

[0034] in They are represented as datum feature and reference feature respectively. They are respectively represented as the number of values ​​of the benchmark feature and the reference feature, represents the observed value of the target variable under the intersection of two category features, Represents the expected value of the intersection of two category features, the expected value The calculation formula is:

[0035] ;

[0036] in is the benchmark characteristic in the contingency table The number of categories, is the reference feature in the contingency table The number of categories, For the reference feature Take the first The sum of the rows corresponding to the values, For the reference feature Take the first The sum of the corresponding columns when there are values. is the sum of the target variable of the baseline characteristic and the reference characteristic in the contingency table;

[0037] Calculate the contingency table correlation coefficient based on the chi-square value. The calculation formula is:

[0038] ;

[0039] The category features and continuous features Substitute the correlation coefficient formula into the contingency table to obtain and .

[0040] Furthermore, the method for constructing a prediction model of a target variable and a categorical feature standard value and a continuous feature standard value includes:

[0041] A neural network prediction model is established. According to the influence of categorical features and continuous features on the target variable, the ReLU function is used as the activation function for the categorical feature standard value, and the Sigmoid function is used as the activation function for the continuous feature. The calculation formula of the target function is:

[0042] ;

[0043] in , , is the preset activation function, For the Category features, is the total number of category features, For the A continuous feature, is the total number of continuous features, , is the bias of different hidden layers of the objective function.

[0044] Based on the same inventive concept, on the other hand, the present invention also provides an artificial intelligence decision optimization system driven by big data, the system comprising a data collection module, a data processing module, a prediction model module, an attribute feature input module, and a decision output module; the data processing module comprises an attribute gain calculation module, a correlation calculation module, and a standard scaling calculation module; the data collection module is connected to the data processing module, the data processing module is connected to the prediction model module, the prediction model module is connected to the attribute feature input module and the decision output module; the data processing module is connected to the attribute gain calculation module, the correlation calculation module, and the standard scaling calculation module;

[0045] A data collection module is used to obtain a sample data set and divide the sample data set into a training set and a validation set, wherein the sample data set includes attribute features, attribute values, and target variables, wherein the attribute features include categorical features and continuous features;

[0046] A data processing module is used to process the data according to a preprocessing method, wherein the preprocessing method includes removing the upper threshold of the sample data set; calculating the attribute gain and correlation of the category feature and converting them into the category feature standard value according to a preset conversion method, and calculating the standard scaling and correlation of the continuous feature and converting them into the continuous feature standard value according to a preset conversion method;

[0047] The prediction model module is used to build a prediction model for the target variable and the standard value of the category feature and the standard value of the continuous feature, bring the training set into the prediction model to train the prediction model parameters, and bring the validation set into the prediction model to optimize the prediction model parameters;

[0048] An attribute feature input module is used to classify attribute features into certain features and uncertain features, set a value range for the uncertain features as a constraint condition according to a preset condition classification method, and set the target variable as a fixed target;

[0049] The decision output module is used to input the constraints and fixed targets into the prediction model, calculate the optimal value of the uncertain features according to the Lagrange multiplier method; obtain a new sample data set for the next preset period and optimize the prediction model parameters according to the new sample data set, recalculate the new optimal value of the uncertain features by inputting new attribute features, and realize artificial intelligence decision optimization.

[0050] Furthermore, the working steps of the data processing module include:

[0051] Calculate the attribute gain and correlation of the category feature, calculate the standard scaling and correlation of the continuous feature; convert the category feature into the category feature standard value according to the first preset ratio, and convert the continuous feature into the continuous feature standard value according to the second preset ratio; convert the category feature standard value The calculation formula is:

[0052] ;

[0053] in is the category feature, is the attribute gain of the category feature, is the correlation degree of the category feature, , are the weight coefficients of category feature attribute gain and category feature association, is the adjustment parameter for the standard value of the category feature;

[0054] Continuous characteristic standard value The calculation formula is:

[0055] ;

[0056] in is a continuous feature, is the standard scaling of continuous features, is the correlation degree of continuous features, , are the weight coefficients of the continuous feature standard scaling and the continuous feature association, is the adjustment parameter of the standard value of the continuous feature.

[0057] Furthermore, the working steps of the attribute gain calculation module and the standard scaling calculation module include:

[0058] Divide the target variable into target categories according to the preset classification method, obtain the total number of entries in the sample data set and the number of entries corresponding to each target category, and the information entropy of the sample data set The calculation formula is:

[0059] ;

[0060] in is a sample data set, is the information entropy of the sample data set, For the target categories, is the total number of target categories, For the The probability of the target category;

[0061] The total number of entries corresponding to the category feature and the number of different values ​​of the target category under the category feature are counted respectively, and the probability of the number of different values ​​of the category feature accounting for the total number of entries corresponding to the category feature is calculated. The conditional entropy of different values ​​of the category feature is obtained respectively, and the conditional entropy of different values ​​of the category feature is summed to obtain the total conditional entropy of the category feature. The total conditional entropy of the category feature The calculation formula is:

[0062] ;

[0063] in is the first Category features, is a category feature The number of categories, The first Value categories, Indicates the value is The number of entries corresponding to the value category, Indicates the total number of entries corresponding to the category feature, Indicates The value category is equal to The probability of

[0064] The information entropy of the sample data set Subtract the total conditional entropy of the categorical features Get category feature attribute gain, category feature attribute gain The calculation formula is:

[0065] ;

[0066] According to the normal distribution characteristics of continuous features, the mean and standard deviation of continuous features are calculated. According to the characteristics of the standard deviation standardization method that reflects the degree of data deviation from the mean, the standard deviation standardization method is negative and then added with a preset constant to obtain the standard scaling value of the continuous feature. The calculation formula is:

[0067] ;

[0068] in is the original value of the continuous feature, is the standard scaling value, is the mean value of the continuous feature, is the standard deviation of the continuous feature, is a preset constant.

[0069] Furthermore, the working steps of the correlation calculation module include:

[0070] The continuous features are divided into stage features according to the preset classification method, and the attribute gain of the stage feature is calculated. The maximum value of the attribute gain of the stage feature and the attribute gain of the category feature is used as the benchmark feature. The other stage features and category features are used as reference features. A contingency table between the reference features and the benchmark features is established and the chi-square value is calculated. The chi-square value The calculation formula is:

[0071] ;

[0072] in They are represented as datum feature and reference feature respectively. They are respectively represented as the number of values ​​of the benchmark feature and the reference feature, represents the observed value of the target variable under the intersection of two category features, Represents the expected value of the intersection of two category features, the expected value The calculation formula is:

[0073] ;

[0074] in is the benchmark characteristic in the contingency table The number of categories, is the reference feature in the contingency table The number of categories, For the reference feature Take the first The sum of the rows corresponding to the values, For the reference feature Take the first The sum of the corresponding columns when there are values. is the sum of the target variable of the baseline characteristic and the reference characteristic in the contingency table;

[0075] Calculate the contingency table correlation coefficient based on the chi-square value. The calculation formula is:

[0076] ;

[0077] The category features and continuous features Substitute the correlation coefficient formula into the contingency table to obtain and .

[0078] Furthermore, the working steps of the prediction model module include:

[0079] A neural network prediction model is established. According to the influence of categorical features and continuous features on the target variable, the ReLU function is used as the activation function for the categorical feature standard value, and the Sigmoid function is used as the activation function for the continuous feature. The calculation formula of the target function is:

[0080] ;

[0081] in , , is the preset activation function, For the Category features, is the total number of category features, For the A continuous feature, is the total number of continuous features, , is the bias of different hidden layers of the objective function.

[0082] (3) Beneficial effects

[0083] Compared with the prior art, the present invention has the following beneficial effects:

[0084] 1. Convert categorical features and continuous features into standard values ​​at the same time so that they can be used simultaneously in the same prediction model to improve the prediction ability of the model.

[0085] 2. When performing feature standardization, the correlation between features is considered, and the feature correlation is stripped away when training the prediction model, thereby improving the accuracy of the prediction model. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] Figure 1 This is a flowchart of an artificial intelligence decision optimization method based on big data drive according to Embodiment 1 of the present invention;

[0087] Figure 2 This is a module block diagram of an artificial intelligence decision optimization system driven by big data according to Example 2 of the present invention. DETAILED DESCRIPTION

[0088] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0089] Before giving examples, it is necessary to explain the application scenarios of the present invention. With the rapid development of IoT devices and Internet platforms, prediction models driven by big data are becoming more and more perfect, but there are still some shortcomings in existing prediction models. First, when processing data, it is often difficult for existing prediction models to include both category features and continuous features as independent variable factors into the model. Second, in the actual data processing process, there may be correlation between category features and continuous features, which may have an adverse effect on the training process of the model. For example, in the commodity recommendation system of the e-commerce platform, attribute features include category features such as commodity category and commodity brand and continuous features such as user evaluation and price index; in the bank credit risk assessment system, attribute features include category features such as marital status and occupation and continuous features such as income and liabilities. Therefore, a new method for processing category features and continuous features is considered, which converts category features and continuous features into standard values ​​together, so that they can be used simultaneously in the same prediction model to improve the collaborative performance of the prediction model. At the same time, the degree of correlation between features is considered when performing feature standardization, thereby improving the accuracy of the prediction model.

[0090] Example 1: Figure 1 As shown, this embodiment provides an artificial intelligence decision optimization method based on big data drive, and the method includes the following steps:

[0091] S1. Obtain a sample data set and divide the sample data set into a training set and a validation set, wherein the sample data set includes attribute features, attribute values, and a target variable, wherein the attribute features include category features and continuous features;

[0092] S2. Processing the data according to a preprocessing method, wherein the preprocessing method includes removing the upper threshold of the sample data set; calculating the attribute gain and correlation of the category feature and converting them into the category feature standard value according to a preset conversion method, calculating the standard scaling and correlation of the continuous feature and converting them into the continuous feature standard value according to a preset conversion method;

[0093] S3, construct a prediction model for the target variable and the standard value of the categorical feature and the standard value of the continuous feature, bring the training set into the prediction model to train the prediction model parameters, and bring the validation set into the prediction model to optimize the prediction model parameters;

[0094] S4, dividing the attribute features into certain features and uncertain features, setting a value range for the uncertain features as a constraint condition according to the preset condition classification method and setting the target variable as a fixed target;

[0095] S5. Input the constraints and fixed targets into the prediction model, and calculate the optimal value of the uncertain feature according to the Lagrange multiplier method; obtain a new sample data set for the next preset period and optimize the prediction model parameters according to the new sample data set, and recalculate the new optimal value of the uncertain feature by inputting new attribute features to achieve artificial intelligence decision optimization.

[0096] For example, in a second-hand house recommendation system, a data set containing category features such as the geographical location and house type of the house and continuous features such as the year of construction and house size is obtained; in the data processing stage, the upper threshold in the sample data set is removed, such as large flats, school district villas and other cases, to avoid the influence of extreme values ​​on model training; then the attribute gain and correlation of the category features are calculated, and these category features are converted into standard values ​​according to the preset method; for continuous features, standard scaling processing is performed and their correlation is calculated, and also converted into standard values; a prediction model is constructed to predict house prices based on the standard values ​​of the category features and the continuous features of the house; the prediction model parameters are obtained using the training set data. Next, the hard conditions of the homebuyer are used as the determining features, the acceptable conditions of the homebuyer are used as the constraints, the budget price of the homebuyer is set, the constraints and the budget price are input into the prediction model, and the house that is closest to the homebuyer's expectations is intelligently selected.

[0097] Furthermore, the method of calculating the attribute gain and correlation of the category feature and converting them into the category feature standard value according to the preset conversion method, and calculating the standard scaling and correlation of the continuous feature and converting them into the continuous feature standard value according to the preset conversion method includes:

[0098] Calculate the attribute gain and correlation of the category feature, calculate the standard scaling and correlation of the continuous feature; convert the category feature into the category feature standard value according to the first preset ratio, and convert the continuous feature into the continuous feature standard value according to the second preset ratio; convert the category feature standard value The calculation formula is:

[0099] ;

[0100] in is the category feature, is the attribute gain of the category feature, is the correlation degree of the category feature, , are the weight coefficients of category feature attribute gain and category feature association, is the adjustment parameter for the standard value of the category feature;

[0101] Continuous characteristic standard value The calculation formula is:

[0102] ;

[0103] in is a continuous feature, is the standard scaling of continuous features, is the correlation degree of continuous features, , are the weight coefficients of the continuous feature standard scaling and the continuous feature association, is the adjustment parameter of the standard value of the continuous feature.

[0104] For example, in the product recommendation system of an e-commerce platform, product category features and continuous features are obtained, such as product category, brand, color, size, etc. and product price, number of reviews, sales volume, etc. The attribute gain of each category feature is calculated, that is, the contribution of the feature to the model prediction target, such as user click-through rate, purchase conversion rate, etc.; the correlation of each category feature is calculated, that is, the correlation between the feature and the prediction target; for continuous features, Z-score standardization is used for standard scaling to eliminate the influence of dimension, the correlation between each continuous feature and the prediction target is calculated, and the category features and continuous features are converted into category feature standard values ​​and continuous feature standard values ​​respectively. Based on the category feature standard values ​​and continuous feature standard values, a prediction model is constructed to predict user behavior, such as user preferences. Input user hard conditions, such as product brand, color, size, etc. and user acceptable conditions, such as price range, evaluation score, etc. and user budget price, and the prediction model intelligently selects the product that is closest to the user's expectations.

[0105] Furthermore, the method of calculating the attribute gain and correlation of the category feature, calculating the standard scaling and correlation of the continuous feature; converting the category feature into the category feature standard value according to the first preset ratio, and converting the continuous feature into the continuous feature standard value according to the second preset ratio includes:

[0106] Divide the target variable into target categories according to the preset classification method, obtain the total number of entries in the sample data set and the number of entries corresponding to each target category, and the information entropy of the sample data set The calculation formula is:

[0107] ;

[0108] in is a sample data set, is the information entropy of the sample data set, For the target categories, is the total number of target categories, For the The probability of the target category;

[0109] The total number of entries corresponding to the category feature and the number of different values ​​of the target category under the category feature are counted respectively, and the probability of the number of different values ​​of the category feature accounting for the total number of entries corresponding to the category feature is calculated. The conditional entropy of different values ​​of the category feature is obtained respectively, and the conditional entropy of different values ​​of the category feature is summed to obtain the total conditional entropy of the category feature. The total conditional entropy of the category feature The calculation formula is:

[0110] ;

[0111] in is the first Category features, is a category feature The number of categories, The first Value categories, Indicates the value is The number of entries corresponding to the value category, Indicates the total number of entries corresponding to the category feature, Indicates The value category is equal to The probability of

[0112] The information entropy of the sample data set Subtract the total conditional entropy of the categorical features Get category feature attribute gain, category feature attribute gain The calculation formula is:

[0113] ;

[0114] According to the normal distribution characteristics of continuous features, the mean and standard deviation of continuous features are calculated. According to the characteristics of the standard deviation standardization method that reflects the degree of data deviation from the mean, the standard deviation standardization method is negative and then added with a preset constant to obtain the standard scaling value of the continuous feature. The calculation formula is:

[0115] ;

[0116] in is the original value of the continuous feature, is the standard scaling value, is the mean value of the continuous feature, is the standard deviation of the continuous feature, is a preset constant.

[0117] For example, a dataset of 1,000 house transaction records is obtained. The target variable is the price of the house, which is divided into three categories: low price (less than or equal to 1 million), medium price (1 million to 5 million), and high price (greater than 5 million). The number of entries in each target category is calculated: 300 low-price, 500 medium-price, and 200 high-price. The information entropy of the sample dataset is for:

[0118] ; "House type" is divided into three categories: "apartment", "villa" and "townhouse". The probability of different values ​​of the target category under each category feature is calculated and the conditional entropy is calculated. The attribute gain is calculated based on the conditional entropy and the information entropy of the sample data set. For continuous features, take "house area" as an example, the house area data is obtained as , calculate the average ; Calculate standard deviation , set the constant This is to allow standard scaling values is not a negative number, use the standard scaling formula to convert the "house area" into a standard value , and finally the prediction model is brought in to facilitate machine training.

[0119] Furthermore, the method of calculating the attribute gain and correlation of the category feature, calculating the standard scaling and correlation of the continuous feature; converting the category feature into the category feature standard value according to the first preset ratio, and converting the continuous feature into the continuous feature standard value according to the second preset ratio includes:

[0120] The maximum value of the attribute gain of the category feature is used as the benchmark feature, and other category features are used as reference features. A contingency table between the reference feature and the benchmark feature is established and the chi-square value is calculated. The calculation formula is:

[0121] ;

[0122] in They are represented as datum feature and reference feature respectively. They are respectively represented as the number of values ​​of the benchmark feature and the reference feature, represents the observed value of the target variable under the intersection of two category features, Represents the expected value of the intersection of two category features, the expected value The calculation formula is:

[0123] ;

[0124] in is the benchmark characteristic in the contingency table The number of categories, is the reference feature in the contingency table The number of categories, For the reference feature Take the first The sum of the rows corresponding to the values, For the reference feature Take the first The sum of the corresponding columns when there are values. is the sum of the target variable of the baseline characteristic and the reference characteristic in the contingency table;

[0125] Calculate the contingency table correlation coefficient based on the chi-square value. The calculation formula is:

[0126] ;

[0127] The continuous features are divided into stage features according to the preset classification method, and the contingency table correlation coefficient of the stage features is calculated according to the calculation formula of the contingency table correlation coefficient .

[0128] For example, in the product recommendation system of an e-commerce platform, the feature Indicates whether the product belongs to the "Electronic Products" category (yes = 1, no = 0), feature Indicates whether the product belongs to the "household appliances" category (yes = 1, no = 0), constructing a The contingency table of Indicates the number of products that are neither electronic products nor household appliances, Indicates the number of products that are not electronic products but are household appliances, Indicates the number of products that are electronic products but not household appliances, represents the number of products that are both electronic products and household appliances, and , It is The row sum of the rows is or , It is The column sum of the columns, i.e. or , is the sum of the cells in the entire table, i.e. , the expected value is 17.5, the chi-square value is approximately equal to 0.74, and the correlation coefficient formula of the contingency table is obtained. .

[0129] Furthermore, the method for constructing a prediction model of a target variable and a categorical feature standard value and a continuous feature standard value includes:

[0130] A neural network prediction model is established. According to the influence of categorical features and continuous features on the target variable, the ReLU function is used as the activation function for the categorical feature standard value, and the Sigmoid function is used as the activation function for the continuous feature. The calculation formula of the target function is:

[0131] ;

[0132] in , , is the preset activation function, For the Category features, is the total number of category features, For the A continuous feature, is the total number of continuous features, , is the bias of different hidden layers of the objective function.

[0133] For example, in a medical health data analysis system, the patient's gender (male or female), age (A1), blood pressure (A2), cholesterol level (A3), body mass index (A4) and other attribute features and feature values ​​are used as the input layer. For categorical features such as gender, the ReLU activation function can be used, and for continuous features such as age, blood pressure, cholesterol level, body mass index, etc., the Sigmoid activation function can be used. Assuming that the gender is male, the value here is 1, age A1 = 45, blood pressure A2 = 90, cholesterol level A3 = 200, and body mass index A4 = 28, the standard scaling values ​​are obtained. , the hidden layer uses the ReLU activation function to calculate the category features ; Use Sigmoid activation function to calculate continuous features , , , , trained neural network models through specialized machine learning library programming.

[0134] Embodiment 2: Based on the same inventive concept, Figure 2As shown, this embodiment also provides an artificial intelligence decision optimization system driven by big data, the system includes a data collection module, a data processing module, a prediction model module, an attribute feature input module, and a decision output module; the data processing module includes an attribute gain calculation module, a correlation calculation module, and a standard scaling calculation module; the data collection module is connected to the data processing module, the data processing module is connected to the prediction model module, the prediction model module is connected to the attribute feature input module and the decision output module; the data processing module is connected to the attribute gain calculation module, the correlation calculation module, and the standard scaling calculation module;

[0135] A data collection module is used to obtain a sample data set and divide the sample data set into a training set and a validation set, wherein the sample data set includes attribute features, attribute values, and target variables, wherein the attribute features include categorical features and continuous features;

[0136] A data processing module is used to process the data according to a preprocessing method, wherein the preprocessing method includes removing the upper threshold of the sample data set; calculating the attribute gain and correlation of the category feature and converting them into the category feature standard value according to a preset conversion method, and calculating the standard scaling and correlation of the continuous feature and converting them into the continuous feature standard value according to a preset conversion method;

[0137] The prediction model module is used to build a prediction model for the target variable and the standard value of the category feature and the standard value of the continuous feature, bring the training set into the prediction model to train the prediction model parameters, and bring the validation set into the prediction model to optimize the prediction model parameters;

[0138] An attribute feature input module is used to classify attribute features into certain features and uncertain features, set a value range for the uncertain features as a constraint condition according to a preset condition classification method, and set the target variable as a fixed target;

[0139] The decision output module is used to input the constraints and fixed targets into the prediction model, calculate the optimal value of the uncertain features according to the Lagrange multiplier method; obtain a new sample data set for the next preset period and optimize the prediction model parameters according to the new sample data set, recalculate the new optimal value of the uncertain features by inputting new attribute features, and realize artificial intelligence decision optimization.

[0140] Furthermore, the working steps of the data processing module include:

[0141] Calculate the attribute gain and correlation of the category feature, calculate the standard scaling and correlation of the continuous feature; convert the category feature into the category feature standard value according to the first preset ratio, and convert the continuous feature into the continuous feature standard value according to the second preset ratio; convert the category feature standard value The calculation formula is:

[0142] ;

[0143] in is the category feature, is the attribute gain of the category feature, is the correlation degree of the category feature, , are the weight coefficients of category feature attribute gain and category feature association, is the adjustment parameter for the standard value of the category feature;

[0144] Continuous characteristic standard value The calculation formula is:

[0145] ;

[0146] in is a continuous feature, is the standard scaling of continuous features, is the correlation degree of continuous features, , are the weight coefficients of the continuous feature standard scaling and the continuous feature association, is the adjustment parameter of the standard value of the continuous feature.

[0147] Furthermore, the working steps of the attribute gain calculation module and the standard scaling calculation module include:

[0148] Divide the target variable into target categories according to the preset classification method, obtain the total number of entries in the sample data set and the number of entries corresponding to each target category, and the information entropy of the sample data set The calculation formula is:

[0149] ;

[0150] in is a sample data set, is the information entropy of the sample data set, For the target categories, is the total number of target categories, For the The probability of the target category;

[0151] The total number of entries corresponding to the category feature and the number of different values ​​of the target category under the category feature are counted respectively, and the probability of the number of different values ​​of the category feature accounting for the total number of entries corresponding to the category feature is calculated. The conditional entropy of different values ​​of the category feature is obtained respectively, and the conditional entropy of different values ​​of the category feature is summed to obtain the total conditional entropy of the category feature. The total conditional entropy of the category feature The calculation formula is:

[0152] ;

[0153] in is the first Category features, is a category feature The number of categories, The first Value categories, Indicates the value is The number of entries corresponding to the value category, Indicates the total number of entries corresponding to the category feature, Indicates Value categories equal to The probability of

[0154] The information entropy of the sample data set Subtract the total conditional entropy of the categorical features Get category feature attribute gain, category feature attribute gain The calculation formula is:

[0155] ;

[0156] According to the normal distribution characteristics of continuous features, the mean and standard deviation of continuous features are calculated. According to the characteristics of the standard deviation standardization method that reflects the degree of data deviation from the mean, the standard deviation standardization method is negative and then added with a preset constant to obtain the standard scaling value of the continuous feature. The calculation formula is:

[0157] ;

[0158] in is the original value of the continuous feature, is the standard scaling value, is the mean value of the continuous feature, is the standard deviation of the continuous feature, is a preset constant.

[0159] Furthermore, the working steps of the correlation calculation module include:

[0160] The continuous features are divided into stage features according to the preset classification method, and the attribute gain of the stage feature is calculated. The maximum value of the attribute gain of the stage feature and the attribute gain of the category feature is used as the benchmark feature. The other stage features and category features are used as reference features. A contingency table between the reference features and the benchmark features is established and the chi-square value is calculated. The chi-square value The calculation formula is:

[0161] ;

[0162] in They are represented as datum feature and reference feature respectively. They are respectively represented as the number of values ​​of the benchmark feature and the reference feature, represents the observed value of the target variable under the intersection of two category features, Represents the expected value of the intersection of two category features, the expected value The calculation formula is:

[0163] ;

[0164] in is the benchmark characteristic in the contingency table The number of categories, is the reference feature in the contingency table The number of categories, For the reference feature Take the first The sum of the rows corresponding to the values, For the reference feature Take the first The sum of the corresponding columns when there are values. is the sum of the target variable of the baseline characteristic and the reference characteristic in the contingency table;

[0165] Calculate the contingency table correlation coefficient based on the chi-square value. The calculation formula is:

[0166] ;

[0167] The category features and continuous features Substitute the correlation coefficient formula into the contingency table to obtain and .

[0168] Furthermore, the working steps of the prediction model module include:

[0169] A neural network prediction model is established. According to the influence of categorical features and continuous features on the target variable, the ReLU function is used as the activation function for the categorical feature standard value, and the Sigmoid function is used as the activation function for the continuous feature. The calculation formula of the target function is:

[0170] ;

[0171] in , , is the preset activation function, For the Category features, is the total number of category features, For the A continuous feature, is the total number of continuous features, , is the bias of different hidden layers of the objective function.

[0172] It should be noted that, regarding the system in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0173] Finally, it should be noted that: Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments, or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An artificial intelligence decision optimization method based on big data drive, characterized in that: The method comprises: S1. Obtain a sample data set and divide the sample data set into a training set and a validation set, wherein the sample data set includes attribute features, attribute values, and a target variable, wherein the attribute features include category features and continuous features; The category features include product category, product brand, product color, and product size; the continuous features include product price, product evaluation, and product sales volume; S2. Processing the data according to a preprocessing method, wherein the preprocessing method includes removing the upper threshold of the sample data set; calculating the attribute gain and correlation of the category feature and converting them into the category feature standard value according to a preset conversion method, calculating the standard scaling and correlation of the continuous feature and converting them into the continuous feature standard value according to a preset conversion method; S3, construct a prediction model for the target variable and the standard value of the categorical feature and the standard value of the continuous feature, bring the training set into the prediction model to train the prediction model parameters, and bring the validation set into the prediction model to optimize the prediction model parameters; The target variables include user click rate and purchase conversion rate; S4, dividing the attribute features into certain features and uncertain features, setting a value range for the uncertain features as a constraint condition according to the preset condition classification method and setting the target variable as a fixed target; Input the category features and continuous features of the products selected by the user; S5. Input the constraints and fixed targets into the prediction model, and calculate the optimal value of the uncertain feature according to the Lagrange multiplier method; obtain a new sample data set for the next preset period and optimize the prediction model parameters according to the new sample data set, and recalculate the new optimal value of the uncertain feature by inputting new attribute features to achieve artificial intelligence decision optimization; The prediction model intelligently recommends products that are most in line with user expectations.

2. The big data-driven artificial intelligence decision optimization method according to claim 1 is characterized in that: The method of calculating the attribute gain and correlation of the category feature and converting them into the category feature standard value according to the preset conversion method, and calculating the standard scaling and correlation of the continuous feature and converting them into the continuous feature standard value according to the preset conversion method includes: Calculate the attribute gain and correlation of the category feature, calculate the standard scaling and correlation of the continuous feature; convert the category feature into the category feature standard value according to the first preset ratio, and convert the continuous feature into the continuous feature standard value according to the second preset ratio; convert the category feature standard value The calculation formula is: ； in is the category feature, is the attribute gain of the category feature, is the correlation degree of the category feature, , are the weight coefficients of category feature attribute gain and category feature association, is the adjustment parameter for the standard value of the category feature; Continuous characteristic standard value The calculation formula is: ； in is a continuous feature, is the standard scaling of continuous features, is the correlation degree of continuous features, , are the weight coefficients of the continuous feature standard scaling and the continuous feature association, is the adjustment parameter of the standard value of the continuous feature.

3. The big data driven artificial intelligence decision optimization method according to claim 2 is characterized in that: The method of calculating the attribute gain and correlation of the category feature, calculating the standard scaling and correlation of the continuous feature; converting the category feature into the category feature standard value according to the first preset ratio, and converting the continuous feature into the continuous feature standard value according to the second preset ratio includes: Divide the target variable into target categories according to the preset classification method, obtain the total number of entries in the sample data set and the number of entries corresponding to each target category, and the information entropy of the sample data set The calculation formula is: ； in is a sample data set, is the information entropy of the sample data set, For the target categories, is the total number of target categories, For the The probability of the target category; The total number of entries corresponding to the category feature and the number of different values ​​of the target category under the category feature are counted respectively, and the probability of the number of different values ​​of the category feature accounting for the total number of entries corresponding to the category feature is calculated. The conditional entropy of different values ​​of the category feature is obtained respectively, and the conditional entropy of different values ​​of the category feature is summed to obtain the total conditional entropy of the category feature. The total conditional entropy of the category feature The calculation formula is: ； in is the first Category features, is a category feature The number of categories, The first Value categories, Indicates the value is The number of entries corresponding to the value category, Indicates the total number of entries corresponding to the category feature, Indicates The value category is equal to The probability of The information entropy of the sample data set Subtract the total conditional entropy of the categorical features Get category feature attribute gain, category feature attribute gain The calculation formula is: ； According to the normal distribution characteristics of continuous features, the mean and standard deviation of continuous features are calculated. According to the characteristics of the standard deviation standardization method that reflects the degree of data deviation from the mean, the standard deviation standardization method is negative and then added with a preset constant to obtain the standard scaling value of the continuous feature. The calculation formula is: ； in is the original value of the continuous feature, is the standard scaling value, is the mean value of the continuous feature, is the standard deviation of the continuous feature, is a preset constant.

4. The big data driven artificial intelligence decision optimization method according to claim 2 is characterized in that: The method of calculating the attribute gain and correlation of the category feature, calculating the standard scaling and correlation of the continuous feature; converting the category feature into the category feature standard value according to the first preset ratio, and converting the continuous feature into the continuous feature standard value according to the second preset ratio includes: The continuous features are divided into stage features according to the preset classification method, and the attribute gain of the stage feature is calculated. The maximum value of the attribute gain of the stage feature and the attribute gain of the category feature is used as the benchmark feature. The other stage features and category features are used as reference features. A contingency table between the reference features and the benchmark features is established and the chi-square value is calculated. The chi-square value The calculation formula is: ； in They are represented as datum feature and reference feature respectively. They are respectively represented as the number of values ​​of the benchmark feature and the reference feature, represents the observed value of the target variable under the intersection of two category features, Represents the expected value of the intersection of two category features, the expected value The calculation formula is: ； in is the benchmark characteristic in the contingency table The number of categories, is the reference feature in the contingency table The number of categories, For the reference feature Take the first The sum of the rows corresponding to the values ​​of For the reference feature Take the first The sum of the corresponding columns when there are values. is the sum of the target variable of the baseline characteristic and the reference characteristic in the contingency table; Calculate the contingency table correlation coefficient based on the chi-square value. The calculation formula is: ； The category features and continuous features Substitute the correlation coefficient formula into the contingency table to obtain and .

5. The big data-driven artificial intelligence decision optimization method according to claim 1 is characterized in that: The method for constructing a prediction model of a target variable, a categorical feature standard value, and a continuous feature standard value comprises: A neural network prediction model is established. According to the influence of categorical features and continuous features on the target variable, the ReLU function is used as the activation function for the categorical feature standard value, and the Sigmoid function is used as the activation function for the continuous feature. The calculation formula of the target function is: ； in , , is the preset activation function, For the Category features, is the total number of category features, For the A continuous feature, is the total number of continuous features, , is the bias of different hidden layers of the objective function.

6. An artificial intelligence decision optimization system driven by big data, characterized in that: The system comprises a data collection module, a data processing module, a prediction model module, an attribute feature input module, and a decision output module; the data processing module comprises an attribute gain calculation module, a correlation calculation module, and a standard scaling calculation module; the data collection module is connected to the data processing module, the data processing module is connected to the prediction model module, the prediction model module is connected to the attribute feature input module and the decision output module; the data processing module is connected to the attribute gain calculation module, the correlation calculation module, and the standard scaling calculation module; A data collection module is used to obtain a sample data set and divide the sample data set into a training set and a validation set, wherein the sample data set includes attribute features, attribute values, and target variables, wherein the attribute features include categorical features and continuous features; The category features include product category, product brand, product color, and product size; the continuous features include product price, product evaluation, and product sales volume; A data processing module, used to process the data according to a preprocessing method, wherein the preprocessing method includes removing an upper threshold of a sample data set; Calculate the attribute gain and correlation of the category feature and convert them into the category feature standard value according to the preset conversion method, calculate the standard scaling and correlation of the continuous feature and convert them into the continuous feature standard value according to the preset conversion method; The prediction model module is used to build a prediction model for the target variable and the standard value of the category feature and the standard value of the continuous feature, bring the training set into the prediction model to train the prediction model parameters, and bring the validation set into the prediction model to optimize the prediction model parameters; The target variables include user click rate and purchase conversion rate; An attribute feature input module is used to classify attribute features into certain features and uncertain features, set a value range for the uncertain features as a constraint condition according to a preset condition classification method, and set the target variable as a fixed target; Used to input the category features and continuous features of the products selected by the user; A decision output module is used to input the constraints and fixed targets into the prediction model, and calculate the optimal value of the uncertain feature according to the Lagrange multiplier method; obtain a new sample data set for the next preset period and optimize the prediction model parameters according to the new sample data set, and recalculate the new optimal value of the uncertain feature by inputting new attribute features to achieve artificial intelligence decision optimization; The prediction model is used to intelligently recommend products that are closest to user expectations.

7. The big data driven artificial intelligence decision optimization system according to claim 6 is characterized in that: The working steps of the data processing module include: Calculate the attribute gain and correlation of the category feature, calculate the standard scaling and correlation of the continuous feature; convert the category feature into the category feature standard value according to the first preset ratio, and convert the continuous feature into the continuous feature standard value according to the second preset ratio; convert the category feature standard value The calculation formula is: ； in is the category feature, is the attribute gain of the category feature, is the correlation degree of the category feature, , are the weight coefficients of category feature attribute gain and category feature association, is the adjustment parameter for the standard value of the category feature; Continuous characteristic standard value The calculation formula is: ； in is a continuous feature, is the standard scaling of continuous features, is the correlation degree of continuous features, , are the weight coefficients of the continuous feature standard scaling and the continuous feature association, is the adjustment parameter of the standard value of the continuous feature.

8. The big data driven artificial intelligence decision optimization system according to claim 7 is characterized in that: The working steps of the attribute gain calculation module and the standard scaling calculation module include: Divide the target variable into target categories according to the preset classification method, obtain the total number of entries in the sample data set and the number of entries corresponding to each target category, and the information entropy of the sample data set The calculation formula is: ； in is a sample data set, is the information entropy of the sample data set, For the target categories, is the total number of target categories, For the The probability of the target category; The total number of entries corresponding to the category feature and the number of different values ​​of the target category under the category feature are counted respectively, and the probability of the number of different values ​​of the category feature accounting for the total number of entries corresponding to the category feature is calculated. The conditional entropy of different values ​​of the category feature is obtained respectively, and the conditional entropy of different values ​​of the category feature is summed to obtain the total conditional entropy of the category feature. The total conditional entropy of the category feature The calculation formula is: ； in is the first Category features, is a category feature The number of categories, The first Value categories, Indicates the value is The number of entries corresponding to the value category, Indicates the total number of entries corresponding to the category feature, Indicates The value category is equal to The probability of The information entropy of the sample data set Subtract the total conditional entropy of the categorical features Get category feature attribute gain, category feature attribute gain The calculation formula is: ； According to the normal distribution characteristics of continuous features, the mean and standard deviation of continuous features are calculated. According to the characteristics of the standard deviation standardization method that reflects the degree of data deviation from the mean, the standard deviation standardization method is negative and then added with a preset constant to obtain the standard scaling value of the continuous feature. The calculation formula is: ； in is the original value of the continuous feature, is the standard scaling value, is the mean value of the continuous feature, is the standard deviation of the continuous feature, is a preset constant.

9. The big data driven artificial intelligence decision optimization system according to claim 7 is characterized in that: The working steps of the association calculation module include: The continuous features are divided into stage features according to the preset classification method, and the attribute gain of the stage feature is calculated. The maximum value of the attribute gain of the stage feature and the attribute gain of the category feature is used as the benchmark feature. The other stage features and category features are used as reference features. A contingency table between the reference features and the benchmark features is established and the chi-square value is calculated. The chi-square value The calculation formula is: ； in They are represented as datum feature and reference feature respectively. They are respectively represented as the number of values ​​of the benchmark feature and the reference feature, represents the observed value of the target variable under the intersection of two category features, Represents the expected value of the intersection of two category features, the expected value The calculation formula is: ； in is the benchmark characteristic in the contingency table The number of categories, is the reference feature in the contingency table The number of categories, For the reference feature Take the first The sum of the rows corresponding to the values ​​of For the reference feature Take the first The sum of the corresponding columns when there are values. is the sum of the target variable of the baseline characteristic and the reference characteristic in the contingency table; Calculate the contingency table correlation coefficient based on the chi-square value. The calculation formula is: ； The category features and continuous features Substitute the correlation coefficient formula into the contingency table to obtain and .

10. The big data driven artificial intelligence decision optimization system according to claim 6, characterized in that: The working steps of the prediction model module include: A neural network prediction model is established. According to the influence of categorical features and continuous features on the target variable, the ReLU function is used as the activation function for the categorical feature standard value, and the Sigmoid function is used as the activation function for the continuous feature. The calculation formula of the target function is: ； in , , is the preset activation function, For the Category features, is the total number of category features, For the A continuous feature, is the total number of continuous features, , is the bias of different hidden layers of the objective function.

Images