Data processing method and device based on artificial intelligence, terminal and storage medium

Abstract

The invention relates to the technical field of artificial intelligence, and provides a data processing method and device based on artificial intelligence, a terminal and a storage medium, and the method comprises the steps: segmenting an important data set selected from an original data set through employing an XGBoost model into a plurality of branch data sets; training and testing a lightGBM model by using each sub-data set to obtain a test passing rate; pre-standardizing the training data and the test data corresponding to the first field in the sub-data set to obtain a new sub-data set; training and testing the lightGBM model by using the new sub-data set to obtain a test passing rate; judging whether the data corresponding to the first field needs to be standardized or not accordingto the passing rate of the two tests; and repeatedly executing the process until whether the data corresponding to the last field in the important data set needs to be standardized or not is judged, and updating the important data set according to all judgment results to obtain a target data set. According to the method, the data set with relatively high stability and relatively high contributiondegree to the prediction model can be selected.

Description

technical field [0001] The present invention relates to the technical field of artificial intelligence, in particular to an artificial intelligence-based data processing method, device, terminal and storage medium. Background technique [0002] In the agent retention / low performance prediction model, some characteristics are affected by seasonality (such as the number and quality of recruits in February) and business aspects (such as the forced promotion of APP, the quality of newcomers in assessment months and non-assessment months), there are The phenomenon of instability across time, that is, the distribution of features (mean, standard deviation, etc.) changes over time, which brings difficulties to model training and prediction across time. [0003] In addition, since the training of the agent retention / low performance prediction model is to train the data of the first 3 to 6 months of the forecast month as a whole training set (for example, the training set of the rete...

AI Technical Summary

Problems solved by technology

However, the quality of the recruited population in each month may be slightly different, which leads to the finding of the optimal split point may not be stable in multiple months of training; the quality and feature distribution of the recruited population in the forecast month and training month are also discrepancies, which creates a potential hazard for the stability of the model's predictions across time

Images