Automatic data exploration method and system based on contribution degree of target label

Abstract

The invention discloses an automatic data exploration method and system based on the contribution degree of a target label, and the method comprises the steps: carrying out the correlation integration of feature data and the target label, and obtaining a data set; inputting the data set into a random forest and cross validation model, an information gain rate algorithm and a random forest algorithm, and determining a sorting table of contribution degrees of each feature field in the feature data relative to a target label; processing the three sort tables in a voting mode to obtain a feature contribution degree sort table; obtaining a preset number of feature fields ranked in the front in the feature contribution degree ranking table, performing correlation check on every two feature fields, and removing the feature fields with correlation exceeding a preset threshold value; and carrying out data distribution analysis according to basic data types of the fields for feature fields left after the relevance test is eliminated. According to the technical scheme, automatic screening of the feature fields is achieved, the feature screening efficiency is greatly improved, and tedious and repeated manual work is omitted.

Description

technical field [0001] The invention relates to the technical field of data mining, in particular to an automatic data exploration method based on target label contribution degree and an automatic data exploration system based on target label contribution degree. Background technique [0002] At present, feature engineering is a very important link in data mining, and it is also the link with the largest workload. It is necessary to select features that contribute to the target label, and at the same time, data exploration and analysis are required to provide input for model construction and model interpretability. Each modeling requires manual processing, including manual writing of SQL statements for statistical analysis, and feature exploration through modeling methods. There are a lot of tedious and repetitive tasks, and the workload is large and the work efficiency is low. SUMMARY OF THE INVENTION [0003] In view of the above problems, the present invention provides ...

AI Technical Summary

Problems solved by technology

Each modeling process requires manual processing, including manually writing SQL statements for statistical analysis and feature exploration through modeling methods. There are a lot of tedious and repetitive tasks, heavy workload, and low work efficiency.

Images