Automatic data exploration method and system based on target label contribution degree

Abstract

The application discloses an automatic data exploration method and system based on target label contribution degree, and the method comprises the following steps: associating and integrating feature data with a target label to obtain a data set; inputting the data set into a random forest and a cross-validation model, an information gain rate algorithm and a random forest algorithm respectively to determine a sorting table of the contribution degrees of respective feature fields in the feature data relative to the target label; processing the three sorting tables in a voting manner to obtain a feature contribution degree sorting table; obtaining a preset number of feature fields with high ranking in the feature contribution degree sorting table, performing correlation test on the feature fields two by two, and removing feature fields with correlation exceeding a preset threshold; and performing data distribution analysis on the feature fields remaining after the correlation test according to the basic data types of the fields. Through the technical scheme, automatic screening of the feature fields is realized, and the efficiency of feature screening is greatly improved, and tedious and repetitive manual work is avoided.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data mining, and in particular to an automatic data exploration method based on target label contribution degree and an automatic data exploration system based on target label contribution degree. BACKGROUND

[0002] At present, feature engineering is a very important link in data mining and also the link with the largest workload, which needs to select features with high contribution to target labels and needs to perform data exploration analysis to provide input for model construction and model interpretability. Manual processing is required in each modeling, including manual preparation of SQL statements for statistical analysis and feature exploration through modeling methods, which involves a large amount of tedious and repetitive work, has a large workload and is low in work efficiency. SUMMARY

[0003] In view of the above problems, the present application provides an automatic data exploration method and system based on target label contribution degree, calculates the contribution degrees of feature data in a data set with respect to a target field through a machine learning model, removes feature fields with high correlation from feature fields with high contribution degrees, and performs data distribution analysis on the remaining feature fields, so as to realize automatic selection of feature fields and greatly improve the efficiency of feature selection, thereby eliminating tedious and repetitive manual work.

[0004] To achieve the above object, the present application provides an automatic data exploration method based on target label contribution degree, comprising:

[0005] Integrating the feature data and the target label to obtain a data set;

[0006] Inputting the data set into a random forest and a cross-validation model to determine a first sorting table of contribution degrees of each feature field in the feature data with respect to the target label;

[0007] Calculating a second sorting table of contribution degrees of each feature field in the feature data with respect to the target label through an information gain rate algorithm;

[0008] Calculating a third sorting table of contribution degrees of each feature field in the feature data with respect to the target label through a random forest algorithm;

[0009] Processing the first sorting table, the second sorting table and the third sorting table in a voting manner to obtain a feature contribution degree sorting table;

[0010] Obtaining a preset number of feature fields with high ranking in the feature contribution degree sorting table, performing correlation test on the feature fields two by two, and removing feature fields with correlation exceeding a preset threshold;

[0011] For the remaining feature fields after the correlation test is eliminated, data distribution analysis is performed according to the basic data type of the field.

[0012] In the technical solution, preferably, the automatic data exploration method based on the contribution degree of the target label further comprises: presenting a data feature distribution graph in a visual manner according to the data distribution analysis result of the feature field.

[0013] In the technical solution, preferably, the automatic data exploration method based on the contribution degree of the target label further comprises:

[0014] According to the preset legitimacy rule, the data set and the target label therein are subjected to legitimacy judgment;

[0015] For the data set determined to have legitimacy, the random forest and cross-validation model, the information gain rate algorithm and the random forest algorithm are respectively constructed, and the contribution degree of the feature field relative to the target label is calculated;

[0016] If it is determined that the data set does not have legitimacy, a warning information is issued and the current program is exited.

[0017] In the technical solution, preferably, the data distribution analysis according to the basic data type of the field specifically comprises:

[0018] For the numerical field, data distribution analysis is performed according to the chi-square binning method;

[0019] For the enumeration type field, data distribution analysis is directly performed.

[0020] The application also proposes an automatic data exploration system based on the contribution degree of the target label, which applies the automatic data exploration method based on the contribution degree of the target label disclosed in any one of the above technical solutions, and comprises:

[0021] A data integration module is configured to associate and integrate the feature data with the target label to obtain a data set;

[0022] A first sorting module is configured to input the data set into the random forest and cross-validation model to determine a first sorting table of the contribution degree of each feature field in the feature data relative to the target label;

[0023] A second sorting module is configured to calculate a second sorting table of the contribution degree of each feature field in the feature data relative to the target label through the information gain rate algorithm;

[0024] A third sorting module is configured to calculate a third sorting table of the contribution degree of each feature field in the feature data relative to the target label through the random forest algorithm;

[0025] A voting sorting module is configured to process the first sorting table, the second sorting table and the third sorting table in a voting manner to obtain a feature contribution degree sorting table.

[0026] A correlation verification module is configured to obtain a preset number of feature fields with a high ranking in the feature contribution degree sorting table, perform correlation verification on the feature fields in pairs, and eliminate feature fields with a correlation exceeding a preset threshold.

[0027] A data analysis module is configured to perform data distribution analysis on the feature fields remaining after the correlation verification.

[0028] In the technical solution described above, preferably, the automated data exploration system based on target label contribution degree further comprises a visual analysis module configured to present a data feature distribution graph in a visual manner based on the data distribution analysis result of the feature fields.

[0029] In the technical solution described above, preferably, the automated data exploration system based on target label contribution degree further comprises a legality judgment module configured to perform legality judgment on the data set and the target label in the data set according to a preset legality rule.

[0030] If the data set is determined to be legal, the random forest and the cross-validation model, the information gain rate algorithm and the random forest algorithm are constructed respectively to calculate the contribution degree of the feature fields relative to the target label.

[0031] If the data set is determined to be illegal, a warning information is issued and the current program is exited.

[0032] In the technical solution described above, preferably, in the data distribution analysis process performed by the data analysis module based on the basic data type of the fields:

[0033] For numerical fields, the data distribution analysis is performed according to the chi-square binning method.

[0034] For enumeration fields, the data distribution analysis is directly performed.

[0035] Compared with the prior art, the present application has the following beneficial effects: the contribution degree of the feature data in the data set relative to the target field is calculated by using a machine learning model, the feature fields with a high correlation and a high contribution degree are eliminated, and data distribution analysis is performed on the remaining feature fields, thereby realizing automated screening of the feature fields, greatly improving the efficiency of feature screening, and eliminating tedious and repetitive manual work. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1A flowchart of an automatic data exploration method based on target label contribution degree according to an embodiment of the present application is disclosed.

[0037] Figure 2 A module diagram of an automatic data exploration system based on target label contribution degree according to an embodiment of the present application is disclosed.

[0038] In the figure, the correspondence between each component and the reference numeral is as follows:

[0039] 11. Data integration module, 12. First sorting module, 13. Second sorting module, 14. Third sorting module, 15. Voting sorting module, 16. Correlation test module, 17. Data analysis module. DETAILED DESCRIPTION

[0040] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0041] The present application will be further described in detail below with reference to the drawings:

[0042] As shown in the figure, according to the automatic data exploration method based on target label contribution degree provided by the present application, the method comprises the following steps: Figure 1

[0043] Integrating and associating the feature data with the target label to obtain a data set;

[0044] Inputting the data set into a random forest and a cross-validation model to determine a first sorting table of the contribution degree of each feature field in the feature data with respect to the target label;

[0045] Calculating a second sorting table of the contribution degree of each feature field in the feature data with respect to the target label by an information gain rate algorithm;

[0046] Calculating a third sorting table of the contribution degree of each feature field in the feature data with respect to the target label by a random forest algorithm;

[0047] Processing the first sorting table, the second sorting table and the third sorting table in a voting manner to obtain a feature contribution degree sorting table;

[0048] Obtaining a preset number of feature fields with high ranking in the feature contribution degree sorting table, performing correlation test on each two of the feature fields, and eliminating the feature fields with correlation exceeding a preset threshold; ​

[0049] For the remaining feature fields after the correlation test is eliminated, data distribution analysis is performed according to the basic data type of the field.

[0050] In this embodiment, the contribution of the feature data in the data set to the target field is calculated by the machine learning model, and the feature fields with higher contribution are eliminated, and data distribution analysis is performed on the remaining feature fields, thereby realizing automatic screening of the feature fields, greatly improving the efficiency of feature screening, and eliminating tedious and repetitive manual work.

[0051] In the above embodiment, preferably, the automatic data exploration method based on the contribution of the target label further comprises: presenting the data feature distribution graph in a visual manner according to the data distribution analysis result of the feature field.

[0052] In the above embodiment, preferably, the automatic data exploration method based on the contribution of the target label further comprises:

[0053] According to the preset legitimacy rule, the legitimacy of the data set and the target label therein is judged;

[0054] For the data set determined to have legitimacy, a random forest and a cross-validation model, an information gain rate algorithm and a random forest algorithm are constructed respectively, and the contribution of the feature field to the target label is calculated;

[0055] If it is determined that the data set does not have legitimacy, a warning information is issued and the current program is exited.

[0056] In the above embodiment, preferably, the data distribution analysis according to the basic data type of the field specifically comprises:

[0057] For numerical fields, data distribution analysis is performed according to the chi-square binning method;

[0058] For enumeration fields, data distribution analysis is directly performed.

[0059] Specifically, according to the automatic data exploration method based on the contribution of the target label disclosed in the above embodiment, in the implementation process, the method is further specifically described by the following specific steps of the embodiment:

[0060] Step 1: Making a data set: associating and integrating the target label with the feature data;

[0061] Step 2: Obtaining a sorting table 1 of the contribution of each feature field to the target label by random forest and cross-validation;

[0062] Step 3: Calculating a sorting table 2 of the contribution of each feature field to the target label by the information gain rate algorithm;

[0063] Step 4: obtain the ranking table 3 of the contribution degree of each feature field with respect to the target label through the random forest algorithm;

[0064] Step 5: process the ranking tables 1, 2 and 3 in a voting manner to obtain a feature contribution degree ranking table;

[0065] Step 6: take the top 25 feature fields, and then perform correlation test on each two, and only keep one for the high correlation features;

[0066] Step 7: perform automatic data distribution exploration, and perform automatic data distribution analysis on the numerical fields according to the chi-square binning method; and directly perform data distribution analysis on the enumeration type;

[0067] Step 8: present the data feature distribution graph in a visual manner.

[0068] As shown in Figure 2 The application further provides an automatic data exploration system based on target label contribution degree, which applies the automatic data exploration method based on target label contribution degree disclosed in any one of the above embodiments, and includes:

[0069] A data integration module 11 is configured to associate and integrate the feature data with the target label to obtain a data set;

[0070] A first ranking module 12 is configured to input the data set into a random forest and a cross-validation model to determine a first ranking table of the contribution degree of each feature field in the feature data with respect to the target label;

[0071] A second ranking module 13 is configured to calculate a second ranking table of the contribution degree of each feature field in the feature data with respect to the target label through an information gain rate algorithm;

[0072] A third ranking module 14 is configured to calculate a third ranking table of the contribution degree of each feature field in the feature data with respect to the target label through a random forest algorithm;

[0073] A voting ranking module 15 is configured to process the first ranking table, the second ranking table and the third ranking table in a voting manner to obtain a feature contribution degree ranking table;

[0074] A correlation test module 16 is configured to obtain a preset number of feature fields ranked at the top in the feature contribution degree ranking table, perform correlation test on each two, and eliminate the feature fields with a correlation exceeding a preset threshold;

[0075] A data analysis module 17 is configured to perform data distribution analysis on the feature fields remaining after the correlation test according to the basic data type of the fields.

[0076] In the above embodiment, preferably, the automated data exploration system based on the contribution degree of the target label further comprises a visual analysis module 18 configured to analyze the data distribution of the feature field and present a data feature distribution graph in a visual manner.

[0077] In the above embodiment, preferably, the automated data exploration system based on the contribution degree of the target label further comprises a legality judgment module 19 configured to perform legality judgment on the data set and the target label therein according to a preset legality rule.

[0078] If the data set is determined to be legal, a random forest and a cross-validation model, an information gain ratio algorithm and a random forest algorithm are constructed respectively to calculate the contribution degree of the feature field relative to the target label;

[0079] If the data set is determined to be illegal, a warning information is sent and the current program is exited.

[0080] In the above embodiment, preferably, in the data distribution analysis process according to the basic data type of the field, the data analysis module 17 is configured to:

[0081] For the numerical field, the data distribution analysis is performed according to the chi-square binning method;

[0082] For the enumeration field, the data distribution analysis is directly performed.

[0083] The functions to be achieved by each module of the automated data exploration system based on the contribution degree of the target label disclosed in the above embodiment correspond to the respective steps in the automated data exploration method based on the contribution degree of the target label disclosed in the above embodiment, and thus will not be described herein.

[0084] The above only represents the preferred embodiments of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. An automatic data exploration method based on target label contribution degree, characterized in that, The method comprises the following steps: correlating and integrating the feature data with the target label to obtain a data set; inputting the data set into a random forest and cross-validation model to determine a first ranking table of the contribution degrees of each feature field in the feature data to the target label; calculating a second ranking table of the contribution degrees of each feature field in the feature data to the target label by an information gain rate algorithm; calculating a third ranking table of the contribution degrees of each feature field in the feature data to the target label by a random forest algorithm; processing the first ranking table, the second ranking table and the third ranking table by a voting method to obtain a feature contribution degree ranking table; obtaining a preset number of feature fields with high ranking in the feature contribution degree ranking table, performing correlation test on the feature fields in pairs, and removing the feature fields with correlation exceeding a preset threshold; performing data distribution analysis on the remaining feature fields after the correlation test according to the basic data types of the fields; presenting a data feature distribution graph in a visual manner according to the data distribution analysis results of the feature fields; performing legality judgment on the data set and the target label therein according to a preset legality rule; for the data set determined to be legal, constructing the random forest and cross-validation model, the information gain rate algorithm and the random forest algorithm to calculate the contribution degrees of the feature fields to the target label; if the data set is determined to be illegal, issuing a warning information and exiting the current program. 2.The target label contribution based automated data exploration method of claim 1, wherein, The data distribution analysis according to the basic data types of the fields specifically comprises: for numerical fields, performing data distribution analysis according to a chi-square binning method; for enumeration fields, directly performing data distribution analysis.

3. An automated data exploration system based on target label contribution, characterized in that, The method comprises the following steps: a data integration module for correlating and integrating feature data with a target label to obtain a data set; a first ranking module for inputting the data set into a random forest and cross-validation model to determine a first ranking table of the contribution degrees of each feature field in the feature data to the target label; a second ranking module for calculating a second ranking table of the contribution degrees of each feature field in the feature data to the target label by an information gain rate algorithm; a third ranking module for calculating a third ranking table of the contribution degrees of each feature field in the feature data to the target label by a random forest algorithm; a voting ranking module for processing the first ranking table, the second ranking table and the third ranking table by a voting method to obtain a feature contribution degree ranking table; a correlation test module for obtaining a preset number of feature fields with high ranking in the feature contribution degree ranking table, performing correlation test on the feature fields in pairs, and removing the feature fields with correlation exceeding a preset threshold; a data analysis module for performing data distribution analysis on the remaining feature fields after the correlation test according to the basic data types of the fields. 4.The target label contribution based automated data exploration system of claim 3, wherein, A visual analysis module is further included for presenting a data feature distribution graph in a visual manner according to the data distribution analysis results of the feature fields. 5.The target label contribution based automated data exploration system of claim 4, wherein, Also include a legal judgment module, the legal judgment module according to the preset legality rules, the data set and the target label therein are judged for legality; If the data set is determined to have legality, the random forest and cross-validation model, the information gain ratio algorithm and the random forest algorithm are respectively constructed, and the contribution degree of the feature field Relative to the target label is calculated; If the data set is determined to be illegal, a warning message is issued and the current program is exited. 6.The target tag contribution-based automated data exploration system of claim 4, wherein, During the data distribution analysis process according to the basic data type of the field: For numerical fields, data distribution analysis is performed according to the chi-square binning method; For enumeration type fields, directly perform data distribution analysis.

Images