Data preprocessing method and device

By performing feature classification and abnormal detection of data, detailed data exploration and abnormal detection reports are generated, and the problems of incomplete data preprocessing and insufficient automation in the existing technology are solved, and the efficiency and accuracy of data preprocessing are achieved. Data projects in multiple fields.

CN113515577BActive Publication Date: 2024-11-15INDUSTRIAL AND COMMERCIAL BANK OF CHINA
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110789650.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-07-13
Publication Date
2024-11-15
Estimated Expiration
2041-07-13

AI Technical Summary

Technical Problem

In the prior art, the data preprocessing process is not comprehensive and lacks automation, resulting in incomplete data cleaning, poor model accuracy and stability, and a lack of universal data preprocessing framework, especially in data exploration and anomaly detection in different fields. Unified technical solutions.

Method used

Provide a data preprocessing method and device, by performing feature classification, exploration and abnormal detection of sample data, generate data exploration and abnormal detection reports, including feature distribution, null value rate, correlation and abnormal sample recognition, and use clustering algorithms Automate output reports to simplify operations and improve the efficiency and accuracy of data preprocessing.

Benefits of technology

It realizes the automation and standardization of data preprocessing, improves the efficiency and accuracy of data preprocessing, provides a universal solution, is suitable for different structured data projects, and can be used as a basis for third-party acceptance testing.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113515577B_ABST
    Figure CN113515577B_ABST
Patent Text Reader

Abstract

The embodiment of the present application provides a data preprocessing method and device, which can also be used in the financial field. The method includes: performing feature classification on the collected sample data, and determining the number of feature types and the overall feature missing rate of the sample data; performing feature exploration and anomaly detection on the sample data of each feature category in the sample data, and determining the feature distribution, feature null value rate, feature correlation and abnormal samples of the sample data of each feature category; generating a data exploration summary, a data exploration result report and anomaly detection result report of the sample data according to the number of feature types and the overall feature missing rate of the sample data and the feature distribution, feature null value rate, feature correlation and sample clustering results of the sample data of each feature category; the present application can effectively improve the efficiency and accuracy of data preprocessing.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of data processing and can also be used in the financial field, and specifically relates to a data preprocessing method and device. Background Art

[0002] Data preprocessing refers to the process of cleaning data based on data exploration and anomaly retrieval results. Data preprocessing is a prerequisite for data analysis and data modeling. Fig.10 Schematic diagram of the flow chart of the data preprocessing method in the prior art.

[0003] A comprehensive and standardized data preprocessing process is very necessary. On the one hand, when analyzing data, a comprehensive exploration of features and samples can fully utilize the value of data to avoid weak conclusions and waste of information. On the other hand, when modeling data, the data determines the upper limit of the model. Large skewness, severely asymmetric distribution features and abnormal samples that are out of the group will interfere with the fitting of the model and affect its accuracy, operating efficiency and stability.

[0004] However, in actual projects, the data preprocessing process is often incomplete and lacks verification. Data exploration and anomaly detection are all conducted by analysts based on experience, which is completely dependent on the analyst's knowledge system and subjective initiative. There are projects that only preprocess data for some features, or completely skip the preprocessing process and directly analyze and model. In addition, there is a lack of supervision mechanism, and there are often too much dirty data, data that deviates from reality, single-dimensional data analysis reports, and poor interpretability / stability of models, which greatly reduces the effectiveness of analysis and modeling.

[0005] The market lacks an automated universal preprocessing framework, and the published patents are mostly concentrated on specific preprocessing dimensions in limited fields, such as patents for abnormal sample detection only in the advertising field. Considering that data cleaning and conversion are closely related to actual business, the content of data exploration and anomaly detection in various fields is basically the same - all of them are feature distribution exploration, missing value statistics, outlier judgment, abnormal sample detection, etc. Therefore, there is a lack of technical solutions focusing on data exploration and anomaly detection in the data preprocessing stage. Summary of the invention

[0006] In order to solve the problems in the prior art, the present application provides a data preprocessing method and device, which can effectively improve the efficiency and accuracy of data preprocessing.

[0007] In order to solve at least one of the above problems, the present application provides the following technical solutions:

[0008] In a first aspect, the present application provides a data preprocessing method, comprising:

[0009] Performing feature classification on the collected sample data, and determining the number of feature types and the overall feature missing rate of the sample data;

[0010] Performing feature exploration and anomaly detection on sample data of each feature category in the sample data to determine feature distribution, feature null value rate, feature correlation and abnormal samples of the sample data of each feature category;

[0011] Based on the number of feature types of the sample data and the overall missing rate of the features, as well as at least one of the feature distribution, feature null value rate, feature correlation, and sample clustering results of the sample data of each feature category, a data exploration summary, a data exploration result report, and anomaly detection result report of the sample data are generated.

[0012] Furthermore, generating a data exploration summary and a data exploration result report of the sample data according to the number of feature types of the sample data and the overall feature missing rate and at least one of the feature distribution, feature null value rate, feature correlation, and sample clustering results of the sample data of each feature category includes:

[0013] Determine sample data overview information in the data exploration summary according to the number of feature types of the sample data;

[0014] Perform variable traversal output on the sample data exploration results whose feature categories are continuous single variable features, discrete features, and date features, and determine the feature distribution information and feature null value rate information in the data exploration result report;

[0015] Calculate the pairwise correlation coefficients of sample data whose feature categories are continuous multivariate features, and determine the feature correlation information in the data exploration result report.

[0016] Furthermore, generating an anomaly detection result report of the sample data according to the number of feature types of the sample data and the overall missing rate of the features and at least one of the feature distribution, feature null value rate, feature correlation, and sample clustering results of the sample data of each feature category includes:

[0017] Determine overall missing statistics in the anomaly detection result report based on each feature whose missing ratio is greater than a threshold value and the corresponding missing value in the overall missing rate of features of the sample data;

[0018] Perform feature exploration and determine abnormal features, and determine an abnormal feature list in the abnormal detection result report based on the abnormal features;

[0019] Determine low-data-volume abnormal sample data in the sample data according to a preset clustering algorithm, and determine an abnormal sample list in an abnormal detection result report according to the low-data-volume abnormal sample data.

[0020] Furthermore, the determining of low data volume abnormal sample data in the sample data according to a preset clustering algorithm further includes:

[0021] The corresponding clustering algorithm is determined based on whether the data volume of the sample data exceeds the preset data volume threshold.

[0022] Furthermore, the sample data exploration results whose feature categories are continuous single variable features are output through variable traversal to determine feature distribution information and feature null value rate information in the data exploration result report, including:

[0023] The sample data exploration results with the feature category of continuous single variable features are output by variable traversal to determine the number of null values, null value rate, number of negative values, number of 0 values, number of positive values, kurtosis, skewness, mean value, standard deviation, quartiles, maximum value, minimum value, variable frequency distribution histogram and kernel density curve in the data exploration result report.

[0024] Furthermore, the variable traversal output of the sample data exploration results whose feature categories are discrete features, and determining the feature distribution information and feature null value rate information in the data exploration result report, include:

[0025] Perform variable traversal output on the sample data exploration results whose feature categories are discrete features, and determine the number of categories, number of null values, null value rate, category with the largest frequency and its frequency proportion, and category frequency distribution bar chart in the data exploration result report.

[0026] Furthermore, the variable traversal output of the sample data exploration results whose feature category is date-type features, and determining the feature distribution information and feature null value rate information in the data exploration result report, include:

[0027] Perform variable traversal output on the sample data exploration results whose feature category is date-type features, determine the number of null values, null value rate, first time, last time, quartile date in the data exploration result report, and output frequency distribution by date distribution.

[0028] In a second aspect, the present application provides a data preprocessing device, comprising:

[0029] A feature pre-classification module is used to perform feature classification on the collected sample data and determine the number of feature types and the overall feature missing rate of the sample data;

[0030] A feature exploration and anomaly detection module, used to perform feature exploration and anomaly detection on the sample data of each feature category in the sample data, and determine the feature distribution, feature null value rate, feature correlation and abnormal samples of the sample data of each feature category;

[0031] A report generation module is used to generate a data exploration summary, a data exploration result report and anomaly detection result report of the sample data based on the number of feature types of the sample data and the overall missing rate of the features, as well as at least one of the feature distribution, feature null value rate, feature correlation and sample clustering results of the sample data of each feature category.

[0032] Furthermore, the report generation module includes:

[0033] A data summary processing unit, used to determine sample data summary information in the data exploration summary according to the number of feature types of the sample data;

[0034] The data detailed exploration unit is used to perform variable traversal output on the sample data exploration results whose feature categories are continuous single variable features, discrete features, and date features, and determine the feature distribution information and feature null value rate information in the data exploration result report;

[0035] The correlation determination unit is used to calculate the pairwise correlation coefficients of sample data whose feature categories are continuous multivariate features, and determine the feature correlation information in the data exploration result report.

[0036] Furthermore, the report generation module includes:

[0037] An anomaly detection unit is used to determine overall missing statistics in an anomaly detection result report according to each feature whose missing ratio is greater than a threshold value in the overall missing rate of features of the sample data and the corresponding missing values;

[0038] A feature anomaly detection unit, used to perform feature exploration and determine abnormal features, and determine an abnormal feature list in the abnormal detection result report according to the abnormal features;

[0039] The sample anomaly detection unit is used to determine low-data-volume abnormal sample data in the sample data according to a preset clustering algorithm, and determine an abnormal sample list in an anomaly detection result report according to the low-data-volume abnormal sample data.

[0040] Furthermore, the report generation module also includes:

[0041] The clustering algorithm selection unit is used to determine the corresponding clustering algorithm according to whether the data volume of the sample data exceeds a preset data volume threshold.

[0042] In a third aspect, the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the data preprocessing method when executing the program.

[0043] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the data preprocessing method when executed by a processor.

[0044] It can be seen from the above technical solution that the present application provides a data preprocessing method and device, which standardizes the data preprocessing process and simplifies the amount of data preprocessing operations by automatically outputting data exploration and anomaly detection reports. It can also be used as a basis for third-party acceptance testing and is universal for different structured data projects, thereby effectively improving the efficiency and accuracy of data preprocessing. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0046] Figure 1 This is one of the flow charts of the data preprocessing method in the embodiment of the present application;

[0047] Figure 2 This is a second flow chart of the data preprocessing method in an embodiment of the present application;

[0048] Figure 3 This is a third flow chart of the data preprocessing method in an embodiment of the present application;

[0049] Figure 4 This is one of the structural diagrams of the data preprocessing device in the embodiment of the present application;

[0050] Figure 5 This is the second structural diagram of the data preprocessing device in the embodiment of the present application;

[0051] Figure 6 This is the third structural diagram of the data preprocessing device in the embodiment of the present application;

[0052] Figure 7 This is the fourth structural diagram of the data preprocessing device in the embodiment of the present application;

[0053] Figure 8 This is an overall flow chart of a data preprocessing method in a specific embodiment of the present application;

[0054] Fig. 9 A schematic diagram of clustering algorithm selection in a specific embodiment of the present application;

[0055] Fig.10This is a flow chart of a data preprocessing method in the prior art of this application;

[0056] Fig.11 A schematic diagram of the data preprocessing method flow chart for comparing the present application with the prior art;

[0057] Fig.12 It is a schematic diagram of the structure of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0059] Considering that data cleaning and conversion in the prior art are closely related to actual business, the contents of data exploration and anomaly detection in various fields are basically the same - they all perform feature distribution exploration, missing value statistics, outlier judgment, abnormal sample detection, etc. Therefore, there is a lack of technical solutions focusing on data exploration and anomaly detection in the data preprocessing stage. This application provides a data preprocessing method and device, see Fig.11 By automatically outputting data exploration and anomaly detection reports, it standardizes the data preprocessing process and simplifies the amount of data preprocessing operations. It can also serve as a basis for third-party acceptance testing and is universal for different structured data projects, thereby effectively improving the efficiency and accuracy of data preprocessing.

[0060] In order to effectively improve the efficiency and accuracy of data preprocessing, the present application provides an embodiment of a data preprocessing method, see Figure 1 , the data preprocessing method specifically includes the following contents:

[0061] Step S101: feature classifying the collected sample data, and determining the number of feature types and the overall feature missing rate of the sample data.

[0062] Optionally, the present application may first pre-classify the collected sample data according to different data feature types, and divide the features into primary key features, label features, continuous features, discrete features, and date features, and then perform an overview analysis to determine the number of feature types and the overall feature missing rate (also known as the null value rate) of the sample data, that is, the overview analysis mainly counts the overall data volume, the number of features of each pre-classified subcategory, and the overall feature missing situation.

[0063] Step S102: performing feature exploration and anomaly detection on the sample data of each feature category in the sample data, and determining the feature distribution, feature null value rate, feature correlation and abnormal samples of the sample data of each feature category.

[0064] Optionally, after completing the above overview analysis, the present application may further perform detailed analysis on the sample data of each feature category in the sample data, namely feature exploration and anomaly detection, wherein the detailed analysis is carried out from two dimensions: feature and sample:

[0065] (1) Feature mining and anomaly detection are carried out according to the pre-classified subcategories in the feature dimension, mainly focusing on three aspects: feature distribution, feature null value rate and feature correlation.

[0066] (2) The sample dimension mainly performs anomaly detection and identifies abnormal samples through machine learning algorithms. The solution ultimately outputs data exploration and anomaly detection reports, reflecting the automated data preprocessing process.

[0067] Thereby, the characteristic distribution, characteristic null value rate, characteristic correlation and abnormal samples of the sample data of each characteristic category can be determined. Preferably, the characteristic correlation can be the correlation of continuous variables.

[0068] Step S103: Generate a data exploration summary, a data exploration result report and anomaly detection result report of the sample data according to the number of feature types of the sample data and the overall missing rate of the features, as well as at least one of the feature distribution, feature null value rate, feature correlation and sample clustering results of the sample data of each feature category.

[0069] Optionally, based on the number of feature types of the sample data and the overall missing rate of the features, as well as at least one of the feature distribution, feature null value rate, feature correlation, and sample clustering results of the sample data of each feature category, the present application can generate a data exploration result report and / or anomaly detection result report for the sample data.

[0070] The data exploration result report may include data overview information, data exploration detail information of continuous features, data exploration detail information of discrete features, and data exploration detail information of date features.

[0071] The anomaly detection result report may include feature anomaly detection summary information (the number of features with abnormal conditions, the number of features involved in various types of anomalies, and feature details), sample anomaly detection summary information (the number and proportion of abnormal samples calculated by the summary algorithm), and may also include feature anomaly detection judgment logic. For example, for the number of data primary keys after deduplication in the data overview analysis module! = the number of rows, it is prompted that "the data primary key is not unique". For another example, for continuous single variable features, the skewness>3 feature is displayed as a distribution abnormality feature, and the maximum value=minimum value feature is displayed. For another example, for continuous multivariate features, the collinearity feature is displayed; for discrete features, the category number=1 feature is displayed; for date features, the first time=last time feature is displayed; for sample anomaly detection, the number and proportion of abnormal samples under each algorithm are displayed.

[0072] From the above description, it can be seen that the data preprocessing method provided in the embodiment of the present application can standardize the data preprocessing process and simplify the amount of data preprocessing operations by automatically outputting data exploration and anomaly detection reports. It can also be used as a basis for third-party acceptance testing and is universal for different structured data projects, thereby effectively improving the efficiency and accuracy of data preprocessing.

[0073] In order to accurately explore the sample data, in one embodiment of the data preprocessing method of the present application, see Figure 2 , the above step S103 may also specifically include the following contents:

[0074] Step S201: Determine sample data summary information in the data exploration summary according to the number of feature types of the sample data.

[0075] Specifically, they may include:

[0076] ①The data is m rows and n columns;

[0077] ②Data label characteristics (with or without), and (if any) statistics for each category;

[0078] ③Data (with or without) primary key, the number of primary keys after deduplication;

[0079] ④ The number of data features (except labels and primary keys): continuous features, discrete features, and date features;

[0080] ⑤q features are missing among the data features (except labels and primary keys).

[0081] Step S202: Perform variable traversal output on the sample data exploration results whose feature categories are continuous single variable features, discrete features, and date features, and determine feature distribution information and feature null value rate information in the data exploration result report.

[0082] Specifically, they may include:

[0083] For continuous single variable features, traverse and output each variable:

[0084] ① This feature: number of null values, null value rate, number of negative values / 0 values / positive values.

[0085] ②The characteristics: kurtosis, skewness, mean, standard deviation, quartile, maximum value, minimum value.

[0086] ③This feature: variable frequency distribution histogram and kernel density curve.

[0087] For discrete features, traverse and output each variable:

[0088] ① The characteristics: number of categories, number of null values, null value ratio, category with the highest frequency and its frequency ratio.

[0089] ②This feature: TOP10 category frequency distribution bar chart.

[0090] For date type features, traverse and output each variable:

[0091] ①The characteristics: number of null values, null value rate, first time, last time, and quartile date.

[0092] ②This feature: Output frequency distribution by date distribution.

[0093] Step S203: Calculate the pairwise correlation coefficients of the sample data whose feature categories are continuous multivariate features, and determine the feature correlation information in the data exploration result report.

[0094] For continuous multivariate features, calculate the pairwise correlation coefficient:

[0095] ① The logarithm of the correlation coefficient greater than 0.7, involving the number of variables.

[0096] ②Correlation diagram with correlation coefficient greater than 0.7.

[0097] ③Correlation diagrams and tables.

[0098] In order to accurately perform anomaly detection on sample data in advance, in one embodiment of the data preprocessing method of the present application, see Figure 3 , the above step S103 may also specifically include the following contents:

[0099] Step S301: determining overall missing statistics in the anomaly detection result report according to each feature whose missing ratio is greater than a threshold value in the overall missing rate of features of the sample data and the corresponding missing values.

[0100] For example, the feature names and values ​​of the features with a missing ratio greater than 70% are displayed.

[0101] Step S302: performing feature exploration and determining abnormal features, and determining an abnormal feature list in the abnormal detection result report according to the abnormal features;

[0102] Step S303: determining low-data-volume abnormal sample data in the sample data according to a preset clustering algorithm, and determining an abnormal sample list in an abnormal detection result report according to the low-data-volume abnormal sample data.

[0103] Specifically include:

[0104] ① Clustering algorithm: select the algorithm, input parameters, output evaluation value, compare the amount of data in each category, and the one with the lowest amount of data is temporarily listed as abnormal samples.

[0105] ② Isolation forest algorithm: input parameters, output evaluation value, and the TOP 10% samples are the list of abnormal samples.

[0106] In order to accurately select the corresponding clustering algorithm according to the sample data volume, in an embodiment of the data preprocessing method of the present application, the above step S103 may also specifically include the following contents:

[0107] The corresponding clustering algorithm is determined based on whether the data volume of the sample data exceeds the preset data volume threshold.

[0108] Optional, see Fig. 9 ,Regarding the selection logic of clustering algorithm: select according to the number of samples. When the number of samples is less than 10,000, KMeans / MeanShift / VBGMM algorithm can be used for clustering, and the proportion of each category can be output. The samples in the category with the lowest data volume can be temporarily listed as abnormal samples. When the sample size is greater than 10,000, MiniBatch KMeans is used, and the logic for determining whether the sample is abnormal is the same as above.

[0109] In some feasible embodiments of the present application, the anomaly detection detailed report may include seven aspects:

[0110] 1. The primary key has duplicate values.

[0111] 2. The missing ratio is greater than the threshold, which is abnormal.

[0112] 3. The skewness of the continuous feature distribution is large and the distribution is abnormal, or the continuous variable value is unique and abnormal.

[0113] 4. Continuous feature collinearity is abnormal.

[0114] 5. The discrete eigenvalue is unique and abnormal.

[0115] 6. The unique feature value of date type is abnormal.

[0116] 7. Outliers are defined as anomalies.

[0117] In order to effectively improve the efficiency and accuracy of data preprocessing, the present application provides an embodiment of a data preprocessing device for implementing all or part of the content of the data preprocessing method, see Figure 4 , the data preprocessing device specifically includes the following contents:

[0118] The feature pre-classification module 10 is used to perform feature classification on the collected sample data and determine the number of feature types and the overall feature missing rate of the sample data.

[0119] The feature exploration and anomaly detection module 20 is used to perform feature exploration and anomaly detection on the sample data of each feature category in the sample data, and determine the feature distribution, feature null value rate, feature correlation and abnormal samples of the sample data of each feature category.

[0120] The report generation module 30 is used to generate a data exploration summary, a data exploration result report and anomaly detection result report of the sample data based on the number of feature types of the sample data and the overall missing rate of the features, as well as at least one of the feature distribution, feature null value rate, feature correlation and sample clustering results of the sample data of each feature category.

[0121] From the above description, it can be seen that the data preprocessing device provided in the embodiment of the present application can standardize the data preprocessing process and simplify the amount of data preprocessing operations by automatically outputting data exploration and anomaly detection reports. It can also be used as a basis for third-party acceptance testing and is universal for different structured data projects, thereby effectively improving the efficiency and accuracy of data preprocessing.

[0122] In order to accurately perform data exploration on sample data, in one embodiment of the data preprocessing device of the present application, see Figure 5 , the report generating module 30 comprises:

[0123] The data summary processing unit 31 is used to determine the sample data summary information in the data exploration summary according to the number of feature types of the sample data.

[0124] The data detailed exploration unit 32 is used to perform variable traversal output on the sample data exploration results whose feature categories are continuous single variable features, discrete features and date features, and determine the feature distribution information and feature null value rate information in the data exploration result report.

[0125] The correlation determination unit 33 is used to calculate the pairwise correlation coefficients of the sample data whose feature categories are continuous multivariate features, and determine the feature correlation information in the data exploration result report.

[0126] In order to accurately perform anomaly detection on sample data in advance, in one embodiment of the data preprocessing device of the present application, see Figure 6 , the report generating module 30 comprises:

[0127] Overview The anomaly detection unit 34 is used to determine the overall missing statistics in the anomaly detection result report according to each feature whose missing ratio is greater than a threshold in the overall missing rate of the features of the sample data and the corresponding missing values.

[0128] A feature anomaly detection unit 35, configured to perform feature exploration and determine abnormal features, and determine an abnormal feature list in the abnormal detection result report according to the abnormal features;

[0129] The sample anomaly detection unit 36 ​​is used to determine low-data-volume abnormal sample data in the sample data according to a preset clustering algorithm, and determine an abnormal sample list in an anomaly detection result report according to the low-data-volume abnormal sample data.

[0130] In order to accurately select the corresponding clustering algorithm according to the sample data volume, in one embodiment of the data preprocessing device of the present application, see Figure 7 , the report generating module 30 further includes:

[0131] The clustering algorithm selection unit 37 is used to determine the corresponding clustering algorithm according to whether the data volume of the sample data exceeds a preset data volume threshold.

[0132] In order to further illustrate the present solution, the present application also provides a specific application example of using the above data preprocessing device to implement the data preprocessing method, see Figure 8 , specifically including the following contents:

[0133] First, pre-classify the data features and divide them into primary key features, label features, continuous features, discrete features, and date features. Then, conduct overview analysis and detailed analysis in turn.

[0134] Among them, the overview analysis mainly counts the overall data volume, the number of features of each pre-classified subcategory, and the overall missing features.

[0135] The detailed analysis is carried out from two dimensions: features and samples:

[0136] (1) Feature mining and anomaly detection are carried out according to the pre-classified subcategories in the feature dimension, mainly focusing on three aspects: feature distribution, feature null value rate and feature correlation.

[0137] (2) The sample dimension mainly performs anomaly detection and identifies abnormal samples through machine learning algorithms. The solution ultimately outputs data exploration and anomaly detection reports, reflecting the automated data preprocessing process.

[0138] Among them, it is worth noting that:

[0139] (1) This application only processes structured data.

[0140] (2) During feature pre-classification, Boolean features are included in the discrete feature list.

[0141] (3) For data categories with labels, when drawing the feature distribution, the distribution results subdivided by label shall be supplemented and listed in the appendix.

[0142] (4) Clustering algorithm selection logic: The selection is based on the number of samples. When the number of samples is less than 10,000, the KMeans / MeanShift / VBGMM algorithm can be used for clustering, and the proportion of each category can be output. The samples in the category with the lowest data volume can be temporarily classified as abnormal samples. When the sample size is greater than 10,000, MiniBatch KMeans is used, and the logic for determining whether a sample is abnormal is the same as above.

[0143] From the above content, it can be seen that this application standardizes the data preprocessing process and simplifies the amount of data preprocessing operations by automatically outputting data exploration and anomaly detection reports. It can also be used as a basis for third-party acceptance testing and is universal for different structured data projects.

[0144] From a hardware perspective, in order to effectively improve the efficiency and accuracy of data preprocessing, the present application provides an embodiment of an electronic device for implementing all or part of the content in the data preprocessing method, and the electronic device specifically includes the following content:

[0145] Processor, memory, communication interface and bus; wherein the processor, memory and communication interface communicate with each other through the bus; the communication interface is used to realize information transmission between the data preprocessing device and related devices such as the core business system, user terminal and related database; the logic controller can be a desktop computer, a tablet computer and a mobile terminal, etc., but the present embodiment is not limited thereto. In the present embodiment, the logic controller can be implemented with reference to the embodiment of the data preprocessing method and the embodiment of the data preprocessing device in the embodiment, and the contents thereof are incorporated herein, and the repeated parts are not repeated.

[0146] It is understandable that the user terminal may include a smart phone, a tablet electronic device, a network set-top box, a portable computer, a desktop computer, a personal digital assistant (PDA), a vehicle-mounted device, a smart wearable device, etc. Among them, the smart wearable device may include smart glasses, a smart watch, a smart bracelet, etc.

[0147] In practical applications, part of the data preprocessing method can be executed on the electronic device side as described above, or all operations can be completed in the client device. The specific selection can be based on the processing capability of the client device and the limitations of the user's usage scenario. This application does not limit this. If all operations are completed in the client device, the client device may also include a processor.

[0148] The client device may have a communication module (i.e., a communication unit) that can communicate with a remote server to achieve data transmission with the server. The server may include a server on the task scheduling center side, and other implementation scenarios may also include a server on an intermediate platform, such as a server on a third-party server platform that has a communication link with the task scheduling center server. The server may include a single computer device, or a server cluster consisting of multiple servers, or a server structure of a distributed device.

[0149] Fig.12 FIG. 9 is a schematic block diagram of the system structure of the electronic device 9600 according to an embodiment of the present application. Fig.12 As shown, the electronic device 9600 may include a central processor 9100 and a memory 9140; the memory 9140 is coupled to the central processor 9100. It is worth noting that Fig.12 is exemplary; other types of structures may also be used to supplement or replace this structure to implement telecommunication functions or other functions.

[0150] In one embodiment, the data preprocessing method function may be integrated into the central processing unit 9100. The central processing unit 9100 may be configured to perform the following control:

[0151] Step S101: feature classifying the collected sample data, and determining the number of feature types and the overall feature missing rate of the sample data.

[0152] Step S102: performing feature exploration and anomaly detection on the sample data of each feature category in the sample data, and determining the feature distribution, feature null value rate, feature correlation and abnormal samples of the sample data of each feature category.

[0153] Step S103: Generate a data exploration summary, a data exploration result report and anomaly detection result report of the sample data according to the number of feature types of the sample data and the overall missing rate of the features, as well as at least one of the feature distribution, feature null value rate, feature correlation and sample clustering results of the sample data of each feature category.

[0154] From the above description, it can be seen that the electronic device provided in the embodiment of the present application standardizes the data preprocessing process and simplifies the amount of data preprocessing operations by automatically outputting data exploration and anomaly detection reports. It can also be used as a basis for third-party acceptance testing and is universal for different structured data projects, thereby effectively improving the efficiency and accuracy of data preprocessing.

[0155] In another embodiment, the data preprocessing device may be configured separately from the central processing unit 9100. For example, the data preprocessing device may be configured as a chip connected to the central processing unit 9100, and the data preprocessing method function may be implemented under the control of the central processing unit.

[0156] like Fig.12 As shown, the electronic device 9600 may also include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It is worth noting that the electronic device 9600 does not necessarily have to include Fig.12 In addition, the electronic device 9600 may also include Fig.12 For components not shown, reference may be made to the prior art.

[0157] like Fig.12 As shown, the central processing unit 9100 is sometimes also referred to as a controller or an operation control, and may include a microprocessor or other processor device and / or logic device. The central processing unit 9100 receives input and controls the operation of various components of the electronic device 9600.

[0158] The memory 9140 may be, for example, one or more of a cache, a flash memory, a hard drive, a removable medium, a volatile memory, a non-volatile memory or other suitable devices. The above-mentioned information related to the failure may be stored, and a program for executing the relevant information may also be stored. The CPU 9100 may execute the program stored in the memory 9140 to implement information storage or processing, etc.

[0159] The input unit 9120 provides input to the central processing unit 9100. The input unit 9120 is, for example, a key or a touch input device. The power supply 9170 is used to provide power to the electronic device 9600. The display 9160 is used to display display objects such as images and texts. The display may be, for example, an LCD display, but is not limited thereto.

[0160] The memory 9140 may be a solid-state memory, such as a read-only memory (ROM), a random access memory (RAM), a SIM card, etc. It may also be a memory that saves information even when the power is off, can be selectively erased, and is provided with more data, examples of which are sometimes referred to as EPROMs, etc. The memory 9140 may also be some other type of device. The memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 may include an application / function storage unit 9142, which is used to store application programs and function programs or processes for executing the operation of the electronic device 9600 through the central processor 9100.

[0161] The memory 9140 may also include a data storage unit 9143 for storing data, such as contacts, digital data, pictures, sounds, and / or any other data used by the electronic device. The driver storage unit 9144 of the memory 9140 may include various drivers for communication functions of the electronic device and / or for executing other functions of the electronic device (such as messaging applications, address book applications, etc.).

[0162] The communication module 9110 is a transmitter / receiver 9110 that sends and receives signals via an antenna 9111. The communication module (transmitter / receiver) 9110 is coupled to the central processor 9100 to provide input signals and receive output signals, which may be the same as the case of a conventional mobile communication terminal.

[0163] Based on different communication technologies, multiple communication modules 9110 may be provided in the same electronic device, such as a cellular network module, a Bluetooth module and / or a wireless LAN module, etc. The communication module (transmitter / receiver) 9110 is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130 to provide an audio output via the speaker 9131 and receive an audio input from the microphone 9132, thereby realizing a common telecommunication function. The audio processor 9130 may include any suitable buffer, decoder, amplifier, etc. In addition, the audio processor 9130 is also coupled to the central processor 9100, so that recording can be performed on the local machine through the microphone 9132, and the sound stored on the local machine can be played through the speaker 9131.

[0164] The embodiments of the present application also provide a computer-readable storage medium capable of implementing all the steps of the data preprocessing method in the above embodiments, where the execution subject is a server or a client. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, all the steps of the data preprocessing method in the above embodiments are implemented by the execution subject being a server or a client. For example, when the processor executes the computer program, the following steps are implemented:

[0165] Step S101: feature classifying the collected sample data, and determining the number of feature types and the overall feature missing rate of the sample data.

[0166] Step S102: performing feature exploration and anomaly detection on the sample data of each feature category in the sample data, and determining the feature distribution, feature null value rate, feature correlation and abnormal samples of the sample data of each feature category.

[0167] Step S103: Generate a data exploration summary, a data exploration result report and anomaly detection result report of the sample data according to the number of feature types of the sample data and the overall missing rate of the features, as well as at least one of the feature distribution, feature null value rate, feature correlation and sample clustering results of the sample data of each feature category.

[0168] From the above description, it can be seen that the computer-readable storage medium provided in the embodiment of the present application standardizes the data preprocessing process and simplifies the amount of data preprocessing operations by automatically outputting data exploration and anomaly detection reports. It can also be used as a basis for third-party acceptance testing and is universal for different structured data projects, thereby effectively improving the efficiency and accuracy of data preprocessing.

[0169] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0170] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (apparatus), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0171] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0172] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0173] The present invention uses specific embodiments to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A data preprocessing method, characterized in that: The method comprises: Performing feature classification on the collected sample data, and determining the number of feature types and the overall feature missing rate of the sample data; Performing feature exploration and anomaly detection on sample data of each feature category in the sample data to determine feature distribution, feature null value rate, feature correlation and abnormal samples of the sample data of each feature category; Generate a data exploration summary, a data exploration result report, and an anomaly detection result report of the sample data according to the number of feature types of the sample data and the overall missing rate of the features, as well as at least one of the feature distribution, feature null value rate, feature correlation, and sample clustering results of the sample data of each feature category; Generating a data exploration summary and a data exploration result report of the sample data according to the number of feature types of the sample data and the overall feature missing rate and at least one of the feature distribution, feature null value rate, feature correlation, and sample clustering results of the sample data of each feature category, includes: Determine sample data overview information in the data exploration summary according to the number of feature types of the sample data; Perform variable traversal output on the sample data exploration results whose feature categories are continuous single variable features, discrete features, and date features, and determine the feature distribution information and feature null value rate information in the data exploration result report; Calculate the pairwise correlation coefficients of sample data whose feature categories are continuous multivariate features, and determine the feature correlation information in the data exploration result report; Determine overall missing statistics in the anomaly detection result report based on each feature whose missing ratio is greater than a threshold value and the corresponding missing value in the overall missing rate of features of the sample data; Perform feature exploration and determine abnormal features, and determine an abnormal feature list in the abnormal detection result report based on the abnormal features; Determine low-data-volume abnormal sample data in the sample data according to a preset clustering algorithm, and determine an abnormal sample list in an abnormal detection result report according to the low-data-volume abnormal sample data.

2. The data preprocessing method according to claim 1, characterized in that: The variable traversal output of the sample data exploration results whose feature categories are continuous single variable features, and determining the feature distribution information and feature null value rate information in the data exploration result report, include: The sample data exploration results with the feature category of continuous single variable features are output by variable traversal to determine the number of null values, null value rate, number of negative values, number of 0 values, number of positive values, kurtosis, skewness, mean value, standard deviation, quartiles, maximum value, minimum value, variable frequency distribution histogram and kernel density curve in the data exploration result report.

3. The data preprocessing method according to claim 1, characterized in that: The variable traversal output of the sample data exploration results whose feature categories are discrete features, and the determination of feature distribution information and feature null value rate information in the data exploration result report, include: Perform variable traversal output on the sample data exploration results whose feature categories are discrete features, and determine the number of categories, number of null values, null value rate, category with the largest frequency and its frequency proportion, and category frequency distribution bar chart in the data exploration result report.

4. The data preprocessing method according to claim 1, characterized in that: The variable traversal output of the sample data exploration results whose feature category is date-type features, and determining the feature distribution information and feature null value rate information in the data exploration result report, include: Perform variable traversal output on the sample data exploration results whose feature category is date-type features, determine the number of null values, null value rate, first time, last time, quartile date in the data exploration result report, and output frequency distribution by date distribution.

5. The data preprocessing method according to claim 1, characterized in that: The step of determining low data volume abnormal sample data in the sample data according to a preset clustering algorithm further includes: The corresponding clustering algorithm is determined based on whether the data volume of the sample data exceeds the preset data volume threshold.

6. A data preprocessing device, characterized in that: include: A feature pre-classification module is used to perform feature classification on the collected sample data and determine the number of feature types and the overall feature missing rate of the sample data; A feature exploration and anomaly detection module, used to perform feature exploration and anomaly detection on the sample data of each feature category in the sample data, and determine the feature distribution, feature null value rate, feature correlation and abnormal samples of the sample data of each feature category; A report generation module, for generating a data exploration summary, a data exploration result report and an anomaly detection result report of the sample data according to the number of feature types of the sample data and the overall missing rate of the features and at least one of the feature distribution, feature null value rate, feature correlation and sample clustering results of the sample data of each feature category; The report generation module includes: A data summary processing unit, used to determine sample data summary information in the data exploration summary according to the number of feature types of the sample data; The data detailed exploration unit is used to perform variable traversal output on the sample data exploration results whose feature categories are continuous single variable features, discrete features, and date features, and determine the feature distribution information and feature null value rate information in the data exploration result report; A correlation determination unit, used to calculate the pairwise correlation coefficients of sample data whose feature categories are continuous multivariate features, and determine feature correlation information in a data exploration result report; The report generation module includes: An anomaly detection unit is used to determine overall missing statistics in an anomaly detection result report according to each feature whose missing ratio is greater than a threshold value in the overall missing rate of features of the sample data and the corresponding missing values; A feature anomaly detection unit, used to perform feature exploration and determine abnormal features, and determine an abnormal feature list in the abnormal detection result report according to the abnormal features; The sample anomaly detection unit is used to determine low-data-volume abnormal sample data in the sample data according to a preset clustering algorithm, and determine an abnormal sample list in an anomaly detection result report according to the low-data-volume abnormal sample data.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the data preprocessing method according to any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the data preprocessing method according to any one of claims 1 to 5 are implemented.

9. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the data preprocessing method according to any one of claims 1 to 5 are implemented.