A method and system for automatically detecting data quality problems without prior knowledge

Abstract

The present invention provides a method and system for automatically detecting data quality problems without prior knowledge, including: automatically identifying abnormal null values ​​of data fields through a trained model of non-null requirements for data fields, and marking them as abnormal data; identifying and establishing foreign key relationships between data tables, and identifying abnormal data with missing foreign keys; automatically identifying actual primary keys in tables, and identifying and marking non-unique primary keys. The method and system for automatically detecting data quality problems without prior knowledge of the present invention integrates multiple automated algorithm systems to inspect data, reducing manpower consumption; in the process of data quality verification, there is no need to manually provide verification rules; it has a learnable characteristic, and its recognition accuracy will become higher and higher as the application time of the model increases; in the process of model recognition, it will automatically generate and discover logical connections between some data, facilitating the development of data inventory work.

Description

Technical Field

[0001] The present invention relates to the technical field of data quality verification, and in particular to a method and system for automatically detecting data quality problems without prior knowledge. Background Art

[0002] Through machine learning and deep learning, and without providing business data rules, using a variety of artificial intelligence algorithms, most types of data problems can be systematically and automatically identified.

[0003] At present, there are problems in the data quality verification of existing big data, such as low efficiency in discovery and over-reliance on business logic for verification. Summary of the invention

[0004] The present invention proposes a method and system for automatically detecting data quality problems without prior knowledge, which can automatically discover suspected problematic data in existing data, improve the efficiency of data quality verification, and strengthen data quality control measures.

[0005] To achieve the above object, the present invention adopts the following technical solutions:

[0006] A method for automatically detecting data quality problems without prior knowledge, comprising:

[0007] S11. Automatically identify abnormal null values ​​in data fields through the trained model of non-null requirements for data fields, and mark them as abnormal data;

[0008] S12, identifying and establishing foreign key relationships between data tables, and identifying abnormal data with missing foreign keys;

[0009] S13. Automatically identify the actual primary key in the table, and identify and mark the non-unique primary key.

[0010] Furthermore, the method further comprises the following steps:

[0011] S21. Automatically analyze the actual data type of the text field and automatically mark abnormal data types;

[0012] S22, automatically identify and mark texts and coding fields with outlier lengths;

[0013] S23, automatically identify the text mode of text and coding fields, and automatically find data that does not conform to the text mode;

[0014] S24. Automatically identify data with accuracy lower than a certain value and mark it as abnormal data.

[0015] Furthermore, the method further comprises the following steps:

[0016] S31, automatically identifying and marking outlier values ​​in numerical fields;

[0017] S32, automatically fitting the calculation relationship between numerical fields, and marking the data that does not conform to the calculation relationship as abnormal;

[0018] S33, intelligently identify the greater than, less than, and equal to relationships between fields, and mark the data that does not conform to the logical relationship as abnormal;

[0019] S34. Intelligently identify the Liqun code value and mark it as abnormal data.

[0020] Furthermore, the method further comprises the following steps:

[0021] S41. Automatically identify tables with highly similar contents, and automatically identify and mark inconsistent data between tables.

[0022] Furthermore, the S11 specifically includes:

[0023] 1). Remove noise, identify enumeration fields, and GroupBy those with less than 10 categories;

[0024] 2). The entire table is vectorized according to the following rules: for non-enumeration fields, the null value is 0 and the non-null value is 1; for enumeration fields, the null value is 0 and the other enumeration fields are 1-9 in sequence;

[0025] 3) Use the principal component analysis algorithm to determine the correlation between fields. If there are k fields, first use the first field as y and the other fields as x. Then use the second field as y and the other fields as x. This process is repeated until all fields have been used as y.

[0026] 4). Process each field in turn:

[0027] a) If the field has no associated field and the null value ratio is lower than a certain threshold, the non-null fields in the field are marked;

[0028] b) For fields with association relationships:

[0029] i. The input is the associated field, and the output is the current field;

[0030] ii. Perform homogenization on samples of all input fields;

[0031] iii. Train the entire data set, and then verify the entire data set after training. If the predicted results are inconsistent with the actual results, it is considered abnormal data.

[0032] Furthermore, the S12 specifically includes:

[0033] 1). Scan the similarity between fields in the entire database, or compare the primary key with all fields;

[0034] 2) Intersect similar fields in pairs. If the proportion of the difference result in one field is higher than the set value, the two fields are considered to be mapped fields.

[0035] 3). Calculate the number of categories after GroupBy of the two fields and score them according to the following conditions:

[0036] a) Number of categories / number of fields, 1 point is added if the ratio is larger

[0037] b) 1 point will be added if the number of categories is large;

[0038] c) If the ratio of difference number of sets to number of fields is small, 1 point will be added;

[0039] The final result with a high score is the primary key, the one with a low score is the foreign key, and the primary and foreign key relationships are recorded;

[0040] 4). Identify missing data for foreign keys.

[0041] Furthermore, the S21 specifically includes:

[0042] 1). Scan all data in the text field field by field, excluding null values, and identify the following:

[0043] i. Those that can be segmented into Chinese words are considered text;

[0044] ii. If Chinese characters cannot be recognized, see if they can be converted into dates;

[0045] iii. If the first character of the unrecognizable string is 0 or contains non-digits or decimal points, it is a code;

[0046] iv. If none of the above is a numerical value;

[0047] The above types are vectorized to 0-3 respectively;

[0048] 2) Cluster the vectorization of a single field. If a type greater than the set value is found, mark the field as this type. If the number reaches the set value, mark it as an unknown field.

[0049] 3). Mark the data in each field that is not of the field type as suspected problematic data.

[0050] On the other hand, the present invention also discloses a system for automatically detecting data quality problems without prior knowledge, comprising the following units:

[0051] The data item missing identification module is used to automatically identify abnormal null values ​​in data fields through the trained model of non-null requirements for data fields, and mark them as abnormal data;

[0052] Data foreign key missing identification module is used to identify and establish foreign key relationships between data tables and identify abnormal data with missing foreign keys;

[0053] The primary key conflict identification module automatically identifies the actual primary key in the table and identifies and marks non-unique primary keys.

[0054] Furthermore, the following units are included:

[0055] Data type error recognition unit, which is used to automatically analyze the actual data type of text fields and automatically mark abnormal data types;

[0056] Data text length anomaly recognition unit, used to automatically identify and mark text and coding fields with outlier lengths;

[0057] Data text pattern anomaly recognition unit, used to automatically recognize the text pattern of text and coding fields, and automatically find data that does not conform to the text pattern;

[0058] The data accuracy error identification unit is used to automatically identify data whose accuracy is lower than a certain value and mark it as abnormal data.

[0059] Furthermore, the following units are included:

[0060] Data range error or dimension error identification unit, used to automatically identify and mark outlier values ​​in numerical fields;

[0061] Data calculation error identification unit, used to automatically fit the calculation relationship between numerical fields and mark data that does not conform to the calculation relationship as abnormal;

[0062] Data logical relationship error recognition unit, used to intelligently identify the greater than, less than, and equal to relationships between fields, and mark data that does not conform to the logical relationship as abnormal;

[0063] Data code error identification unit, used to intelligently identify the Liqun code value and mark it as abnormal data;

[0064] The cross-table data inconsistency identification unit is used to automatically identify tables with highly similar contents, and automatically identify and mark inconsistent data between tables.

[0065] It can be seen from the above technical solutions that the automatic detection method for data quality problems without prior knowledge of the present invention verifies data through an automated algorithm system, reducing manpower consumption; in the process of data quality verification, there is no need to manually provide verification rules; it has a learnable characteristic, and its recognition accuracy will become higher and higher as the model application time increases; in the process of model recognition, it will automatically generate and discover logical connections between some data, which is convenient for the development of data inventory work. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 It is a schematic diagram of the steps of the method of the present invention;

[0067] Figure 2 is a flow chart of a method for identifying missing data items of the present invention;

[0068] Figure 3 This is an application scenario of the data item missing identification method of the present invention;

[0069] Figure 4 It is a flow chart of the data foreign key missing identification algorithm of the present invention;

[0070] Figure 5 This is the application scenario of the data foreign key missing identification algorithm of the present invention;

[0071] Figure 6 is a flow chart of the primary key conflict identification algorithm of the present invention;

[0072] Figure 7 This is an application scenario of the primary key conflict identification algorithm of the present invention;

[0073] Figure 8 is a flow chart of a data type error identification algorithm of the present invention;

[0074] Fig. 9 This is an application scenario of the data type error identification algorithm of the present invention;

[0075] Fig.10 It is a flow chart of the data text length anomaly identification algorithm of the present invention;

[0076] Fig.11 This is the application scenario of the data text length anomaly recognition algorithm of the present invention;

[0077] Fig.12 is a flow chart of the data text pattern anomaly recognition algorithm of the present invention;

[0078] Fig.13 This is the application scenario of the data text pattern anomaly recognition algorithm of the present invention;

[0079] Fig.14 is a flow chart of the data accuracy error identification algorithm of the present invention;

[0080] Fig.15 This is the application scenario of the data accuracy error identification algorithm of the present invention;

[0081] Fig.16 is a flow chart of a data range error or dimension error identification algorithm of the present invention;

[0082] Fig.17 , Fig.18This is an application scenario of the data range error or dimension error identification algorithm of the present invention;

[0083] Fig.19 is a flow chart of a data calculation error identification algorithm of the present invention;

[0084] Fig. 20 This is an application scenario of the data calculation error identification algorithm of the present invention;

[0085] Fig.21 is a flow chart of the data logic relationship error identification algorithm of the present invention;

[0086] Fig. 22 This is an application scenario of the data logic relationship error recognition algorithm of the present invention;

[0087] Fig.23 is a flow chart of a data code error identification algorithm of the present invention;

[0088] Fig.24 This is an application scenario of the data code error recognition algorithm of the present invention;

[0089] Fig.25 is a flow chart of the cross-table data inconsistency identification algorithm of the present invention;

[0090] Fig.26 This is an application scenario of the cross-table data inconsistency identification algorithm of the present invention. DETAILED DESCRIPTION

[0091] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments.

[0092] like Figure 1 As shown, the method for automatically detecting data quality problems without prior knowledge described in this embodiment includes:

[0093] S11. Automatically identify abnormal null values ​​in data fields through the trained model of non-null requirements for data fields, and mark them as abnormal data;

[0094] S12, identifying and establishing foreign key relationships between data tables, and identifying abnormal data with missing foreign keys;

[0095] S13, automatically identifying the actual primary key in the table, and identifying and marking the non-unique primary key;

[0096] S21. Automatically analyze the actual data type of the text field and automatically mark abnormal data types;

[0097] S22, automatically identify and mark texts and coding fields with outlier lengths;

[0098] S23, automatically identify the text mode of text and coding fields, and automatically find data that does not conform to the text mode;

[0099] S24, automatically identify data with accuracy lower than a certain value and mark it as abnormal data;

[0100] S31, automatically identifying and marking outlier values ​​in numerical fields;

[0101] S32, automatically fitting the calculation relationship between numerical fields, and marking the data that does not conform to the calculation relationship as abnormal;

[0102] S33, intelligently identify the greater than, less than, and equal to relationships between fields, and mark the data that does not conform to the logical relationship as abnormal;

[0103] S34, intelligently identify the Liqun code value and mark it as abnormal data;

[0104] S41. Automatically identify tables with highly similar contents, and automatically identify and mark inconsistent data between tables.

[0105] Specifically, the embodiment of the present invention proposes twelve algorithms for the four aspects of data integrity, compliance, accuracy, and consistency, as follows:

[0106] 1. To address the problem of data integrity, the integrity sniffer includes three algorithms: data item missing identification algorithm, data foreign key missing identification algorithm, and primary key conflict identification algorithm.

[0107] 11) Data item missing identification algorithm: Automatically identify data field empty exceptions. The process is as follows Figure 2 , through deep learning algorithms to train models for non-empty requirements for data fields in different scenarios, automatically identify abnormal empty values ​​in data fields, and mark empty values ​​in fields that rarely have empty values ​​as abnormal data. Application scenarios such as Figure 3 .

[0108] Specific implementation steps:

[0109] 1. Remove noise, identify enumeration fields, and group by less than 10 categories;

[0110] 2. The entire table is vectorized according to the following rules: for non-enumeration fields, the null value is 0 and the non-null value is 1; for enumeration fields, the null value is 0 and the values ​​of other enumeration fields are 1-9 in sequence;

[0111] 3. Use the principal component analysis algorithm to determine the correlation between fields. If there are k fields, first use the first field as y and the other fields as x. Then use the second field as y and the other fields as x. This process is repeated until all fields have been processed as y. (Note: In this vectorization process, if there is a relationship between fields, the correlation coefficient should be very large, and a threshold needs to be determined through experiments.)

[0112] 4. Process each field in turn:

[0113] a) If the field has no associated field and the null value ratio is lower than a certain threshold, the non-null fields in the field are marked;

[0114] b) For fields with association relationships:

[0115] i. The input is the associated field, and the output is the current field;

[0116] ii. Perform homogenization on the samples of all input fields (for example, if there are two associated fields A and B, A has two types of values ​​and B has three types of values, then combine them into six types of samples, remove the samples with zero data in the six types, and then sort the number of valid classes from large to small, discard the classes with too small numbers (how to define too small numbers depends on experiments, such as <100), take the smallest class from the classes that have not been discarded, and use its number as the basis to randomly extract the same amount of data from other valid types);

[0117] iii. Train the entire data set, and then verify the entire data set after training. If the predicted results are inconsistent with the actual results, the data is considered abnormal.

[0118] 12) Data foreign key missing identification algorithm: Intelligently identify and establish foreign key relationships between data tables, and identify abnormal data with missing foreign keys. The process is as follows Figure 4 Intelligently identify similar fields between all tables, and intersect all similar fields. If the intersection result is extremely high in one field, it is considered that the two are in a foreign key relationship, and automatically identify abnormal data with missing foreign keys based on the dependent foreign key relationship. Application scenarios include Figure 5 .

[0119] Specific implementation steps:

[0120] 1. Scan the similarity between fields in the entire database, or compare the primary key with all fields;

[0121] 2. Intersect similar fields in pairs. If the difference result accounts for a very high proportion of one field (this threshold requires experimentation), the two fields are considered to be mapping fields.

[0122] 3. Calculate the number of categories after GroupBy of the two fields and score them according to the following conditions:

[0123] a) Number of categories / number of fields: 1 point will be added if the ratio is larger;

[0124] b) 1 point will be added if the number of categories is large;

[0125] c) If the ratio of difference number of sets to number of fields is small, 1 point will be added;

[0126] The final result with a high score is the primary key, the one with a low score is the foreign key, and the primary and foreign key relationships are recorded;

[0127] 4. Identify data with missing foreign keys.

[0128] 13) Primary key conflict identification algorithm: Automatically identify the actual primary key in the table and identify non-unique primary keys. The process is as follows: Figure 6 Cluster all values ​​in the field. If the clustering result is extremely discrete, the field is considered to be the primary key, and non-unique values ​​in the field are automatically identified. Application scenarios include Figure 7 .

[0129] Specific implementation steps:

[0130] 1. GroupBY the fields;

[0131] 2. Cluster the GroupBy results;

[0132] 3. If the clustering results converge to 1, the field is considered to be the primary key and non-unique values ​​are identified.

[0133] At the same time, this embodiment also includes the following steps:

[0134] 2. To address data compliance issues, the compliance sniffer includes four algorithms: data type error recognition algorithm, data text length anomaly recognition algorithm, data text pattern anomaly recognition algorithm, and data accuracy error recognition algorithm;

[0135] 21) Data type error identification algorithm: intelligently analyze the actual data type of text fields and automatically mark abnormal data types. The process is as follows: Figure 8 Through named entity recognition technology, we build recognition models for data types such as text (such as name, address, etc.), numeric value, code (list ID, enumeration code, etc.), date, etc., automatically mark type labels for each field, and identify abnormal data in the field that does not conform to the format based on the field type label. Application scenarios include Fig. 9 .

[0136] Specific implementation steps:

[0137] 1. Scan all data (excluding null values) in text fields (excluding enumeration fields) field by field and identify the following:

[0138] i. Those that can be segmented into Chinese words are considered text;

[0139] ii. If Chinese characters cannot be recognized, see if they can be converted into dates;

[0140] iii. If the first character of the unrecognizable string is 0 or contains non-digits or decimal points, it is a code;

[0141] iv. If none of the above is a numerical value;

[0142] The above types are vectorized to 0-3 respectively;

[0143] 2. Cluster the vectorization of a single field. If a very large type is found, mark the field as that type. If the types are similar in number (the specific threshold needs to be determined through experiments), mark it as an unknown field.

[0144] 3. Mark the data in each field that is not of the field type as suspected problematic data.

[0145] 22) Data text length anomaly identification algorithm: Automatically identify text and coding fields with outlier lengths. The process is as follows Fig.10 Cluster the lengths of all data in text and encoding fields (excluding enumeration fields), find data with particularly outlier lengths, and mark them as abnormal data. Application scenarios include Fig.11 .

[0146] 23) Data text pattern anomaly recognition algorithm: intelligently recognize the text pattern of text and coding fields, and automatically find data that does not conform to the text pattern. The process is as follows Fig.12 Through Chinese word segmentation and named entity recognition technology, we can intelligently identify the composition patterns of text and coding fields, and through deep learning algorithms, we can train models to recognize text patterns in different scenarios. Application scenarios include Fig.13 .

[0147] The implementation steps are as follows:

[0148] 1. Use the Chinese word segmentation algorithm to segment the text and obtain the part of speech (especially to see if you can find a more precise part of speech, such as distinguishing between addresses and names. If that is not possible, you can also distinguish between nouns, quantifiers, etc.);

[0149] 2. Vectorize different types of parts of speech, numbers, letters, special characters, etc. For example, 0 means unrecognizable, numbers are 1, letters are 2, - is 3..., nouns are 5, and quantifiers are 6. For example, "SKY-Chimelong Community-3rd Building 308 Room" is vectorized as 2(SKY)3(-)5(Chimelong)5(Community)3(-)1(3)6(Building)1(308)6(Room);

[0150] 3. The other fields of the table are vectorized according to step 2 of the method for missing values;

[0151] 4. Take one text type field as y and the other fields as x, and find the association between the text fields according to step 3 of the missing value method until all text type fields are taken as y;

[0152] 5. If there is no associated field, perform clustering to find out the incompatible patterns and identify them as anomalies; if there is an associated field, perform training and identification according to step 4 of the missing value method.

[0153] 24) Data accuracy error identification algorithm: Automatically identify rarely occurring data accuracy and mark it as abnormal data. The process is as follows Fig.14 Automatically classify the accuracy of all data in numeric fields and text fields marked as numeric values, select the accuracy classifications with a very small number, and mark them as abnormal data. Application scenarios include Fig.15 .

[0154] 3. To address the data accuracy issue, the accuracy sniffer includes four algorithms: data range error or dimension error recognition algorithm, data calculation error recognition algorithm, data logical relationship error recognition algorithm, and data code error recognition algorithm;

[0155] 31) Data range error or dimension error identification algorithm: Automatically identify outlier values ​​in numerical fields. The process is as follows Fig.16 Cluster all data in the numeric field, select the data with very few and serious outliers, and mark them as abnormal data. Application scenarios include Fig.17 , 18 .

[0156] 32) Data calculation error identification algorithm: intelligently fit the calculation relationship between numerical fields and mark the data that does not conform to the calculation relationship as abnormal. The process is as follows Fig.19 First, use the principal component analysis algorithm to find the fields with correlation relationships, then use the multidimensional fitting algorithm to fit the calculation formula between the related fields, and finally use the calculation formula to perform simulation calculations, marking the data that deviates greatly from the calculation results as abnormal data. Application scenarios such as Fig. 20 .

[0157] method:

[0158] 1. All numeric fields and numeric text fields are digitized, and all other fields are vectorized according to step 2 of the missing value method;

[0159] 2. Follow step 3 in Missing Values ​​to find the associated fields;

[0160] 3. Process the related field sets one by one (the number of numeric fields in the field set must be greater than 2), arrange and combine the other non-numeric fields in each set, and perform the next step separately for each combination;

[0161] 4. Perform sliding fitting on the numerical fields in the field set, using a multidimensional fitting algorithm (difficult, requires research, but there is definitely a solution), fit a specific calculation formula, and then perform calculations after fitting the formula to see how far the calculation results deviate from the actual results. Large deviations are marked as abnormal data (how large the deviation is needs to be determined by experiments).

[0162] 33) Data logical relationship error identification algorithm: intelligently identify the greater than, less than and equal to relationships between fields, and mark the data that does not conform to the logical relationship as abnormal. The process is as follows Fig.21 Differences are made between numeric and date fields. Through deep learning algorithms, a model is trained to identify the positive and negative differences in different scenarios. Based on the model, abnormal data in the field that does not conform to logical relationships is found. Application scenarios include Fig. 22 .

[0163] Specific implementation steps:

[0164] 1. All numeric fields, date fields, numeric text fields, and date text fields are digitized, and all other fields are vectorized according to step 2 of the missing value method;

[0165] 2. Follow step 3 in Missing Values ​​to find the associated fields;

[0166] 3. Process the related field sets one by one (the number of numeric fields in the field set must be greater than or equal to 2, or the number of date fields must be greater than or equal to 2), arrange and combine the other non-numeric fields in each set, and perform the next step separately for each combination; 4.

[0168] a) The dates and values ​​in the field set are combined in pairs (date sum date combination, value sum value combination), the two fields are subtracted, and the calculation and settlement results are vectorized according to the formula that less than 0 is 0, equal to 0 is 1, and greater than 0 is 2;

[0169] b) Homogenize the samples of each combination according to 4.2.2 of the missing value method;

[0170] c) Train the entire data set, and then verify the entire data set after training. If the predicted results are inconsistent with the actual results, it is considered abnormal data.

[0171] 34) Data code error identification algorithm: intelligently identify the Liqun code value and mark it as abnormal data. The process is as follows Fig.23Cluster the fields identified as enumeration types, identify the extremely small number of codes, and mark them as abnormal data. Application scenarios include Fig.24 .

[0172] Fourth, in response to the problem of data consistency, the consistency sniffer includes an algorithm: cross-table data inconsistency identification algorithm;

[0173] 41) Cross-table data inconsistency identification algorithm: intelligently identify tables with highly similar contents and automatically identify inconsistent data between tables. The process is as follows Fig.25 Based on the foreign key results of intelligent identification, the data tables with foreign key relationships are compared for similarity. For highly similar tables, the data between the tables is compared one by one, and inconsistent data is automatically identified. Application scenarios include Fig.26 .

[0174] It can be seen from the above technical solution that the automatic detection method for data quality problems without prior knowledge of the present invention integrates the above 12 automated algorithm systems to check the data separately, thereby reducing manpower consumption; in the process of data quality verification, there is no need to manually provide verification rules; it has a learnable characteristic, and its recognition accuracy will become higher and higher as the model application time increases; in the process of model recognition, it will automatically generate and discover logical connections between some data, which is convenient for the development of data inventory work.

[0175] On the other hand, the embodiment of the present invention also discloses a system for automatically detecting data quality problems without prior knowledge, including the following units:

[0176] The data item missing identification module is used to automatically identify abnormal null values ​​in data fields through the trained model of non-null requirements for data fields, and mark them as abnormal data;

[0177] Data foreign key missing identification module is used to identify and establish foreign key relationships between data tables and identify abnormal data with missing foreign keys;

[0178] The primary key conflict identification module automatically identifies the actual primary key in the table and identifies and marks non-unique primary keys;

[0179] Data type error recognition unit, which is used to automatically analyze the actual data type of text fields and automatically mark abnormal data types;

[0180] Data text length anomaly recognition unit, used to automatically identify and mark text and coding fields with outlier lengths;

[0181] Data text pattern anomaly recognition unit, used to automatically recognize the text pattern of text and coding fields, and automatically find data that does not conform to the text pattern;

[0182] Data accuracy error identification unit, used to automatically identify data accuracy below a certain value and mark it as abnormal data;

[0183] Data range error or dimension error identification unit, used to automatically identify and mark outlier values ​​in numerical fields;

[0184] Data calculation error identification unit, used to automatically fit the calculation relationship between numerical fields and mark data that does not conform to the calculation relationship as abnormal;

[0185] Data logical relationship error recognition unit, used to intelligently identify the greater than, less than, and equal to relationships between fields, and mark data that does not conform to the logical relationship as abnormal;

[0186] Data code error identification unit, used to intelligently identify the Liqun code value and mark it as abnormal data;

[0187] The cross-table data inconsistency identification unit is used to automatically identify tables with highly similar contents, and automatically identify and mark inconsistent data between tables.

[0188] It is understandable that the system provided by the embodiment of the present invention corresponds to the method provided by the embodiment of the present invention, and the explanation, examples and beneficial effects of the relevant contents can refer to the corresponding parts in the above method.

[0189] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0190] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0191] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0192] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0193] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for automatically detecting data quality problems without prior knowledge, characterized in that: The following steps are involved: S11. Automatically identify abnormal null values ​​of data fields through the trained model of non-null requirements of data fields and mark them as abnormal data; The S11 specifically includes: 1). Remove noise, identify enumeration fields, and GroupBy those with less than 10 categories; 2). The entire table is vectorized according to the following rules: non-enumeration fields: null value is 0, non-null value is 1; enumeration fields, null value is 0, other enumeration types are 1-9; 3) Use the principal component analysis algorithm to determine the correlation between fields. If there are k fields, first use the first field as y and the other fields as x. Then use the second field as y and the other fields as x. This process is repeated until all fields have been used as y. 4). Process each field in turn: a) If the field has no associated field and the null value rate is lower than the set threshold, the non-null fields in the field are marked; b) For fields with association relationships: i. The input is the associated field, and the output is the current field; ii. Perform homogenization on samples of all input fields; iii. Train the entire data set, and then verify the entire data set after training. If the predicted results are inconsistent with the actual results, the data is considered abnormal. S12, identifying and establishing foreign key relationships between data tables, and identifying abnormal data with missing foreign keys; The S12 specifically includes: 1). Scan the similarity between fields in the entire database, or compare the primary key with all fields; 2). Perform set differences on similar fields. If the proportion of the difference result in one field is higher than the set value, the two fields are considered to be mapping fields. 3). Calculate the number of categories after GroupBy of the two fields and score them according to the following conditions: a) Number of categories / number of fields, 1 point is added if the ratio is larger b) 1 point will be added if the number of categories is large; c) If the ratio of difference number of sets to number of fields is smaller, 1 point will be added; The final result with a high score is the primary key, the one with a low score is the foreign key, and the primary and foreign key relationships are recorded; 4). Identify data with missing foreign keys; S13. Automatically identify the actual primary key in the table, and identify and mark the non-unique primary key.

2. The method for automatically detecting data quality problems without prior knowledge according to claim 1, characterized in that: The following steps are also included: S21. Automatically analyze the actual data type of text fields and automatically mark abnormal data types; S22, automatically identify and mark texts and coding fields with outlier lengths; S23, automatically identify the text mode of text and coding fields, and automatically find data that does not conform to the text mode; S24, automatically identify data whose accuracy is lower than the set value and mark it as abnormal data.

3. The method for automatically detecting data quality problems without prior knowledge according to claim 2, characterized in that: The following steps are also included: S31, automatically identifying and marking outlier values ​​in numerical fields; S32, automatically fitting the calculation relationship between numerical fields, and marking the data that does not conform to the calculation relationship as abnormal; S33, intelligently identify the greater than, less than, and equal to relationships between fields, and mark the data that does not conform to the logical relationship as abnormal; S34. Intelligently identify outlier code values ​​and mark them as abnormal data.

4. The method for automatically detecting data quality problems without prior knowledge according to claim 3, characterized in that: The following steps are also included: S41. Automatically identify tables with highly similar contents, and automatically identify and mark inconsistent data between tables.

5. The method for automatically detecting data quality problems without prior knowledge according to claim 2, characterized in that: The S21 specifically includes: 1). Scan all data in the text field field by field, excluding null values, and identify the following: i. If Chinese characters can be recognized, they are considered as text; ii. If Chinese characters cannot be recognized, see if they can be converted into dates; iii. If the first character is 0 or contains non-digits or decimal points, it is a code. iv. If none of the above is a numerical value; The above types are vectorized to 0-3 respectively; 2). Cluster the vectorization of a single field. If a type greater than the set value is found, mark the field as this type. If the number reaches the set value, mark it as an unknown field. 3). Mark the data in each field that is not of the field type as suspected problematic data.

6. An automatic detection system for data quality problems without prior knowledge, characterized in that: The following units are included: The data item missing identification module is used to automatically identify abnormal null values ​​in data fields through the trained model of non-null requirements for data fields, and mark them as abnormal data; specifically, it includes: 1). Remove noise, identify enumeration fields, and GroupBy those with less than 10 categories; 2). The entire table is vectorized according to the following rules: non-enumeration fields: null value is 0, non-null value is 1; enumeration fields, null value is 0, other enumeration types are 1-9; 3) Use the principal component analysis algorithm to determine the correlation between fields. If there are k fields, first use the first field as y and the other fields as x. Then use the second field as y and the other fields as x. This process is repeated until all fields have been used as y. 4). Process each field in turn: a) If the field has no associated field and the null value rate is lower than the set threshold, the non-null fields in the field are marked; b) For fields with association relationships: i. The input is the associated field, and the output is the current field; ii. Perform homogenization on samples of all input fields; iii. Train the entire data set, and then verify the entire data set after training. If the predicted results are inconsistent with the actual results, the data is considered abnormal. The data foreign key missing identification module is used to identify and establish foreign key relationships between data tables and identify abnormal data with missing foreign keys; specifically, it includes: 1). Scan the similarity between fields in the entire database, or compare the primary key with all fields; 2). Perform set differences on similar fields. If the proportion of the difference result in one field is higher than the set value, the two fields are considered to be mapping fields. 3). Calculate the number of categories after GroupBy of the two fields and score them according to the following conditions: a) Number of categories / number of fields, 1 point is added if the ratio is larger b) 1 point will be added if the number of categories is large; c) If the ratio of difference number of sets to number of fields is smaller, 1 point will be added; The final result with a high score is the primary key, the one with a low score is the foreign key, and the primary and foreign key relationships are recorded; 4). Identify data with missing foreign keys; The primary key conflict identification module automatically identifies the actual primary key in the table and identifies and marks non-unique primary keys.

7. The automatic detection system for data quality problems without prior knowledge according to claim 6 is characterized in that: Also includes the following units: Data type error recognition unit, which is used to automatically analyze the actual data type of text fields and automatically mark abnormal data types; Data text length anomaly recognition unit, used to automatically identify and mark text and coding fields with outlier lengths; Data text pattern anomaly recognition unit, used to automatically recognize the text pattern of text and coding fields, and automatically find data that does not conform to the text pattern; The data accuracy error identification unit is used to automatically identify data whose accuracy is lower than the set value and mark it as abnormal data.

8. The automatic detection system for data quality problems without prior knowledge according to claim 6 is characterized in that: Also includes the following units: Data range error or dimension error identification unit, used to automatically identify and mark outlier values ​​in numerical fields; Data calculation error identification unit, used to automatically fit the calculation relationship between numerical fields and mark data that does not conform to the calculation relationship as abnormal; Data logical relationship error recognition unit, used to intelligently identify the greater than, less than, and equal to relationships between fields, and mark data that does not conform to the logical relationship as abnormal; Data code error identification unit, used to intelligently identify outlier code values ​​and mark them as abnormal data; The cross-table data inconsistency identification unit is used to automatically identify tables with highly similar contents, and automatically identify and mark inconsistent data between tables.

Images