Multi-rule collaborative data cleaning system, storage medium and device

Abstract

Multi-rule cooperative data cleaning system, storage medium and equipment relate to the field of computer data processing. The present application is to solve the problem that the existing data cleaning method is difficult to realize automatic data cleaning due to frequent logical conflicts and high error rate. The present application comprises: a rule storage unit for storing repair rules; a rule modification unit for modifying repair rules; a rule conflict detection unit for detecting whether the rules in the computing sequence conflict; a rule repair unit for repairing conflicting rules; a rule sequence calculation unit for calculating rule sequence; an error data detection unit for judging the cause of data error and recommending a repair method; an error data repair unit for repairing error data according to the recommended repair method and rule sequence; and a constraint modification unit for repairing data using the recommended repair method and sending the repair method to the rule library module. The present application is used for data cleaning.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer data processing, in particular to a multi-rule collaborative data cleaning system, a storage medium and equipment. BACKGROUND

[0002] In the information age, dirty data is everywhere. If the data user analyzes and utilizes the dirty data, the wrong conclusion will be obtained, thereby having a negative effect on the development of things. According to a statistical data, only the wrong date data causes tens of billions of dollars of loss to the American enterprises every year. In order to cope with the proliferation of dirty data, data cleaning plays an increasingly important role in the analysis and management of data.

[0003] At present, data cleaning is the main method for the analysis and management of data by the person skilled in the art. Data cleaning is mainly used to detect abnormal data (such as error data, missing data and inconsistent data) existing in the data, and solve the inconsistency problem through filling in missing values, smoothing noise data, identifying or deleting outlier data and the like. There are three methods for data cleaning: data cleaning based on integrity constraints, rule-based data cleaning and human-computer collaborative data cleaning. Among them, the rule-based data cleaning method plays an important role in the field of data cleaning because of its simple implementation and significant cleaning effect. However, when facing actual problems, there are often many rules, and the effect may be very different according to different cleaning sequences. The error of rule execution sequence often leads to the reduction of data cleaning quality. Therefore, it is crucial for the quality of data cleaning to generate correct and consistent rule execution sequence. In recent years, researchers have obtained some research results on rule-based data cleaning based on rule library and rules. However, with the increasing complexity of business rules and the sharp increase of data volume, logical conflicts frequently occur and the error rate is high in the rule execution process, which brings great obstacles and troubles to the implementation of automatic data cleaning. SUMMARY

[0004] The present application aims to solve the problem that the existing data cleaning method is difficult to realize automatic data cleaning due to frequent logical conflicts and high error rate, and proposes a multi-rule collaborative data cleaning system, a storage medium and equipment.

[0005] The multi-rule collaborative data cleaning system comprises a rule library module, a rule management module and a data cleaning module.

[0006] The rule library module comprises a rule storage unit and a rule modification unit.

[0007] The rule storage unit is used for storing repair rules and sending the repair rules to the rule management module.

[0008] The rule modification unit is configured to modify the repair rule stored in the rule storage unit;

[0009] The modification of the repair rule includes adding a predicate and deleting an error predicate.

[0010] The rule management module includes a rule conflict detection unit, a rule repair unit, and a rule order calculation unit.

[0011] The rule conflict detection unit is configured to detect whether the rules being calculated for order in the rule order calculation unit have conflicts, and to pass the rules having conflicts to the rule repair unit.

[0012] The rule repair unit is configured to repair the conflicting rules, and to send the repaired rule set to the rule order calculation unit, and to send the information of the repaired rules to the rule library module.

[0013] The rule order calculation unit is configured to calculate the order of the rules in the repaired rule set.

[0014] The data cleaning module includes an error data detection unit, an error data repair unit, and a constraint modification unit.

[0015] The error data detection unit is configured to determine the cause of data errors, and to recommend a repair method according to the error cause, and to send the error cause, the error data, and the recommended repair method to the error data repair unit or the constraint modification unit.

[0016] The cause of the data errors includes data itself errors and constraint errors.

[0017] The error data repair unit is configured to repair the error data according to the recommended repair method and the order of the rules obtained by the rule order calculation unit, and to obtain cleaned data.

[0018] The constraint modification unit is configured to determine whether to repair the error data by using the repair method recommended by the error data detection unit, and to send the repair method of the repair constraint to the rule library module.

[0019] Further, the system further includes a rule mining unit.

[0020] The rule mining unit is configured to learn rules from the cleaned data set to obtain new rules, and to save the new rules in the rule storage unit for the next round of data cleaning.

[0021] Preferably, the repair rule includes a functional dependency FD, a metric functional dependency MFD, a numeric functional dependency NDF, and a denial constraint DC.

[0022] Preferably, the rule conflict includes: the predicate of the rule contains a relationship, and the rule is repeated.

[0023] Preferably, the method for repairing the rule includes: deleting the rule and modifying the rule.

[0024] Preferably, the rule sequence calculation unit is used to calculate the sequence of the rules in the repaired rule set, and a partial order set algorithm is used to realize.

[0025] Further, the error data detection unit is used to judge the error cause of the data and recommend a repair method according to the error cause, and specifically, the error cause of the data and the repair method are obtained as a training set by user selection and verification, and the trained classifier is obtained by training the probability classifier using the training set.

[0026] S1, obtaining the error cause of the data and the repair method as a training set by user selection and verification, and obtaining the trained classifier by training the probability classifier using the training set.

[0027] S2, inputting the error data into the trained classifier to obtain the probability of the error cause of the data and each repair method.

[0028] The error cause type includes: data itself error and constraint error.

[0029] S3, obtaining the recommended repair method according to the probability of each repair method by using a repair search algorithm.

[0030] Preferably, the repair method includes: editing a rule, repairing a rule and a Sherlock rule.

[0031] A storage medium, the storage medium stores at least one instruction, the at least one instruction is loaded by a processor and executes the multi-rule coordinated data cleaning system.

[0032] A multi-rule coordinated data cleaning device, the device includes a processor and a memory, the memory stores at least one instruction, the at least one instruction is loaded by the processor and executed to realize the multi-rule coordinated data cleaning system.

[0033] The beneficial effects of the present application are:

[0034] The rule sorting method provided by the present application reduces logical errors in data cleaning, the rule conflict detection unit reduces error cleaning, solves the problem of logical conflict, and improves the efficiency of data cleaning; the present application also modifies the rules in the data cleaning process, continuously provides an environment for data cleaning, and thus provides a basis for realizing fully automated data cleaning. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 The flowchart of the present application. DETAILED DESCRIPTION

[0036] Specific implementation one: as Figure 1 The multi-rule coordination data cleaning system of the embodiment includes a rule library module, a rule management module, and a data cleaning module.

[0037] The rule library module includes a rule storage unit and a rule modification unit.

[0038] The rule storage unit is configured to store repair rules and send a set of repair rules to the rule management module. The repair rules include FD (functional dependency), MFD (metric functional dependency), NDF (numeric functional dependency), DC (denial constraint), and the like.

[0039] The rule modification unit is configured to modify the repair rules stored in the rule storage unit. The modification methods include adding more predicates and deleting incorrect predicates.

[0040] The rule management module includes a rule conflict detection unit, a rule repair unit, and a rule order calculation unit.

[0041] The rule conflict detection unit is configured to detect whether the rules being calculated for order in the rule order calculation unit have conflicts and pass the rules having conflicts to the rule repair unit. The conflicts include a containing relationship between the predicates of the rules and rule duplication.

[0042] The rule repair unit is configured to repair the rules having conflicts and send a set of repaired rules to the rule order calculation unit. The rule repair unit also feeds back information about the repaired rules to the rule library module. The repair methods include deleting rules and modifying rules.

[0043] The rule order calculation unit is configured to calculate the order of the rules in the set of repaired rules.

[0044] The data cleaning module includes an error data detection unit, an error data repair unit, a constraint modification unit, and a rule mining unit.

[0045] The error data detection unit is configured to determine the causes of data errors and recommend repair methods according to the causes. The error data detection unit also sends error data and the recommended repair methods to the error data repair unit or the constraint modification unit according to the causes. The causes of errors include data itself errors and constraint errors.

[0046] The error data detection unit sends data having data itself errors to the error data repair unit and sends data having constraint errors to the constraint modification unit.

[0047] The error data repair unit is configured to repair error data according to the recommended repair methods and the order of rules obtained by the rule order calculation unit to obtain cleaned data.

[0048] Wherein, the user manually selects which repair method to use;

[0049] The constraint modification unit determines whether to use the repair method recommended by the error data detection unit for repair, and sends the repair method of the repair constraint to the rule base module;

[0050] The rule mining unit is used to learn the rules in the cleaned data and obtain new rules for the next round of data cleaning;

[0051] The rule base module classifies and manages the rules, which helps us to extract the required rules for output. Considering that the rules may not be enough and new rules need to be learned from the data, the rule base can add new rules to the rule base. The rules stored in the rule base are not immutable, because considering that the rules may not be completely correct, the rules may be modified according to the characteristics of the data, including adding more predicates, deleting incorrect predicates, etc.

[0052] Specific implementation method two: the rule order calculation unit is used to calculate the order of rules in the repair rule set, specifically: using the partial order set idea to sort the rules of the locally optimizable part to generate a rule chain; for example, for function dependency, there are ψ1:X→Y, ψ2:Y→Z, ψ3:Z→G, we consider ψ1 as the pre-condition rule of ψ2, and ψ3 as the post-condition rule of ψ2. For rules that have a connection, we ensure that when a rule is used for cleaning, its pre-condition rules have all been used, and its post-condition rules cannot be cleaned before it. Find the rules with partial order relations first, form a rule set, and use the partial order set idea to generate several rule chains. When cleaning, the rules must be cleaned in the order of the rule chain. For rules that do not generate rule chains, their order does not matter.

[0053] The traditional method uses sequential execution, but the sequential execution method arranges all rules in a certain order and executes them sequentially, without considering the logical relationship between rules, resulting in rule logic conflicts. In fact, rules do not necessarily only have a front and back connection, but also have parallel rules, which are divided into selection and branching types. Therefore, we use the partial order set idea to obtain rule order calculation.

[0054] Specific implementation method three: the error data detection unit is used to determine the cause of the data error and recommend a repair method according to the error cause, and according to the error cause, the error data is sent and the repair method is sent to the error data repair unit or the constraint modification unit, specifically:

[0055] S1, obtain the data error cause and repair method selected and verified by the user as a training set, and train the probability classifier using the training set to obtain a trained classifier;

[0056] S2, inputting the error data into the trained classifier to obtain data error causes and probabilities of each repair method;

[0057] The error cause types include data itself error and constraint error;

[0058] S3, using a repair search algorithm to obtain a recommended repair method according to the probability of each repair method;

[0059] The repair methods include an edit rule, a repair rule and a Sherlock rule;

[0060] The edit rule uses the relationship master data and the interaction with the user to ensure the reliable data repair, the repair rule encodes the correct value in the rule to perform the automatic data cleaning, and the Sherlock rule can also automatically mark and repair the data;

[0061] The repair search algorithm specifically searches for possible modification conditions on the error data or the constraint, modifies the data or the constraint to make the current data as little as possible to violate the constraint, and satisfies the constraint until the constraint is satisfied. Finding a condition that satisfies the constraint indicates that a repair method is found, and all possible repair methods are found when the search is completed. The cost model is used to calculate the cost of each repair method, and the lower the cost, the higher the probability of correct search.

[0062] Different repair types have different search spaces. In the case of data itself error, only the data is searched; in the case of constraint error, only the constraint is searched; and in the case of both errors, both the data and the constraint are searched;

[0063] S4, according to the data error causes, sending the data itself error and the recommended repair method to an error data repair unit, and sending the constraint error and the recommended repair method to a constraint modification unit;

[0064] In the present embodiment, the probability classifier that predicts the type of repair is initially trained using the repairs selected and validated by the user. These repairs provide a baseline for the classifier that represents the type of modification consistent with the user's and application's preferences. As the data and constraints change, inconsistencies can arise that need to be resolved. Once the classifier is trained, it can predict the type of repair needed to handle the violation. The properties of the violation are described by computing a set of statistics, such as the number of violating tuples and the number of violating functional dependencies. The classifier generates assertions and computes the probability of each type of repair. These repair predictions are passed to a repair search algorithm that narrows the search space of repairs based on the classifier's recommendations. The repair search algorithm includes a cost model that determines which repairs are the best solution to resolve the inconsistency. The repair search algorithm recommends a set of data or constraint repairs to the user, who ultimately decides which repairs to apply. The applied repairs are then used to retrain the classifier, and the process is repeated.

[0065] Embodiment five: the constraint modification suggestion providing unit is configured to provide a modification constraint modification suggestion according to a constraint error cause, and feed back the suggestion to the rule base module, in particular:

[0066] According to the probability of the repair type, a repair search algorithm is used to recommend a set of data or constraint repairs to the user, and the data or constraint selected by the user is used to retrain the classifier;

[0067] The repair search algorithm includes a cost model that determines which repairs are the best solution to resolve the inconsistency.

[0068] The modified constraint is returned to the database to replace the original constraint. Only this step can be manually involved to decide not to replace, so it is described as a constraint modification suggestion. If there is no manual involvement, the modified constraint is directly replaced by the original constraint.

[0069] Embodiment six: a storage medium, the storage medium storing at least one instruction, the at least one instruction being loaded and executed by a processor to implement the multi-rule collaborative data cleaning system.

[0070] Embodiment seven: a multi-rule collaborative data cleaning device, the device including a processor and a memory, the memory storing at least one instruction, the at least one instruction being loaded and executed by the processor to implement the multi-rule collaborative data cleaning system.

Claims

1. A multi-rule coordinated data cleaning system, characterized in that The system comprises a rule base module, a rule management module and a data cleaning module. The rule base module comprises a rule storage unit and a rule modification unit. The rule storage unit is configured to store repair rules and send the repair rules to the rule management module. The repair rules comprise function dependency (FD), metric function dependency (MFD), numeric function dependency (NDF) and denial constraint (DC). The rule modification unit is configured to modify the repair rules stored in the rule storage unit. The modification of the repair rules comprises adding a predicate and deleting an incorrect predicate. The rule management module comprises a rule conflict detection unit, a rule repair unit and a rule order calculation unit. The rule conflict detection unit is configured to detect whether the rules being calculated for order in the rule order calculation unit are in conflict and pass the rules in conflict to the rule repair unit. The rule repair unit is configured to repair the rules in conflict and send the repaired rule set to the rule order calculation unit and send the information of the repair rules to the rule base module. The rule order calculation unit is configured to calculate the order of the rules in the repaired rule set. The rules that can be optimized locally are sorted using the idea of partial order set to generate rule chains. The rules with partial order relationship are found out first to form a rule set. The rules without rule chains are not considered for their order. The data cleaning module comprises an incorrect data detection unit, an incorrect data repair unit and a constraint modification unit. The incorrect data detection unit is configured to determine the cause of data errors and recommend a repair method according to the cause of the errors. The cause of the data errors comprises data itself error and constraint error.

2. The multi-rule coordinated data cleansing system of claim 1, wherein: The incorrect data repair unit is configured to repair the incorrect data according to the recommended repair method and the rule order calculated by the rule order calculation unit to obtain cleaned data. The constraint modification unit is configured to determine whether to repair the incorrect data using the repair method recommended by the incorrect data detection unit and send the repair method of the repair constraint to the rule base module.

3. The multi-rule coordinated data cleansing system of claim 2, wherein: The system further comprises a rule mining unit.

4. The multi-rule coordinated data cleansing system of claim 3, wherein: The rule mining unit is configured to learn rules from the cleaned data set to obtain new rules and save the new rules in the rule storage unit for the next round of data cleaning.

5. The multi-rule coordinated data cleansing system of any one of claims 1 to 4, wherein: The rule conflict comprises predicate inclusion relationship and rule repetition. The repair method of the rules comprises deleting rules and modifying rules. The specific process of determining the cause of data errors and recommending a repair method by the incorrect data detection unit comprises the following steps. S1: Obtain the data error cause and repair method selected and verified by a user as a training set and train a probability classifier using the training set to obtain a trained classifier. S2: Input the incorrect data into the trained classifier to obtain the probability of each repair method for the data error cause. The error cause types include: data itself error and constraint error; S3, adopting a repair search algorithm to obtain a recommended repair mode according to the probability of each repair mode.

6. The multi-rule coordinated data cleansing system of claim 5, wherein: The repair modes include: an editing rule, a repair rule and a Sherlock rule.

7. A storage medium characterized by: The storage medium has at least one instruction stored therein, and the at least one instruction is loaded and executed by the processor to realize the functions implemented by the multi-rule coordinated data cleaning system according to any one of claims 1 to 6.

8. A data cleansing device with multi-rule coordination, characterized in that, The device includes a processor and a memory, and the memory has at least one instruction stored therein, and the at least one instruction is loaded and executed by the processor to realize the functions implemented by the multi-rule coordinated data cleaning system according to any one of claims 1 to 6.

Images