An automatic data cleaning method based on DeepDive

Abstract

The invention discloses an automatic data cleaning method based on DeepDive, and the method comprises the steps: (1) comparing an original data scale with a set threshold, and carrying out the randomsampling when the scale exceeds the threshold, so as to obtain sampling data; (2) learning from the original data or the sampling data to obtain a Bayesian network between attributes; (3) converting the Bayesian network obtained by learning into a first-order predicate logic rule; (4) calculating the weight of a first-order predicate logic rule by using a mutual information theory, and convertingthe first-order predicate logic rule with the weight into a Markov logic network; (5) generating a DeepDive rule based on the Markov logic network; (6) carrying out probability reasoning of error/missing data based on DeepDive, and obtaining the probability of taking different values by the attributes of the tuples; And (7) using the reasoning result for cleaning the original dirty data. The method can be used for automatically cleaning the data without existing data quality modes/rules and manual intervention, and the data cleaning efficiency and quality can be effectively improved.

Description

technical field [0001] The invention relates to the technical field of data processing, in particular to an automatic data cleaning method based on DeepDive. Background technique [0002] Dirty data is very common in the real world, and cleaning of dirty data is a long-standing problem. In the era of big data, the importance of data cleaning is even more prominent. Detecting errors from dirty data and repairing them is one of the main challenges in the field of data analysis. Poor data quality may lead to greatly reduced accuracy of analysis results. Generally speaking, data cleaning consists of two stages: the first stage is error detection, which detects errors or abnormal data contained in it; the second stage is error repair, which repairs errors or abnormalities contained in the data. Most of the existing data cleaning methods use the existing constraint rules or patterns for error detection and repair, which mainly have the following limitations: (1) Existing methods...

AI Technical Summary

Problems solved by technology

[0004] In order to solve the problem of automatic data cleaning in the absence of existing patterns / rules and manual participation, the present invention proposes an automatic data cleaning method based on DeepDive, thereby solving the aforementioned problems in the prior art

Images