Unsupervised data automatic cleaning method

Abstract

The invention discloses an unsupervised data automatic cleaning method. The method comprises the following steps of: A, learning a data model: learning a dependency relationship among attributes fromoriginal data which may contain invalid data, and finding out a hidden non-absolute or relatively weak dependency relationship to obtain a data model represented in a Bayesian network form; B, generating a data cleaning rule; generating a data cleaning rule after obtaining the original data or a complete data model sampled by the original data, and specifically generating a predicate and a first-order predicate rule; c, generating a Markov logic network based on the predicates generated in the step B and the first-order predicate rule; and D, generating an inference rule based on the Markov logic network generated in the step C and cleaning data based on an inference result. According to the method, the data quality of each business system of a company can be effectively improved under the condition that a large amount of manpower and material resources are not consumed, and a management layer is helped to make correct decisions.

Description

technical field [0001] The invention relates to the technical field of data management, in particular to an unsupervised automatic data cleaning method. Background technique [0002] Data in the real world usually needs to be cleaned (the data that needs to be cleaned is defined as dirty data below), such as may contain inconsistent, noisy, incomplete or repeated values. In business, erroneous data can cause great financial losses. For example, wrong customer information may lead to wrong delivery of the goods purchased by the company, which not only increases the delivery cost of the company, but also has a relatively large negative impact on the image of the company for a long time. [0003] Among the existing data cleaning methods, some methods require heavy manual participation in the data cleaning process, such as providing suggestions for cleaning or confirming repairs, etc.; although some methods do not require manual participation in the cleaning process, they need ...

AI Technical Summary

Problems solved by technology

In business, erroneous data can cause large financial losses

For example, wrong customer information may lead to wrong delivery of the goods purchased by the company, which not only increases the delivery cost of the company, but also has a relatively large negative impact on the image of the company for a long time

[0003] Among the existing data cleaning methods, some methods require heavy manual participation in the data cleaning process, such as providing suggestions for cleaning or confirming repairs, etc.; although some methods do not require manual participation in the cleaning process, they need to be formulated in advance. rule

Existing data cleaning methods are not applicable when data rules are unknown or labor costs are unaffordable

Images