Distributed data cleaning system and method based on data analysis

Abstract

The invention relates to a distributed data cleaning system based on data analysis, which comprises a multivariate heterogeneous database, a storage user model, metadata elements and source data elements corresponding to the metadata elements, and at least one processing unit operable to: extract a user model, metadata elements, and source data elements of the multivariate heterogeneous database;select initial metadata elements; select at least more than one data attribute item of the user model as a relationship parameter and a preset weighted value corresponding to the relationship parameter, and extract a metadata set Q; clean the metadata elements in the metadata set Q. According to the invention, multiple categories of metadata sets can be screened on the basis of selection of the user model matching relationship parameters, the targeted cleaning rules can be selected for each category of metadata sets, the cleaning speed is increased, high flexibility and practicability are achieved, and the controllability of data cleaning is improved.

Description

technical field [0001] The invention belongs to the technical field of artificial intelligence data processing, and in particular relates to a distributed data cleaning system based on data analysis. Background technique [0002] A heterogeneous database system is a collection of related multiple database systems, which can realize data sharing and transparent access. Several database systems already existed before joining the heterogeneous database system. They have their own database management systems and external database systems. Each component has its own autonomy. While realizing data sharing, each database system still has its own application characteristics, integrity control and security control; [0003] ----The goal of the heterogeneous database system is to realize the merging and sharing of data information resources, hardware equipment resources and human resources between different databases. The key point is to establish a global data model or a global exte...

AI Technical Summary

Problems solved by technology

[0005] On the basis of heterogeneous databases, the diversification of data sources is added to form a multivariate heterogeneous database. Compared with ordinary heterogeneous databases, multivariate heterogeneous databases have more data characteristics of diversified sources, so the degree of multivariate complexity of data is more complex. On the upper level, if such a multivariate heterogeneous database is cleaned directly by cleaning tools, the predetermined cleaning rules cannot be generally applied to the data in the multivariate heterogeneous database. The cleaning of complex and multivariate data will lead to loss of data attributes and damage The integrity of the data, the cleaning speed is slow, and the cleaning effect is also unpredictable

Images