A distributed heterogeneous data cleaning system based on visual management

Abstract

The invention belongs to the technical field of data processing, and discloses a distributed heterogeneous data cleaning system based on visual management. The source data cleaning modules are set in parallel, and each heterogeneous data cleaning module can run in parallel on an independent server or in the heterogeneous data cleaning process, and the data cleaning visualization management terminal performs unified management and scheduling on the heterogeneous data cleaning modules; The cleaning module includes an original database, a test target library, a database reading preprocessing middleware, an ETL conversion module, a data checking middleware and a log module. The invention can realize distributed heterogeneous data cleaning and improve cleaning and conversion processing speed.

Description

technical field [0001] The present invention relates to the technical field of data processing, and more particularly, to a distributed heterogeneous data cleaning system based on visual management. Background technique [0002] Data cleaning is the core of government intensification, data warehouse and data mining. It is the basis of government data migration. The complexity of heterogeneous data leads to slow data cleaning and error-prone. Due to the wide range of data sources in ETL technology, These data sources may be stored on different hardware or different operating systems, so there will inevitably be some "dirty data" in these data sources. The purpose of data cleaning is to find and eliminate those data that do not meet the specifications, which is important for ensuring data. The high quality of the data warehouse has a very important impact on the correctness of the data warehouse and subsequent data mining and decision analysis. [0003] The heterogeneity of d...

AI Technical Summary

Problems solved by technology

[0003] The heterogeneous sources of data lead to difficulties in data standardization and structuring, and it is difficult for a set of codes to synchronously process data from different sources. For example, differences in the structure of Oracle and Mysql databases lead to different types of data such as dates that require different logical processing and conversion into target data. During the conversion, dirty data may be written into the target database due to bugs in programs such as ETL, such as spelling mistakes, repeated information, missing data, etc., which directly lead to data quality that cannot meet the requirements. In addition, traditional cleaning methods cannot be parallelized on a large scale Handle and manage the data processing process, the data needs to be processed serially, and the cleaning and conversion processing speed is slow, which are unavoidable in the traditional processing methods

Images