Distributed heterogeneous data cleaning system based on visual management

Abstract

The invention belongs to the technical field of data processing. The invention discloses a distributed heterogeneous data cleaning system based on visual management. The heterogeneous data cleaning system comprises a plurality of heterogeneous data cleaning modules, a data cleaning visual management end and a formal library, wherein the plurality of heterogeneous data cleaning modules are arrangedin parallel; each heterogeneous data cleaning module can run in parallel in an independent server or a heterogeneous data cleaning process; the data cleaning visual management end performs unified management scheduling on the heterogeneous data cleaning modules; and the heterogeneous data cleaning module comprises an original database, a test target library, database reading preprocessing middleware, an ETL conversion module, data checking middleware and a log module. The distributed heterogeneous data cleaning system can achieve distributed heterogeneous data cleaning and improve the cleaning conversion processing speed.

Description

technical field [0001] The present invention relates to the technical field of data processing, and more specifically, the present invention relates to a distributed heterogeneous data cleaning system based on visual management. Background technique [0002] Data cleaning is the core of government intensification, data warehouse and data mining. It is the basis of government data migration, and the complexity of heterogeneous data makes data cleaning slow and error-prone, and because the data sources in ETL technology are very extensive, These data sources may be stored on different hardware or different operating systems, so it is inevitable that there will be some "dirty data" in these data sources. The high quality has a very important impact on the correctness of the data warehouse and subsequent data mining and decision analysis. [0003] The heterogeneous sources of data lead to difficulties in data standardization and structuring, and it is difficult for a set of cod...

AI Technical Summary

Problems solved by technology

[0003] The heterogeneous sources of data lead to difficulties in data standardization and structuring, and it is difficult for a set of codes to synchronously process data from different sources. For example, differences in the structure of Oracle and Mysql databases lead to different types of data such as dates that require different logical processing and conversion into target data. During the conversion, dirty data may be written into the target database due to bugs in programs such as ETL, such as spelling mistakes, repeated information, missing data, etc., which directly lead to data quality that cannot meet the requirements. In addition, traditional cleaning methods cannot be parallelized on a large scale Handle and manage the data processing process, the data needs to be processed serially, and the cleaning and conversion processing speed is slow, which are unavoidable in the traditional processing methods

Images