Data preprocessing method for heterogeneous data sources

Abstract

The invention provides a data preprocessing method for heterogeneous data sources. The method comprises the following steps: reading heterogeneous data from a plurality of heterogeneous data sources; preprocessing the heterogeneous data based on a preprocessing rule library to obtain normalized data; storing the normalized data in a database to be applied to data integration, data mining and / or online analysis processing of enterprises. By adopting the data preprocessing method, politics and law service data can be shared; the method has high generality, and is easy to extend; three times of progressive preprocessing are performed on the data, and a processing process can be traced, so that a processing rule is easy to modify, the data processing efficiency and processing accuracy are increased, and the data can be stored in a unified way based on an error log modification and extraction rule to provide services to the outside.

Description

technical field [0001] The invention relates to the technical field of data processing, in particular to a data preprocessing method for heterogeneous data sources. Background technique [0002] When building an information system, even with good design and planning, there is no guarantee that the quality of the stored data will meet the user's requirements in all cases. There is a need for metadata to represent data quality. The four indicators of data consistency, correctness, completeness and minimality are defined in a formal method. According to the extent to which these indicators are satisfied in the information system, the data quality analysis and model in data engineering are put forward, and it is believed that there are many candidate data quality measurement indicators. Users should choose some of them according to the needs of the application. Indicators are divided into two categories: data quality indicators and data quality parameters. The former is objec...

AI Technical Summary

Problems solved by technology

They all provide some data preprocessing functions, but there are also great limitations: (1) lack of versatility, such as DTS is only applicable to Windows platform, and can only use ODBC to connect various data sources; (2) lack of easy Usability and scalability, although OWB has a good effect on the preprocessing of names and addresses, but the process is too cumbersome and difficult to use, and it is difficult for users to write their own customized programs to adapt to data preprocessing in specific fields ;(3) The preprocessing function is limited, and only certain types of "dirty data" can be preprocessed

The existing data preprocessing method only preprocesses the data once, but cannot preprocess the data progressively and retrospectively. The data processing efficiency is low, the accuracy is poor, and the processing rules are difficult to modify, let alone based on wrong Log modification and extraction rules and unified storage of data to provide services

Images