Massive multi-source heterogeneous data ETL method and system supporting interface adaptation

Abstract

The invention discloses a massive multi-source heterogeneous data ETL method and system supporting interface adaptation. The method comprises a data extraction step of setting basic information of data sources and a target database, adaptively matching corresponding ETL tools for different data sources and performing parameter setting on the ETL tools, a data conversion step of finishing ETL operation control execution and scheduling management, performing buffer storage and management on extracted data and finishing processing of data cleaning and conversion and the like, a data loading stepof carrying out quality inspection on converted data objects and updating and loading the data inspected to be correct into the target database according to table structure output defined by a data model, and a data monitoring step of performing monitoring management on an ETL operation execution process, an operation resource usage condition and a system operation condition. The proper ETL tool is adaptively matched; the extraction and conversion of massive data are achieved; and efficient execution and orderly management of ETL operation are realized.

Description

technical field [0001] The invention relates to the field of ETL management, in particular to an ETL method and system for massive multi-source heterogeneous data supporting interface adaptation. Background technique [0002] At present, the industry has accumulated a large amount of data, and the volume, type and change of data are increasing rapidly. However, big data has not been fully utilized, and the huge value contained in it has yet to be tapped. Big data often has multi-source heterogeneous characteristics and comes from different and dispersed business systems. There are structured data, semi-structured data, unstructured data and other types, which are difficult to extract and convert into required data. In the big data environment, the data presents the characteristics of large capacity, multiple styles, and frequent interactions. With the continuous increase of collected data, the data processing logic is gradually complicated, and it is faced with the transmiss...

AI Technical Summary

Problems solved by technology

In the big data environment, the data presents the characteristics of large capacity, multiple styles, and frequent interactions. With the continuous increase of collected data, the data processing logic is gradually complicated, and it is faced with the transmission efficiency of massive multi-source heterogeneous data between different databases. question

[0003] Traditional ETL tools are expensive, highly dependent on specific business, and have a centralized architecture, that is, design, operation and management are all concentrated on one server, and the requirements for hardware are very high

In the traditional ETL management mode, generally according to the attributes of the source database and the target database, manually determine the ETL tool, and set the ETL task process, set parameters, and start the task. This manual ETL management mode has a complicated process and consumes a lot of manpower and time. And it cannot meet the ETL operation requirements of massive multi-source heterogeneous data

Images