ETL (Extract Transform Load)-based data optimization method and equipment

Abstract

The embodiment of the invention provides an ETL (Extract Transform Load)-based data optimization method and ETL-based data optimization equipment. The method comprises the following steps of: previously arranging a plurality of data processing units according to a data extract, transform and load process ETL; previously setting a communication mechanism for the data processing units; acquiring instruction information including source data input by a user; constructing a data processing flow corresponding to the instruction information according to the source data; and optimizing the data processing flow according to the data processing units and the preset communication mechanism. By previously setting the data processing units and the communication mechanism, simplified optimization, branch parallel optimization and parallel optimization between records of data are realized, the processing efficiency of data optimization is increased, and hardware resources are saved.

Description

technical field [0001] The present invention relates to data processing technology, in particular to the processing technology in the process of data migration and conversion, specifically an ETL-based data optimization method and equipment. Background technique [0002] In the process of enterprise information construction, it often involves processing a large amount of scattered and heterogeneous data, and the process of data extraction, transformation, and loading (Extract-Transform-Load, ETL) is an essential part of the process. In the prior art, there are mainly the following ways to realize the ETL process: [0003] 1. Hardcoded [0004] Hard coding is an independently running program compiled by high-level languages ​​(such as C, C++) or scripts, or a dynamic link library embedded in the ETL framework. The advantage of this method is flexibility, as long as the data type and processing logic supported by the programming language used can be realized in the ETL proce...

AI Technical Summary

Problems solved by technology

[0011] These implementations usually have specific data processing logic for specific external data and loading targets, and these logics are solidified in an ETL program; so such implementations can only be used in specific ETL scenarios, and in other scenarios , it is impossible to use the previous results all the way to the new scene, or to reuse them, and can only complete a new realization for the new specific scene;

[0012] 2. Poor maintainability and usability

Maintaining such an ETL process involves the management of a large number of "scripts" or "codes", which is very confusing, and this puts forward quite high requirements for the technical level of the implementers, otherwise it will be difficult to achieve

[0014] 3. No metadata management

[0015] Some technical solutions, such as "hard coding" and "stored procedures", lack the process of storing and managing metadata, which makes it very difficult to run, track and analyze the ETL process, as well as maintain and adjust later

[0016] 4. Inefficiency

With the development, the amount of data in various industry systems is increasing. The ETL process usually has to deal with massive data, and the real-time requirements are getting higher and higher. Therefore, higher and higher requirements are put forward for the efficiency of the ETL process. Requirements, traditional processing methods can no longer meet the requirements

[0018] 5. It takes up a lot of resources

This leads to a large amount of memory, CPU, and bandwidth resources being occupied

Images