A big data-oriented etl method and device

Abstract

The invention provides a big-data-oriented ETL (Extraction-Transformation-Loading) method and device. The method comprises the following steps: obtaining a parameter configuration file; according to the parameter configuration file, carrying out parameter configuration on data to be synchronized; and through an ETL method, synchronizing the data subjected to the parameter configuration. A data importing and exporting function among various data sources can be realized, and various data transformation operations based on hdfs data is realized through the simple configuration of a visual interface.

Description

technical field [0001] The present invention relates to the technical field of ETL, in particular to a big data-oriented ETL method and device. Background technique [0002] In recent years, the Internet has developed rapidly, social networks have gradually matured, cloud computing and Internet of Things applications have become more abundant, and software and hardware technologies have continued to innovate. These changes have enabled governments and enterprises to obtain more data than ever before, and have Computing power and methods to analyze these massive amounts of data. At present, big data has become an economic asset with great value like currency and gold. By analyzing and mining the potential information contained in these data, it can promote the development of enterprises and society. This information contains huge commercial value, but the meaningful data for data mining only accounts for a small part of it. Therefore, in order to avoid wasting unnecessary ti...

AI Technical Summary

Problems solved by technology

However, the use of these open source projects requires a certain programming foundation and there is no convenient UI interface, which is still inconvenient for data analysts.

[0005] Although traditional ETL tools, such as kettle and talend, have made some expansions in the field of bigdata and provided a graphical drag-and-drop interface, the cost of learning and using is high, and the execution efficiency of departmental components in a big data environment is not ideal

Moreover, the ETL open source tools currently used in the industry do not have a visual graphical interface or the graphical interface is inconvenient to use, and the cost of learning and using it for non-technical personnel is relatively high.

However, the existing big data data extraction tools like Sqoop have lengthy execution scripts, require operators to have a certain basis for use, and have high requirements for operators.

In addition, for traditional ETL tools, although kettle and talend have made some expansions in the field of bigdata and provided a graphical drag-and-drop interface, the cost of learning and using is high, and the execution efficiency of departmental components in a big data environment is not ideal

Images