Method for realizing ETL conversion processing by utilizing programmable function expression based on XML description in big data scene

Abstract

The invention relates to a method for realizing ETL (Extensible Transform and Load) conversion processing by utilizing a programmable function expression based on XML (Extensive Makeup Language) description in a big data scene, which comprises the following steps of: loading a control file, and segmenting the type of a data file; loading the data file according to a preset batch amount, traversingdata lines, and processing the data lines one by one; performing data analysis; carrying out field type verification; and selecting a control file, operating the data and outputting data, and storingthe data output data in various storage systems. By the adoption of the method for achieving ETL conversion processing through the programmable function expression based on XML description in the bigdata scene, data analysis work is submitted to XML analysis control, different configuration files are loaded according to different logistics enterprises, and different logistics data can be cleanedinto unified standard data. According to the technical scheme, a large amount of program encoding work is reduced, different data can be controlled and maintained, the performance is close to the excellent native performance, and the data processing performance is improved.

Description

technical field [0001] The present invention relates to the field of computer technology, in particular to the field of big data processing, and specifically refers to a method for implementing ETL conversion processing by using programmable functions based on XML description in a big data scene. Background technique [0002] ETL is the abbreviation of Extract-Transform-Load in English, which is used to describe the process of extracting, transforming, and loading data from the source to the destination. ETL is an important part of building a data warehouse. Users extract the required data from the data source, after data cleaning and processing, and finally load the data into the target data warehouse according to the pre-defined model to do various business intelligence. analysis or for master data management systems. [0003] Specifically: [0004] Data extraction: read data from various original business systems or unstructured documents; [0005] Data conversion: con...

AI Technical Summary

Problems solved by technology

[0014] 1. The process of data import and SQLETL extraction consumes a lot of IO performance and computing resources;

[0015] 2. Unable to adapt to big data scenarios. Most databases are stored in separate databases and tables. When faced with the need to store billions or more data, they are often incapable;

[0016] 3. The expressive power of SQL is insufficient, and it cannot handle some complex logic;

[0017] 4. From the basic data warehouse to the theme database, it often needs to go through multiple associations and multiple cleaning services, and the intermediate tables take up a lot of storage space; the special architecture implemented by high-level programming language cannot be reused, and can only solve the problem in specific scenarios. ETL requirements

Images