ETL (extract-transform-load) data processing method, device and system

A data processing and data technology, applied in electrical digital data processing, special data processing applications, instruments, etc., can solve the problems of reduced data processing performance, affecting data processing rate, unable to provide ETL data processing horizontal expansion capability, etc. Reliability, speed and efficiency gains

Inactive Publication Date: 2013-12-18
杭州勒卡斯广告策划有限公司
View PDF0 Cites 21 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0006] The technical problem to be solved in this application is to provide a kind of ETL data processing method, device and system, in order to solve the ETL data processing scheme in the prior art, in order to support the data processing of multiple data sources, need multiple and to be processed The corresponding driver / import / export tool support of the data source, thus reducing the speed of ETL data processing and technical problems affecting the efficiency of ETL data processing; and used to solve the existing ETL data processing solutions that cannot provide ETL data The horizontal expansion capability of processing makes the data processing performance of single-point ETL tools significantly lower when faced with massive data, which affects the technical problem of data processing rate

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • ETL (extract-transform-load) data processing method, device and system
  • ETL (extract-transform-load) data processing method, device and system
  • ETL (extract-transform-load) data processing method, device and system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0060] The following will clearly and completely describe the technical solutions in the embodiments of the application with reference to the drawings in the embodiments of the application. Apparently, the described embodiments are only some of the embodiments of the application, not all of them. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the scope of protection of this application.

[0061] refer to figure 1 , which shows a flow chart of Embodiment 1 of an ETL data processing method provided by the present application, the method may include the following steps:

[0062] Step 101: Obtain source data in a data source, and convert the source data into comma-separated value CSV format source data.

[0063] Wherein, the data source includes one or more of multiple (N kinds) data sources such as data source MySQL, data source Oracle, data source PostgreSQL, data sou...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses ETL (extract-transform-load) data processing method, device and system. The method includes: acquiring source data from a data source, and converting the source data into source data in the CSV (comma-separated value) format; establishing an ETL task according to the source data in the CSV format, and triggering an ETL tool to run the ETL task; transmitting running results of the ETL task to a target data warehouse. The data, to be processed, of the data source is uniformly converted in the CSV format and is then extracted, transformed and loaded; loading results are disposed in the target data warehouse; in case of varieties of data sources, adding a conversion method of CSV-formatted data is required only; accordingly, setting of various drive/import/export tools causes decreased processing speed of ETL data in the prior art is avoided, and processing speed and efficiency of the ETL data can be higher.

Description

technical field [0001] The present application relates to the field of data integration middleware, in particular to an ETL data processing method, device and system. Background technique [0002] Information is an important resource of modern enterprises and the basis for enterprises to use scientific management and decision-making analysis. Enterprises use ETL (Extraction-Transformation-Loading, extraction, transformation, loading) technical means to extract heterogeneous data from different data sources to the temporary middle layer for cleaning, transformation, and integration processing, that is, data extraction, data conversion, and data Loading and other processing, and put the processed data into the target data warehouse. [0003] At present, data processing tools such as DataStage are generally used to realize ETL data processing. The DataStage is a suite of integrated tools designed to simplify and automate the process of data extraction, transformation, and mai...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F17/30
Inventor 李晨
Owner 杭州勒卡斯广告策划有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap