Universal ETL tool process model generation method based on workflow

A process model and workflow technology, applied in the field of data processing, can solve problems such as poor reusability and poor versatility, and achieve the effect of reducing development costs and improving development efficiency

Active Publication Date: 2017-11-24
NANJING GUODIAN NANZI POWER GRID AUTOMATION CO LTD
View PDF4 Cites 12 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

However, no matter which one is mentioned above, there are several

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Universal ETL tool process model generation method based on workflow
  • Universal ETL tool process model generation method based on workflow
  • Universal ETL tool process model generation method based on workflow

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0043] In order to make the object, technical solution and advantages of the present invention more clear, the present invention will be further described in detail below in conjunction with the examples. It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention.

[0044] The application principle of the present invention will be described in detail below in conjunction with the accompanying drawings.

[0045] A process model generation method based on a workflow-based general ETL tool, comprising the following steps:

[0046] S1: On the basis of the traditional ETL conceptual model, establish a workflow-based ETL model tool, including the job master control module and the extraction, conversion, and loading modules;

[0047] The job master control module is used to complete the scheduling work of all ETL jobs, realize the timing, queuing, parallel scheduling, process control, and lo...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a universal ETL tool process model generation method based on workflow. Abstraction is conducted based on an ETL conceptual model, control of whole ETL and regular execution and dispatching of tasks are completed, the ETL model based on the workflow is built, a directed graph of an ETL process is built based on the ETL model, and through eight State transformation modes of integration transformation, decomposition, mergence, serialization, parallelization, restore point addition, division and replication, the ETL process is transformed into a new ETL process with a higher execution efficiency through appropriate transformation. According to the ETL system conceptual model different from traditional ETL models based on the workflow, steps of underlying concrete data extraction, transformation and loading are shielded, and when business is changed, a user only needs to modify an ETL flow chart. The development efficiency is greatly improved, the development cost is reduced, and popularization of data warehouses and data integration are really achieved.

Description

technical field [0001] The invention belongs to the technical field of data processing, and in particular relates to a method for generating a process model of a workflow-based general ETL tool. Background technique [0002] In recent years, with the rise of e-commerce and the rapid development of the information technology industry, the existence of information islands has become more and more common. The global annual investment in application system integration and data integration is 300 billion US dollars, and this data is still growing at an annual rate of 28%. At the same time, the increasingly hot system integration and data warehouse market makes more and more Many ETL tools have emerged. [0003] Traditional ETL design methods and ETL tools have become less and less suitable for changing needs, because traditional ETL design and development not only require familiarity with the structure of data tables, rules for processing data, and very good programming skills, ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F17/30
CPCG06F16/254
Inventor 周文欢张天宝徐衍吴同凯方鹏初日辉徐苏君蔡雷鸣花子岚陈梵
Owner NANJING GUODIAN NANZI POWER GRID AUTOMATION CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap