ETL (Extract Transform and Load) scheduling method and device for data processing and control logic separation

A technology of control logic and data processing, applied in the field of data processing, can solve problems such as increasing the complexity of scheduling ETL and reducing the efficiency of ETL scheduling, and achieves the effects of easy popularization, simple and convenient implementation, and wide application range

Active Publication Date: 2018-07-10
CHINA MOBILE SUZHOU SOFTWARE TECH CO LTD +1
View PDF4 Cites 1 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

This will greatly increase the complexity of scheduling ETL and reduce the scheduling efficiency of ETL

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • ETL (Extract Transform and Load) scheduling method and device for data processing and control logic separation
  • ETL (Extract Transform and Load) scheduling method and device for data processing and control logic separation
  • ETL (Extract Transform and Load) scheduling method and device for data processing and control logic separation

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0045] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention.

[0046] In a specific embodiment of the invention, figure 2 It is a schematic flow diagram of an implementation flow of an ETL scheduling method in which data processing and control logic are separated in an embodiment of the present invention. Such as figure 2 As shown, the ETL scheduling method for separating data processing and control logic may include the following steps:

[0047] Step 201, searching for the current ETL task in all published ETL tasks.

[0048]In a specific embodiment of the present invention, all published ETL tasks may be pre-saved in a pre-set Redis database. Specifically, all published ETL tasks can be saved in the preset Redis database in the form of "task identifier Key + task content Value". When scheduling the current ETL task, you can first ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The embodiment of the invention discloses an ETL (Extract Transform and Load) scheduling method and device for data processing and control logic separation. The method comprises the following steps that: in all released ETL tasks, searching a current ETL task; when the current ETL task is not searched in all released ETL tasks, obtaining a current control flow corresponding to the current ETL task, wherein the current control flow at least comprises a current data flow corresponding to the current ETL tasks; and according to the current control flow, scheduling the current ETL task. By use ofthe method, the scheduling process of the ETL task can be simplified, and in addition, the scheduling efficiency of the ETL task can be improved.

Description

technical field [0001] The invention relates to data processing technology, in particular to an ETL scheduling method and device for separating data processing and control logic. Background technique [0002] With the development of big data technology, there are more and more distributed data storage systems. Big data applications generally need to integrate multiple different data storage systems to build data warehouses for different applications. The traditional data extraction, transformation and loading ETL refers to the process of extracting, transforming and loading the data in the data source to the data warehouse in the process of building the data warehouse. [0003] figure 1 It is a schematic diagram of the composition structure of the ETL task in the prior art. Such as figure 1 As shown, the data processing process of the ETL task is: first read the basic object configuration in the configuration database, extract data from the source database in turn accordi...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F17/30
CPCG06F16/254
Inventor 胡国靖王国飞李光瑞汪远航罗志成
Owner CHINA MOBILE SUZHOU SOFTWARE TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap