Method and apparatus for implementing ETL scheduling

Abstract

The invention discloses a method for scheduling ETL, comprising: determining the triggering mode, operation sequence and mutual dependency relationship of each subtask flow included in the task flow based on preset configuration file aiming at an optional task flow when implementing plural task flows included in the ETL scheduling; triggering corresponding subtask flows in turn according to the set triggering mode and implementing the triggered subtask flow according to set sequence, wherein after determining that at least a subtask flow is implemented, starting to implement other subtask flows depending on at least a subtask flow and other triggered subtasks based on the dependency relationship between subtask flows. Thus, subtask flow in each task flow has clear service logic and service function so as to effectively enhance the implementing efficiency of the ETL scheduling flow. The invention also discloses an apparatus for implementing ETL scheduling.

Description

technical field [0001] The present application relates to the field of computers, in particular to a process control method and device. Background technique [0002] Data Warehouse (DW) is a subject-oriented, integrated, relatively stable data collection that reflects historical changes and is used to support management decisions. A data warehouse is an independent data environment, and data extraction, transformation and loading (Extraction-Transformation-Loading, ETL) is an important part of building a data warehouse. [0003] ETL is used to extract data from distributed and heterogeneous data sources (for example, relational data, flat data files, etc.) Loaded into the data warehouse, so that the built data warehouse becomes the basis of online analytical processing and data mining. Technically, ETL mainly involves several aspects such as association, transformation, increment, scheduling and monitoring. Usually, the data in the data warehouse does not require real-tim...

AI Technical Summary

Problems solved by technology

Moreover, the JBPM engine core adopts the token-based sequential transfer method, which cannot meet the actual application requirements in ETL process scheduling such as "task fallback" and "task jump forward".

[0007] On the other hand, the process description language (JPDL) used by the JBPM engine cannot describe the dependency relationship between parent and child processes, and can only rely on programmers to extend the node class in the JAVA program to achieve similar functions. In this way, in the task program execution In the process, it is necessary to repeatedly create sub-processes by extending the class, which seriously reduces the execution efficiency of the parent process.

At the same time, the visual management of the overall ETL scheduling process is not provided in the JBPM process designer, which increases the technical difficulty of process design and later maintenance

Images