Method for processing multiple precursors and multiple subsequent nodes in Oozie workflow

A workflow and successor node technology, applied in the direction of electrical digital data processing, program control design, multi-program device, etc., can solve the problems of multiple successors, multiple predecessors of task nodes, etc.

Pending Publication Date: 2020-01-03
INSPUR SOFTWARE CO LTD
View PDF3 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0010] In order to solve the problem of multiple predecessors and multiple successors of task nodes and realize the free construction of Oozie's DAG process, the present invention proposes a processing method for multiple predecessors and multiple successor nodes in the Oozie workflow

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Method for processing multiple precursors and multiple subsequent nodes in Oozie workflow
  • Method for processing multiple precursors and multiple subsequent nodes in Oozie workflow
  • Method for processing multiple precursors and multiple subsequent nodes in Oozie workflow

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0037] In this embodiment, oozie 4.3.1 is adopted, and the element type is added to the xsd of oozie-client, a sub-project of oozie. Since there are many xsd versions of oozie, oozie-workflow-0.4.xsd is selected here.

[0038] The oozie-core project of Oozie defines MultiNextNodeDef and MultiPrevNodeDef, the MultiNextNodeDef is a multi-successor task processing node, and the MultiPrevNodeDef is a multi-precursor task processing node.

[0039] (1) Code 1 defines the element type MULTI_PREV and MULTI_NEXT defines the element type MIX_IN

[0040]

[0041]

[0042] (2) Code 2 declares MULTI_PREV and MULTI_NEXT elements

[0043]

[0044]

[0045] (3) Finally, add mix-in to workflow-app, as shown in code 3.

[0046] Code 3 adds MULTI-PREV and MULTI-NEXT elements to workflow-app

[0047]

[0048]

[0049] In the MultiNextNodeDef, the executor is defined as MultiNextActionExecutor and the internal class SignalXCommand, and the SignalXCommand is used to add newly star...

Embodiment 2

[0060] In this embodiment, taking two main task nodes (Task A1 and Task A2), and each main task node has two sub-task nodes (Task B1~Task B4) as an example, the traditional Oozie workflow and the Oozie The workflow of the processing method of multiple predecessors and multiple successor nodes in the workflow is compared and verified.

[0061] attached figure 1 It is a schematic diagram of the traditional Oozie workflow. In the figure, although Fork / Join appear in pairs, but the two tasks separated by TaskA1, Task B1 is merged into Join J1, and Task B2 is merged into Join J2; any node in Task B1~TaskB4 will be punished when the execution is completed. , so there is no guarantee that Task B1 to Task B4 will be executed successfully in this process.

[0062] attached figure 2 It is a workflow schematic diagram of the processing method of multiple predecessors and multiple successor nodes in the Oozie workflow. In the figure, the MULTI-NEXT node is used to replace the Fork no...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention particularly relates to a method for processing multiple precursors and multiple subsequent nodes in an Oozie workflow. The method comprises the following steps that a multi-subsequent task processing node is arranged behind each main task node, and sub-task nodes of each main task node are arranged behind multi-subsequent task processing nodes; a multi-precursor task processing node is arranged in front of an end node, and all sub-task nodes needing to be combined are connected to the same multi-precursor task processing node. Smooth execution of the multiple precursor task nodes and the multiple subsequent task nodes can be ensured. Original functions of the Oozie can be compatible, normal execution of an original Oozie workflow is guaranteed, and the problem that multiple precursor task nodes and multiple subsequent task nodes cannot be smoothly executed in the Oozie building workflow is solved.

Description

technical field [0001] The invention relates to the technical field of big data processing, in particular to a method for processing multiple predecessors and multiple successor nodes in an Oozie workflow. Background technique [0002] Oozie is a service component based on a workflow engine, which is specially designed for executing HadoopMap / Reduce tasks or Pig jobs through process orchestration. Oozie realizes the process execution of tasks by organizing many executable tasks (Map / Reduce, Pig, etc.) into a workflow in a DAG manner. [0003] Oozie's workflow is defined through hPDL language (an XML process definition language similar to JBOSS JBPM). When Oozie executes a task, it will submit the task to a remote system (such as Hadoop cluster, etc.) for execution; when the task is executed, it will obtain the execution status of the task through the callback of the remote system; then, Oozie will continue to execute The next process task. [0004] There are two types of ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F9/50
CPCG06F9/5027G06F2209/508
Inventor 张旭赵志宏周庆勇王建华
Owner INSPUR SOFTWARE CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap