A method for implementing a highly configurable workflow with high degree of freedom

Abstract

The present invention discloses a method for implementing a highly configurable workflow with high degree of freedom, including functions such as custom task scheduling attributes, task dependencies, span cycle dependencies, message configuration, and task re-running. The data is analyzed and collected through Hadoop and Spark technologies to achieve complex offline computing scheduling, providing efficient and convenient support for workflow scheduling methods.

Description

Technical Field

[0001] The present invention relates to the field of computers, and in particular to a method for realizing a highly configurable workflow with a high degree of freedom. Background Art

[0002] As people's lives become more digital and intelligent, data analysis plays an increasingly important role. Due to the diversity and complexity of the business, it is often necessary to combine multiple data analysis tasks into a larger analysis task, that is, to execute them in the form of a workflow.

[0003] The scheduling and execution of data analysis workflows are affected by multiple factors such as data size, storage location, business processes, computer operations, data transmission, etc., which are directly related to the stability, reliability, and efficiency of the entire task. To effectively optimize the process, it is necessary to comprehensively consider multiple factors and design a scientific and reasonable scheduling optimization method.

[0004] The currently popular scheduling methods generally have the advantages of being simple and easy to understand, but they also have the following shortcomings:

[0005] (1) In the current scheduling method, a master node often manages the task execution of each node. Once a node program fails, the entire data analysis task will fail.

[0006] (2) Task failure cannot be promptly reported. When a task fails, relevant personnel cannot be notified in time by email or other means, resulting in untimely response and low efficiency.

[0007] (3) Lack of retry function. Due to the ever-changing environment, the success of task operation is affected by various factors, such as network jitter, server service downtime, etc., which may lead to task failure.

[0008] (4) Lack of operation logs. During the task operation, the task was affected by business anomalies, resulting in task failure. There is no unified log monitoring page, making it difficult to troubleshoot the problem. Summary of the invention

[0009] In view of the problems existing in the prior art, the present invention provides a method for implementing a workflow with high degree of freedom and high configurability, which solves at least one of the above problems.

[0010] The technical solution of the present invention is as follows: a method for implementing a highly configurable workflow with high degree of freedom, comprising the following steps:

[0011] S1. Create a new workflow data analysis task based on the big data computing framework;

[0012] S2. After the workflow data analysis task is successfully created in step S1, the basic information of the task attributes and the historical version submission information of the task can be viewed;

[0013] S3. Set workflow task scheduling attributes, which include scheduling attributes, inter-task dependencies, and cross-cycle dependencies; the scheduling attributes include the workflow effective time, scheduling cycle, and specific running time, which are used to control the workflow to run periodically within a specified time;

[0014] S4. View the workflow task dependency view;

[0015] S5. Workflow message configuration, which can set whether the workflow message is sent, including the task timeout, alarm interval, and notification method. In the case of task failure, it is used to send notifications to relevant personnel in a timely manner to monitor the task;

[0016] S6. The workflow task is saved and run;

[0017] S7. Re-run the failed task of the workflow node.

[0018] Furthermore, after the workflow task in step S6 is saved and executed, the running status of the workflow scheduling can be viewed on the monitoring page, where the running status is waiting to be executed, running, successful execution, and failed execution.

[0019] Furthermore, in step S7, the failed task of the workflow node is re-run: during the running of the workflow task, if it fails due to network fluctuations or abnormal server system, the task can be re-triggered and re-run when the network or server system is normal later.

[0020] Furthermore, in step S1, the task types of various nodes created in the workflow data analysis task include data synchronization, Hive, Pyspark, shell, spark, sparksql, virtual nodes, and workflows; the workflow information is organized through the connection between nodes.

[0021] Furthermore, the step S2 includes the following sub-steps:

[0022] S21. Basic information of task attributes, including task name, task ID, task type, responsible person, creation time, last modification time, last modification person and corresponding description;

[0023] S22. The historical version submission information of the task includes the submission time, submitter and corresponding description.

[0024] Furthermore, the cross-cycle dependency in step S3 includes the following types, which can be selected as needed:

[0025] (1) Not dependent on the previous scheduling cycle;

[0026] (2) Self-dependence, waiting for the previous cycle to end before continuing to run;

[0027] (3) Self-dependence, waiting for the success of the previous scheduling cycle before continuing to run;

[0028] (4) Wait for the previous cycle of the downstream task to end before continuing to run;

[0029] (5) Wait for the previous cycle of the downstream task to succeed before continuing to run.

[0030] Furthermore, the notification sending method in step S5 includes email and enterprise WeChat.

[0031] Furthermore, the big data computing framework is a hadoop framework or a spark framework.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] 1. The present invention is based on the big data Hadoop and Spark frameworks and supports different task types for each node in the workflow.

[0034] 2. Monitor the running status of each node of the workflow in real time. Through the monitoring page, you can view the running status of each node of the workflow.

[0035] 3. Efficient processing of abnormal tasks. Through message configuration, if a task fails, the relevant personnel will be informed by email or other notification methods. The relevant personnel can efficiently locate and analyze the problem by checking the task operation log.

[0036] 4. The operations of workflow nodes do not interfere with each other, realizing complex offline computing scheduling. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a newly created workflow node;

[0038] Figure 2 is the task attribute information;

[0039] Figure 3 Set the scheduling properties for the task;

[0040] Figure 4 For inter-task dependencies;

[0041] Figure 5 is span cycle dependency;

[0042] Figure 6Message configuration for task operation;

[0043] Figure 7 Rerun failed tasks for workflow nodes. DETAILED DESCRIPTION

[0044] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings and embodiments of the present invention.

[0045] The present invention comprises the following steps S1 to S4:

[0046] S1. Create a new workflow data analysis task.

[0047] In the embodiment of the present invention, an example of a new workflow data analysis task is as follows:

[0048] Create a new task named xq_gzl_1016_2, set the task type to workflow, and do not use it as a template. The workflow task is stored in the test_workflow directory. Figure 1 shown.

[0049] S2, task attribute information;

[0050] After the task is created successfully, you can view the task attribute information, such as Figure 2 shown.

[0051] S3. Set workflow task scheduling attributes, including scheduling attributes, inter-task dependencies, and cross-cycle dependencies;

[0052] S31, the scheduling attribute of the task is set as follows Figure 3 shown.

[0053] The purpose of setting the scheduling properties of the workflow is to better control the operation of the workflow tasks. Through the workflow's effective date, scheduling cycle, and specific running time, the workflow can be guaranteed to run periodically within the specified time.

[0054] First, according to the scheduling attributes, set the workflow running cycle. By setting the effective date of the workflow task, you can control the execution cycle of the task. In the effective date of the workflow task, you can set the scheduling cycle of the task running, where the scheduling cycle is hour, day, week, month. After setting the scheduling cycle, you can set the specific running time of the workflow scheduling.

[0055] S32. Dependencies between tasks:

[0056] Before this workflow scheduling task can run, it must rely on other workflow scheduling tasks. Figure 4 shown.

[0057] S33, span cycle dependence:

[0058] This test does not depend on the previous scheduling cycle, so select the first one and operate as follows Figure 5 shown.

[0059] S4. View the workflow task dependency view;

[0060] This example does not involve the dependency view. If there are dependencies, the dependencies will be displayed.

[0061] S5, workflow message configuration;

[0062] Configure the message configuration for task operation. If a task fails in the later stage, notify the relevant personnel via email or corporate WeChat for timely processing. The operation is as follows Figure 6 shown.

[0063] S6. The workflow task is saved and run;

[0064] Workflow runs, and runs tasks through the task types used by workflow nodes. Task types include data synchronization, Hive, Pyspark, shell, spark, sparksql, virtual nodes, and workflows.

[0065] S7, re-run the failed task of the workflow node;

[0066] If a task on a node fails due to environmental factors, you can rerun the task on the monitoring page.

[0067] Operation Figure 7 shown.

[0068] The present invention is based on big data hadoop and spark frameworks and supports different task types for each node in the workflow. It can monitor the running status of each node in the workflow in real time. Through the monitoring page, the running status of each node in the workflow can be viewed. The operation of each node does not interfere with each other, and complex offline computing scheduling is realized.

[0069] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for implementing a highly configurable workflow with high degree of freedom, characterized in that: The following steps are involved: S1. Create a new workflow data analysis task based on a big data computing framework; the big data computing framework is a hadoop framework or a spark framework; In step S1, the task types of various nodes created in the workflow data analysis task include data synchronization, Hive, Pyspark, shell, spark, sparksql, virtual nodes, and workflows; the workflow information is formed by connecting nodes to each other; S2. After the workflow data analysis task is successfully created in step S1, the basic information of the task attributes and the historical version submission information of the task can be viewed; S3. Setting workflow task scheduling attributes, wherein the workflow task scheduling attributes include scheduling attributes, inter-task dependencies, and cross-cycle dependencies; The scheduling attributes include the effective time, scheduling period and specific running time of the workflow, which are used to control the workflow to run periodically within the specified time; S4. View the workflow task dependency view; S5. Workflow message configuration, setting whether the workflow message is sent, including the task timeout, alarm interval, and notification sending method. In the case of task failure, it is used to send notifications to relevant personnel in a timely manner to monitor the task; S6. The workflow task is saved and run; S7. Re-run the failed task of the workflow node.

2. The method for implementing a highly configurable workflow with high degree of freedom according to claim 1, characterized in that: After the workflow task is saved and run in step S6, the running status of the workflow scheduling can be viewed on the monitoring page, where the running status is waiting to run, running, running successfully, and running failed.

3. The method for implementing a highly configurable workflow with high degree of freedom according to claim 1, characterized in that: In step S7, the failed task of the workflow node is re-run: during the running of the workflow task, if it fails due to network fluctuations or server system abnormalities, the task can be re-triggered and re-run when the network or server system is normal later.

4. The method for implementing a highly configurable workflow with high degree of freedom according to claim 1, characterized in that: The step S2 comprises the following sub-steps: S21. Basic information of task attributes, including task name, task ID, task type, responsible person, creation time, last modification time, last modification person and corresponding description; S22. The historical version submission information of the task includes the submission time, submitter and corresponding description.

5. The method for implementing a highly configurable workflow with high degree of freedom according to claim 1, characterized in that: The cross-cycle dependencies in step S3 include the following, which can be selected as needed: (1) Not dependent on the previous scheduling cycle; (2) Self-dependence, waiting for the previous cycle to end before continuing to run; (3) Self-dependence, waiting for the success of the previous scheduling cycle before continuing to run; (4) Wait for the previous cycle of the downstream task to end before continuing to run; (5) Wait for the previous cycle of the downstream task to succeed before continuing to run.

6. The method for implementing a highly configurable workflow with high degree of freedom according to claim 1, characterized in that: The notification sending method in step S5 includes email and enterprise WeChat.

Images