A bloodline-based task scheduling optimization method, system, device and medium

Abstract

The application discloses a kind of methods, systems, equipment and media based on blood relationship task scheduling optimization, related to task scheduling optimization technical field, which comprises the following steps: step A: the task needing automatic configuration workflow in distributed task scheduling platform is parsed by blood relationship, and task data blood relationship is obtained;Step B: the task data blood relationship obtained in step A is used to construct task dependency graph DAG by blood relationship;Step C: based on the task dependency graph DAG constructed in step B, task scheduling execution is carried out.The system comprises: blood relationship analysis unit, task dependency graph construction unit and task scheduling execution unit.The application reduces the workload and error rate of manual configuration task dependency by blood relationship analysis, and the size of the number of tasks contained is also improved by the automatically generated directed acyclic graph DAG.

Description

Technical Field

[0001] This invention relates to the field of task scheduling optimization technology, specifically to a lineage-based task scheduling optimization method, system, device, and medium. Background Technology

[0002] The Apache DolphinScheduler is a distributed, decentralized, and easily scalable visual directed acyclic graph (DAG) workflow task scheduling platform. It aims to resolve complex big data task dependencies and provide applications with data and various operation (OPS) orchestration relationships; addressing the challenges of complex data extraction, transformation, loading, and ETL dependencies, and the inability to monitor task health. DolphinScheduler assembles tasks in a streaming manner using a DAG, enabling real-time monitoring of task execution status and supporting operations such as retrying, resuming failed tasks at specified nodes, pausing, resuming, and terminating tasks. It includes an AlertServer process providing alert services, an ApiServer process providing web services for users to edit task workflows, a MasterServer process responsible for scheduling workflows, and WorkerServer processes responsible for actually executing tasks, communicating with each other via Netty.

[0003] DolphinScheduler's DAG API is a web service provided by the user interaction service ApiServer. However, in large enterprises, especially in today's era where data is a valuable asset, data tasks are increasing. If workflows are created directly using the DAG API provided by ApiServer, manual editing becomes difficult when the number of tasks reaches dozens, as task dependencies require manual checking and configuration. Treating workflows as node configurations significantly increases workflow complexity, particularly hindering cross-departmental collaborative development. Therefore, this invention proposes a method for automatically generating task dependencies and constructing a Directed Acyclic Graph (DAG) to address these issues. Summary of the Invention

[0004] The technical problem this invention aims to solve is that existing task scheduling methods suffer from complex and difficult manual construction of Directed Acyclic Graphs (DAGs), resulting in low construction efficiency and a high rate of manual configuration errors. The purpose of this invention is to provide a lineage-based task scheduling optimization method, system, device, and medium. Lineage resolution reduces the workload and error rate of manual task configuration, while the automatically generated DAG increases the number of tasks it can contain.

[0005] This invention is achieved through the following technical solution:

[0006] In a first aspect, the present invention provides a lineage-based task scheduling optimization method, which includes the following steps:

[0007] Step A: Perform lineage analysis on the tasks in the distributed task scheduling platform that require automatic workflow configuration to obtain the task data lineage;

[0008] Step B: Based on the task data lineage obtained in Step A, construct a task dependency graph (DAG) using the lineage relationships;

[0009] Step C: Based on the task dependency graph DAG constructed in Step B, task scheduling and execution are performed, thereby realizing automatic workflow configuration and reducing the workload and human error rate of manual workflow configuration in the case of large number of tasks.

[0010] This invention is applied to the distributed task scheduling platform Apache DolphinScheduler. This invention improves the efficiency of constructing directed acyclic graphs (DAGs), reduces manual configuration errors, and increases the number of tasks per node in a directed acyclic graph (DAG).

[0011] Furthermore, the specific steps of step A are as follows:

[0012] Step A1: When there are n tasks in the distributed task scheduling platform, assuming that task0 to taskm are tasks that need to be automatically configured as virtual workflows;

[0013] Step A2: Perform lineage analysis on tasks from task0 to taskm to obtain the task data lineage. The task data lineage includes input data information and output data information. Specifically, the output data information hash table (hashtableout) is obtained by storing the task as the key and the output data table information as the value (key-value pairs). The input data information hash table (hashtablein) is obtained by storing the input data information as the key and the task as the value (key-value pairs). The task definition without any input data information is stored in the starting set (startset).

[0014] Furthermore, the data information structures in both the input and output data information contain information for any number of data table entries, including the data source name, database name, and data table name.

[0015] Furthermore, the bloodline analysis is as follows:

[0016] If the task requiring automatic workflow configuration is an SQL-type task, then the task data lineage is obtained by parsing the SQL.

[0017] If the task requiring automatic workflow configuration is a non-SQL task (i.e., other tasks), the task data lineage is obtained by manually parsing the configuration.

[0018] Furthermore, the specific steps of step B are as follows:

[0019] Step B1: Construct a directed acyclic graph (DAG) with tasks from task0 to taskm as nodes and task dependencies as edges;

[0020] Step B2: Use the task set startset obtained in step A2 as the current task set currenttasks for traversal, and create the next task set nexttasks for traversal;

[0021] Step B3: For a specific task in the current task set currenttasks, obtain the output data information outdatai of that task;

[0022] Step B4: Traverse the data entries in the output data information outdatai;

[0023] Step B5: For the data table entries in the output data information outdatai, use the data table entries as keys to obtain the downstream task taskj from the input data information hashtablein in step A2;

[0024] Step B6: Add a directed edge from the task node to the downstream task node in the directed acyclic graph (DAG) from Step B1, and add the downstream task node to the next traversal task set nexttasks.

[0025] Step B7: Repeat steps B5 to B6 until the data table entries in the output data information outdatai have been traversed and processed.

[0026] Step B8: Repeat steps B3 to B7 until all tasks in the current task set currenttasks have been processed.

[0027] Step B9: Clear the current task set currenttasks and swap the current task set currenttasks with the next task set nexttasks;

[0028] Step B10: Repeat steps B3 to B9 until the next traversal task set nexttasks is empty. At this point, the task dependency graph DAG is completed.

[0029] Furthermore, the task dependencies are as follows:

[0030] The fact that taskx depends on tasksky is equivalent to taskx's input data containing tasksky's output data; where taskx and tasksky are task nodes in a directed acyclic graph (DAG).

[0031] Furthermore, the specific steps of step C are as follows:

[0032] Step C1: From the task dependency graph DAG constructed in step B, select the task node with an in-degree of 0, generate a task object TaskInstance based on the task node and set the triggering method to lineage triggering, and save it to the database; at the same time, put the task node with an in-degree of 0 into the priority scheduling queue of the master node MasterServer to be executed.

[0033] Step C2: Based on the load balancing algorithm, the MasterServer selects the WorkerServer to execute the lineage task;

[0034] Step C3: After the lineage task is completed, the Master node receives the completion notification from the Worker node and deletes the lineage task associated with the task dependency graph DAG node in step B.

[0035] Step C4: Traverse the successor nodes of the node tasks deleted in C3 and check if their in-degree is 0; if so, repeat step C1.

[0036] Step C5: When there are no nodes in the task dependency graph DAG in step B, the entire lineage task scheduling and execution is completed.

[0037] Secondly, the present invention provides a lineage-based task scheduling optimization system, which supports the aforementioned lineage-based task scheduling optimization method; the system includes:

[0038] The lineage resolution unit is used to resolve the lineage of tasks that require automatic workflow configuration in the distributed task scheduling platform to obtain the task data lineage.

[0039] The task dependency graph construction unit is used to construct a task dependency graph (DAG) based on the task data lineage obtained from the lineage resolution unit.

[0040] The task scheduling and execution unit is used to schedule and execute tasks based on the task dependency graph DAG constructed by the task dependency graph construction unit, thereby realizing automatic workflow configuration and reducing the workload and human error rate of manual workflow configuration in the case of large number of tasks.

[0041] Thirdly, the present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the aforementioned lineage-based task scheduling optimization method.

[0042] Fourthly, the present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned lineage-based task scheduling optimization method.

[0043] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0044] 1. This invention performs lineage analysis on tasks requiring automatic workflow configuration in a distributed task scheduling platform to obtain task data lineage; it then constructs a task dependency graph (DAG) based on the lineage relationships; and finally, it schedules and executes tasks based on the constructed DAG, thereby achieving automatic workflow configuration and reducing the workload and human error rate of manual workflow configuration in the case of large-scale tasks. Specifically, when step B10 is completed, the entire task dependency graph (DAG) has already been constructed, eliminating the need for manual configuration of task dependencies and creation of the task dependency graph (DAG); the judgment in step C3 is to ensure that the execution of the entire task dependency graph (DAG) does not conflict with other manually constructed DAGs in the system.

[0045] 2. This invention is applied to the distributed task scheduling platform Apache DolphinScheduler. This invention improves the efficiency of constructing directed acyclic graphs (DAGs), reduces manual configuration errors, and increases the number of tasks per node in a directed acyclic graph (DAG). Attached Figure Description

[0046] The accompanying drawings, which are included to provide a further understanding of embodiments of the invention and form part of this application, do not constitute a limitation thereof. In the drawings:

[0047] Figure 1 This is a flowchart of a bloodline-based task scheduling optimization method according to Embodiment 1 of the present invention;

[0048] Figure 2 This is a block diagram of a bloodline-based task scheduling optimization system according to Embodiment 2 of the present invention. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the embodiments and accompanying drawings. The illustrative embodiments and descriptions of the present invention are only used to explain the present invention and are not intended to limit the present invention.

[0050] Example 1

[0051] like Figure 1 As shown, Figure 1 This is a flowchart of a lineage-based task scheduling optimization method according to Embodiment 1 of the present invention. The present invention provides a lineage-based task scheduling optimization method, which is applied to the distributed task scheduling platform Apache DolphinScheduler. This invention improves the efficiency of constructing Directed Acyclic Graphs (DAGs), reduces manual configuration errors, and simultaneously increases the number of tasks per node in the DAG.

[0052] The method includes the following steps:

[0053] Step A: Perform lineage analysis on the tasks in the distributed task scheduling platform that require automatic workflow configuration to obtain the task data lineage;

[0054] Specifically, the steps in step A are as follows:

[0055] Step A1: When there are n tasks in the distributed task scheduling platform, assuming that task0 to taskm are tasks that need to be automatically configured as virtual workflows;

[0056] Step A2: Perform lineage analysis on tasks 0 to taskm to obtain the task data lineage. The task data lineage includes input data information (indata1, ..., indatai, ..., indatam) and output data information (outdata1, ..., outdatai, ..., outdatam). The input data information indatai and the output data information outdatai correspond to the lineage of the task. Both the input and output data information contain any number of data item entries (dataitem). Each data item includes the data source name, database name, and table name. For example, indatai contains two data items, dataitemi1 and dataitemi2, where dataitemi1 corresponds to the data source database tables indatasourcei1, indatabasei1, and intablei1, respectively.

[0057] The method of this invention stores the task as the key and the output data table information as the value (key-value pairs) in a hash table structure to obtain the output data information hash table hashtableout; it also stores the input data information as the key and the task as the value (key-value pairs) in a hash table structure to obtain the input data information hash table hashtablein; and it stores the task definition without any input data information in the starting set startset.

[0058] In this embodiment, bloodline analysis is as follows:

[0059] If the task requiring automatic workflow configuration is an SQL-type task, then the task data lineage is obtained by parsing the SQL.

[0060] If the task requiring automatic workflow configuration is a non-SQL task (i.e., other tasks), the task data lineage is obtained by manually parsing the configuration.

[0061] Step B: Based on the task data lineage obtained in Step A, construct a task dependency graph (DAG) using the lineage relationships;

[0062] Specifically, the steps in step B are as follows:

[0063] Step B1: Construct a directed acyclic graph (DAG) with tasks from task0 to taskm as nodes and task dependencies as edges. The task dependencies are defined as follows: taskx depends on tasksky if the input data of taskx contains the output data of tasksky. Here, taskx and tasksky are task nodes in the DAG, meaning they belong to any one of the tasks from task0 to taskm. If the execution of taskx depends on the successful execution of tasksky, then there exists a directed edge in the DAG pointing from tasksky to taskx.

[0064] Step B2: Use the task set startset obtained in step A2 as the current task set currenttasks for traversal, and create the next task set nexttasks for traversal;

[0065] Step B3: For the task task in the current task set currenttasks, use the task task as the key to retrieve the value of the output data information hash table hashtableout from step A2, which is to retrieve the output data information outdatai of the task task.

[0066] Step B4: Since the input data for task task may be multiple data table items, iterate through the data table items in the output data information outdatai;

[0067] Step B5: For the data item dataitemj in the output data information outdatai, use dataitemj as the key to obtain the downstream task taskj from the input data information hash table hashtablein in step A2;

[0068] Step B6: Add a directed edge from the task node (taski task node) to the downstream task node (taskj task node) in the directed acyclic graph (DAG) in step B1, and add the downstream task node (taskj task node) to the next traversal task set nexttasks.

[0069] Step B7: Repeat steps B5 to B6 until the data table entries in the output data information outdatai have been traversed and processed.

[0070] Step B8: Repeat steps B3 to B7 until all tasks in the current task set currenttasks have been processed.

[0071] Step B9: Clear the current task set currenttasks and swap the current task set currenttasks with the next task set nexttasks;

[0072] Step B10: Repeat steps B3 to B9 until the next traversal task set nexttasks is empty. At this point, the task dependency graph DAG is completed.

[0073] Step C: Based on the task dependency graph DAG constructed in Step B, task scheduling and execution are performed, thereby realizing automatic workflow configuration and reducing the workload and human error rate of manual workflow configuration in the case of large number of tasks.

[0074] Specifically, the steps in step C are as follows:

[0075] Step C1: From the task dependency graph DAG constructed in step B, select the task node with an in-degree of 0, generate a task object TaskInstance based on the task node and set the triggering method to lineage triggering, and save it to the database; at the same time, put the task node with an in-degree of 0 into the priority scheduling queue of the master node MasterServer to be executed.

[0076] Step C2: Based on the load balancing algorithm, the MasterServer selects the WorkerServer to execute the lineage task;

[0077] Step C3: After the lineage task is completed, the Master node receives the completion notification from the Worker node and deletes the lineage task associated with the task dependency graph DAG node in step B.

[0078] Step C4: Traverse the successor nodes of the node tasks deleted in C3 and check if their in-degree is 0; if so, repeat step C1.

[0079] Step C5: When there are no nodes in the task dependency graph DAG in step B, the entire lineage task scheduling and execution is completed.

[0080] Based on the above implementation steps, the entire task dependency graph (DAG) is constructed upon completion of step B10, thus eliminating the need for manual configuration of task dependencies and DAG creation. The judgment in step C3 ensures that the execution of the entire task dependency graph DAG does not conflict with other manually constructed DAGs already in the system. This invention is applied to the distributed task scheduling platform Apache DolphinScheduler. This invention improves the efficiency of DAG construction, reduces manual configuration errors, and simultaneously increases the number of tasks per node in the DAG.

[0081] Example 2

[0082] like Figure 2 As shown, the difference between this embodiment and Embodiment 1 is that this embodiment provides a lineage-based task scheduling optimization system, which is used to implement the lineage-based task scheduling optimization method described in Embodiment 1;

[0083] Figure 2 This is a block diagram of a lineage-based task scheduling optimization system according to Embodiment 2 of the present invention. The system includes:

[0084] The lineage resolution unit is used to resolve the lineage of tasks that require automatic workflow configuration in the distributed task scheduling platform to obtain the task data lineage.

[0085] The task dependency graph construction unit is used to construct a task dependency graph (DAG) based on the task data lineage obtained from the lineage resolution unit.

[0086] The task scheduling and execution unit is used to schedule and execute tasks based on the task dependency graph DAG constructed by the task dependency graph construction unit, thereby realizing automatic workflow configuration and reducing the workload and human error rate of manual workflow configuration in the case of large number of tasks.

[0087] The execution process of each unit can be carried out according to the steps of the bloodline-based task scheduling optimization method described in Example 1, and will not be repeated in this example.

[0088] This invention belongs to the field of task scheduling optimization and solves the problems of configuring task dependencies and the large workload of workflows in traditional technologies. This invention uses a lineage resolution unit to resolve the lineage of tasks that require automatic workflow configuration in a distributed task scheduling platform to obtain task data lineage; a task dependency graph construction unit uses the task data lineage obtained by the lineage resolution unit to construct a task dependency graph (DAG) based on the lineage relationships; and a task scheduling execution unit performs task scheduling and execution based on the constructed task dependency graph (DAG), thereby realizing automatic workflow configuration and reducing the workload and human error rate of manual workflow configuration in the case of large number of tasks.

[0089] Meanwhile, the present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the aforementioned lineage-based task scheduling optimization method.

[0090] Meanwhile, the present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the aforementioned lineage-based task scheduling optimization method.

[0091] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A task scheduling optimization method based on lineage, characterized in that, The method includes the following steps: Step A: Perform lineage analysis on the tasks in the distributed task scheduling platform that require automatic workflow configuration to obtain the task data lineage; Step B: Based on the task data lineage obtained in Step A, construct a task dependency graph (DAG) using the lineage relationships; Step C: Based on the task dependency graph DAG constructed in Step B, perform task scheduling and execution; The specific steps of step A are as follows: Step A1: When there are n tasks in the distributed task scheduling platform, assuming that task0~taskm are tasks that need to be automatically configured as virtual workflows; Step A2: Perform lineage analysis on tasks 0 to 100kJ to obtain the task data lineage; the task data lineage includes input data information and output data information; wherein, the task is stored as the key and the output data table information is stored as the value in a hash table structure to obtain the output data information hash table hashtableout; the input data information is stored as the key and the task as the value in a hash table structure to obtain the input data information hash table hashtablein; and tasks without any input data information are stored in the starting set startset; The bloodline analysis is as follows: If the task requiring automatic workflow configuration is an SQL-type task, then the task data lineage is obtained by parsing the SQL. If the task requiring automatic workflow configuration is a non-SQL task, the task data lineage is obtained by manually parsing the configuration. The specific steps of step C are as follows: Step C1: From the task dependency graph DAG constructed in step B, select the task node with an in-degree of 0, generate a task object TaskInstance based on the task node and set the triggering method to lineage triggering, and save it to the database; at the same time, put the task node with an in-degree of 0 into the priority scheduling queue of the master node MasterServer to be executed. Step C2: Based on the load balancing algorithm, the MasterServer selects the WorkerServer to execute the lineage task; Step C3: After the lineage task is completed, the Master node receives the completion notification from the Worker node and deletes the lineage task associated with the task dependency graph DAG node in step B. Step C4: Traverse the successor nodes of the node tasks deleted in C3 and check if their in-degree is 0; if so, repeat step C1. Step C5: When there are no nodes in the task dependency graph DAG in step B, the entire lineage task scheduling and execution is completed.

2. The task scheduling optimization method based on lineage according to claim 1, characterized in that, The data information structures in both the input and output data information contain information for any number of data table entries, which include the data source name, database name, and data table name.

3. The task scheduling optimization method based on lineage according to claim 1, characterized in that, The specific steps of step B are as follows: Step B1: Construct a directed acyclic graph (DAG) with tasks from task0 to taskm as nodes and task dependencies as edges; Step B2: Use the task set startset obtained in step A2 as the current task set currenttasks for traversal, and create the next task set nexttasks for traversal; Step B3: For a specific task in the current task set currenttasks, obtain the output data information outdatai of that task; Step B4: Traverse the data entries in the output data information outdatai; Step B5: For the data table entries in the output data information outdatai, use the data table entries as keys to obtain the downstream task taskj from the input data information hashtablein in step A2; Step B6: Add a directed edge from the task node to the downstream task node in the directed acyclic graph (DAG) from Step B1, and add the downstream task node to the next traversal task set nexttasks. Step B7: Repeat steps B5 to B6 until the data table entries in the output data information outdatai have been traversed and processed. Step B8: Repeat steps B3 to B7 until all tasks in the current task set currenttasks have been processed. Step B9: Clear the current task set currenttasks and swap the current task set currenttasks with the next task set nexttasks; Step B10: Repeat steps B3 to B9 until the next traversal task set nexttasks is empty. At this point, the task dependency graph DAG is completed.

4. The task scheduling optimization method based on lineage according to claim 3, characterized in that, The task dependencies are as follows: The fact that taskx depends on tasksky is equivalent to taskx's input data containing tasksky's output data; where taskx and tasksky are task nodes in a directed acyclic graph (DAG).

5. A kinship-based task scheduling optimization system, characterized in that, This system is used to implement a lineage-based task scheduling optimization method as described in any one of claims 1 to 4; the system includes: The lineage resolution unit is used to resolve the lineage of tasks that require automatic workflow configuration in the distributed task scheduling platform to obtain the task data lineage. The task dependency graph construction unit is used to construct a task dependency graph (DAG) based on the task data lineage obtained from the lineage resolution unit. The task scheduling and execution unit is used to schedule and execute tasks based on the task dependency graph DAG constructed by the task dependency graph construction unit.

6. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements a lineage-based task scheduling optimization method as described in any one of claims 1 to 4.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements a lineage-based task scheduling optimization method as described in any one of claims 1 to 4.

Images