Task processing method, device, server and storage medium

Abstract

The present disclosure relates to a task processing method, device, server and storage medium, the method comprising: obtaining a set of pending task identifiers; the set of pending task identifiers includes multiple pending task identifiers; obtaining the pending tasks corresponding to the pending task identifiers from the initial task dependency graph; determining new task dependencies between the pending tasks; constructing a new task dependency graph based on the pending tasks and the new task dependencies; calling the new task dependency graph to execute the pending tasks. In this embodiment, the new task dependency graph only includes the pending tasks corresponding to the pending task identifiers, thereby shortening the task execution link, allowing the tasks to be completed in a timely manner, thereby improving the task processing efficiency.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a task processing method, device, server, and storage medium. Background Art

[0002] The computation, analysis, and processing of big data are typically accomplished by numerous independent processes, each of which performs specific data processing logic. These processes are called tasks. In actual big data workflow processing, tasks often have sequential dependencies, also known as lineage relationships.

[0003] For example, to process Task A, Task B must first be completed. Because Task A's processing depends on the results of Task B's processing, the dependency relationship between Task A and Task B is: Task A depends on Task B. Due to the dependencies between tasks, to ensure the normal processing of the entire task in the workflow, tasks must be executed in an orderly and efficient manner according to the dependencies. Therefore, before each task is executed, a task instance lineage graph is constructed based on its task metadata. The task instance lineage graph organizes the dependencies between task instances for each task, and the task instances are executed sequentially according to the task instance lineage graph to complete the task execution.

[0004] However, in the existing workflow scheduling system, the task re-run function is implemented through the task instance lineage graph. All downstream tasks from the task instance lineage graph are obtained according to the root node selected by the user, and the root node and all downstream tasks are re-run. However, in practice, users only care about the execution of some of the tasks. If all tasks are re-run, the task execution chain is long, resulting in the task not being completed in time, resulting in low task processing efficiency. Summary of the Invention

[0005] The present disclosure provides a task processing method, device, server, and storage medium to at least address the problem in related technologies of long task execution links, which results in delayed task completion and low task processing efficiency. The technical solutions of the present disclosure are as follows:

[0006] According to a first aspect of an embodiment of the present disclosure, there is provided a task processing method, comprising:

[0007] Obtaining a set of pending task identifiers; the set of pending task identifiers includes a plurality of pending task identifiers;

[0008] Obtaining the pending task corresponding to the pending task identifier from the initial task dependency graph;

[0009] Determining new task dependencies between the tasks to be processed;

[0010] Constructing a new task dependency graph based on the tasks to be processed and the new task dependency;

[0011] The new task dependency graph is called to execute the pending tasks.

[0012] In one embodiment, obtaining the to-be-processed task corresponding to the to-be-processed task identifier from the initial task dependency graph includes:

[0013] Traverse the task identifiers corresponding to the tasks in the initial task dependency graph in sequence;

[0014] Obtaining a task corresponding to a task identifier that is consistent with the task identifier to be processed as the current task;

[0015] Obtaining the task identifier of the current downstream task corresponding to the current task;

[0016] When the task identifier of the current downstream task is consistent with the identifier of the task to be processed, the current task and the current downstream task are determined as tasks to be processed, and the step of sequentially traversing the task identifiers corresponding to the tasks in the initial task dependency graph is returned;

[0017] When the task identifier of the current downstream task is inconsistent with the identifier of the task to be processed, the current downstream task is used as the current task, and the process returns to the step of obtaining the task identifier of the current downstream task corresponding to the current task.

[0018] In one embodiment, before obtaining the to-be-processed task corresponding to the to-be-processed task identifier from the initial task dependency graph, the method further includes:

[0019] Obtaining a task dependency model diagram, wherein the task dependency model diagram includes a task identifier corresponding to the task, a scheduling period, and a dependency formula; the dependency formula is used to represent the task dependency relationship between the task and the upstream task corresponding to the task;

[0020] Get the input start task ID and target time interval;

[0021] Obtaining a task corresponding to a task identifier that is consistent with the starting task name as the current task;

[0022] Taking the current task and the downstream task corresponding to the current task as the target task;

[0023] generating task execution data of the target task according to the scheduling period and the target time interval of the target task;

[0024] Determining the data dependency relationship between the task execution data according to the dependency relationship formula;

[0025] The task execution data are combined according to the data dependency to obtain the initial task dependency graph.

[0026] In one embodiment, generating the task execution data of the target task according to the scheduling period and the target time interval of the target task includes:

[0027] Determining at least one task execution time according to the scheduling period of the target task and the target time interval;

[0028] The target task and the task execution time are combined to obtain at least one task execution data.

[0029] In one embodiment, obtaining the task dependency model diagram includes:

[0030] Obtaining task metadata; the task metadata includes a task identifier, a scheduling period, and upstream dependency data corresponding to the task; the upstream dependency data includes an upstream task identifier and an upstream dependency descriptor of an upstream task corresponding to the task;

[0031] Generating a dependency relationship formula of the task according to the upstream dependency descriptor and the scheduling period of the upstream task;

[0032] A task dependency model diagram is generated according to the task identifier corresponding to the task and the dependency formula.

[0033] In one embodiment, the upstream dependency descriptor includes an operator and a dependency time, and generating the dependency relationship of the task according to the upstream dependency descriptor and the scheduling period of the upstream task includes:

[0034] Obtaining a bitmap; the bitmap includes assignment locations for the scheduling period, the operator, and the dependency time;

[0035] According to the conversion relationship of the scheduling period, the scheduling period of the upstream task is converted into a first result value; according to the conversion relationship of the operator, the operator is converted into a second result value; and according to the conversion relationship of the dependency time, the dependency time is converted into a third result value;

[0036] The first result value, the second result value and the third result value are sequentially filled into the corresponding assignment positions to obtain the dependency relationship formula of the task.

[0037] In one embodiment, calling the new task dependency graph to execute the pending task includes:

[0038] Extracting the new task dependency from the new task dependency graph;

[0039] According to the new task dependency, each of the tasks to be processed is executed in sequence to obtain the corresponding task processing results.

[0040] According to a second aspect of an embodiment of the present disclosure, there is provided a task processing device, including:

[0041] A task identifier acquisition unit is configured to acquire a set of pending task identifiers; the set of pending task identifiers includes a plurality of pending task identifiers;

[0042] A task acquisition unit is configured to execute the task to be processed corresponding to the task to be processed identifier from the initial task dependency graph;

[0043] A dependency determination unit, configured to determine a new task dependency between the tasks to be processed;

[0044] A dependency graph construction unit is configured to construct a new task dependency graph according to the tasks to be processed and the new task dependency relationships;

[0045] The task execution unit is configured to execute and call the new task dependency graph to execute the task to be processed.

[0046] In one embodiment, the task acquisition unit is further configured to execute the step of sequentially traversing the task identifiers corresponding to the tasks in the initial task dependency graph; acquiring the task corresponding to the task identifier consistent with the task identifier to be processed as the current task; acquiring the task identifier of the current downstream task corresponding to the current task; when the task identifier of the current downstream task is consistent with the task identifier to be processed, determining the current task and the current downstream task as tasks to be processed, and returning to the step of sequentially traversing the task identifiers corresponding to the tasks in the initial task dependency graph; when the task identifier of the current downstream task is inconsistent with the task identifier to be processed, determining the current downstream task as the current task, and returning to the step of acquiring the task identifier of the current downstream task corresponding to the current task.

[0047] In one embodiment, the device also includes an initial task dependency graph acquisition unit, which is configured to execute acquisition of a task dependency model graph, wherein the task dependency model graph includes a task identifier, a scheduling period, and a dependency formula corresponding to the task; the dependency formula is used to characterize the task dependency relationship between the task and the upstream task corresponding to the task; obtain the input starting task identifier and target time interval; obtain the task corresponding to the task identifier consistent with the starting task name as the current task; use the current task and the downstream task corresponding to the current task as the target task; generate the task execution data of the target task according to the scheduling period and the target time interval of the target task; determine the data dependency between the task execution data according to the dependency formula; and combine the task execution data according to the data dependency to obtain the initial task dependency graph.

[0048] In one embodiment, the initial task dependency graph acquisition unit is further configured to determine at least one task execution time based on the scheduling period of the target task and the target time interval; and combine the target task with the task execution time to obtain at least one task execution data.

[0049] In one embodiment, the initial task dependency graph acquisition unit is further configured to execute the acquisition of task metadata; the task metadata includes the task identifier, scheduling period and upstream dependency data corresponding to the task; the upstream dependency data includes the upstream task identifier and upstream dependency descriptor of the upstream task corresponding to the task; based on the upstream dependency descriptor and the scheduling period of the upstream task, the dependency formula of the task is generated; based on the task identifier corresponding to the task and the dependency formula, a task dependency model graph is generated.

[0050] In one embodiment, the upstream dependency descriptor includes an operator and a dependency time, and the initial task dependency graph acquisition unit is further configured to execute an acquisition bitmap; the bitmap includes assignment positions for the scheduling period, the operator and the dependency time; according to the conversion relationship of the scheduling period, the scheduling period of the upstream task is converted into a first result value; according to the conversion relationship of the operator, the operator is converted into a second result value, and according to the conversion relationship of the dependency time, the dependency time is converted into a third result value; the first result value, the second result value and the third result value are filled into the corresponding assignment positions in sequence to obtain the dependency formula of the task.

[0051] In one embodiment, the task execution unit is further configured to extract the new task dependency from the new task dependency graph; and execute each of the to-be-processed tasks in sequence according to the new task dependency to obtain corresponding task processing results.

[0052] According to a third aspect of an embodiment of the present disclosure, a server is provided, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the task processing method as in the implementation of the first aspect.

[0053] According to a fourth aspect of an embodiment of the present disclosure, a storage medium is provided, comprising: when instructions in the storage medium are executed by a processor of a server, the server is enabled to execute the task processing method as in the implementation of the first aspect.

[0054] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, which includes a computer program stored in a readable storage medium, and at least one processor of a device reads and executes the computer program from the readable storage medium, so that the device performs the task processing method described in any one of the embodiments of the first aspect.

[0055] The technical solutions provided by the embodiments of the present disclosure bring at least the following beneficial effects:

[0056] A set of pending task identifiers, including multiple pending task identifiers, is obtained, and the pending tasks corresponding to the pending task identifiers are obtained from an initial task dependency graph. New task dependencies between the pending tasks are then determined, and a new task dependency graph is constructed based on the pending tasks and the new task dependencies. Finally, the new task dependency graph is called to execute the pending tasks. In this embodiment, a new task dependency graph can be extracted from the initial task dependency graph based on the obtained pending task identifiers. Because the new task dependency graph only includes the pending tasks corresponding to the pending task identifiers, the task execution chain is shortened, enabling tasks to be completed in a timely manner, thereby improving task processing efficiency.

[0057] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The accompanying drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute an improper limitation of the present disclosure.

[0059] Figure 1The figure is a diagram showing an application environment of a task processing method according to an exemplary embodiment.

[0060] Figure 2 The figure is a flowchart showing a task processing method according to an exemplary embodiment.

[0061] Figure 3 The figure is a schematic diagram showing a task dependency graph according to an exemplary embodiment.

[0062] Figure 4 The present invention is a flowchart showing the steps of obtaining to-be-processed tasks corresponding to to-be-processed task identifiers from an initial task dependency graph according to an exemplary embodiment.

[0063] Figure 5 The figure is a schematic diagram of a task dependency model diagram according to an exemplary embodiment.

[0064] Figure 6 The figure is a schematic diagram showing task metadata according to an exemplary embodiment.

[0065] Figure 7 The figure is a schematic diagram showing a dependency relationship according to an exemplary embodiment.

[0066] Figure 8 The present invention is a flowchart showing a method of rebuilding a task dependency graph according to an exemplary embodiment.

[0067] Figure 9 The figure is a flowchart of obtaining a downstream node according to an exemplary embodiment.

[0068] Figure 10 The figure is a schematic diagram showing a task dependency graph before and after reconstruction according to an exemplary embodiment.

[0069] Figure 11 The figure is a block diagram of a task processing device according to an exemplary embodiment.

[0070] Figure 12 The figure shows the internal structure of a server according to an exemplary embodiment. DETAILED DESCRIPTION

[0071] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0072] It should be noted that the terms "first," "second," and the like in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the numbers used in this manner are interchangeable where appropriate so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure as detailed in the appended claims.

[0073] The task processing method provided by the present disclosure can be applied to Figure 1 In the application environment shown, the terminal 110 interacts with the server 120 via a network, and the server 120 obtains a set of pending task identifiers uploaded by the terminal 110; the set of pending task identifiers includes multiple pending task identifiers; the pending tasks corresponding to the pending task identifiers are obtained from the initial task dependency graph; new task dependencies between the pending tasks are determined; a new task dependency graph is constructed based on the pending tasks and the new task dependencies; the new task dependency graph is called to execute the pending tasks. The terminal 110 can be, but is not limited to, various smart phones, tablet computers, or laptop computers, and the server 120 can be implemented as an independent server or a server cluster consisting of multiple servers.

[0074] Figure 2 is a flowchart of a task processing method according to an exemplary embodiment. Figure 2 As shown, the task processing method is used in the server and includes the following steps.

[0075] In step S210 , a set of pending task identifiers is obtained; the set of pending task identifiers includes a plurality of pending task identifiers.

[0076] Among them, pending tasks refer to tasks that need to be processed, specifically tasks that need to be rerun. Of course, they can also be adjusted according to actual conditions and are not specifically limited here.

[0077] Among them, the pending task identifier refers to the identification information of the pending task, such as the task name, task serial number, etc.; in actual scenarios, the pending task identifier can be a specified task name, specifically the task name corresponding to the task that the user wants to retain for rerunning, which can be entered into the server by the user through the terminal.

[0078] Specifically, the server obtains a set of pending task identifiers uploaded by the terminal, or obtains a set of pending task identifiers including multiple pending task identifiers from a local database; in this way, by obtaining a set of pending task identifiers including multiple pending task identifiers, it is convenient to subsequently obtain the pending tasks corresponding to the pending task identifiers from the initial task dependency graph.

[0079] For example, the terminal responds to the user's task selection operation on the task processing interface, obtains the set of pending task identifiers selected by the user, and the set of pending task identifiers includes multiple pending task identifiers selected by the user; generates a task processing request based on the set of pending task identifiers selected by the user, and sends the task processing request to the corresponding server; the server parses the received task processing request and obtains the set of pending task identifiers.

[0080] In step S212, the to-be-processed task corresponding to the to-be-processed task identifier is obtained from the initial task dependency graph.

[0081] There is a one-to-one correspondence between pending task identifiers and pending tasks, and different pending task identifiers correspond to different pending tasks. It should be noted that the pending tasks obtained from the initial task dependency graph have corresponding task execution data, specifically task instances.

[0082] Among them, the initial task dependency graph includes tasks and task dependencies between tasks. Task dependencies are used to represent the relationship between upstream tasks and downstream tasks. In actual scenarios, the initial task dependency graph can refer to a task instance lineage graph. The task instance lineage graph refers to a graph composed of multiple task instance lineages (with dependencies); task instance lineage refers to a graph that adds dependencies between task instances on the basis of task lineage, making the dependencies more detailed and specific; task lineage refers to the dependency between tasks. This dependency is relatively simple and can reflect a one-to-one relationship, but it cannot well express a one-to-many or many-to-one relationship. For example, if the downstream is a daily task instance and the upstream is a 24-hour task instance, the task lineage cannot be well expressed, so task instance lineage is needed to express this relationship.

[0083] A task is the smallest entity scheduled for execution in a workflow's task scheduling system. It supports jobs in various types, including Hive, MapReduce, and Spark. The initial task dependency graph in this embodiment is a DAG (Directed Acyclic Graph), a directed graph without cycles. When executing a task, the task execution data corresponding to the task is executed according to the initial task dependency graph to complete the task processing.

[0084] In actual scenarios, the task instance lineage graph includes task instances and instance dependencies between task instances, where a task instance refers to a scheduling entity that executes a task at a specified routine time, and the routine time (ExecutionDate) refers to a specific moment in the corresponding scheduling cycle. A task instance has its corresponding task name and routine time. The task name (taskid) is a unique identifier used to identify the task instance. The scheduling cycle refers to the routine scheduling of the task according to certain rules, such as the time of day, day of the week, etc. The instance dependency relationship represents the dependency relationship between task instances. For example, if task instance A depends on task instance B, then task instance B is the upstream task of task instance A, and task instance A needs to depend on the execution result of task instance B.

[0085] For example, referring to Figure 3 Each node in the task instance lineage graph represents a combination of a task instance and an executionDate (routine time). For example, in the diagram, node1_1, node1_2, and node1_3 are task instances of task1 corresponding to different routine times. Node2 is a task instance of task1 corresponding to a single routine time. Node2 depends on node1. Therefore, during the actual scheduling process, node2 will not be scheduled until the task instances corresponding to node1_1, node1_2, and node1_3 have been successfully scheduled.

[0086] Specifically, the server traverses the initial task dependency graph based on the pending task identifier, and filters out the pending tasks corresponding to the pending task identifier from the initial task dependency graph. In this way, only the pending tasks corresponding to the pending task identifier are obtained from the initial task dependency graph, without the need to obtain other tasks, thereby improving the efficiency of obtaining pending tasks and facilitating the subsequent determination of new task dependencies between pending tasks. For example, the pending task identifier is the task identifier corresponding to the task that the user wants to retain for rerunning, which can be input into the server by the user through the terminal. After obtaining it, the server can obtain the corresponding pending task from the task instance lineage graph based on the pending task identifier.

[0087] In step S214 , new task dependencies between tasks to be processed are determined.

[0088] The new task dependency relationship refers to a task dependency relationship constructed based on the determined tasks to be processed.

[0089] Specifically, the server obtains a preset dependency building instruction and, based on the preset dependency building instruction and the pending tasks obtained from the initial task dependency graph, generates task dependencies between the pending tasks as new task dependencies. This reconstructing of new task dependencies between the pending tasks facilitates subsequent execution of the corresponding pending tasks based on the new task dependencies, eliminating the need to execute other tasks. This simplifies the task processing process and further improves task processing efficiency.

[0090] For example, after determining pending tasks, new task dependencies between the pending tasks will be re-determined, where the new task dependencies can be determined based on the original task dependencies. For example, assume that the initial task dependency graph includes Task A, Task B, and Task C, where Task A is the upstream task of Task B, and Task B is the upstream task of Task C. If the pending tasks are Pending Task A and Pending Task C, then the new task dependency between the pending tasks is Pending Task C depends on Pending Task A.

[0091] In step S216 , a new task dependency graph is constructed based on the tasks to be processed and the new task dependency relationships.

[0092] The new task dependency graph refers to a task dependency graph consisting of pending tasks and new task dependency relationships between pending tasks, such as Figure 3 .

[0093] Specifically, the server obtains a preset task dependency graph construction instruction and, based on the preset task dependency graph construction instruction, constructs a corresponding task dependency graph based on the pending tasks and the new task dependencies between the pending tasks, as a new task dependency graph. In this way, by constructing a new task dependency graph that only includes the pending tasks and the new task dependencies between the pending tasks, it is easier to subsequently call the new task dependency graph and execute the corresponding pending tasks without executing other tasks, thereby shortening the task processing chain and further improving task processing efficiency.

[0094] In step S218, the new task dependency graph is called to execute the tasks to be processed.

[0095] For example, since the new task dependency represents the task dependencies between pending tasks, the pending tasks can be reassembled according to the new task dependency to obtain a new task dependency graph that describes the task dependencies between the pending tasks. When executing a task, the new task dependency graph is invoked to execute the corresponding pending task. In this way, since the new task dependency graph only includes the pending tasks corresponding to the pending task identifiers, when executing a task based on the new task dependency graph, only the pending tasks are processed, thereby improving the flexibility of task processing.

[0096] In the above-described task processing method, a set of pending task identifiers including multiple pending task identifiers is obtained, and the pending tasks corresponding to the pending task identifiers are obtained from the initial task dependency graph; then, new task dependencies between the pending tasks are determined, and a new task dependency graph is constructed based on the pending tasks and the new task dependencies; finally, the new task dependency graph is called to execute the pending tasks. In this embodiment, a new task dependency graph can be extracted from the initial task dependency graph based on the obtained pending task identifiers. Since the new task dependency graph only includes the pending tasks corresponding to the pending task identifiers, the task execution chain is shortened, allowing tasks to be completed in a timely manner, thereby improving task processing efficiency.

[0097] As an optional embodiment of the present disclosure, refer to Figure 4 In step S212, the to-be-processed task corresponding to the to-be-processed task identifier is obtained from the initial task dependency graph, which may specifically include the following steps:

[0098] In step S410, task identifiers corresponding to tasks in the initial task dependency graph are traversed in sequence.

[0099] In step S412, a task corresponding to a task identifier that is consistent with the identifier of the task to be processed is obtained as the current task.

[0100] In step S414 , the task identifier of the current downstream task corresponding to the current task is obtained.

[0101] In step S416, when the task identifier of the current downstream task is consistent with the identifier of the task to be processed, the current task and the current downstream task are determined as tasks to be processed, and the process returns to the step of sequentially traversing the task identifiers corresponding to the tasks in the initial task dependency graph.

[0102] In step S418 , when the task identifier of the current downstream task is inconsistent with the task identifier to be processed, the current downstream task is taken as the current task, and the task identifier of the current downstream task corresponding to the current task is returned.

[0103] The user can specify one or more pending tasks that need to be re-run, and the pending task identifiers can be one or more.

[0104] For example, when constructing a new task dependency graph for tasks to be processed, the initial task dependency graph is first obtained, and then the task identifiers corresponding to the tasks in the initial task dependency graph are traversed in sequence; if there is a task corresponding to a task identifier that is consistent with the identifier of the task to be processed, the task is used as the current task, and the task identifier of the current downstream task of the current task is obtained; wherein, if the task identifier corresponding to the current downstream task is consistent with the identifier of the task to be processed, the current task and the current downstream task are used as tasks to be processed, and the process returns to step S410 until the task identifiers corresponding to all tasks in the initial task dependency graph are traversed; if the task identifier corresponding to the current downstream task is inconsistent with the identifier of the task to be processed, the downstream task corresponding to the current downstream task is continued to be obtained, and it is determined whether the task identifier of the downstream task corresponding to the current downstream task is consistent with the identifier of the task to be processed, until a downstream task whose task identifier is consistent with the identifier of the task to be processed is found, or the traversal of all downstream tasks of the current task is completed.

[0105] The technical solution provided by the embodiment of the present disclosure is conducive to screening out tasks with task identifiers consistent with the identifiers of the tasks to be processed from the initial task dependency graph by traversing the task identifiers corresponding to the tasks in the initial task dependency graph in sequence, avoiding omission or wrong selection, thereby making the determined tasks to be processed more accurate, and thus making the new task dependencies between the tasks to be processed determined subsequently more accurate; at the same time, only the tasks to be processed corresponding to the identifiers of the tasks to be processed are obtained from the initial task dependency graph, thereby achieving the purpose of selecting the tasks to be processed, thereby improving the flexibility of task processing.

[0106] As an optional implementation of the present disclosure, tasks have corresponding task priorities. After obtaining the pending tasks corresponding to the pending task identifiers from the initial task dependency graph, the method further includes: treating tasks whose priorities reach the specified priority level as pending tasks.

[0107] In this embodiment, the corresponding pending tasks can be obtained according to the input pending task identifiers, and the initial task dependency graph can be reconstructed based on the obtained pending tasks, so that the new task dependency graph is more concise and efficient; in addition, in order to ensure the normal execution of tasks, this embodiment can also treat tasks with higher priorities in the initial task dependency graph as pending tasks, which can ensure that important tasks are not cut out, thereby ensuring the success rate of task execution.

[0108] As an optional implementation of the present disclosure, in step S212, before obtaining the pending task corresponding to the pending task identifier from the initial task dependency graph, the following contents are also included: obtaining a task dependency model graph, the task dependency model graph including the task identifier, scheduling period and dependency formula corresponding to the task; the dependency formula is used to characterize the task dependency relationship between the task and the upstream task corresponding to the task; obtaining the input starting task identifier and target time interval; obtaining the task corresponding to the task identifier consistent with the starting task name as the current task; taking the current task and the downstream task corresponding to the current task as the target task; generating the task execution data of the target task according to the scheduling period and target time interval of the target task; determining the data dependency between the task execution data according to the dependency formula; combining the task execution data according to the data dependency to obtain the initial task dependency graph.

[0109] Among them, the task dependency model diagram describes the task dependency relationship between tasks. The task dependency model diagram includes task identifiers (such as task names), scheduling cycles and dependency formulas corresponding to multiple tasks. Among them, the task identifier (taskid) is a unique identifier used to identify the task. The scheduling cycle refers to the routine scheduling of the task according to certain rules, such as execution at a certain time of day, execution on a certain day of the week, etc. The dependency formula is used to describe the task dependency relationship between tasks, and the task dependency relationship can reflect the one-to-one relationship between tasks. For example, if task A depends on task B, then task B is the upstream task of task A, and task A needs to depend on the execution result of task B.

[0110] For example, the task dependency model diagram can be as follows: Figure 5 The task instance lineage model diagram shown in the figure, where nodes node1, node2, node3, node4, and node5 represent the task entities corresponding to tasks task1, task2, task3, task4, and task5. The task dependency relationship between nodes is reflected by depExp (dependency formula), that is, Figure 3 In depExp1, depExp2, depExp3, depExp4, and depExp5, for example, depExp1 can reflect the task dependency relationship between task1 and task2, depExp2 can reflect the task dependency relationship between task1 and task4, depExp3 can reflect the task dependency relationship between task1 and task3, depExp4 can reflect the task dependency relationship between task2 and task5, and depExp5 can reflect the task dependency relationship between task3 and task5.

[0111] Among them, the starting task identifier refers to the task identifier corresponding to the first task that needs to be executed, such as the starting task name, and the target time interval is the time interval during which the task needs to be executed; specifically, the target time interval includes the start time startDate and the end time endDate. For example, the user can specify to start execution from the task corresponding to the third node, and the time interval for executing the task is within the previous three days. In this embodiment, the user can input parameters according to actual needs, that is, input the starting task identifier and the target time interval, so that the task dependency diagram can be obtained based on the starting task identifier, the target time interval and the task dependency model diagram.

[0112] For example, first find the task identifier that is consistent with the input starting task identifier from the task dependency model diagram, and then use the task corresponding to the task identifier as the current task. According to the above, there is a dependency relationship between tasks. After determining the first task, that is, the current task, the downstream task corresponding to the current task will be obtained and used together as the target task of this embodiment.

[0113] The dependency formula is used to represent the task dependency relationship between a task and its corresponding upstream task. The data dependency relationship is used to represent the relationship between task execution data, that is, the upstream task execution data and the downstream task execution data. After obtaining the task execution data corresponding to the target task, the data dependency relationship between the task execution data can be derived based on the dependency formula. The task execution data specifically refers to the task instance, and the data dependency relationship specifically refers to the instance dependency relationship between task instances.

[0114] For example, suppose that task execution data 1 is obtained based on target task 1, and task execution data 2 and task execution data 3 are obtained based on target task 2. There is a task dependency relationship in which target task 1 depends on target task 2. Then, a data dependency relationship can be obtained in which task execution data 1 depends on task execution data 2 and task execution data 3. Clearly, the data dependency relationships between task execution data are generated based on the task dependency relationships between tasks, making the data dependency relationships more detailed and specific.

[0115] Since data dependency represents the dependency between task execution data, the task execution data can be combined according to the data dependency to obtain a task dependency graph that describes the data dependency between all task execution data. When executing a task, the corresponding task execution data will be executed according to the task dependency graph.

[0116] The technical solution provided by the embodiment of the present disclosure can obtain a task dependency graph for any target time interval through a full task dependency model graph; since the task dependency model graph abstractly describes the dependencies between tasks, it occupies less memory resources compared to the task dependency graph that describes the data dependencies between task execution data, thereby saving server resources and further improving the utilization of server resources; in addition, the task dependency model graph of this embodiment can be reused, thereby improving data utilization.

[0117] As an optional implementation of the present disclosure, task execution data of the target task is generated according to the scheduling period and target time interval of the target task, including: determining at least one task execution time according to the scheduling period and target time interval of the target task; combining the target task with the task execution time to obtain at least one task execution data.

[0118] The task execution time refers to the specific time when the task is executed, specifically the routine time.

[0119] Specifically, the server can determine one or more task execution times based on the scheduling period of the target task and the target time interval input by the user. After obtaining the task execution time, the task execution time can be used as the execution time of the target task, thereby obtaining at least one task execution data of the target task.

[0120] For example, assuming that the input target time interval is the first three days, if the scheduling period of the target task is 4 o'clock every day, three routine times can be obtained. Combining the three routine times with the target task can obtain three task instances.

[0121] The technical solution provided by the embodiment of the present disclosure generates task execution data of the target task within the target time interval according to the scheduling period and target time interval of the target task, which is conducive to improving the accuracy of the task execution data of the determined target task, thereby ensuring the success rate of task execution, and at the same time avoiding the defect of missing the task execution data of the target task, resulting in task execution failure and the need to re-execute the task, which leads to extended task processing time.

[0122] As an optional implementation of the present disclosure, obtaining a task dependency model diagram specifically includes the following contents: obtaining task metadata; the task metadata includes the task identifier, scheduling period and upstream dependency data corresponding to the task; the upstream dependency data includes the upstream task identifier and upstream dependency descriptor of the upstream task corresponding to the task; generating the task dependency formula based on the upstream dependency descriptor and the scheduling period of the upstream task; generating the task dependency model diagram based on the task identifier and dependency formula corresponding to the task.

[0123] Among them, task metadata is the basic content of a task and the data source for the entire task dependency model diagram. Refer to Figure 6 , task metadata may include the task name (taskId), the type of the task (taskType, such as a hive task or a spark task, etc.), the scheduling period of the task (cronExp, which is a cron rule expression, such as specifying what time to run every day / week), the creation time of the task (createTime), and the upstream dependent data (upstreams) of the task. The upstream dependent data includes the previous task name of the upstream task and the dependency descriptor. The dependency descriptor can be empty, indicating that the upstream and downstream tasks are in the same period.

[0124] Among them, the format of the task dependency descriptor is:

[0125] {"op": "[or|empty|and|max|min]", "value": "[p_date - m|p_date|p_date - m,p_date - n]"}.

[0126] Specifically, the dependency descriptor is divided into two parts:

[0127] (1) op (operator): The default value is "empty", which is the same as "and", and all must be satisfied. "or" requires at least one to be satisfied. "max" obtains the maximum value within the interval, and "min" obtains the minimum value within the interval.

[0128] (2) value (dependency time): Corresponds to a time or a time interval.

[0129] Assume that both the upstream and downstream tasks are daily tasks. The dependency descriptor of the downstream task depending on the upstream task can be:

[0130] "{p_date - n}" and "and{p_date - n}" are equivalent: depend on the specified dependency time.

[0131]

[0131] "{p_date - m,p_date - n}" (m < n): Depend on all dependency times within the specified interval.

[0132] "or{p_date - m,p_date - n}" (m < n): Depend on any dependency time within the specified interval;

[0133] "max{p_date - m,p_date - n}" (m < n): Depend on the largest (ready) dependency time that meets the conditions within the specified interval.

[0134] "max{p_date - n}" (<= p_date - n): Depend on the largest dependency time that meets the conditions.

[0135] Specifically, the dependency formula between tasks can be derived based on the dependency descriptors and scheduling cycles corresponding to the tasks. After obtaining the dependency formula of the tasks, the dependency formula and task identifiers of the tasks can be saved together in the task instance lineage model diagram template according to the task dependency relationship, thereby obtaining the task dependency model diagram.

[0136] The technical solution provided by the embodiment of the present disclosure, by obtaining a task dependency model diagram, is conducive to subsequently obtaining an initial task dependency diagram through the task dependency model diagram, and then obtaining the pending tasks corresponding to the pending task identifiers from the initial task dependency diagram, and only obtaining the pending tasks that the user cares about, which is conducive to improving the flexibility of task re-running and making effective use of server resources.

[0137] As an optional implementation of the present disclosure, the upstream dependency descriptor includes an operator and a dependency time, and a dependency formula of the task is generated according to the upstream dependency descriptor and the scheduling period of the upstream task, including: obtaining a bitmap; the bitmap includes assignment positions for the scheduling period, operator and dependency time; according to the conversion relationship of the scheduling period, the scheduling period of the upstream task is converted into a first result value; according to the conversion relationship of the operator, the operator is converted into a second result value, and according to the conversion relationship of the dependency time, the dependency time is converted into a third result value; the first result value, the second result value and the third result value are filled into the corresponding assignment positions in sequence to obtain the dependency formula of the task.

[0138] In this embodiment, the dependency descriptor can be converted into a dependency expression. The dependency expression is a simplified representation of the dependency descriptor. A bitmap (corresponding to a long integer, occupying 8 bytes) can be used to express complex task dependencies. The use of a bitmap greatly saves storage space and reduces memory usage. At the same time, the bitmap bit processing is efficient and fast.

[0139] The bitmap sets corresponding assignment positions for the scheduling cycle, operator and dependency time respectively. After converting the scheduling cycle, operator and dependency time according to the corresponding conversion relationships, the corresponding first result value, second result value and third result value can be obtained; the first result value, second result value and third result value are filled in the corresponding assignment positions in turn to obtain the dependency relationship formula of this embodiment.

[0140] For example, see Figure 7 , the assignment meaning of each bit in the 8-byte (64-bit) dependency expression (i.e., dependency expression) is:

[0141] 1. According to the scheduling period of the upstream task, the calculated result value is assigned to the 0-1 position of the dependency expression.

[0142] 2. Based on the second part of the dependency descriptor (value, dependency time), the calculated result value is assigned to bits 2-60 and 63 of the dependency expression.

[0143] 3. According to the first part of the dependency descriptor (op, operator), the calculated result data is assigned to bits 61-62 of the dependency expression.

[0144] Among them, the 63rd bit of the dependency expression indicates whether it depends on the past task or the future task, where 0 indicates dependence on the past and 1 indicates dependence on the future.

[0145] Specifically, the dependency descriptor conversion bit dependency expression can be as follows:

[0146] Example 1: Dependency descriptor "{p_date-0}", the derived dependency expression: 000...000 (64 zeros), the corresponding integer value is 0, indicating that the two tasks are executed in the same cycle.

[0147] Example 2: The dependency descriptor "{p_date-1}" can be derived into the dependency expression: 000...01000, which corresponds to an integer value of 8, indicating that the downstream task depends on the previous scheduling cycle of the upstream task.

[0148] Example 3: The dependency descriptor "{p_date-4, p_date-1}" can be derived into the dependency expression: 00...01111000, which corresponds to an integer value of 120, indicating that the downstream task depends on the first four consecutive scheduling periods of the upstream task.

[0149] The technical solution provided by the embodiment of the present disclosure expresses complex task dependencies through simple dependency formulas, which is conducive to saving space storage, thereby reducing server memory usage, and making server processing calculations more efficient and faster, further improving the processing efficiency of subsequent tasks; at the same time, by generating task dependency formulas, it is conducive to subsequently combining the task identifiers corresponding to the tasks to generate a task dependency model diagram.

[0150] As an optional implementation of the present disclosure, in step S218, a new task dependency graph is called to execute the tasks to be processed, which specifically includes the following contents: extracting new task dependencies from the new task dependency graph; executing each task to be processed in sequence according to the new task dependencies to obtain the corresponding task processing results.

[0151] Specifically, the server traverses the new task dependency graph in sequence, extracts new task dependencies from the new task dependency graph, and executes the pending tasks corresponding to each pending task identifier in sequence according to the pending task identifiers in the new task dependency, to obtain the corresponding task processing results.

[0152] The technical solution provided by the embodiment of the present disclosure shortens the task execution link and enables tasks to be completed in a timely manner, thereby improving task processing efficiency, because the new task dependency graph only includes the pending tasks corresponding to the pending task identifiers. At the same time, only the pending tasks in the new task dependency graph are processed, and there is no need to process other tasks, so that server resources are effectively utilized, thereby improving server resource utilization.

[0153] In order to enable those skilled in the art to better understand the implementation process of reconstructing the task instance blood relationship graph in this embodiment, the following uses a specific example to illustrate. Figure 8 , the overall reconstruction process is divided into three parts:

[0154] Part 1: Combine the task instance lineage model graph with the user-specified root node name taskId, startDate (start time), endDate (end time), and the filter set (target task name) to generate the original task instance lineage graph.

[0155] Part 2: Convert the original task instance lineage graph into a map storage format to obtain map1 (key: task instance, value: the direct downstream task instance corresponding to the task instance), and use the "nearest downstream algorithm" to find the mapping relationship between the filtered target task instance and its corresponding downstream task instance to obtain map2 (where the taskId in the key and value both belong to the user-specified taskid filter set).

[0156] Part 3: Generate the final, new task instance lineage graph based on map2.

[0157] Among them, for the second part, the specific implementation process is as follows:

[0158] 1. Generate a mapping table from each node to its direct downstream node based on the task instance lineage graph:

[0159] map1:Map <TaskInstance,set <taskinstance>>; Among them, TaskInstance is the task instance, set <taskinstance>It is the downstream task instance of this task instance;

[0160] 2. Initialize the mapping table from the "to be filtered" node to its direct downstream nodes:

[0161] map2:Map <TaskInstance,set <taskinstance>>

[0162] 3. Traverse map1 (key, value) in sequence;

[0163] 4. Determine whether the traversal is complete; if so, start executing the third part; if not, execute step 5;

[0164] 5. Get the key corresponding to a key-value pair in map1, type TaskInstance;

[0165] 6. Determine whether the key.taskId (task name) corresponding to the key is in the filtered set; if so, execute step 7; if not, execute step 3;

[0166] 7. Use the "Get Nearest Downstream Algorithm" to find the direct downstream node of the key. The direct downstream node must also be in the filtered set.

[0167] 8. Update (key,"direct downstream node") into map2.

[0168] Among them, for the third part, the specific implementation process is as follows:

[0169] 1. Initialize a new task instance lineage graph object graph, type is graph<TaskInstance,String> ;

[0170] 2. Traverse map2 (key, value) in sequence;

[0171] 3. Determine whether the traversal is complete; if so, obtain a new task instance lineage graph; if not, execute step 4;

[0172] 4. Get the value corresponding to the key-value pair in map2, type Set <taskinstance>;

[0173] 5. Traverse the downstream nodes e in value in turn, type TaskInstance;

[0174] 6. Determine whether the traversal is complete; if so, proceed to step 2; if not, proceed to step 6;

[0175] 7. Generate two nodes, e and key, and save the dependency of e->key to the graph, where e is the downstream node and key is the upstream node.

[0176] For step 7 of the second part, refer to Figure 9 The specific implementation process is as follows:

[0177] 1. Given a task instance ti: (type) TaskInstance;

[0178] 2. Initialize the direct downstream node set result:Set of ti <taskinstance>;

[0179] 3. Initialize the temporary storage stack: Stack <taskinstance>;

[0180] 4.map1 gets the direct downstream node set of ti downs:Set <taskinstance>

[0181] 5. Determine whether downs is empty; if so, return result (get an empty set of direct downstream nodes); if not, execute step 6;

[0182] 6. Push all downs into the stack;

[0183] 7. Determine whether stack is empty; if so, return result (get an empty set of direct downstream nodes); if not, execute step 8;

[0184] 8. Pop the task instance d at the top of the stack;

[0185] 9. Determine whether the taskId of d is in the filtered set. If so, add d to the set of direct downstream nodes (result.add(d)). If not, execute step 10.

[0186] 10.map1 gets the direct downstream set of d newDowns:Set <taskinstance>, push all newDowns into the stack and execute step 7.

[0187] Reference Figure 10 On the left is the original task instance lineage graph constructed by the task instance lineage model graph. The user wants to retain (filter) the task instances involved in tasks 1, 5, 7, 9, and 10. After the reconstruction processing of this embodiment, the reconstructed task instance lineage graph on the right is obtained. It can be seen that the task instance lineage graph reconstructed by this embodiment shortens the task execution link, so the task can be completed in time, which improves work efficiency.

[0188] In existing workflow scheduling systems, rerunning task instances is implemented using a task instance lineage graph. Based on a user-selected root node, all downstream task instances corresponding to the task instance lineage graph are retrieved, and the root node and all its corresponding downstream task instances are rerun. This implementation of rerunning all downstream task instances is applicable in certain scenarios, but is not universally applicable. Simultaneously rerunning all downstream task instances can lead to the following issues: Failure to meet hierarchical guarantees: Tasks are divided according to importance. When resources are limited, only task instances corresponding to high-priority tasks are allowed to execute, while task instances corresponding to low-priority tasks are blocked. Rerunning all downstream task instances may cause some tasks to fail, impacting the execution of other task instances and failing to meet hierarchical guarantees. Resource waste: When rerunning a large number of downstream task instances, the inability to filter out low-priority task instances means that all downstream task instances must be executed, requiring a significant amount of rerunning resources. Data generation delays: When rerunning a large number of downstream task instances, users may only be interested in a small number of tasks. Task instances corresponding to these tasks may not be executed in a timely manner due to resource constraints or lengthy links, resulting in delayed data generation.

[0189] The application of this embodiment can have the following advantages: increasing the flexibility of task re-running: users can customize the selection of specified tasks to re-run (filtering low-priority task instances), among which high-priority task instances can also be re-run, so that task instances of tasks that users care about and important tasks can be re-run; effective use of resources; reconstructing the task instance lineage graph to effectively crop the original task instance lineage graph, avoiding the re-running of all downstream task instances, and effectively utilizing resources; ensuring timely data output; constructing a task instance lineage graph based on the tasks that users care about, greatly reducing the length of the execution link, and re-running tasks are executed in a timely manner, thereby further ensuring data output.

[0190] It should be understood that although Figure 2 and Figure 4 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Figure 2 and Figure 4 At least part of the steps may include multiple steps or multiple stages. These steps or stages are not necessarily performed at the same time, but can be performed at different times. The order of execution of these steps or stages is not necessarily one by one, but can be performed in turn or alternately with other steps or at least part of the steps or stages in other steps.

[0191] Figure 11 FIG. 1 is a block diagram of a task processing device according to an exemplary embodiment. Figure 11 The device includes a task identification acquisition unit 1110, a task acquisition unit 1112, a dependency determination unit 1114, a dependency graph construction unit 1116 and a task execution unit 1118.

[0192] The task identifier acquisition unit 1110 is configured to acquire a set of pending task identifiers; the set of pending task identifiers includes a plurality of pending task identifiers;

[0193] The task acquisition unit 1112 is configured to execute the task to be processed corresponding to the task to be processed identifier from the initial task dependency graph;

[0194] The dependency determination unit 1114 is configured to determine a new task dependency between the tasks to be processed;

[0195] The dependency graph construction unit 1116 is configured to construct a new task dependency graph according to the to-be-processed tasks and the new task dependency relationships;

[0196] The task execution unit 1118 is configured to execute and call the new task dependency graph to execute the task to be processed.

[0197] In one embodiment, the task acquisition unit 1112 is further configured to execute the step of sequentially traversing the task identifiers corresponding to the tasks in the initial task dependency graph; acquiring the task corresponding to the task identifier that is consistent with the identifier of the task to be processed as the current task; acquiring the task identifier of the current downstream task corresponding to the current task; when the task identifier of the current downstream task is consistent with the identifier of the task to be processed, determining the current task and the current downstream task as the tasks to be processed, and returning to the step of sequentially traversing the task identifiers corresponding to the tasks in the initial task dependency graph; when the task identifier of the current downstream task is inconsistent with the identifier of the task to be processed, determining the current downstream task as the current task, and returning to the step of acquiring the task identifier of the current downstream task corresponding to the current task.

[0198] In one embodiment, the task processing device also includes an initial task dependency graph acquisition unit, which is configured to execute and acquire a task dependency model graph, wherein the task dependency model graph includes a task identifier, a scheduling period, and a dependency formula corresponding to the task; the dependency formula is used to characterize the task dependency relationship between the task and the upstream task corresponding to the task; obtain the input starting task identifier and target time interval; obtain the task corresponding to the task identifier consistent with the starting task name as the current task; use the current task and the downstream task corresponding to the current task as the target task; generate task execution data of the target task according to the scheduling period and target time interval of the target task; determine the data dependency between the task execution data according to the dependency formula; and combine the task execution data according to the data dependency to obtain the initial task dependency graph.

[0199] In one of the embodiments, the initial task dependency graph acquisition unit is further configured to determine at least one task execution time based on the scheduling period and target time interval of the target task; and combine the target task with the task execution time to obtain at least one task execution data.

[0200] In one embodiment, the initial task dependency graph acquisition unit is further configured to execute the acquisition of task metadata; the task metadata includes the task identifier, scheduling period and upstream dependency data corresponding to the task; the upstream dependency data includes the upstream task identifier and upstream dependency descriptor of the upstream task corresponding to the task; based on the upstream dependency descriptor and the scheduling period of the upstream task, the dependency formula of the task is generated; based on the task identifier and dependency formula corresponding to the task, a task dependency model graph is generated.

[0201] In one embodiment, the upstream dependency descriptor includes an operator and a dependency time, and the initial task dependency graph acquisition unit is further configured to execute an acquisition bitmap; the bitmap includes assignment positions for the scheduling period, operator and dependency time; according to the conversion relationship of the scheduling period, the scheduling period of the upstream task is converted into a first result value; according to the conversion relationship of the operator, the operator is converted into a second result value, and according to the conversion relationship of the dependency time, the dependency time is converted into a third result value; the first result value, the second result value and the third result value are filled in the corresponding assignment positions in sequence to obtain the dependency formula of the task.

[0202] In one embodiment, the task execution unit 1118 is further configured to extract new task dependencies from the new task dependency graph; and execute each to-be-processed task in sequence according to the new task dependency to obtain corresponding task processing results.

[0203] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0204] Figure 12 1 is a block diagram of a device 1200 for executing the above-mentioned task processing method according to an exemplary embodiment. For example, the device 1200 may be a server. Figure 12 The device 1200 includes a processing component 1220, which further includes one or more processors and a memory resource represented by a memory 1222 for storing instructions, such as applications, that can be executed by the processing component 1220. The application stored in the memory 1222 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1220 is configured to execute the instructions to perform the task processing method described above.

[0205] The device 1200 may also include a power supply component 1224 configured to perform power management of the device 1200, a wired or wireless network interface 1226 configured to connect the device 1200 to a network, and an input / output (I / O) interface 1228. The device 1200 may operate based on an operating system stored in the memory 1222, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, or the like.

[0206] In an exemplary embodiment, a storage medium including instructions is also provided, such as memory 1222 including instructions, which can be executed by a processor of device 1200 to perform the above method. The storage medium can be a non-transitory computer-readable storage medium, such as a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0207] In an exemplary embodiment, a computer program product is also provided, which includes a computer program stored in a readable storage medium, and at least one processor of a device reads and executes the computer program from the readable storage medium, so that the device performs the task processing method described in any embodiment of the present disclosure.

[0208] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the following claims.

[0209] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.< / taskinstance> < / taskinstance> < / taskinstance> < / taskinstance> < / taskinstance> < / taskinstance> < / taskinstance> < / taskinstance>

Claims

1. A task processing method, characterized in that: include: Obtain a set of pending task identifiers; the set of pending task identifiers includes multiple pending task identifiers; Obtaining the to-be-processed task corresponding to the to-be-processed task identifier from the initial task dependency graph; the initial task dependency graph is determined according to the task dependency formula in the task dependency model graph; the task dependency formula is obtained by sequentially filling the first result value of the scheduling period of the upstream task corresponding to the task, the second result value of the operator of the upstream task corresponding to the task, and the third result value of the dependency time of the upstream task corresponding to the task into the assignment position of the scheduling period of the upstream task, the assignment position of the operator of the upstream task, and the assignment position of the dependency time of the upstream task included in the bitmap; Determining new task dependencies between the tasks to be processed; Constructing a new task dependency graph according to the tasks to be processed and the new task dependency; The new task dependency graph is called to execute the tasks to be processed.

2. The task processing method according to claim 1, characterized in that: The step of obtaining the to-be-processed task corresponding to the to-be-processed task identifier from the initial task dependency graph includes: Traverse the task identifiers corresponding to the tasks in the initial task dependency graph in sequence; Acquire a task corresponding to a task identifier that is consistent with the task identifier to be processed as the current task; Obtaining a task identifier of a current downstream task corresponding to the current task; When the task identifier of the current downstream task is consistent with the identifier of the task to be processed, the current task and the current downstream task are determined as tasks to be processed, and the step of sequentially traversing the task identifiers corresponding to the tasks in the initial task dependency graph is returned; When the task identifier of the current downstream task is inconsistent with the task identifier to be processed, the current downstream task is used as the current task, and the step of obtaining the task identifier of the current downstream task corresponding to the current task is returned.

3. The task processing method according to claim 1, characterized in that: Before obtaining the to-be-processed task corresponding to the to-be-processed task identifier from the initial task dependency graph, the method further includes: Obtaining a task dependency model diagram, wherein the task dependency model diagram includes a task identifier corresponding to the task, a scheduling period, and a dependency formula; the dependency formula is used to characterize the task dependency relationship between the task and an upstream task corresponding to the task; Get the input start task ID and target time interval; Acquire a task corresponding to a task identifier that is consistent with the start task identifier as the current task; Taking the current task and the downstream task corresponding to the current task as the target task; Generate task execution data of the target task according to the scheduling period and the target time interval of the target task; Determine the data dependency relationship between the task execution data according to the dependency relationship formula; The task execution data are combined according to the data dependency relationship to obtain the initial task dependency relationship graph.

4. The task processing method according to claim 3, characterized in that: The step of generating the task execution data of the target task according to the scheduling period and the target time interval of the target task includes: Determining at least one task execution time according to the scheduling period of the target task and the target time interval; The target task is combined with the task execution time to obtain at least one task execution data.

5. The task processing method according to claim 3, characterized in that: The step of obtaining the task dependency model diagram includes: Acquire task metadata; the task metadata includes a task identifier, a scheduling period, and upstream dependency data corresponding to the task; the upstream dependency data includes an upstream task identifier and an upstream dependency descriptor of an upstream task corresponding to the task; Generate a dependency relation formula of the task according to the upstream dependency descriptor and the scheduling period of the upstream task; A task dependency model diagram is generated according to the task identifier corresponding to the task and the dependency formula.

6. The task processing method according to claim 5, characterized in that: The upstream dependency descriptor includes an operator and a dependency time, and the generating of the dependency relationship of the task according to the upstream dependency descriptor and the scheduling period of the upstream task includes: Obtaining a bitmap; the bitmap comprising assignment locations for the scheduling period, the operator, and the dependent time; According to the conversion relationship of the scheduling period, the scheduling period of the upstream task is converted into a first result value; according to the conversion relationship of the operator, the operator is converted into a second result value; and according to the conversion relationship of the dependent time, the dependent time is converted into a third result value; The first result value, the second result value and the third result value are sequentially filled into the corresponding assignment positions to obtain the dependency relationship formula of the task.

7. The task processing method according to any one of claims 1 to 6, characterized in that: The calling of the new task dependency graph to execute the task to be processed includes: Extracting the new task dependency relationship from the new task dependency graph; According to the new task dependency, each of the tasks to be processed is executed in sequence to obtain the corresponding task processing results.

8. A task processing device, characterized in that: include: A task identifier acquisition unit is configured to execute acquisition of a set of task identifiers to be processed; the set of task identifiers to be processed includes a plurality of task identifiers to be processed; The task acquisition unit is configured to execute the acquisition of the pending task corresponding to the pending task identifier from the initial task dependency graph; the initial task dependency graph is determined according to the task dependency formula in the task dependency model graph; the task dependency formula is obtained by sequentially filling the first result value of the scheduling period of the upstream task corresponding to the task, the second result value of the operator of the upstream task corresponding to the task, and the third result value of the dependency time of the upstream task corresponding to the task into the assignment position of the scheduling period of the upstream task, the assignment position of the operator of the upstream task, and the assignment position of the dependency time of the upstream task included in the bitmap; A dependency determination unit, configured to determine a new task dependency between the tasks to be processed; A dependency graph construction unit is configured to construct a new task dependency graph according to the to-be-processed tasks and the new task dependency relationships; The task execution unit is configured to execute and call the new task dependency graph to execute the task to be processed.

9. The task processing device according to claim 8, characterized in that: The task acquisition unit is configured to sequentially traverse the task identifiers corresponding to the tasks in the initial task dependency graph; acquire the task corresponding to the task identifier that is consistent with the task identifier to be processed as the current task; Obtain the task identifier of the current downstream task corresponding to the current task; when the task identifier of the current downstream task is consistent with the identifier of the task to be processed, determine the current task and the current downstream task as the tasks to be processed, and return to the step of sequentially traversing the task identifiers corresponding to the tasks in the initial task dependency graph; when the task identifier of the current downstream task is inconsistent with the identifier of the task to be processed, take the current downstream task as the current task, and return to the step of obtaining the task identifier of the current downstream task corresponding to the current task.

10. The task processing device according to claim 8, characterized in that: The device also includes an initial task dependency graph acquisition unit, which is configured to execute acquisition of a task dependency model graph, wherein the task dependency model graph includes a task identifier, a scheduling period, and a dependency formula corresponding to the task; the dependency formula is used to characterize the task dependency relationship between the task and the upstream task corresponding to the task; obtain an input start task identifier and a target time interval; obtain a task corresponding to a task identifier consistent with the start task identifier as the current task; and use the current task and the downstream task corresponding to the current task as the target task; Generate task execution data of the target task according to the scheduling period and the target time interval of the target task; determine data dependency between the task execution data according to the dependency formula; combine the task execution data according to the data dependency to obtain the initial task dependency graph.

11. The task processing device according to claim 10, characterized in that: The initial task dependency graph acquisition unit is further configured to determine at least one task execution time according to the scheduling period of the target task and the target time interval; and combine the target task with the task execution time to obtain at least one task execution data.

12. The task processing device according to claim 10, characterized in that: The initial task dependency graph acquisition unit is configured to execute the acquisition of task metadata; the task metadata includes a task identifier, a scheduling period and upstream dependency data corresponding to the task; the upstream dependency data includes an upstream task identifier and an upstream dependency descriptor of an upstream task corresponding to the task; a dependency formula of the task is generated according to the upstream dependency descriptor and the scheduling period of the upstream task; and a task dependency model graph is generated according to the task identifier corresponding to the task and the dependency formula.

13. The task processing device according to claim 12, characterized in that: The upstream dependency descriptor includes an operator and a dependency time, and the initial task dependency graph acquisition unit is further configured to execute and acquire a bitmap; the bitmap includes assignment positions for the scheduling period, the operator and the dependency time; according to the conversion relationship of the scheduling period, the scheduling period of the upstream task is converted into a first result value; according to the conversion relationship of the operator, the operator is converted into a second result value, and according to the conversion relationship of the dependency time, the dependency time is converted into a third result value; The first result value, the second result value and the third result value are sequentially filled into the corresponding assignment positions to obtain the dependency relationship formula of the task.

14. The task processing device according to any one of claims 8 to 13, characterized in that: The task execution unit is further configured to extract the new task dependency from the new task dependency graph; and execute each of the tasks to be processed in sequence according to the new task dependency to obtain a corresponding task processing result.

15. A server, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the task processing method according to any one of claims 1 to 7.

16. A storage medium, characterized in that: When the instructions in the storage medium are executed by a processor of the server, the server is enabled to execute the task processing method as described in any one of claims 1 to 7.

17. A computer program product, the program product comprising a computer program, characterized in that: The computer program is stored in a readable storage medium, and at least one processor of the device reads and executes the computer program from the readable storage medium, so that the device performs the task processing method according to any one of claims 1 to 7.

Images