Task processing method and computer program product

Abstract

The invention relates to a task processing method and a computer program product. The method is applied to a task processing device, the task processing device comprises an entry module, an analysis module and an execution module, and the method comprises the steps that the entry module obtains a workflow file of a workflow and obtains a workflow tree based on the workflow file; the analysis module obtains all sub-task nodes of the task node for each task node, determines executable sub-task nodes from all the sub-task nodes of the task node when the task state of the task node is a completed state, and pushes the executable sub-task nodes to the execution module; when the execution module determines that the task nodes meet the execution conditions, the execution module pushes the task nodes to an execution queue to be executed, polls and checks the task information of the task nodes, updates the task states of the task nodes according to the task information, and pushes the executed task nodes to the analysis module. By adopting the method, the task execution efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a task processing method and a computer program product. Background Art

[0002] A workflow is an operational process consisting of multiple basic tasks, each with its own sequence and data-transfer dependencies. Workflow scheduling is a key technology for coordinating multi-task computations, directly impacting the efficiency of task computations.

[0003] In traditional technology, workflow scheduling technology analyzes the sequence of tasks and then drives task execution and status updates through the workflow scheduler.

[0004] However, driving task execution through a workflow scheduler requires managing each task and performing periodic polling on each task, resulting in low task execution efficiency. Summary of the Invention

[0005] Based on this, it is necessary to provide a task processing method and a computer program product that can improve task execution efficiency in response to the above technical problems.

[0006] In a first aspect, the present application provides a task processing method, which is applied to a task processing device, the task processing device including an entry module, a parsing module, and an execution module, the method comprising:

[0007] The entry module obtains the workflow file of the workflow and obtains a workflow tree based on the workflow file; wherein the workflow tree includes multiple task nodes, each task node includes a task state, a pre-dependency and a post-dependency, the pre-dependency includes all parent task nodes of the task node, and the post-dependency includes all child task nodes of the task node;

[0008] The parsing module obtains all subtask nodes of each task node for each task node. When the task status of the task node is completed, it determines the executable subtask nodes from all subtask nodes of the task node and determines whether the executable subtask nodes meet the dependency conditions. If so, it pushes the executable subtask nodes to the execution module.

[0009] The execution module pushes each task node in the execution module to the execution queue for execution when it determines that the task node meets the execution conditions, and patrols to check the task information of the task node, updates the task status of the task node according to the task information, and after determining that the task node is completed, pushes the completed task node to the parsing module.

[0010] In one embodiment, the parsing module includes a parsing coroutine and a parsing channel corresponding to the parsing coroutine; the parsing module obtains all subtask nodes of the task node for each task node, including: the parsing coroutine obtains the task node from the corresponding parsing channel; the parsing coroutine searches for all subtask nodes of the task node based on the task node based on the workflow tree; correspondingly, when the task status of the task node is completed, the executable subtask node is determined from all subtask nodes of the task node, and it is judged whether the executable subtask node meets the dependency condition. If so, the executable subtask node is pushed to the execution module, including: when the parsing coroutine determines that the task status of the task node is completed, the executable subtask node is determined from all subtask nodes of the task node, and it is judged whether the executable subtask node meets the dependency condition. If so, the executable subtask node is pushed to the execution module.

[0011] In one embodiment, the method further includes: when the parsing coroutine determines that the task status of the task node is in an incomplete state, updating the task status of all subtask nodes of the task node to skip; and / or, when the parsing coroutine determines that the task status of the task node is in a completed state and the executable subtask node does not meet the dependency condition, updating the task status of the executable subtask node to skip.

[0012] In one embodiment, the parsing coroutine is based on the workflow tree, and after obtaining all subtask nodes of the task node according to the task node, the method further includes: the parsing coroutine determines whether there is an executable subtask node among all the subtask nodes of the task node; if so, the parsing coroutine executes the step of determining whether the executable subtask node meets the dependency condition; if not, the parsing coroutine determines the workflow status of the workflow based on the task status of all task nodes in the workflow tree, and when the workflow status is the completed state, updates the workflow status of the workflow stored in the database.

[0013] In one embodiment, the execution module includes an initialization coroutine, an initialization queue and an execution channel; the method further includes: the initialization coroutine obtains a task node from the initialization queue and determines whether the task node is in the execution module; if not, if the task node meets the binding condition, the initialization coroutine binds the task node to the label and pushes the bound task node to the execution channel.

[0014] In one embodiment, the execution module includes an execution coroutine, and the execution module pushes each task node in the execution module to an execution queue for execution when determining that the task node meets the execution condition, including: the execution coroutine obtains the bound task node from the execution channel, wherein the bound task node is the task node in the execution module; the execution coroutine determines whether the input data of the bound task node meets the input condition; if so, determines that the bound task node meets the execution condition, and pushes the bound task node to the execution queue for execution.

[0015] In one embodiment, the method further includes: if the bound task node does not meet the execution condition, updating the task status of the bound task node to skip; deleting the label of the bound task node, and pushing the task node after the label is deleted to the parsing module.

[0016] In one embodiment, after pushing the bound task node to the execution queue for execution, the method further includes: if the bound task node fails to be submitted, updating the task status of the bound task node to failure; deleting the label of the bound task node, and pushing the task node after the label is deleted to the parsing module.

[0017] In one embodiment, after the entry module obtains the workflow file and obtains the workflow tree based on the workflow file, the method further includes: the entry module determines whether there is a task node to be executed in the workflow tree; if so, the entry module stores the workflow tree in the memory and pushes the task node to be executed to the execution module; if not, the entry module determines the workflow status of the workflow based on the task status of all task nodes in the workflow tree, and when the workflow status is completed, updates the workflow status of the workflow stored in the database.

[0018] In a second aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any one of the methods described in the first aspect when executing the computer program.

[0019] In a third aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any one of the methods described in the first aspect.

[0020] In a fourth aspect, the present application further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of any one of the methods described in the first aspect.

[0021] The above-mentioned task processing method and computer program product are applied to a task processing device, wherein an entry module in the task processing device obtains a workflow file of a workflow and obtains a workflow tree based on the workflow file; wherein the workflow tree includes multiple task nodes, each task node includes a task status, a pre-dependency and a post-dependency, the pre-dependency includes all parent task nodes of the task node, and the post-dependency includes all child task nodes of the task node; then the parsing module in the task processing device obtains all child task nodes of the task node for each task node, and when the task status of the task node is a completed state, determines the executable child task nodes from all child task nodes of the task node, and judges the executable child tasks Whether the node meets the dependency conditions, if so, the executable subtask node is pushed to the execution module; the execution module in the task processing device pushes the task node to the execution queue for execution for each task node in the execution module when it determines that the task node meets the execution conditions, and patrols to check the task information of the task node, updates the task status of the task node according to the task information, and after determining that the task node is executed, pushes the completed task node to the parsing module. In this way, after the user submits the workflow file to the task processing device, there is no need for the workflow scheduler to drive the task execution, but a self-driven mode of task driving task and task updating task status, thereby improving the task execution efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments of the present application or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying any creative work.

[0023] Figure 1 is a structural block diagram of a task processing device in one embodiment;

[0024] Figure 2 1 is a flowchart of a task processing method in one embodiment;

[0025] Figure 3 A schematic diagram of a process for an entry module to perform tasks in one embodiment;

[0026] Figure 4 A schematic diagram of a process for executing tasks by a parsing module in one embodiment;

[0027] Figure 5 A schematic diagram of a flow chart of an execution module executing a task in one embodiment;

[0028] Figure 6Flow chart for another embodiment to execute module to execute task;

[0029] Figure 7 Internal structure diagram of computer device in one embodiment. DETAILED DESCRIPTION

[0030] In order to make the purposes, technical solutions and advantages of the present application more clear, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0031] A workflow is a running flow composed of multiple basic tasks, and the multiple tasks have a sequence and a data transmission dependency relationship. The workflow scheduling technology is a key technology for solving the coordinated computation of multiple tasks, and directly affects the efficiency of task computation.

[0032] At present, the workflow scheduling technology has made some progress in process driving, process analysis, data flow transmission and the like, but for users, there is a greater necessity for high efficiency and high expansion. Therefore, it is of great significance to improve the process driving to save system resources, improve efficiency, and support more flexible data definition and more inter-task relationship dependencies.

[0033] In the conventional technology, the workflow scheduling technical solution is one of the following two kinds. One is to analyze the sequence of tasks through a DAG (Directed Acyclic Graph, directed acyclic graph), and then to drive the task execution and state update through a workflow scheduler. The other is to analyze the sequence of tasks recursively, and directly execute the task in the recursive function, and finally update the state by the upper calling function.

[0034] However, the task execution driven by the workflow scheduler needs to manage each task, and needs to perform a timing polling on each task, resulting in low task execution efficiency and difficult state management. The recursive execution of tasks cannot achieve true concurrent processing, and there is still a sequence in logical processing. In addition, the current workflow task execution dependency is only dependent on the successful execution of the previous task, and cannot execute the current task when the previous task fails to execute or does not execute.

[0035] Therefore, it is necessary to propose effective technical means to solve the above problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes can not be described again in some embodiments. The technical solutions of the present application and how the technical solutions of the present application solve the above technical problems will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0036] The task processing method provided by the embodiments of the present application can be applied to, for example,Figure 1 The task processing device shown includes an entry module, a parsing module and an execution module, wherein the parsing module and the execution module push task nodes to each other to realize the processing of the workflow.

[0037] In one embodiment, Figure 2 As shown, a task processing method is provided, which is applied to Figure 1 The task processing device in FIG. 1 is described, including the following steps 201 to 203. In which:

[0038] In step 201, the entry module obtains the workflow file of the workflow and obtains a workflow tree based on the workflow file; wherein the workflow tree includes multiple task nodes, each task node includes a task status, a pre-dependency and a post-dependency, the pre-dependency includes all parent task nodes of the task node, and the post-dependency includes all child task nodes of the task node.

[0039] The workflow file may be a YAML format file, which is generated by a user through Python, node, or written; the workflow file includes multiple tasks and input and output data of the entire workflow.

[0040] The entry module obtains the workflow file of the workflow, and the entry module may receive the workflow file submitted by the user through an interface, thereby obtaining the workflow file.

[0041] The workflow tree is obtained based on the workflow file. The entry module can parse the workflow file, obtain the sequential dependency relationship between multiple tasks, and construct a workflow tree for the tasks with dependencies, thereby obtaining the workflow tree. Among them, parsing the workflow file can be parsing the sequential dependency relationship of multiple tasks according to DAG. A task node in the workflow tree represents a task. The task node includes task status, pre-dependencies and post-dependencies. The task status includes success, failure, cancellation or skipping. In addition, the task node can also include a task identifier (i.e., task ID). Sending task nodes between the entry module, parsing module and execution module can refer to sending the task identifier of the task node, thereby improving data transmission efficiency and thus improving workflow processing efficiency.

[0042] In an optional embodiment, after obtaining the workflow tree, the entry module determines whether there is a task node to be executed in the workflow tree; if so, the entry module stores the workflow tree in the memory and pushes the task node to be executed to the execution module; if not, the entry module determines the workflow status of the workflow based on the task status of all task nodes in the workflow tree, and when the workflow status is the completed state, updates the workflow status of the workflow stored in the database, or updates the workflow status of the workflow and stores it in the database.

[0043] The entry module determines whether there is a task node to be executed in the workflow tree, including: the entry module traverses all task nodes in the workflow tree, and if there is a task node whose task state is a non-completion state and all parent task nodes of the task node have a completion state, it is determined that there is a task node to be executed, or if there is a task node whose task state is a non-completion state and the task node has no parent task node, it is determined that there is a task node to be executed.

[0044] The completion state includes "success", "failure" and "cancel"; the non-completion state includes other states except the completion state, such as "pause", "skip", "to be run" and "running" and the like.

[0045] The entry module stores the workflow tree into the memory, which can be that the entry module stores the workflow tree in the memory in the form of key-value pair, that is, stores the workflow tree into the memory map. After storing the workflow tree into the memory, the task parameters are parsed, and the task node to be executed is pushed to the execution module; specifically, the execution module includes an initialization queue, and the entry module pushes the task node to be executed into the initialization queue of the execution module.

[0046] The entry module determines the workflow state of the workflow according to the task state of all task nodes in the workflow tree, and the specific determination manner is related to the workflow itself, which is not limited here. In one example, the task state of all task nodes is traversed, and when there is a task node whose task state is failure, it is determined that the workflow state is failure. In another example, the task state of the parent task node does not necessarily have an impact on the execution of the child task node, and when determining the workflow state of the workflow, the task parameters and the task state of the task node need to be combined to determine; for example, the parent task nodes of the task node C are task node B1, task node B2 and task node B3, the input data of the task node C can be the output of the task node B1 and the output of the task node B2, or the output of the task node B1 and the output of the task node B3, therefore, when the task state of the task node B2 is failure and the task state of the task node B3 is success, or when the task state of the task node B2 is success and the task state of the task node B3 is failure, the finally determined workflow state can also be success.

[0047] When the workflow state is a completion state, the workflow state of the workflow stored in the database is updated; when it is determined that the workflow state is a non-completion state, for example, the workflow is paused for processing due to a reason, no processing is performed.

[0048] The content executed by the above entry module is referred to Figure 3As shown, after the entry module acquires the workflow file of the workflow, the execution of the workflow is started, that is, the entry module parses the workflow file, constructs the workflow tree, and then determines whether there is a task node to be executed in the workflow tree; if yes, the workflow tree is stored in the memory map, the task parameters are parsed, and the task node to be executed is pushed to the execution module; if no, the workflow state is calculated, and it is determined whether the workflow state is a completed state; if the workflow state is the completed state, the workflow state of the workflow is updated and stored in the database, or the workflow state of the workflow stored in the database is updated; if the workflow state is not the completed state (i.e., a non-completed state), the execution of the workflow is ended.

[0049] In step 202, the parsing module acquires all subtask nodes of each task node, determines executable subtask nodes from all subtask nodes of the task node when the task state of the task node is the completed state, and determines whether the executable subtask nodes meet the dependency condition; if yes, the subtask nodes are pushed to the execution module.

[0050] The parsing module includes a parsing coroutine and a parsing channel corresponding to the parsing coroutine, and it can be understood that the number of parsing coroutines is multiple, and the number of parsing channels is also multiple, and the multiple parsing coroutines and the multiple parsing channels are one-to-one corresponding.

[0051] The execution module pushes the task nodes for which the execution is completed, the task nodes that do not meet the execution condition, and the task nodes that fail to be submitted for execution in the execution queue to the parsing module, and the parsing module includes multiple parsing channels, so that the parsing module distributes the task nodes to the corresponding parsing channels by using a scheduling algorithm, and each parsing coroutine acquires the task nodes from the corresponding parsing channel.

[0052] In an optional embodiment, the parsing module acquires all subtask nodes of each task node, including: the parsing coroutine acquires the task node from the corresponding parsing channel; the parsing coroutine finds all subtask nodes of the task node based on the workflow tree according to the task node; and correspondingly, executable subtask nodes are determined from all subtask nodes of the task node, and it is determined whether the executable subtask nodes meet the dependency condition; if yes, the subtask nodes are pushed to the execution module, including: the parsing coroutine determines executable subtask nodes from all subtask nodes of the task node when the task state of the task node is the completed state, and it is determined whether the executable subtask nodes meet the dependency condition; if the dependency condition is met, the executable subtask nodes are pushed to the execution module.

[0053] Before the parsing coroutine obtains the task node from the corresponding parsing channel, the method further includes: in a case where the corresponding parsing channel is not blocked, the parsing coroutine directly reads the corresponding parsing channel; in a case where the corresponding parsing channel is blocked, the parsing coroutine waits until the parsing channel is not blocked before reading the corresponding parsing channel; after reading the corresponding parsing channel, if the parsing channel is in a closed state, the parsing coroutine exits; if the parsing channel is in a non-closed state (i.e., an open state), the parsing coroutine performs the step of obtaining the task node from the corresponding parsing channel.

[0054] After the parsing coroutine obtains the task node, the parsing coroutine finds all child task nodes of the task node based on the workflow tree according to the task node. Specifically, the DFS (Depth-First Search) algorithm can be used to find all child task nodes of the task node. The DFS algorithm is an algorithm for traversing or searching a tree or graph. The DFS algorithm searches the branches of the tree as deeply as possible. When all edges of the task node v have been explored, the search backtracks to the starting node of the edge that discovered the task node v. This process continues until all nodes reachable from the source node have been discovered. If there are still undiscovered nodes, one of them is selected as the source node and the above process is repeated. The entire process is repeated until all nodes have been visited.

[0055] In an example, the workflow tree is stored in the memory, and the task node obtained by the parsing coroutine is only a task identifier of the task node. Therefore, the parsing coroutine can find the task node from the workflow tree based on the task identifier of the task node, so as to find the post-depending task included in the task node, i.e., to obtain all child task nodes of the task node.

[0056] After all child task nodes of the task node are found, the method further includes: the parsing coroutine obtains a task state of the task node. When the task state of the task node is in a non-completion state (such as skipping, waiting to run, running, etc.), the parsing coroutine updates the task states of all child task nodes of the task node to skipping, and updates the task states of the child task nodes to the memory and the database. Then, the task node is pushed to the parsing module, and the parsing coroutine continues to read the corresponding parsing channel or waits in a case where the corresponding parsing channel is blocked.

[0057] When the task status of a task node is completed (such as failed, successful, canceled, etc.), the parsing coroutine determines whether there is an executable subtask node among all the subtask nodes of the task node; if so, the parsing coroutine executes the steps of determining whether the executable subtask node meets the dependency conditions; if not, the parsing coroutine determines the workflow status of the workflow based on the task status of all task nodes in the workflow tree, and when the workflow status is completed, updates the workflow status of the workflow stored in the database, and then the parsing coroutine continues to read the corresponding parsing channel or waits if the corresponding parsing channel is blocked.

[0058] It is understood that the parsing coroutine determines whether there are executable subtask nodes among all subtask nodes of the task node. This can be done by determining whether there are subtask nodes with an incomplete task status (e.g., skipped, pending, running, etc.). If there is at least one subtask node with an incomplete task status, then it is determined that there are executable subtask nodes. If there are no subtask nodes with an incomplete task status, then it is determined that there are no executable subtask nodes. It should be noted that subtask nodes with a completed task status are not processed.

[0059] The parsing coroutine determines whether the executable subtask node meets the dependency condition. The parsing coroutine can find all parent task nodes of the executable subtask node based on the workflow tree. When the task status of all parent task nodes of the executable subtask node are completed, it is determined that the executable subtask node meets the dependency condition; if the task status of at least one parent task node of the executable subtask node is not completed, it is determined that the executable subtask node does not meet the dependency condition.

[0060] When the executable subtask node meets the dependency conditions, the executable subtask node is pushed to the execution module, specifically to the initialization queue in the execution module, and then the parsing coroutine continues to read the corresponding parsing channel or waits when the corresponding parsing channel is blocked.

[0061] When an executable subtask node does not meet the dependency conditions, the task status of the executable subtask node is updated to skipped, and the task status of the executable subtask node is updated to the memory and database. The executable task node is then pushed to the parsing module, and the parsing coroutine continues to read the corresponding parsing channel or waits if the corresponding parsing channel is blocked. It can be understood that for such executable subtask nodes that do not meet the dependency conditions, the parent task node with an incomplete task status will ultimately make the decision.

[0062] The content executed by the above parsing module refers to Figure 4 shown.

[0063] In step 203, the execution module pushes each task node in the execution module to the execution queue for execution when it determines that the task node meets the execution conditions, and checks the task information of the task node in a round-robin manner, updates the task status of the task node according to the task information, and after determining that the task node is executed, pushes the completed task node to the parsing module.

[0064] It should be noted that whether it is a task node pushed by the entry module to the execution module or a subtask node pushed by the parsing module to the execution module, for the execution module, they are all task nodes, and the steps executed for task nodes from different sources are the same.

[0065] In an optional embodiment, the execution module includes an initialization coroutine, an initialization queue, and an execution channel; the task nodes and subtask nodes pushed to the execution module by the entry module and the parsing module are entered into the execution module's initialization queue. The method further includes: the initialization coroutine retrieves the task node from the initialization queue (the task nodes in the initialization queue originate from the entry module and the parsing module) and determines whether the task node is in the execution module; if not, if the task node meets the binding conditions, the initialization coroutine binds the task node to a label and pushes the bound task node into the execution channel.

[0066] Reference Figure 5 As shown, the execution module also includes a queue channel, and the initialization coroutine can obtain the task node from the initialization queue through the queue channel. Specifically, before the initialization coroutine obtains the task node from the initialization queue, the method also includes: when the queue channel is not blocked, the initialization coroutine directly reads the queue channel; when the queue channel is blocked, the initialization coroutine waits until the queue channel is not blocked and then reads the queue channel; after reading the queue channel, if the queue channel is in a closed state, the initialization coroutine exits; if the queue channel is in a non-closed state (that is, an open state), the initialization coroutine executes the step of obtaining the task node from the initialization queue.

[0067] After the initialization coroutine obtains a task node from the initialization queue, it determines whether the task node is in the execution module. This can be done by determining whether the task node has a label attached to it. If the task node has a label attached to it, it is determined to be in the execution module. If the task node does not have a label attached to it, it is determined not to be in the execution module. The label refers to context information and is revocable.

[0068] If the task node is not in the execution module, it is determined whether the task node meets the binding condition. If the task status of the task node is in an uncompleted state, it is determined that the task node meets the binding condition. If the task status of the task node is in a completed state or a paused state, it is determined that the task node does not meet the binding condition.

[0069] If a task node meets the binding conditions, the initialization coroutine binds the task node to the label and pushes the bound task node to the execution channel. The bound task node is also stored in the task map. The number of execution channels can be one, ensuring that task nodes pushed by the parsing module are executed in order.

[0070] In an optional embodiment, the execution module includes an execution coroutine, and the number of execution coroutines can be multiple; the execution module pushes the task node to the execution queue for execution for each task node in the execution module when it determines that the task node meets the execution condition, including: the execution coroutine obtains the bound task node from the execution channel, wherein the bound task node is the task node in the execution module; the execution coroutine determines whether the input data of the bound task node meets the input condition; if so, determines that the bound task node meets the execution condition, and pushes the bound task node to the execution queue for execution.

[0071] Reference Figure 6 As shown, before the executing coroutine obtains the bound task node from the execution channel, the method further includes: when the execution channel is not blocked, the executing coroutine directly reads the execution channel; when the execution channel is blocked, the executing coroutine waits until the execution channel is unblocked and then reads the execution channel; after reading the execution channel, if the execution channel is in a closed state, the executing coroutine exits; if the execution channel is in a non-closed state (that is, an open state), the executing coroutine executes the step of obtaining the bound task node from the execution channel.

[0072] After the execution coroutine obtains the bound task node, it determines whether the input data of the bound task node meets the input conditions. It can be understood that when executing the current task, the output data of the previous task is required, that is, the input data of the current task. When the input data of the current task is missing or there is an error, the current task is not executable. Therefore, it is necessary to determine whether its input data meets the input conditions before the current task is executed. The input conditions can be, for example, that the target variable in the input data must be greater than zero, and / or the target data in the input data must exist, so that when the target variable in the input data is greater than zero, and / or the target data exists in the input data, it is determined that the input data of the bound task node meets the input conditions, otherwise, the input conditions are not met.

[0073] If the input data of the bound task node does not satisfy the input condition, it is determined that the bound task node does not satisfy the execution condition. If the input data of the bound task node satisfies the input condition, it is determined that the bound task node satisfies the execution condition.

[0074] When the bound task node does not satisfy the execution condition, the task state of the bound task node is updated to skip, the task state of the bound task node is updated to the memory and the database, then the label of the bound task node is deleted, and the task node after the label is deleted is pushed to the parsing module. Specifically, the label of the bound task node is deleted from the task map.

[0075] When the bound task node satisfies the execution condition, the bound task node is pushed to the execution queue for execution. It should be noted that the bound task node in the execution queue is submitted to an execution environment, such as a thread pool, a coroutine pool, or other execution frameworks. The task corresponding to the bound task node is operated and processed in the execution environment. However, when there is a lack of resources in the execution environment, a task configuration error, or the like, the submission of the bound task node may fail.

[0076] If the bound task node fails to be submitted, the task state of the bound task node is updated to failed, then the label of the bound task node is deleted, and the task node after the label is deleted is pushed to the parsing module. Specifically, the label of the bound task node is deleted from the task map. It should be noted that when the task state of a task node is failed, the task node is considered to be in a completed state, and thus the child task nodes of the task node can continue to be executed.

[0077] If the bound task node is successfully submitted, the task corresponding to the bound task node is executed in the execution environment to generate task information, which includes data generated after the task is executed and a task state. The execution coroutine performs round-robin query on the task information of the bound task node, analyzes the task information to obtain an analysis result, and updates the task state of the bound task node and the task information to the memory and the database according to the analysis result. It is determined whether the task corresponding to the bound task node is completed, if not, the step of round-robin checking the task information of the bound task node is continued, if yes, the task is settled, the label of the executed bound task node is deleted, and the task node after the label is deleted is pushed to the parsing module, so that the parsing module analyzes whether the child task nodes of the task node are executable, thereby realizing automatic flow transfer of the process. The label of the executed bound task node is deleted, specifically, the label of the bound task node is deleted from the task map.

[0078] The above-mentioned task processing method is applied to a task processing device, in which an entry module in the task processing device obtains a workflow file of a workflow and obtains a workflow tree based on the workflow file; wherein the workflow tree includes a plurality of task nodes, each task node includes a task status, a pre-dependency and a post-dependency, the pre-dependency includes all parent task nodes of the task node, and the post-dependency includes all child task nodes of the task node; then the parsing module in the task processing device obtains all child task nodes of the task node for each task node, and when the task status of the task node is a completed state, determines an executable child task node from all child task nodes of the task node, and determines whether the executable child task node is If the dependency conditions are met, the executable subtask nodes are pushed to the execution module; the execution module in the task processing device pushes the task nodes to the execution queue for execution when it determines that the task nodes meet the execution conditions for each task node in the execution module, and patrols to check the task information of the task nodes, updates the task status of the task nodes according to the task information, and after determining that the task nodes are executed, pushes the completed task nodes to the parsing module. In this way, after the user submits the workflow file to the task processing device, there is no need for the workflow scheduler to drive the task execution, but a self-driven mode in which tasks drive tasks and tasks update the task status, thereby improving the task execution efficiency.

[0079] In addition, all tasks are executed based on coroutines, which not only reduces resource overhead but also further improves task execution efficiency. Furthermore, tasks can depend on the failure, cancellation, or skip status of predecessor tasks in the DAG, and the execution authority of the current task can be determined based on the output of dependent tasks.

[0080] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0081] Based on the same inventive concept, embodiments of the present application also provide a task processing device for implementing the aforementioned task processing method. The implementation solution provided by this device is similar to the implementation solution described in the aforementioned method. Therefore, the specific limitations in one or more of the following task processing device embodiments can be found in the above-mentioned limitations on the task processing method and will not be further elaborated here.

[0082] In an exemplary embodiment, Figure 1 As shown, the task processing device includes an entry module, a parsing module and an execution module. The entry module is used to execute the method steps executed by the entry module in the above method embodiment; the parsing module is used to execute the method steps executed by the parsing module in the above method embodiment; and the execution module is used to execute the method steps executed by the execution module in the above method embodiment.

[0083] Each module in the task processing device described above may be implemented in whole or in part through software, hardware, or a combination thereof. Each module may be embedded in or independent of a processor in a computer device in the form of hardware, or may be stored in a memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0084] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 7As shown. The computer device includes a processor, memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals via wired or wireless means, and the wireless means can be implemented via Wi-Fi, a mobile cellular network, near-field communication (NFC), or other technologies. When executed by the processor, the computer program implements a task processing method. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse.

[0085] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0086] In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps of the method described in the above method embodiment when executing the computer program.

[0087] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described in the above method embodiment are implemented.

[0088] In one embodiment, a computer program product is provided, including a computer program, which implements the steps of the method described in the above method embodiment when executed by a processor.

[0089] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), programmable logic devices (PLDs), quantum computing-based data processing logic devices, artificial intelligence (AI) processors, and the like.

[0090] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0091] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A task processing method, characterized in that: Applied to a task processing device, the task processing device includes an entry module, a parsing module, and an execution module, and the method includes: The entry module obtains a workflow file of the workflow and obtains a workflow tree based on the workflow file; wherein the workflow tree includes a plurality of task nodes, each of which includes a task state, a pre-dependency, and a post-dependency, wherein the pre-dependency includes all parent task nodes of the task node, and the post-dependency includes all child task nodes of the task node; The parsing module obtains all the subtask nodes of each task node, and when the task status of the task node is completed, determines the executable subtask node from all the subtask nodes of the task node, and determines whether the executable subtask node meets the dependency condition. If so, pushes the executable subtask node to the execution module; The execution module pushes each task node in the execution module to the execution queue for execution when it determines that the task node meets the execution conditions, and patrols to check the task information of the task node, updates the task status of the task node according to the task information, and after determining that the task node is completed, pushes the completed task node to the parsing module.

2. The method according to claim 1, characterized in that The parsing module includes a parsing coroutine and a parsing channel corresponding to the parsing coroutine; The parsing module obtains all the subtask nodes of each task node, including: The parsing coroutine obtains the task node from the corresponding parsing channel; The parsing coroutine searches for all the subtask nodes of the task node based on the task node based on the workflow tree; Correspondingly, when the task status of the task node is completed, determining the executable subtask node from all the subtask nodes of the task node, and judging whether the executable subtask node meets the dependency condition, and if so, pushing the executable subtask node to the execution module, including: When the parsing coroutine determines that the task status of the task node is completed, it determines the executable subtask node from all the subtask nodes of the task node, and judges whether the executable subtask node meets the dependency condition. If the dependency condition is met, the executable subtask node is pushed to the execution module.

3. The method according to claim 2, characterized in that The method further comprises: When the parsing coroutine determines that the task status of the task node is in an incomplete state, the task status of all the subtask nodes of the task node is updated to skip; and / or, When the parsing coroutine determines that the task status of the task node is completed and the executable subtask node does not meet the dependency condition, the task status of the executable subtask node is updated to skip.

4. The method according to claim 2, characterized in that After the parsing coroutine searches for and obtains all the subtask nodes of the task node based on the workflow tree, the method further includes: The parsing coroutine determines whether there is an executable subtask node among all the subtask nodes of the task node; If so, the parsing coroutine executes the step of determining whether the executable subtask node meets the dependency condition; If not, the parsing coroutine determines the workflow status of the workflow according to the task status of all the task nodes in the workflow tree, and updates the workflow status of the workflow stored in the database when the workflow status is completed.

5. The method according to claim 1, wherein The execution module includes an initialization coroutine, an initialization queue, and an execution channel; the method further includes: The initialization coroutine obtains the task node from the initialization queue and determines whether the task node is in the execution module; If not, when the task node meets the binding condition, the initialization coroutine binds the task node to the label and pushes the bound task node to the execution channel.

6. The method according to claim 5, characterized in that The execution module includes an execution coroutine. When determining that each task node in the execution module meets an execution condition, the execution module pushes the task node to an execution queue for execution, including: The execution coroutine obtains the bound task node from the execution channel, wherein the bound task node is the task node in the execution module; The execution coroutine determines whether the input data of the bound task node meets the input conditions; If so, it is determined that the bound task node meets the execution condition, and the bound task node is pushed to the execution queue for execution.

7. The method according to claim 6, characterized in that The method further comprises: If the bound task node does not meet the execution condition, updating the task status of the bound task node to skip; The tag of the bound task node is deleted, and the task node after the tag is deleted is pushed to the parsing module.

8. The method according to claim 6, characterized in that After pushing the bound task node to an execution queue for execution, the method further includes: If the bound task node fails to be submitted, the task status of the bound task node is updated to failure; The tag of the bound task node is deleted, and the task node after the tag is deleted is pushed to the parsing module.

9. The method according to claim 1, characterized in that After the entry module obtains the workflow file and obtains the workflow tree based on the workflow file, the method further includes: The entry module determines whether there is the task node to be executed in the workflow tree; If so, the entry module stores the workflow tree in memory and pushes the task node to be executed to the execution module; If not, the entry module determines the workflow status of the workflow according to the task status of all the task nodes in the workflow tree, and updates the workflow status of the workflow stored in the database when the workflow status is completed.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images