Methods, apparatus, electronic devices, and storage media for DAG task orchestration

Abstract

This invention provides a method, apparatus, electronic device, and storage medium for DAG task orchestration. The method for DAG task orchestration includes: defining a DAG scheduling dependency graph; defining a DAG object and instantiating the DAG object through the DAG scheduling dependency graph; constructing a DAG graph of links, traversing from the root node of the DAG graph of links, and cutting each traversal path into several context chains, with the cutting condition being any one of the following three: the in-degree of the current node is 1 and the out-degree of the last node in the current context chain is 1; the in-degree of the current node is 0; the in-degree of the current node is 1 and the out-degree is 0; the in-degree of the current node is 1 and the out-degree is 0; defining a thread pool and allocating a thread pool for each context chain; defining asynchronous tasks based on the thread pool; submitting tasks to the thread pool for execution and obtaining the final result.

Description

Technical Field

[0001] This invention relates to the field of directed acyclic graph (DAG) tasks. Background Technology

[0002] Directed Acyclic Graph (DAG) tasks can be applied to multiple scenarios. For example, in search and recommendation scenarios, DAG graphs can be used to execute strategies. In algorithm platform task scheduling, DAG graphs formed by concatenating operators can be used to schedule execution. In crowd selection tasks, queries can also be parsed into DAG graphs, with each node representing specific intersection, union, and complement execution logic.

[0003] Currently, there are very few DAG scheduling frameworks in the industry that return dependencies. Existing methods focus on processing one node per thread, and the general steps are as follows:

[0004] 1. Define the DAG scheduling flowchart, such as... Figure 1 As shown, there are 13 nodes in total;

[0005] 2. Define the computation logic for each node to form an asynchronous execution method;

[0006] 3. Define a thread pool;

[0007] 4. By analyzing the relationships between nodes, and based on thread pools, one thread is allocated to execute each stage, with 13 thread pools allocated to 13 nodes;

[0008] 5. The organization returns dependencies and synchronously generates the final result.

[0009] The main drawbacks of existing technologies are as follows:

[0010] First: Each node uses a thread, which leads to frequent thread switching and cannot make full use of thread resources;

[0011] Second: Currently, the industry's DAG scheduling frameworks are quite heavy, making it difficult to introduce them into projects in a lightweight manner. Summary of the Invention

[0012] The following provides an overview of various exemplary technical solutions. Some simplifications and omissions may be made in this overview to highlight and illustrate certain aspects of the various exemplary technical solutions, but this does not limit the scope of the invention. Detailed descriptions of exemplary technical solutions sufficient to allow those skilled in the art to generate and use the concepts of the present invention will be presented in subsequent sections.

[0013] The technical solution of this invention provides a method for DAG task orchestration, comprising: defining a DAG scheduling dependency graph; defining a DAG object and instantiating the DAG object through the DAG scheduling dependency graph; constructing a DAG graph of links, traversing from the root node of the DAG graph of links, and cutting each traversal path into several context chains, wherein the cutting condition is any one of the following three: the in-degree of the current node is 1 and the out-degree of the last node of the current context chain is 1; the in-degree of the current node is 0; the in-degree of the current node is 1 and the out-degree is 0; the in-degree of the current node is 1 and the out-degree is 0; defining a thread pool and allocating a thread pool for each context chain; defining asynchronous tasks based on the thread pool; submitting tasks to the thread pool for execution and obtaining the final result.

[0014] Optionally, it also includes: defining asynchronous tasks using Guava.

[0015] Optionally, it also includes: filling all context chains into the DAG graph of the initially constructed graph chains according to the upstream and downstream relationships on the original graph.

[0016] Optionally, it also includes: iterating through all root nodes until all root nodes have been traversed, forming a new DAG graph of graph chains.

[0017] Optionally, it also includes: traversing the graph chain using topological sorting.

[0018] Optionally, it also includes: continuously submitting tasks asynchronously based on the results of the traversal.

[0019] Another technical solution of the present invention also provides a DAG task orchestration apparatus, comprising: a DAG graph cutting module, configured to: define a DAG scheduling dependency graph; define a DAG object, and instantiate the DAG object through the DAG scheduling dependency graph; construct a DAG graph of the graph chain, traverse from the root node of the DAG graph of the graph chain, and cut each traversal path into several context chains, wherein the cutting condition is any one of the following three: the in-degree of the current node is 1 and the out-degree of the last node of the current context chain is 1; the in-degree of the current node is 0; the in-degree of the current node is 1 and the out-degree is 0; and the in-degree of the current node is 1 and the out-degree is 0; and a task processing module, configured to: define a thread pool, allocate a thread pool for each context chain; define asynchronous tasks based on the thread pool; submit tasks to the thread pool for execution, and obtain the final result.

[0020] Optionally, the task processing module also includes Guava, which is used to define asynchronous tasks.

[0021] Optionally, the task processing module is also configured to fill all context chains into the DAG graph of the initially constructed graph chains according to the upstream and downstream relationships on the original graph.

[0022] Optionally, the task processing module is also configured to: loop through all root nodes until all root nodes have been traversed, forming a new DAG graph of the graph chain.

[0023] Optionally, the task processing module is also configured to traverse the graph chain using topological sorting.

[0024] Optionally, the task processing module is also configured to continuously submit tasks asynchronously based on the results of the traversal.

[0025] Another technical solution of the present invention provides an electronic device, including: a processor, a memory, and a computer program stored in the memory and running thereon, wherein the processor executes the computer program to implement the steps of the method described in any of the above technical solutions.

[0026] Another technical solution of the present invention provides a computer-readable storage medium, wherein the computer program, when executed by a processor, implements the steps of the method described in any of the above technical solutions.

[0027] The technical solution of the present invention has the following main beneficial effects:

[0028] First: It is more lightweight compared to existing DAG execution frameworks;

[0029] Second: The graph-based partitioning method, which places as many tasks as possible in a single thread, greatly reduces thread switching and results in higher performance.

[0030] Third: It supports multiple scenarios such as DAG task-dependent execution of algorithm platforms, parallel execution of DAG for audience selection, and execution of DAG strategies for search and recommendation, and has a wide range of applications. Attached Figure Description

[0031] To better understand the various exemplary embodiments, reference can be made to the accompanying drawings, in which:

[0032] Figure 1 A flowchart illustrating a prior art method for DAG task orchestration is shown.

[0033] Figure 2 A flowchart illustrating the DAG task orchestration method provided in the embodiment is shown;

[0034] Figure 3 A schematic diagram of some steps in the DAG task orchestration method provided in the embodiment is shown;

[0035] Figure 4 A schematic diagram of the structure of the DAG task orchestration apparatus provided in the embodiment is shown.

[0036] For ease of understanding, the same reference numerals have been used to refer to elements having substantially the same or similar structure and / or substantially the same or similar function. Detailed Implementation

[0037] The description and figures illustrate the principles of the invention. Therefore, it will be understood that those skilled in the art will be able to design various arrangements, which, although not explicitly described or shown herein, embody the principles of the invention and are included within its scope. Furthermore, all examples cited herein are primarily intended for illustrative purposes to aid the reader in understanding the principles of the invention and the concepts provided by the inventors to deepen understanding in the field, and all examples should be considered as not being limited to such specific cited examples and conditions. Additionally, as used herein, unless otherwise indicated (e.g., “or additionally” or “or in an alternative”), the term “or” means non-exclusive or (i.e., and / or). Moreover, the various embodiments described herein are not necessarily mutually exclusive, as some embodiments can be combined with one or more other embodiments to form new embodiments.

[0038] Terminology Explanation:

[0039] DAG (Directed Acyclic Graph): Generally refers to a directed acyclic graph in graph theory.

[0040] Task orchestration: Asynchronous scheduling and execution of tasks.

[0041] The first embodiment provides a method for orchestrating DAG tasks. Figure 2 A flowchart illustrating a DAG task orchestration method provided in an embodiment is shown, the method comprising:

[0042] S101: Defines a DAG scheduling dependency graph, which can be configured via drag and drop in the background;

[0043] S102: Define a DAG object by parsing the DAG scheduling dependency graph configured in the background drag and drop, and instantiate the DAG object;

[0044] S103: Construct the DAG graph of the graph chain. Starting from the root node of the DAG graph of the graph chain, traverse each traversal path and cut each traversal path. The cutting condition is any one of the following three: the in-degree of the current node is 1 and the out-degree of the last node of the current context chain is 1; the in-degree of the current node is 0; the in-degree of the current node is 1 and the out-degree is 0; cut the nodes of each path into several context chains.

[0045] In the original case (see) Figure 3The left half of the graph will allocate a thread to each node for execution. In high-concurrency scenarios, frequent switching between threads can lead to unnecessary performance loss. The embodiment uses a DAG graph segmentation method to divide the graph into... Figure 3 The left half is divided into the right half of the GraphChain (a new type of node), and each GraphChain is assigned a corresponding thread. This ensures that a node with a single connection can be computed in a single thread, greatly reducing the performance loss caused by switching between threads.

[0046] The specific stage cutting process is as follows:

[0047] Step 1: Construct a Directed Acyclic Graph (DAG) of the GraphChain, consisting of multiple nodes. Nodes can be either models or policies, similar to... Figure 3 The left half.

[0048] Step 2: Starting from the root node of the above graph, we iterate through it using a depth-first traversal approach. Each root node will correspond to a traversal path. For example, in... Figure 3 In the left half, the root nodes are Model 1, Strategy 4, and Model 2. The traversed paths can include: the first path is Model 1, Strategy 3, Strategy 5, Strategy 8, Strategy 13, Strategy 11, Strategy 12; the second path is Strategy 4, Strategy 6, Strategy 8, Strategy 13, Strategy 11, Strategy 12; and the third path is Model 2, Strategy 7, Strategy 9 or Strategy 10, Strategy 11, Strategy 12. Each node includes in-degree and out-degree. The in-degree is the number of paths pointing to that node, and the out-degree is the number of paths from that node to other nodes.

[0049] Step 3: For each traversal path, perform a split based on any one of the following three conditions: First, the current node's in-degree is 1 and the last node in the current context's chain has an out-degree of 1; second, the current node's in-degree is 0; or third, the current node's in-degree is 1 and its out-degree is 0. If any of these conditions are met, add the node to the context's chain. The current context's chain refers to the chain formed by traversing the nodes preceding the current node.

[0050] The following are examples, such as Figure 3 As shown on the left half, the first path is traversed: Model 1, Strategy 3, Strategy 5, Strategy 8, Strategy 13, Strategy 11, Strategy 12.

[0051] When traversing the root node model1, if the current node has an in-degree of 0, satisfying the second condition mentioned above, then model1 is added to the chain of the first context, forming the first node in the chain of the first context.

[0052] When traversing node strategy 3, the current node has an in-degree of 1, the current context chain is model 1, the last node of the current context chain is model 1, and the out-degree of the last node of the current context chain, model 1, is 1. If the first condition is met, then strategy 3 is added to the context chain. Now the first context chain has two nodes, namely model 1 and strategy 3.

[0053] When traversing node strategy 5, the current node has an in-degree of 1, the current context chain consists of model 1 and strategy 3, the last node in the current context chain is strategy 3, and the out-degree of the last node in the current context chain, strategy 3, is 1. If the first condition is met, it is added to the context chain. Now the first context chain has three nodes, namely model 1, strategy 3 and strategy 5.

[0054] When traversing node strategy 8, the current node has an in-degree of 2, the current context chain consists of model 1, strategy 3, and strategy 5, the last node in the current context chain is strategy 5, and the out-degree of the last node in the current context chain, strategy 5, is 1. If any of the above conditions are not met, a new context chain is formed, which is the second context chain.

[0055] When traversing node strategy 13, the current node has an in-degree of 1, the current context chain is strategy 8, the last node of the current context chain is strategy 8, and the out-degree of the last node of the current context chain, strategy 8, is 1. If the first condition is met, it is added to the chain of this context. Now there are two nodes in the second context chain, namely strategy 8 and strategy 13.

[0056] When traversing node strategy 11, the current node has an in-degree of 3, the current context chain is model 8 and strategy 13, the last node of the current context chain is strategy 13, and the out-degree of the last node of the current context chain, strategy 13, is 1. If any of the above conditions are not met, a new context chain is formed, namely the third context chain.

[0057] When traversing node strategy 12, the current node has an in-degree of 1 and an out-degree of 0, which satisfies the third condition mentioned above. Therefore, it is added to the chain of this context. Now there are two nodes in the third context chain, namely strategy 11 and strategy 12.

[0058] After traversing the first path, three context chains are formed, as shown in the gray boxes. Each gray box represents a context chain. The first context chain includes model 1, strategy 3, and strategy 5; the second context chain includes strategy 8 and strategy 13; and the third context chain includes strategy 11 and strategy 12.

[0059] Traverse the second path: strategy 4, strategy 6, strategy 8, strategy 13, strategy 11, strategy 12; the principle is the same as the first path. It can be seen that strategy 4 in the second path satisfies the second condition, and strategy 6 satisfies the first condition. Therefore, strategy 4 and strategy 6 form the fourth context chain. The other nodes in the second path, strategy 8, strategy 13, strategy 11, and strategy 12, are the same as the nodes in the first path, so there is no need to calculate them repeatedly.

[0060] Traverse the third path: Model 2, Strategy 7, Strategy 9 or Strategy 10, Strategy 11, Strategy 12; the principle is the same, Model 2 satisfies the second condition, Strategy 7 satisfies the first condition, and Strategy 9 or Strategy 10 and Strategy 11 do not satisfy any of the conditions. Therefore, the two nodes of Strategy 2 and Strategy 7 form the fifth context chain, Strategy 9 forms the sixth context chain, and Strategy 10 forms the seventh context chain. Strategy 11 and Strategy 12 have already been calculated and do not need to be calculated again.

[0061] In summary, after traversing all paths, a total of seven context chains were formed.

[0062] Step 4: Fill all the traversed context chains into the DAG graph of the initially defined GraphChain according to the upstream and downstream relationships on the original graph.

[0063] Step 5: Loop through all root nodes until all root nodes have been traversed, thus generating a new GraphChain-DAG, as shown below. Figure 3 The right half of the diagram shows that each gray box represents a GraphChain node, and there are a total of seven GraphChain nodes.

[0064] S104: Define a thread pool, allocating one thread pool for each context chain (each GraphChain node). Figure 3 The left half originally required 13 thread pools, now Figure 3 The right half only requires 7 thread pools.

[0065] S105: An asynchronous task submitter can be defined based on Google's Guava, and the execution process of the DAG can use Guava's Futures. Asynchronous tasks can be defined based on thread pools. For example, the first context chain (including Model 1, Strategy 3, and Strategy 5) can act as a thread pool, working concurrently with the thread pools of the fourth context chain (Strategies 4 and 6) and the fifth context chain (including Strategy 2 and 7).

[0066] The specific execution process is as follows:

[0067] Step 1: Traverse the GraphChain using topological sorting, for example... Figure 3 The seven GraphChains on the right half.

[0068] Step 2: Based on the results of the traversal, continuously submit tasks asynchronously.

[0069] The core code for asynchronous execution is as follows:

[0070]

[0071]

[0072] S106: Submit the task to the thread pool for execution and obtain the final result.

[0073] Compared to open-source DAG execution frameworks, it is more lightweight. The graph-based approach places as many tasks as possible in a single thread, greatly reducing thread switching and resulting in higher performance.

[0074] Since most search recommendation strategies can be assigned to a single thread, the overall strategy DAG execution can be improved by 20%+ compared to other solutions.

[0075] It can also support scenarios such as DAG task-dependent execution of algorithm platforms, parallel execution of DAG for crowd selection, and execution of DAG strategies for search and recommendation.

[0076] The second embodiment also provides an apparatus for DAG task orchestration, such as Figure 4 As shown, the device for DAG task orchestration includes: a DAG graph cutting module 401 and a task processing module 402.

[0077] The DAG graph cutting module 401 is configured to: define a DAG scheduling dependency graph; define a DAG object, instantiate the DAG object through the DAG scheduling dependency graph; construct a DAG graph of the graph chain, traverse from the root node of the DAG graph of the graph chain, and cut each traversal path into several context chains, with the cutting condition being any one of the following three: the in-degree of the current node is 1 and the out-degree of the last node of the current context chain is 1; the in-degree of the current node is 0; the in-degree of the current node is 1 and the out-degree of the current node is 0.

[0078] The specific stage cutting process is as follows:

[0079] Step 1: Construct a Directed Acyclic Graph (DAG) of the GraphChain, consisting of multiple nodes. Nodes can be either models or policies, similar to... Figure 3 The left half.

[0080] Step 2: Starting from the root node of the above graph, we iterate through it using a depth-first traversal approach. Each root node will correspond to a traversal path. For example, in... Figure 3 In the left half, the root nodes are Model 1, Strategy 4, and Model 2. The traversed paths can include: the first path is Model 1, Strategy 3, Strategy 5, Strategy 8, Strategy 13, Strategy 11, Strategy 12; the second path is Strategy 4, Strategy 6, Strategy 8, Strategy 13, Strategy 11, Strategy 12; and the third path is Model 2, Strategy 7, Strategy 9 or Strategy 10, Strategy 11, Strategy 12. Each node includes in-degree and out-degree. The in-degree is the number of paths pointing to that node, and the out-degree is the number of paths from that node to other nodes.

[0081] Step 3: For each traversal path, perform a split based on any one of the following three conditions: First, the current node's in-degree is 1 and the last node in the current context's chain has an out-degree of 1; second, the current node's in-degree is 0; or third, the current node's in-degree is 1 and its out-degree is 0. If any of these conditions are met, add the node to the context's chain. The current context's chain refers to the chain formed by traversing the nodes preceding the current node.

[0082] The following are examples, such as Figure 3 As shown on the left half, the first path is traversed: Model 1, Strategy 3, Strategy 5, Strategy 8, Strategy 13, Strategy 11, Strategy 12.

[0083] When traversing the root node model1, if the current node has an in-degree of 0, satisfying the second condition mentioned above, then model1 is added to the chain of the first context, forming the first node in the chain of the first context.

[0084] When traversing node strategy 3, the current node has an in-degree of 1, the current context chain is model 1, the last node of the current context chain is model 1, and the out-degree of the last node of the current context chain, model 1, is 1. If the first condition is met, then strategy 3 is added to the context chain. Now the first context chain has two nodes, namely model 1 and strategy 3.

[0085] When traversing node strategy 5, the current node has an in-degree of 1, the current context chain consists of model 1 and strategy 3, the last node in the current context chain is strategy 3, and the out-degree of the last node in the current context chain, strategy 3, is 1. If the first condition is met, it is added to the context chain. Now the first context chain has three nodes, namely model 1, strategy 3 and strategy 5.

[0086] When traversing node strategy 8, the current node has an in-degree of 2, the current context chain consists of model 1, strategy 3, and strategy 5, the last node in the current context chain is strategy 5, and the out-degree of the last node in the current context chain, strategy 5, is 1. If any of the above conditions are not met, a new context chain is formed, which is the second context chain.

[0087] When traversing node strategy 13, the current node has an in-degree of 1, the current context chain is strategy 8, the last node of the current context chain is strategy 8, and the out-degree of the last node of the current context chain, strategy 8, is 1. If the first condition is met, it is added to the chain of this context. Now there are two nodes in the second context chain, namely strategy 8 and strategy 13.

[0088] When traversing node strategy 11, the current node has an in-degree of 3, the current context chain is model 8 and strategy 13, the last node of the current context chain is strategy 13, and the out-degree of the last node of the current context chain, strategy 13, is 1. If any of the above conditions are not met, a new context chain is formed, namely the third context chain.

[0089] When traversing node strategy 12, the current node has an in-degree of 1 and an out-degree of 0, which satisfies the third condition mentioned above. Therefore, it is added to the chain of this context. Now there are two nodes in the third context chain, namely strategy 11 and strategy 12.

[0090] After traversing the first path, three context chains are formed, as shown in the gray boxes. Each gray box represents a context chain. The first context chain includes model 1, strategy 3, and strategy 5; the second context chain includes strategy 8 and strategy 13; and the third context chain includes strategy 11 and strategy 12.

[0091] Traverse the second path: strategy 4, strategy 6, strategy 8, strategy 13, strategy 11, strategy 12; the principle is the same as the first path. It can be seen that strategy 4 in the second path satisfies the second condition, and strategy 6 satisfies the first condition. Therefore, strategy 4 and strategy 6 form the fourth context chain. The other nodes in the second path, strategy 8, strategy 13, strategy 11, and strategy 12, are the same as the nodes in the first path, so there is no need to calculate them repeatedly.

[0092] Traverse the third path: Model 2, Strategy 7, Strategy 9 or Strategy 10, Strategy 11, Strategy 12; the principle is the same, Model 2 satisfies the second condition, Strategy 7 satisfies the first condition, and Strategy 9 or Strategy 10 and Strategy 11 do not satisfy any of the conditions. Therefore, the two nodes of Strategy 2 and Strategy 7 form the fifth context chain, Strategy 9 forms the sixth context chain, and Strategy 10 forms the seventh context chain. Strategy 11 and Strategy 12 have already been calculated and do not need to be calculated again.

[0093] In summary, after traversing all paths, a total of seven context chains were formed.

[0094] Step 4: Fill all the traversed context chains into the DAG graph of the initially defined GraphChain according to the upstream and downstream relationships on the original graph.

[0095] Step 5: Loop through all root nodes until all root nodes have been traversed, thus generating a new GraphChain-DAG, as shown below. Figure 3 The right half of the diagram shows that each gray box represents a GraphChain node, and there are a total of seven GraphChain nodes.

[0096] The task processing module 402 is configured to: define a thread pool, allocate a thread pool for each context chain; define asynchronous tasks based on the thread pool; submit tasks to the thread pool for execution, and obtain the final result.

[0097] An asynchronous task submitter can be defined based on Google's Guava, and the execution process of the DAG can utilize Guava's Futures. Asynchronous tasks can be defined based on thread pools. For example, the first context chain (including Model 1, Strategy 3, and Strategy 5) can act as a thread pool, working concurrently with the thread pools of the fourth context chain (Strategies 4 and 6) and the fifth context chain (including Strategy 2 and 7).

[0098] The specific execution process is as follows:

[0099] Step 1: Traverse the GraphChain using topological sorting, for example... Figure 3 The seven GraphChains on the right half.

[0100] Step 2: Based on the results of the traversal, continuously submit tasks asynchronously.

[0101] The core code for asynchronous execution is as follows:

[0102]

[0103]

[0104] Optionally, the task processing module also includes Guava, which is used to define asynchronous tasks.

[0105] A third embodiment also provides an electronic device, including: a processor, a memory, and a computer program stored in the memory and running thereon. When the processor executes the computer program, it implements the steps of any of the methods described in the above embodiments, such as steps S101 to S106; or when the processor executes the computer program, it implements the functions of each module / unit in the above embodiments, such as the functions of units 201 to 202. The computer program can be divided into one or more modules / units, which are stored in the memory and executed by the processor. The one or more modules / units can be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the electronic device.

[0106] The electronic device can be a mobile terminal such as a smartphone, or a computing device such as a desktop computer, laptop, PDA, or cloud server. The electronic device may include, but is not limited to, a processor and memory, and may include more or fewer components, or a combination of certain components. For example, the electronic device may also include input / output devices, network access devices, buses, etc. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor. The memory may be an internal storage unit of the electronic device, such as a hard drive or RAM. The memory may also be an external storage device of the electronic device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, FlashCard, etc. Furthermore, the memory may include both internal storage units of the electronic device and external storage devices.

[0107] The fourth embodiment also provides a computer-readable storage medium, wherein the computer program, when executed by a processor, implements the steps of the methods of any of the above embodiments.

[0108] In the various embodiments of this application, the functional units can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units can be implemented in hardware or as software functional units. If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added to or subtracted according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0109] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0110] In the above embodiments, the descriptions of each embodiment have their own emphasis. Parts not described in detail or in a particular embodiment can be referred to in the relevant descriptions of other embodiments. Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0111] In the embodiments provided in this application, it should be understood that the disclosed systems, electronic devices, and methods can be implemented in other ways. For example, the system and electronic device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of systems or units may be electrical, mechanical, or other forms. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the solution in this embodiment according to actual needs.

[0112] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for DAG task orchestration, characterized in that, include: Define a DAG scheduling dependency graph; Define a DAG object, schedule the dependency graph through the DAG, and instantiate the DAG object; To construct the DAG graph of the graph chain, start from the root node of the DAG graph of the graph chain and traverse all root nodes in a loop until all root nodes have been traversed, forming a new DAG graph of the graph chain. Each node includes in-degree and out-degree. In-degree is the number of paths pointing to that node, and out-degree is the number of paths pointing from that node to other nodes. Each traversal path is divided into several context chains, with the following three conditions being any one of the following: the in-degree of the current node is 1 and the out-degree of the last node in the current context chain is 1; the in-degree of the current node is 0; or the in-degree of the current node is 1 and the out-degree is 0. Define a thread pool and allocate a thread pool for each context chain. Define asynchronous tasks based on thread pools; Submit the task to the thread pool for execution and obtain the final result.

2. The method for DAG task orchestration according to claim 1, characterized in that, Also includes: Use Guava to define asynchronous tasks.

3. The method for DAG task orchestration according to claim 1, characterized in that, Also includes: Fill all context chains into the DAG graph of the initially constructed graph chain according to their original upstream and downstream relationships.

4. The method for DAG task orchestration according to claim 1, characterized in that, Also includes: The graph chains are traversed using topological sorting.

5. The method for DAG task orchestration according to claim 1, characterized in that, Also includes: Based on the results of the traversal, tasks are submitted asynchronously continuously.

6. A device for DAG task orchestration, characterized in that, include: The DAG graph cutting module is configured to: define a DAG scheduling dependency graph; Define a DAG object, schedule the dependency graph using the DAG, and instantiate the DAG object; construct the DAG graph of the graph chain, starting from the root node of the DAG graph of the graph chain and traversing it in a loop until all root nodes have been traversed, forming a new DAG graph of the graph chain. Each node includes an in-degree and an out-degree. The in-degree is the number of paths pointing to that node, and the out-degree is the number of paths from that node to other nodes; divide each traversal path into several context chains, with the dividing condition being any one of the following three: the in-degree of the current node is 1 and the out-degree of the last node in the current context chain is 1; the in-degree of the current node is 0; or the in-degree of the current node is 1 and the out-degree is 0. The task processing module is configured to: define a thread pool and allocate a thread pool for each context chain; Based on a thread pool, define asynchronous tasks; submit tasks to the thread pool for execution, and obtain the final results.

7. The apparatus for DAG task orchestration according to claim 6, characterized in that, The task processing module also includes Guava, which is used to define asynchronous tasks.

8. The apparatus for DAG task orchestration according to claim 6, characterized in that, The task processing module is also configured to fill all context chains into the DAG graph of the initially constructed graph chain according to their original upstream and downstream relationships.

9. The apparatus for DAG task orchestration according to claim 6, characterized in that, The task processing module is also configured to traverse the graph chain using topological sorting.

10. The apparatus for DAG task orchestration according to claim 6, characterized in that, The task processing module is also configured to continuously submit tasks asynchronously based on the results of the traversal.

11. An electronic device, characterized in that, include: A processor, a memory, and a computer program stored in the memory that runs on the memory, wherein the processor, when executing the computer program, implements the method of any one of claims 1-5.

12. A computer-readable storage medium, characterized in that, When a computer program is executed by a processor, it implements the method of any one of claims 1-5.

Images