Task scheduling method and related system

Abstract

The invention provides a task scheduling method, which is applied to the technical field of artificial intelligence (AI), and comprises the following steps: obtaining a computational graph, the computational graph comprises a plurality of operators, each of the plurality of operators represents a task, according to the dependency relationship of the plurality of operators in the computational graph, sorting the plurality of operators to obtain an ordered operator sequence, the ordered operator sequence comprises a first operator and a second operator depending on the first operator, the first operator is arranged in front of the first operator, the plurality of operators are grouped according to the ordered operator sequence to obtain a plurality of operator groups, and the plurality of operator groups are sequentially scheduled to equipment according to the priority of the plurality of operator groups. According to the method, the sorted operators are grouped, and task issuing is performed by taking the operator groups as the granularity, so that effective scheduling under limited concurrency of the computational graph is realized, reduction of memory reuse probability and reduction of cache hit rate caused by overlarge concurrency are avoided, and the computational efficiency and performance of the AI model are improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence (AI) technology, and in particular to a task scheduling method, system, computing device cluster, chip, computer-readable storage medium, and computer program product. Background Art

[0002] With the rapid development of artificial intelligence (AI) technology, AI models are being used more and more widely in many fields. For example, AI models can be applied to image recognition, intelligent question and answer, or recommendation systems. In order to ensure the accuracy and efficiency of AI model predictions, AI models usually need to be trained with a large amount of data to optimize the parameters of the AI ​​model. In addition, the reasoning process of the AI ​​model also needs to quickly and accurately process and parse new input data to provide prediction results or make decisions. How to optimize the performance of AI models and improve the efficiency and accuracy of training or reasoning has become a key issue in the industry.

[0003] At present, the mainstream solution to optimize the performance of AI models is to optimize the task scheduling in the computation graph. As a major data structure of the AI ​​model, the computation graph describes the dependencies between various tasks. Specifically, each node in the computation graph represents a task, and each edge represents the data dependency between tasks. According to the data dependency, the execution order of tasks can be reasonably arranged, thereby achieving the performance optimization of the AI ​​model.

[0004] However, existing task scheduling methods are difficult to meet the high demands of AI models for computing efficiency and performance. Summary of the invention

[0005] The present application provides a task scheduling method, which implements effective scheduling under limited concurrency of the computation graph by sorting multiple operators in the computation graph according to their dependencies and then grouping them, and dispatching tasks based on operator groups, thereby avoiding excessive concurrency leading to a decrease in the probability of memory reuse and a decrease in cache hit rate, and improving the computing efficiency and performance of the AI ​​model. The present application also provides a task scheduling system, a computing device cluster, a chip, a computer-readable storage medium, and a computer program product corresponding to the above method.

[0006] In the first aspect, the present application provides a task scheduling method. The method can be performed by a task scheduling system. The task scheduling system can be a software system, which can be used as part of an AI software framework or as an independent computing library for developers to download and integrate into applications developed by developers. The above-mentioned software system can be deployed in a chip or computing device cluster, and the chip or computing device cluster executes the program code of the software system to execute the task scheduling method of the present application. The task scheduling system can also be a hardware system, such as a chip or computing device cluster with a task scheduling function, and the task scheduling method of the present application is executed when the hardware system is running.

[0007] Specifically, the task scheduling system can obtain a calculation graph, which includes multiple operators, each of which represents a task. Then the task scheduling system can sort the multiple operators according to the dependency relationship of the multiple operators in the calculation graph to obtain an ordered operator sequence. For operators with dependencies in the ordered operator sequence, the dependencies are located before the operators. For example, the ordered operator sequence includes a first operator and a second operator that depends on the first operator, and the first operator is arranged before the second operator. Then the task scheduling system can group the multiple operators according to the ordered operator sequence to obtain multiple operator groups. The task scheduling system can sequentially schedule the multiple operator groups to devices (devices used to execute operators, such as accelerators, acceleration cards, and other devices) according to the priorities of the multiple operator groups.

[0008] In this method, by sorting the operators, subsequent operators can safely use the calculation results of the previous operators, thereby ensuring the correctness of the entire calculation process. The sorted operators are grouped, and tasks are issued at the granularity of operator groups to achieve effective scheduling under limited concurrency of the calculation graph, avoiding excessive concurrency that leads to a decrease in the probability of memory reuse and a decrease in cache hit rate, thereby improving the computing efficiency and performance of the AI ​​model.

[0009] In some possible implementations, the multiple operator groups include a first operator group and a second operator group, and the priority of the first operator group is higher than the priority of the second operator group. Accordingly, the task scheduling system may first schedule the operators in the first operator group to the device, and when the scheduling of the operators in the first operator group is completed, schedule the operators in the second operator group to the device.

[0010] This method schedules low-priority operator groups after high-priority operator groups are scheduled, achieving effective scheduling under limited concurrency of the computational graph, avoiding excessive concurrency that leads to a decrease in memory reuse probability and cache hit rate, and improving the computational efficiency and performance of the AI ​​model.

[0011] In some possible implementations, the task scheduling system can sequentially schedule multiple operator groups to devices according to the priorities of the multiple operator groups through a priority queue. Specifically, the task scheduling system can add operators in a ready state in multiple operator groups to the priority queue. Operators in a ready state include operators without dependencies or operators whose dependent operators have been executed. The task scheduling system can take out the target operator from the priority queue and schedule the target operator to the device. Among them, the priority of the target operator is higher than the remaining operators in the priority queue. For example, the target operator can be the operator with the highest priority in the priority queue. When the target operator is executed, the task scheduling system checks whether the operator on which the subsequent operator of the target operator depends has been executed. If so, the subsequent operator is added to the priority queue to continue to take out new operators from the priority queue for scheduling.

[0012] This method adds ready operators to the priority queue, and then controls the operators based on the priority queue to perform task scheduling at the granularity of operator groups, thereby achieving limited concurrency of the computational graph and improving the computational efficiency and performance of the AI ​​model.

[0013] In some possible implementations, the task scheduling system can obtain the group ID of the operator group to which the target operator belongs. When the group ID of the operator group to which the target operator belongs is equal to the group ID of the currently scheduled operator group, the task scheduling system schedules the target operator to the device. In this way, by identifying the group ID of the operator group to which the operator in the priority queue belongs, it is ensured that the operator of one operator group is sent before sending the operator of the next operator group. Through limited concurrency, it is avoided that the probability of memory reuse and the cache hit rate are reduced due to excessive concurrency.

[0014] In some possible implementations, the task scheduling system can also update the number of operators to be scheduled in the currently scheduled operator group. When the number of operators to be scheduled is equal to 0, the currently scheduled operator group is switched to the next operator group. In this way, the concurrency in the operator scheduling process can be controlled, and the next operator group can be scheduled after the currently scheduled operator group is scheduled, so as to avoid excessive concurrency leading to a decrease in the probability of memory reuse and a decrease in cache hit rate, thereby improving the computing efficiency and performance of the AI ​​model.

[0015] In some possible implementations, the task scheduling system can sort multiple operators according to the dependencies of multiple operators in the calculation graph through a critical path strategy or an inverse depth-first search strategy to obtain an ordered sequence of operators. The core of the critical path is to find the longest path length from each operator to the output node, and use the longest path length to assign a priority to each operator. Priority refers to the priority level of execution of each operator in the calculation graph. Operators with higher priorities will be given priority to obtain computing resources for calculation. When the demand or goal is performance priority, the task scheduling system can sort multiple operators according to the dependencies of multiple operators in the calculation graph through a critical path strategy to obtain an ordered sequence of operators. In this way, nodes with higher priorities can be traversed earlier while ensuring the execution order of operators with dependencies, thereby ensuring the performance of the AI ​​model. When the demand or goal is memory priority, the task scheduling system can sort multiple operators according to the dependencies of multiple operators in the calculation graph through an inverse depth-first search strategy to obtain an ordered sequence of operators. The core of the inverse depth-first search strategy is to reverse the calculation graph first so that all edges are in opposite directions. Then, a depth-first search is performed on the inverted computational graph, and the resulting traversal order is reversed to obtain the topological sorting of the original graph. This ensures that the memory reuse rate is maximized while ensuring the execution order of operators with dependencies.

[0016] In some possible implementations, the task scheduling system may group multiple operators according to an ordered operator sequence by a uniform grouping strategy to obtain multiple operator groups. Specifically, the task scheduling system may set the capacity of the operator group, which may be a constant N, indicating the number of operators in the operator group. The task scheduling system may group multiple operators in an ordered operator queue according to the capacity of the operator group to obtain multiple operator groups. Among them, uniform grouping is relatively simple, easy to implement, and can reduce the complexity of task scheduling.

[0017] In some possible implementations, the task scheduling system may also obtain at least one of the computational complexity, memory consumption or load characteristics of multiple operators. Then the task scheduling system may group multiple operators according to the ordered operator sequence and in combination with at least one of the computational complexity, memory consumption or load characteristics of multiple operators to obtain multiple operator groups. Memory consumption includes the memory size occupied by the input and output (IO) of the operator, and the load characteristics are used to describe the load type of the operator, for example, the load characteristics may include computationally intensive or IO intensive. In some examples, the task scheduling system may obtain the memory consumption of multiple operators, and when the memory size occupied by the input and output (for example, input tensor and output tensor) of M consecutive operators exceeds the memory capacity C, the task scheduling system may add the next operator to the new operator group. In other examples, the task scheduling system may obtain the load characteristics of multiple operators, and when grouping multiple operators according to the ordered operator sequence, the load characteristics of multiple operators are combined for grouping, so that the computationally intensive operators and IO intensive operators of the same operator group are balanced to avoid affecting the execution efficiency of operators within the group.

[0018] This method groups operators by combining the computational complexity, memory consumption, or load characteristics of multiple operators, which can improve the accuracy of operator grouping and enhance the performance and efficiency of the AI ​​model.

[0019] In some possible implementations, the task scheduling system may also obtain the scheduling results and the performance values ​​corresponding to the scheduling results, and then update the scheduling parameters according to the scheduling results and the performance values. The scheduling parameters include at least one of the grouping parameters or the sorting parameters. In this way, by optimizing the subsequent task scheduling through historical scheduling data, the task scheduling system can better adapt to different tasks and operating environments, and improve the performance of subsequent task scheduling.

[0020] In a second aspect, the present application provides a task scheduling system. The system comprises:

[0021] An acquisition module, used to acquire a computation graph, wherein the computation graph includes a plurality of operators, each of the plurality of operators representing a task;

[0022] A sorting module, configured to sort the multiple operators in the computation graph according to the dependency relationship between the multiple operators to obtain an ordered operator sequence, wherein the ordered operator sequence includes a first operator and a second operator that depends on the first operator, and the first operator is arranged before the second operator;

[0023] A grouping module, used for grouping the multiple operators according to the ordered operator sequence to obtain multiple operator groups;

[0024] The scheduling module is used to schedule the multiple operator groups to the devices in sequence according to the priorities of the multiple operator groups.

[0025] In some possible implementations, the multiple operator groups include a first operator group and a second operator group, the priority of the first operator group is higher than the priority of the second operator group, and the scheduling module is specifically configured to:

[0026] Scheduling operators in the first operator group to devices;

[0027] When the scheduling of operators in the first operator group is completed, the operators in the second operator group are scheduled to the device.

[0028] In some possible implementations, the scheduling module is specifically used to:

[0029] Adding operators in the plurality of operator groups that are in a ready state to a priority queue, wherein the operators in a ready state include operators without dependencies or operators whose dependent operators have all been executed;

[0030] Taking out a target operator from the priority queue and scheduling the target operator to a device, wherein the priority of the target operator is higher than that of the remaining operators in the priority queue;

[0031] When the target operator is executed, check whether the operator on which the subsequent operator of the target operator depends is executed. If so, add the subsequent operator to the priority queue to continue to take out new operators from the priority queue for scheduling.

[0032] In some possible implementations, the scheduling module is specifically used to:

[0033] Get the group ID of the operator group to which the target operator belongs;

[0034] When the group identifier of the operator group to which the target operator belongs is equal to the group identifier of the currently scheduled operator group, the target operator is scheduled to the device.

[0035] In some possible implementations, the scheduling module is further configured to:

[0036] Update the number of operators to be scheduled in the currently scheduled operator group;

[0037] When the number of operators to be scheduled is equal to 0, the currently scheduled operator group is switched to the next operator group.

[0038] In some possible implementations, the sorting module is specifically used to:

[0039] According to the dependency relationship of the multiple operators in the computation graph, the multiple operators are sorted by a critical path strategy or an inverse depth-first search strategy to obtain an ordered operator sequence.

[0040] In some possible implementations, the grouping module is specifically used to:

[0041] According to the ordered operator sequence, the multiple operators are grouped by a uniform grouping strategy to obtain multiple operator groups.

[0042] In some possible implementations, the grouping module is specifically used to:

[0043] Obtaining at least one of computational complexity, memory consumption, or load characteristics of the multiple operators;

[0044] According to the ordered operator sequence, in combination with at least one of computational complexity, memory consumption or load characteristics of the multiple operators, the multiple operators are grouped to obtain multiple operator groups.

[0045] In some possible implementations, the scheduling module is further configured to:

[0046] Obtaining a scheduling result and a performance value corresponding to the scheduling result;

[0047] According to the scheduling result and the performance value, a scheduling parameter is updated, where the scheduling parameter includes at least one of a grouping parameter or a sorting parameter.

[0048] In a third aspect, the present application provides a computing device cluster. The computing device cluster includes at least one computing device, and the at least one computing device includes at least one processor and at least one memory. The at least one processor and the at least one memory communicate with each other. The at least one processor is used to execute instructions stored in the at least one memory, so that the computing device or the computing device cluster executes the task scheduling method described in the first aspect or any implementation of the first aspect.

[0049] In a fourth aspect, the present application provides a chip. The chip includes a processor and a memory, and the processor may be a central processing unit on the host side. The processor executes the computer-readable instructions so that the chip executes the task scheduling method as described in the first aspect or any implementation of the first aspect.

[0050] In a fifth aspect, the present application provides a computer-readable storage medium, in which instructions are stored, and the instructions instruct a computing device or a computing device cluster to execute the task scheduling method described in the above-mentioned first aspect or any implementation of the first aspect.

[0051] In a sixth aspect, the present application provides a computer program product comprising instructions, which, when executed on a computing device or a computing device cluster, enables the computing device or the computing device cluster to execute the task scheduling method described in the first aspect or any one of the implementations of the first aspect.

[0052] Based on the implementations provided in the above aspects, this application can also be further combined to provide more implementations. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical method of the embodiments of the present application, the drawings required for use in the embodiments are briefly introduced below.

[0054] Figure 1 A hardware structure diagram of a server provided in an embodiment of the present application;

[0055] Figure 2 A flowchart of a task scheduling method provided in an embodiment of the present application;

[0056] Figure 3 A schematic diagram of the structure of a calculation graph provided in an embodiment of the present application;

[0057] Figure 4 A schematic diagram of a communication deadlock provided in an embodiment of the present application;

[0058] Figure 5 A flowchart of a running task delivery provided in an embodiment of the present application;

[0059] Figure 6 A schematic diagram of the structure of a task scheduling system provided in an embodiment of the present application;

[0060] Figure 7 A schematic diagram of the structure of a computing device provided in an embodiment of the present application;

[0061] Figure 8 A schematic diagram of the structure of a computing device cluster provided in an embodiment of the present application;

[0062] Fig. 9 A schematic diagram of the structure of another computing device cluster provided in an embodiment of the present application;

[0063] Fig.10 A schematic diagram of the structure of a computing device cluster provided in an embodiment of the present application. DETAILED DESCRIPTION

[0064] The terms "first" and "second" in the embodiments of the present application are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features.

[0065] First, some technical terms involved in the embodiments of the present application are introduced.

[0066] Computational Graph is a data structure used to describe AI models (such as deep learning models). The computational graph represents the computational process in the AI ​​model as a directed graph. The nodes in the directed graph represent operators (operation, referred to as OP), and the edges in the directed graph represent the data dependencies between operators (also referred to as dependencies), which indicate the flow path of the data. Operators are used to perform computational operations or communication operations. For example, computational operators can be used to perform computational operations such as addition, subtraction, multiplication, and division, and communication operators can be used to perform aggregate communication operations. Each operator can represent a task. Accordingly, the execution process of the computational graph includes the scheduling process of the operators in the computational graph (i.e., the computational graph task scheduling process) and the execution process.

[0067] Computational graph task scheduling, also referred to as task scheduling, refers to scheduling the operators (or tasks represented by operators) in the computational graph to the hardware devices that execute the tasks according to the scheduling strategy. The hardware devices are specifically hardware resources, including but not limited to central processing units (CPUs), graphics processing units (GPUs) or neural network processing units (NPUs), tensor processing units (TPUs). Scheduling strategies refer to the rules or strategies that determine the allocation of tasks represented by operators to hardware resources for execution.

[0068] By optimizing task scheduling, the operating efficiency of AI models can be effectively improved, the waste of computing resources can be reduced, and computing costs can be reduced. This is especially important for AI models with limited resources and huge computing requirements. In addition, task scheduling optimization can ensure that tasks are completed as quickly as possible while meeting task dependencies (i.e., dependencies between tasks), thereby shortening the training or reasoning time of AI models and improving overall efficiency. In addition, through carefully designed and optimized scheduling strategies, the stability and reliability of AI models can be improved, avoiding system performance degradation or calculation errors caused by task scheduling problems.

[0069] At present, the task scheduling methods widely used in the industry include graph-based scheduling and execution mechanisms. Typical implementations of graph-based scheduling and execution mechanisms include graph kernels. Graph Kernel technology is a new programming model implemented on the computing platform that allows developers to build and execute the entire computational graph directly at the hardware level, rather than just a single operator. In this model, compiled operators (called Kernels) are no longer executed independently, but are organized in a unified graph. In the graph, the dependencies between kernels are represented by edges, which means that the execution of a kernel can depend on the completion of one or more other kernels.

[0070] By organizing kernels in a graph, Graph Kernel can achieve a higher level of parallelism. Specifically, if all dependencies of a kernel (such as all kernels that the kernel depends on) are satisfied, then the kernel can start executing immediately without waiting for other unrelated kernels. At any time, as long as there are available hardware resources, a kernel can start executing, which can achieve higher hardware utilization.

[0071] However, in the graph-based scheduling and execution mechanism, the operator concurrency is relatively large, and the execution order of independent operators is dynamically determined, which often reduces the opportunity for memory reuse and leads to increased memory consumption. Taking the training process of an AI model as an example, memory consumption can be roughly divided into two categories: (1) static memory usage used to save information such as weight parameters, gradient information, and optimizer status. This part of memory usage is usually proportional to the scale of the AI ​​model. In addition, the tensors (Tensor) occupying this part of memory have the characteristic of persistence, and their life cycle is the entire stage from the beginning of AI model training to convergence; (2) memory usage of new Tensors dynamically generated by the AI ​​model during forward and reverse operation. The life cycle of this type of Tensor is from the moment the Tensor is generated to the moment when the Tensor is no longer needed. This part of memory usage is dynamic memory usage. Among them, dynamic memory usage can have the opportunity to be reused. In the graph-based scheduling and execution mechanism, if multiple independent operators are executed simultaneously, then the operators executed simultaneously need to retain their own data in memory (such as new tensors dynamically generated during the forward or reverse operation process), and cannot share the memory space of the completed operators. This will not only increase the overall memory consumption, but may also cause memory fragmentation and reduce memory usage efficiency.

[0072] The graph-based scheduling mechanism usually adopts a hungry mode, that is, to keep the hardware device in a saturated state as much as possible. In this mode, operators that are ready early may be executed first, so a large number of other operators may be inserted between two dependent operators A and B for execution. Due to the limited capacity of the cache, the output results of operator A originally in the cache may be eliminated. When executing B, the output results of A need to be reloaded from the memory into the cache, which greatly reduces the cache hit rate and affects the overall execution efficiency.

[0073] Whether it is an increase in memory consumption or a decrease in cache hit rate, it can lead to a decrease in the computing efficiency and performance of the AI ​​model, making it difficult to meet the AI ​​model's high demands for computing efficiency and performance.

[0074] In view of this, the present application provides a task scheduling method. The method can be performed by a task scheduling system. The task scheduling system can be a software system, which can be used as part of an AI software framework, or as an independent computing library for developers to download and integrate into applications developed by developers. The above software system can be deployed in a chip (for example, a CPU) or a computing device cluster, and the chip or computing device cluster executes the program code of the software system to execute the task scheduling method of the present application. The task scheduling system can also be a hardware system, such as a chip or a computing device cluster with a task scheduling function, and the task scheduling method of the present application is executed when the hardware system is running.

[0075] Specifically, the task scheduling system can obtain a calculation graph, which includes multiple operators, each of which represents a task. Then the task scheduling system can sort the multiple operators according to the dependency relationship of the multiple operators in the calculation graph to obtain an ordered operator sequence. For operators with dependencies in the ordered operator sequence, the dependencies are located before the operators. For example, the ordered operator sequence includes a first operator and a second operator that depends on the first operator, and the first operator is arranged before the second operator. Then the task scheduling system can group the multiple operators according to the ordered operator sequence to obtain multiple operator groups. The task scheduling system can sequentially schedule the multiple operator groups to devices (devices used to execute operators) according to the priorities of the multiple operator groups.

[0076] In this method, by sorting the operators, subsequent operators can safely use the calculation results of the previous operators, thereby ensuring the correctness of the entire calculation process. The sorted operators are grouped, and tasks are issued at the granularity of operator groups to achieve effective scheduling under limited concurrency of the calculation graph, avoiding excessive concurrency that leads to a decrease in the probability of memory reuse and a decrease in cache hit rate, thereby improving the computing efficiency and performance of the AI ​​model.

[0077] The task scheduling method of the present application is a task scheduling method based on policy control, which can be widely used in the optimization of various AI models. Among them, the AI ​​model can be a model that requires real-time or near real-time processing tasks, including but not limited to AI models for applications such as autonomous driving and video processing. The task scheduling method based on policy control can effectively schedule and manage tasks to meet strict time constraints. In addition, the method can be applied to the training phase or reasoning phase of the AI ​​model to accelerate the training process or reasoning process of the AI ​​model and achieve efficient use of computing resources. It should be noted that the task scheduling method can also be extended to other application scenarios that require task scheduling, including but not limited to big data processing, scientific computing, and graphics rendering, which are not limited in this application.

[0078] In some possible implementations, the task scheduling method can be applied to multi-core computing devices. On GPUs, TPUs, or other hardware devices with multiple computing cores, the use of a policy-based task scheduling method can effectively allocate and manage the computing resources of each core, thereby improving the overall computing efficiency. In other possible implementations, the task scheduling method can also be applied to large-scale parallel computing architectures, such as parallel computing architectures formed by multiple computing devices. In large-scale parallel computing environments, such as distributed systems or cluster computing environments, the above-mentioned task scheduling method can maximize the degree of parallelism while ensuring the correctness of the algorithm, thereby shortening the overall execution time of the task.

[0079] For ease of description, a multi-core computing device is used as an example server for illustration below.

[0080] See also Figure 1 The hardware structure diagram of a server shown in FIG. 1 is a server 10 that can be a single-machine multi-card architecture server. Figure 1 The server 10 shown can also form a server cluster (also called a computing device cluster) with a multi-machine and multi-card architecture. It should be noted that the "machine" in the above x machine x card refers to a host, and the "card" refers to an acceleration card (also called an accelerator) and other devices.

[0081] The server 10 includes a host 12 and a device 14. The host 12 and the device 14 are connected. The host 12 includes a processor and a memory (i.e., a host memory). The processor may be a CPU, and the memory may be a dual in-line memory module (DIMM). The DIMM may be a double data rate (DDR) type, for example, the memory may be a DDR4 DIMM. Figure 1 In the example, the host 12 includes 4 CPUs and 4 DDR4 DIMM groups, each CPU is connected to 1 DDR4 DIMM group, and each DDR4 DIMM group includes 8 DDR4 DIMMs. Multiple CPUs of the host 12 can be connected to form a hydra mesh.

[0082] Optionally, the host 12 further includes an interface, such as one or more of a Serial Advanced Technology Attachment (SATA) interface, a new generation non-volatile memory express (NVMe) interface, and a Gigabit Ethernet (GE) interface. The host 12 may also include a memory. The memory may include a SATA-supported memory or an NVMe-supported memory, such as a mechanical hard disk that supports SATA, or a solid state drive (SSD) that supports NVMe.

[0083] Device 14 includes an accelerator card. Figure 1 In the example, the accelerator card included in device 14 may be an NPU. Figure 1 The device 14 includes 8 NPUs for illustration. In other possible implementations of the embodiment of the present application, the device 14 may also include more accelerator cards, such as more types of accelerator cards, or a greater number of accelerator cards.

[0084] Figure 1 This is only an example of a computing device that executes the task scheduling method of the present application. In other possible implementations of the present application embodiment, the computing device may also be a server with a single-machine single-card architecture, and this embodiment does not limit this.

[0085] In order to make the technical solution of the present application clearer and easier to understand, the task scheduling method of the present application is described in detail below in conjunction with an embodiment.

[0086] See also Figure 2 The flowchart of a task scheduling method shown in FIG. 1 is a flowchart of a task scheduling method, which can be executed by a server and specifically includes the following steps:

[0087] S202: The server obtains a computation graph.

[0088] The computation graph includes multiple operators, each of which represents a task. Depending on the operator type, the task can be different. For example, the operator can be a computation operator or a communication operator, and accordingly, the task can be a computation task or a communication task (such as a cluster communication task).

[0089] For ease of understanding, this application also provides an example of a computational graph. Figure 3As shown in the figure, the computational graph is essentially a directed acyclic graph (DAG). The rectangular parts in the figure correspond to operators. The different colors or grayscales of the rectangular parts are used to represent different operator types. The operators represented by rectangles of different colors or grayscales can be executed on different hardware units. The first row in the rectangular part represents the identifier (ID) of the i-th operator, and the content in the brackets of the second row represents the time t required for the operator to complete execution. i The ellipse in the figure represents the output of each operator, such as the name and size of the tensor output by the operator. The arrow indicates that a tensor is the input or output of a certain operator. For example, operator A refers to T A1 The arrow of the node represents the Tensor T A1 is an output of operator A, and T A1 The arrow from the node to operator B represents Tensor T A1 is one of the inputs of operator B.

[0090] S204. The server sorts the multiple operators according to the dependencies of the multiple operators in the computation graph to obtain an ordered operator sequence.

[0091] The execution order of operators in the computation graph needs to satisfy the topological sorting rules. Topological sorting is a graph theory concept used to sort nodes on a directed acyclic graph (DAG) so that for each directed edge (u, v), node u is ranked before node v. Based on this, the server can sort multiple operators according to the dependencies of multiple operators in the computation graph to obtain an ordered operator sequence. Among them, the operators with dependencies in the ordered operator sequence have their dependencies ranked before the operators. For example, the ordered operator sequence includes a first operator and a second operator, and the second operator depends on the first operator, that is, the first operator is a dependency of the second operator, and the first operator is ranked before the first operator.

[0092] In specific implementation, the server can select different sorting strategies or a combination of sorting strategies according to requirements to sort multiple operators and obtain an ordered operator sequence. The following are examples of different sorting strategies.

[0093] One sorting strategy is the critical path strategy. When the demand or goal is performance first, the server can sort multiple operators according to the dependencies of multiple operators in the calculation graph through the critical path strategy to obtain an ordered sequence of operators. The core of the critical path is to find the longest path length from each operator to the output node, and use the longest path length to assign a priority to each operator. Priority refers to the priority level of execution of each operator in the calculation graph. Operators with higher priorities will be given priority to obtain computing resources for calculation. For example, the server can determine the priority of operators based on the dependencies of multiple operators and the longest path length from the operator to the output node. On the premise of ensuring the execution order of operators with dependencies, nodes with higher priorities (such as longer paths to the output node) can be sorted first. In other words, nodes with higher priorities can be traversed earlier.

[0094] In specific implementation, the server can initialize the longest path length of all operators to 0, and then use the dynamic programming strategy to traverse towards the output operator for each operator and update the longest path length of each operator. For any operator, its longest path length should be equal to the longest longest path length of all its subsequent operators plus 1. The server can then determine the priority of each operator based on the dependency relationship and the longest path length between operators, sort multiple operators based on the priority, and obtain an ordered sequence of operators. Under the premise of satisfying the topological sorting rules, operators with larger longest path lengths are sorted first and traversed first.

[0095] Another sorting strategy is the inverse depth-first search (DFS) strategy. When the demand or goal is memory priority, the server can sort multiple operators in the computation graph according to their dependencies using the inverse DFS strategy to obtain an ordered sequence of operators. The core of the inverse DFS strategy is to first reverse the computation graph so that all edges are in the opposite direction. Then, a depth-first search is performed on the inverted computation graph, and the resulting traversal order is reversed to obtain a topological sort of the original graph.

[0096] In specific implementation, the server can reverse the computation graph G1 to obtain the computation graph G2. For example, if there is an edge from operator A to operator B in the computation graph G1, then in G2, the direction of this edge will point from operator B to operator A. The server then performs a depth-first search on the computation graph G2. The server can start the search from the output node, and the order obtained at the end of the traversal is the inverse topological sort of the computation graph G1. The server reverses the inverse topological sort of the computation graph G1 to obtain the topological sort of the computation graph G1.

[0097] S206. The server groups multiple operators according to the ordered operator sequence to obtain multiple operator groups.

[0098] Operator Grouping can be to group operators with the same or similar priorities into the same operator group to facilitate efficient task scheduling. Operators in each operator group have the same or similar priorities. This grouping method helps to effectively execute operators in parallel during the subsequent scheduling process, while also ensuring the correctness of execution.

[0099] Specifically, the server can set the priority according to the topologically sorted index. In some examples, the server can directly use the topologically sorted index as the priority of the operator. In other words, the operator with a higher ranking has a higher priority. In other examples, the server can also compare the topologically sorted index with at least one threshold to obtain the priority of the corresponding operator. When different operators fall into the same value interval (a value interval determined by at least one threshold) in the topologically sorted index, the different operators have the same priority. When different operators fall into different value intervals in the topologically sorted indexes, the priority of the operator with a smaller index is higher than the priority of the operator with a larger index.

[0100] Similar to sorting, the server supports multiple grouping strategies or a combination of grouping strategies to group multiple operators. The following are examples of different grouping strategies.

[0101] One grouping strategy may be uniform grouping. For example, the server may set the capacity of each operator group, which may be a constant N, indicating the number of operators in the operator group. The server may group multiple operators in the ordered operator queue according to the capacity of the operator group to obtain multiple operator groups. For example, when the computation graph includes L*N operators, the operators in the ordered operator queue may be evenly divided into L operator groups, each of which includes N consecutive operators in the ordered operator queue.

[0102] Another grouping strategy may be grouping based on operator characteristics. Specifically, the computational complexity, memory consumption or load characteristics of different operators may be different, wherein the memory consumption includes the memory size occupied by the input and output (IO) of the operator, and the load characteristics are used to describe the load type of the operator, for example, the load characteristics may include computationally intensive or IO intensive. In order to ensure execution efficiency and performance, the server may group the multiple operators according to the ordered operator sequence, combined with at least one of the computational complexity, memory consumption or load characteristics of the multiple operators, to obtain multiple operator groups.

[0103] For example, the server can obtain the memory consumption of multiple operators. When the memory size occupied by the input and output of M consecutive operators (such as input tensor and output tensor) exceeds the capacity C, the server can add the next operator to the new operator group.

[0104] For another example, the server can obtain the load characteristics of multiple operators. When grouping multiple operators according to an ordered operator sequence, the load characteristics of multiple operators are combined for grouping, so that the computationally intensive operators and IO-intensive operators in the same operator group are balanced, avoiding affecting the execution efficiency of operators within the group.

[0105] It should be noted that the server can also support more scheduling strategies when performing task scheduling, including sorting strategies and grouping strategies. For example, users can customize scheduling strategies based on the computational complexity, memory consumption (including IO requirements), and load characteristics of operators. Based on the policy-controlled computational graph task scheduling method, this application may consider combining hardware optimization methods with scheduling strategies. For example, this application supports optimizing scheduling strategies based on the characteristics of specific hardware (such as cache size, memory bandwidth, etc.).

[0106] Furthermore, the server can also record the group identifier (ID) of the operator group to which each operator belongs. In this way, in the subsequent execution process, the server can quickly find the corresponding operator according to the group ID of the operator group to which the operator belongs, thereby improving the efficiency of task scheduling. Similarly, the server can also calculate the number of operators in the subgroup, so that in the subsequent execution process, the execution of each operator group can be effectively managed and controlled.

[0107] The above S202 to S206 are the steps that need to be performed in the compile state (Compile-time) of the method. The compile state refers to the stage of program or model compilation, and the operations performed in the compile state include but are not limited to topological sorting, setting priorities, etc. The above processing steps in the compile state are conducive to efficient and accurate scheduling of operators in the calculation graph in the run state (Run-time), thereby improving the running performance of the entire AI model.

[0108] S208. The server sequentially schedules the multiple operator groups to the device according to the priorities of the multiple operator groups.

[0109] The server host can use operator groups as the granularity and schedule multiple operator groups to devices in order according to the priorities of the operator groups, so that the devices can execute the tasks corresponding to the operators in the operator groups according to the priorities. The priority of the operator group can be determined according to the priorities of the operators in the operator group. For example, if the operators in an operator group have the same priority, the priority of the operator group can be represented by the priorities of the operators.

[0110] Specifically, the multiple operator groups may include a first operator group and a second operator group, wherein the priority of the first operator group is higher than that of the second operator group. Accordingly, the server may schedule the tasks corresponding to the operators in the first operator group to the device first; when the scheduling of the operators in the first operator group is completed, the server may schedule the tasks corresponding to the operators in the second operator group to the device.

[0111] In some possible implementations, the server can add operators in a ready state in multiple operator groups to a priority queue, and implement task dispatching with operator groups as granularity from the priority queue. Among them, operators in a ready state include operators without dependencies or operators whose dependent operators have all been executed. In the initial stage, operators in a ready state are usually operators without dependencies. The server can take out the target operator from the priority queue and schedule (dispatch) the target operator to the device. Among them, the priority of the target operator is higher than the remaining operators in the priority queue. In some examples, the target operator can be the operator with the highest priority in the priority queue. When the target operator is executed, the server can check whether the operator on which the subsequent operator of the target operator depends is executed. In other words, the server checks whether the completion of the execution of the target operator makes the subsequent operator ready. If so, the server can add the subsequent operator to the priority queue to continue to take out new operators (such as new target operators) from the priority queue for task scheduling.

[0112] When taking out the target operator from the priority queue and sending it to the device, the server can obtain the group ID of the operator group to which the target operator belongs. When the group ID of the operator group to which the target operator belongs is equal to the group ID of the currently scheduled operator group, it indicates that the target operator is an operator in the currently scheduled operator group, and the server sends the target operator to the device. When the group ID of the operator group to which the target operator belongs is not equal to the group ID of the currently scheduled operator group, the server can do nothing and wait until the operators in the currently scheduled operator group are scheduled before scheduling the target operator.

[0113] Furthermore, the server may check the number of operators to be scheduled in the currently scheduled operator group. When the number of operators to be scheduled is equal to 0, it indicates that all operators in the currently scheduled operator group have been scheduled, and the server may switch the currently scheduled operator group to the next operator group. The next operator group may be the operator group with the highest priority arranged after the currently scheduled operator group, such as the first operator group arranged after the currently scheduled operator group. When switching the operator group, the number of operators to be scheduled may also be initialized to the number of operators in the switched operator group. Furthermore, when the number of operators to be scheduled is not equal to 0, it indicates that the operators in the currently scheduled operator group have not yet been scheduled, and there are operators with higher priorities in the currently scheduled operator group that have not yet been queued.

[0114] In some possible implementations, the server may also obtain the scheduling results and the performance values ​​corresponding to the scheduling results (for example, reasoning time, memory consumption, and computing resource utilization). The performance values ​​corresponding to the scheduling results may be used as feedback signals. Accordingly, the server may update the scheduling parameters based on the scheduling results and the performance values. Among them, the scheduling parameters include at least one of the grouping parameters (such as the capacity C of the memory) or the sorting parameters (such as the capacity of the operator group). By continuously learning and optimizing the scheduling strategy through the scheduling results and the real-time feedback performance values, the task scheduling system can better adapt to different tasks and operating environments.

[0115] The above S208 is a step that needs to be executed in the running state of the method. The server can schedule and execute the operators in the calculation graph through a high priority first strategy based on the topological sorting, priority, grouping information, group ID and other data generated in the compiled state.

[0116] Based on the above description, it can be seen that the task scheduling method provided in this application realizes limited concurrent task scheduling by grouping the sorted operators and issuing tasks based on the operator group granularity, thereby increasing memory reuse opportunities, solving the problem of low cache hit rate, and improving the end-to-end performance and memory utilization of the AI ​​model.

[0117] Moreover, the graph-based scheduling mechanism may lead to communication deadlock problems. Specifically, see Figure 4 , devices with different ranks, such as accelerator card 0 (denoted as rank 0) and accelerator card 1 (denoted as rank 1), all run multiple threads, including thread 0 (thread 0) and thread 1 (thread 1). However, in the graph-based scheduling mechanism, the concurrent progress on different devices is inconsistent, which may lead to different order of communication operators, and eventually lead to deadlock. For example, the communication operator executed by thread 0 on accelerator card 0 and the communication operator executed by thread 1 on accelerator card 1 are both waiting for the data sent by the other party, but neither of them sends their own data, then a deadlock will occur. This can cause the entire calculation process to stop. In order to solve the communication deadlock problem, at most one communication operator can be configured in the calculation graph, which will reduce the scale of the calculation graph and reduce the chance of concurrent execution of operators of different types. The task scheduling method of the present application groups the communication operators in the calculation graph, and each operator group can include at most one communication operator. Since the order of the operator groups is determined, the order of communication operators can be consistent, thereby solving the communication deadlock problem and ensuring execution efficiency.

[0118] Next, the execution process of the running state is described in detail in conjunction with another embodiment.

[0119] See also Figure 5As shown in the flowchart of another task scheduling method, the server can set the number of operator groups to be m and the number of operators in the kth group to be N k , k=(1,2,…,m), the group ID of operator i is g i , index g c Indicates the group ID of the currently scheduled operator group, R c Indicates the currently scheduled operator group g c The number of operators to be scheduled in, that is, the operator group g currently scheduled c The server can then execute the following steps:

[0120] Phase 1: Initialization.

[0121] Specifically, the server may set a pointer, which may point to the currently scheduled operator group. During initialization, the pointer may point to the group with the highest priority (in this embodiment, it is assumed that the group ID of the operator group with the highest priority is m), and then the server may complete the following initialization steps:

[0122] a) Initialize an empty priority queue, denoted as Q;

[0123] b)g c =m;

[0124] c) R c =N gc .

[0125] Phase 2: Ready operators join the queue.

[0126] Specifically, the server can put operators in the ready state (also called ready operators) into the priority queue Q according to their priorities calculated in the compiled state. The ready state of an operator means that the operator has no dependencies, or all dependencies of the operator have been executed. In the initial stage, ready operators are operators that do not depend on the output results of other operators. A computation graph has at least one operator in the ready state in the initial stage.

[0127] Phase 3: Operator scheduling.

[0128] Specifically, the server can select the operator i with the highest current priority from the priority queue Q for distribution according to the priority of the operator. Among them, the operators stored in the priority queue are all in the ready state, and the operators will be executed on the Device after being distributed. The distribution of operators can be divided into the following two situations:

[0129] a) If the group ID g of the operator group to which operator i belongs i Not equal to g c , it returns directly without sending operator i temporarily;

[0130] b) If the group ID g of the operator group to which operator i belongs i Equal to g c , take out operator i from the priority queue, send operator i for execution, and then update R c ←R c -1.

[0131] Phase 4: Operator execution is completed.

[0132] Specifically, the completion of each operator execution may trigger a scheduling interrupt, and the server may respond to the scheduling interrupt to check whether all subsequent operators of the operator have become ready. If an operator is ready, the server may mark the state of the subsequent operator as ready and add the subsequent operator to the priority queue Q.

[0133] Phase 5: Operator group switching.

[0134] When all operators of an operator group have been issued, the server can start issuing the next priority group. Specifically, the server can point the pointer to the next priority operator group. The specific switching logic is:

[0135] a) If R c =0, indicating the operator group g currently scheduled c The number of operators that have not been issued is 0, which means that the operator group needs to be switched. The server can update g c and R c , the update criterion is: g c ←g c -1, When c When <0, it indicates that all operators have been sent to the device for execution. Otherwise, it jumps to phase 3 for execution.

[0136] b) If R c ≠0, indicating the currently scheduled operator group g c There are operators with higher priority that have not entered the priority queue, so there is no need to switch operator groups. The server repeats phases 2 to 5 until g c =0, it indicates that all operator groups have been sent.

[0137] In this method, when all operators are executed, it indicates that the entire computation graph has been executed and the server can return to the end.

[0138] The above-mentioned algorithm flow in the running state monitors the execution status of each group of operators to ensure that the next group of operators is issued only after all operators in the high-priority group have been issued, thereby achieving effective scheduling under limited concurrency of the computational graph, thereby improving the execution performance of the entire AI model and meeting the needs of different AI models. At the same time, this method also solves the problems of poor universality of the flow-based scheduling execution mechanism, frequent operator startup operations, high synchronization overhead, and inability to parallelize due to inter-operator dependencies, as well as the problems of increased memory consumption, reduced cache hit rate, and communication deadlock in the graph-based scheduling execution mechanism.

[0139] Based on the above-mentioned task scheduling method, the present application also provides a task scheduling system. Figure 6 As shown, the system 600 includes:

[0140] An acquisition module 602 is used to acquire a computation graph, wherein the computation graph includes a plurality of operators, each of the plurality of operators representing a task;

[0141] A sorting module 604, configured to sort the multiple operators in the computation graph according to the dependency relationship of the multiple operators to obtain an ordered operator sequence, wherein the ordered operator sequence includes a first operator and a second operator that depends on the first operator, and the first operator is arranged before the second operator;

[0142] A grouping module 606, configured to group the plurality of operators according to the ordered operator sequence to obtain a plurality of operator groups;

[0143] The scheduling module 608 is used to schedule the multiple operator groups to the devices in sequence according to the priorities of the multiple operator groups.

[0144] Exemplarily, the acquisition module 602 , the sorting module 604 , the grouping module 606 , and the scheduling module 608 may be implemented by hardware or software.

[0145] Among them, when implemented by software, the acquisition module 602, the sorting module 604, the grouping module 606, and the scheduling module 608 can be applications running on computer devices, such as computing engines, etc. The above-mentioned computing programs can also be virtualized and provided to users through virtualization services. Virtualization services may include virtual machine (VM) services, bare metal server (BMS) services, and container services. Among them, the VM service can be a service that virtualizes a virtual machine (VM) resource pool on multiple physical hosts through virtualization technology to provide users with VMs on demand for use. The BMS service is a service that virtualizes a BMS resource pool on multiple physical hosts to provide users with BMS on demand for use. The container service is a service that virtualizes a container resource pool on multiple physical hosts to provide users with containers on demand for use. VM is a simulated virtual computer, that is, a logical computer. BMS is a high-performance computing service that can be elastically scalable, and its computing performance is no different from that of a traditional physical machine, and it has the characteristics of secure physical isolation. Container is a kernel virtualization technology that can provide lightweight virtualization to achieve the purpose of isolating user space, processes and resources. It should be understood that the VM service, BMS service and container service in the above virtualization services are only used as specific examples. In actual applications, virtualization services can also be other lightweight or heavyweight virtualization services, which are not specifically limited here.

[0146] When implemented by hardware, the acquisition module 602, the sorting module 604, the grouping module 606, and the scheduling module 608 may include at least one computing device. Alternatively, the acquisition module 602, the sorting module 604, the grouping module 606, and the scheduling module 608 may also be implemented by using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0147] In some possible implementations, the multiple operator groups include a first operator group and a second operator group, and the priority of the first operator group is higher than the priority of the second operator group. The scheduling module 608 is specifically used to schedule the operators in the first operator group to the device, and when the scheduling of the operators in the first operator group is completed, the operators in the second operator group are scheduled to the device.

[0148] In some possible implementations, the scheduling module 608 is specifically configured to:

[0149] Adding operators in the plurality of operator groups that are in a ready state to a priority queue, wherein the operators in a ready state include operators without dependencies or operators whose dependent operators have all been executed;

[0150] Taking out a target operator from the priority queue and scheduling the target operator to a device, wherein the priority of the target operator is higher than that of the remaining operators in the priority queue;

[0151] When the target operator is executed, check whether the operator on which the subsequent operator of the target operator depends is executed. If so, add the subsequent operator to the priority queue to continue to take out new operators from the priority queue for scheduling.

[0152] In some possible implementations, the scheduling module 608 is specifically configured to:

[0153] Get the group ID of the operator group to which the target operator belongs;

[0154] When the group identifier of the operator group to which the target operator belongs is equal to the group identifier of the currently scheduled operator group, the target operator is scheduled to the device.

[0155] In some possible implementations, the scheduling module 608 is further configured to:

[0156] Update the number of operators to be scheduled in the currently scheduled operator group;

[0157] When the number of operators to be scheduled is equal to 0, the currently scheduled operator group is switched to the next operator group.

[0158] In some possible implementations, the sorting module 604 is specifically configured to:

[0159] According to the dependency relationship of the multiple operators in the computation graph, the multiple operators are sorted by a critical path strategy or an inverse depth-first search strategy to obtain an ordered operator sequence.

[0160] In some possible implementations, the grouping module 606 is specifically configured to:

[0161] According to the ordered operator sequence, the multiple operators are grouped by a uniform grouping strategy to obtain multiple operator groups.

[0162] In some possible implementations, the grouping module 606 is specifically configured to:

[0163] Obtaining at least one of computational complexity, memory consumption, or load characteristics of the multiple operators;

[0164] According to the ordered operator sequence, in combination with at least one of computational complexity, memory consumption or load characteristics of the multiple operators, the multiple operators are grouped to obtain multiple operator groups.

[0165] In some possible implementations, the scheduling module 608 is further configured to:

[0166] Obtaining a scheduling result and a performance value corresponding to the scheduling result;

[0167] According to the scheduling result and the performance value, a scheduling parameter is updated, where the scheduling parameter includes at least one of a grouping parameter or a sorting parameter.

[0168] The present application also provides a computing device 700. Figure 7 As shown, computing device 700 includes: bus 702, processor 704, accelerator 705, memory 706 and communication interface 708. Processor 704, memory 706 and communication interface 708 communicate through bus 702. Computing device 700 can be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in computing device 700.

[0169] The bus 702 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 The bus 702 may include a path for transmitting information between various components of the computing device 700 (eg, the memory 706, the processor 704, and the communication interface 708).

[0170] The processor 704 may include a central processing unit (CPU), which is usually located on the host side. The accelerator 705 may include any one or more of a graphics processing unit (GPU), a neural network processing unit (NPU), and a tensor processing unit (TPU), and the accelerator 705 is usually located on the device side.

[0171] The memory 706 may include a volatile memory, such as a random access memory (RAM). The memory 706 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD) or a solid state drive (SSD). The memory 706 stores executable program code, and the processor 704 executes the executable program code to implement the aforementioned task scheduling method. Specifically, the memory 706 stores instructions for the task scheduling system 600 to execute the task scheduling method.

[0172] The communication interface 708 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 700 and other devices or communication networks.

[0173] The embodiment of the present application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smart phone.

[0174] like Figure 8 As shown, the computing device cluster includes at least one computing device 700. The memory 706 in one or more computing devices 700 in the computing device cluster may store the same instructions of the task scheduling system 600 for executing the task scheduling method.

[0175] In some possible implementations, one or more computing devices 700 in the computing device cluster may also be used to execute part of the instructions of the task scheduling system for executing the task scheduling method. In other words, a combination of one or more computing devices 700 may jointly execute the instructions of the task scheduling system 600 for executing the task scheduling method.

[0176] It should be noted that the memory 706 in different computing devices 700 in the computing device cluster may store different instructions for executing part of the functions of the task scheduling system.

[0177] Fig. 9 A possible implementation is shown. Fig. 9 As shown, two computing devices 700A and 700B are connected via a communication interface 708. The memory in computing device 700A stores instructions for executing the functions of acquisition module 602, sorting module 604, and grouping module 606. The memory in computing device 700B stores instructions for executing the functions of scheduling module 608. In other words, the memories 706 of computing devices 700A and 700B jointly store instructions for the task scheduling system 600 to execute the task scheduling method.

[0178] Fig. 9 The connection mode between the computing device clusters shown can be considered that the task scheduling method provided by the present application needs to perform sorting and grouping in the compilation state, and perform task scheduling according to the grouping results in the running state. Therefore, it is considered to hand over the functions implemented by the scheduling module 608 to the computing device 700B for execution.

[0179] It should be understood that Fig. 9 The functions of the computing device 700A shown in FIG. 7 may also be completed by multiple computing devices 700. Similarly, the functions of the computing device 700B may also be completed by multiple computing devices 700.

[0180] In some possible implementations, one or more computing devices in the computing device cluster may be connected via a network, which may be a wide area network or a local area network. Fig.10 A possible implementation is shown. Fig.10 As shown, two computing devices 700C and 700D are connected via a network. Specifically, they are connected to the network via a communication interface in each computing device. In this type of possible implementation, the memory 706 in the computing device 700C stores instructions for executing the functions of the acquisition module 602, the sorting module 604, and the grouping module 606. At the same time, the memory 706 in the computing device 700D stores instructions for executing the functions of the scheduling module 608.

[0181] Fig.10 The connection method between the computing device clusters shown can be considered to be that the task scheduling method provided in this application needs to perform sorting and grouping in the compilation state, and perform task scheduling according to the grouping results in the running state. Therefore, it is considered to hand over the functions implemented by the scheduling module 608 to the computing device 700D for execution.

[0182] It should be understood that Fig.10 The functions of the computing device 700C shown in FIG. 700A may also be completed by multiple computing devices 700. Similarly, the functions of the computing device 700D may also be completed by multiple computing devices 700.

[0183] The embodiment of the present application also provides a chip. The chip includes a processor and a memory, and the processor may be a CPU on the host side. In specific implementation, the processor may execute the computer-readable instructions so that the chip executes the task scheduling method of the aforementioned embodiment.

[0184] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by the computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the computing device to execute the above-mentioned application to the task scheduling system 600 for executing the task scheduling method.

[0185] The embodiment of the present application also provides a computer program product including instructions. The computer program product may be software or a program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes the above-mentioned task scheduling method.

[0186] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A task scheduling method, characterized in that: The method comprises: Obtain a computation graph, wherein the computation graph includes a plurality of operators, each of the plurality of operators representing a task; sorting the multiple operators according to the dependency relationship of the multiple operators in the computation graph to obtain an ordered operator sequence, wherein the ordered operator sequence includes a first operator and a second operator that depends on the first operator, and the first operator is arranged before the second operator; Grouping the multiple operators according to the ordered operator sequence to obtain multiple operator groups; The multiple operator groups are sequentially scheduled to devices according to the priorities of the multiple operator groups.

2. The method according to claim 1, characterized in that The multiple operator groups include a first operator group and a second operator group, the priority of the first operator group is higher than the priority of the second operator group, and the sequentially scheduling the multiple operator groups to the devices according to the priorities of the multiple operator groups includes: Scheduling operators in the first operator group to devices; When the scheduling of operators in the first operator group is completed, the operators in the second operator group are scheduled to the device.

3. The method according to claim 1 or 2, characterized in that: The step of sequentially scheduling the multiple operator groups to devices according to the priorities of the multiple operator groups includes: Adding operators in the plurality of operator groups that are in a ready state to a priority queue, wherein the operators in a ready state include operators without dependencies or operators whose dependent operators have all been executed; Taking out a target operator from the priority queue and scheduling the target operator to a device, wherein the priority of the target operator is higher than that of the remaining operators in the priority queue; When the target operator is executed, check whether the operator on which the subsequent operator of the target operator depends is executed. If so, add the subsequent operator to the priority queue to continue to take out new operators from the priority queue for scheduling.

4. The method according to claim 3, characterized in that The taking out the target operator from the priority queue and scheduling the target operator to the device includes: Get the group ID of the operator group to which the target operator belongs; When the group identifier of the operator group to which the target operator belongs is equal to the group identifier of the currently scheduled operator group, the target operator is scheduled to the device.

5. The method according to claim 3, characterized in that: The method further comprises: Update the number of operators to be scheduled in the currently scheduled operator group; When the number of operators to be scheduled is equal to 0, the currently scheduled operator group is switched to the next operator group.

6. The method according to any one of claims 1 to 5, characterized in that: The step of sorting the multiple operators according to the dependency relationship of the multiple operators in the computation graph to obtain an ordered operator sequence includes: According to the dependency relationship of the multiple operators in the computation graph, the multiple operators are sorted by a critical path strategy or an inverse depth-first search strategy to obtain an ordered operator sequence.

7. The method according to any one of claims 1 to 6, characterized in that: The grouping the multiple operators according to the ordered operator sequence to obtain multiple operator groups includes: According to the ordered operator sequence, the multiple operators are grouped by a uniform grouping strategy to obtain multiple operator groups.

8. The method according to any one of claims 1 to 6, characterized in that: The grouping the multiple operators according to the ordered operator sequence to obtain multiple operator groups includes: Obtaining at least one of computational complexity, memory consumption, or load characteristics of the multiple operators; According to the ordered operator sequence, in combination with at least one of computational complexity, memory consumption or load characteristics of the multiple operators, the multiple operators are grouped to obtain multiple operator groups.

9. The method according to any one of claims 1 to 8, characterized in that: The method further comprises: Obtaining a scheduling result and a performance value corresponding to the scheduling result; According to the scheduling result and the performance value, a scheduling parameter is updated, where the scheduling parameter includes at least one of a grouping parameter or a sorting parameter.

10. A task scheduling system, characterized in that: The system comprises: An acquisition module, used to acquire a computation graph, wherein the computation graph includes a plurality of operators, each of the plurality of operators representing a task; A sorting module, configured to sort the multiple operators in the computation graph according to the dependency relationship between the multiple operators to obtain an ordered operator sequence, wherein the ordered operator sequence includes a first operator and a second operator that depends on the first operator, and the first operator is arranged before the second operator; A grouping module, used for grouping the multiple operators according to the ordered operator sequence to obtain multiple operator groups; The scheduling module is used to schedule the multiple operator groups to the devices in sequence according to the priorities of the multiple operator groups.

11. The system according to claim 10, characterized in that The multiple operator groups include a first operator group and a second operator group, the priority of the first operator group is higher than the priority of the second operator group, and the scheduling module is specifically used for: Scheduling operators in the first operator group to devices; When the scheduling of operators in the first operator group is completed, the operators in the second operator group are scheduled to the device.

12. The system according to claim 10 or 11, characterized in that The scheduling module is specifically used for: Adding operators in the plurality of operator groups that are in a ready state to a priority queue, wherein the operators in a ready state include operators without dependencies or operators whose dependent operators have all been executed; Taking out a target operator from the priority queue and scheduling the target operator to a device, wherein the priority of the target operator is higher than that of the remaining operators in the priority queue; When the target operator is executed, check whether the operator on which the subsequent operator of the target operator depends is executed. If so, add the subsequent operator to the priority queue to continue to take out new operators from the priority queue for scheduling.

13. The system according to claim 12, characterized in that The scheduling module is specifically used for: Get the group ID of the operator group to which the target operator belongs; When the group identifier of the operator group to which the target operator belongs is equal to the group identifier of the currently scheduled operator group, the target operator is scheduled to the device.

14. The system according to claim 12, characterized in that The scheduling module is also used for: Update the number of operators to be scheduled in the currently scheduled operator group; When the number of operators to be scheduled is equal to 0, the currently scheduled operator group is switched to the next operator group.

15. The system according to any one of claims 10 to 14, characterized in that The sorting module is specifically used for: According to the dependency relationship of the multiple operators in the computation graph, the multiple operators are sorted by a critical path strategy or an inverse depth-first search strategy to obtain an ordered operator sequence.

16. The system according to any one of claims 10 to 15, characterized in that The grouping module is specifically used for: According to the ordered operator sequence, the multiple operators are grouped by a uniform grouping strategy to obtain multiple operator groups.

17. The system according to any one of claims 10 to 15, characterized in that The grouping module is specifically used for: Obtaining at least one of computational complexity, memory consumption, or load characteristics of the multiple operators; According to the ordered operator sequence, in combination with at least one of computational complexity, memory consumption or load characteristics of the multiple operators, the multiple operators are grouped to obtain multiple operator groups.

18. The system according to any one of claims 10 to 17, characterized in that The scheduling module is also used for: Obtaining a scheduling result and a performance value corresponding to the scheduling result; According to the scheduling result and the performance value, a scheduling parameter is updated, where the scheduling parameter includes at least one of a grouping parameter or a sorting parameter.

19. A computing device cluster, characterized in that: The computing device cluster includes at least one computing device, and the at least one computing device includes at least one processor and at least one memory, wherein the at least one memory stores computer-readable instructions; the at least one processor executes the computer-readable instructions so that the computing device cluster performs the task scheduling method as described in any one of claims 1 to 9.

20. A chip, characterized in that: The chip includes a processor and a memory, and the processor executes the computer-readable instructions so that the chip performs the task scheduling method according to any one of claims 1 to 9.

21. A computer-readable storage medium, characterized in that: Comprising computer-readable instructions; the computer-readable instructions are used to implement the task scheduling method described in any one of claims 1 to 9.

22. A computer program product, characterized in that Comprising computer-readable instructions; the computer-readable instructions are used to implement the task scheduling method described in any one of claims 1 to 9.

Images