Task scheduling method, scheduling device, electronic device, storage medium and product

Abstract

The present application discloses a task scheduling method, a scheduling device, an electronic device, a storage medium and a product, and relates to the field of task scheduling technology. When receiving at least one task to be executed, the task scheduling method determines a first mapping relationship between multiple operators in each task and multiple accelerators, and establishes a global computation graph based on the first mapping relationship corresponding to each task. Then, according to the global computation graph, each operator is sent to the corresponding accelerator respectively, and the assigned operator task is completed based on the operator library of each accelerator itself. Then, at least one task is executed and completed in combination with the data dependency relationship between multiple operators, breaking the matching restriction between tasks and accelerator resources, and solving the technical problems of low accelerator resource utilization and poor task execution efficiency existing in related technologies, thereby achieving the technical effect of improving accelerator resource utilization and scheduling flexibility, and improving task execution efficiency.

Description

Technical Field

[0001] The present application relates to the field of task scheduling technology, and in particular to a task scheduling method, a scheduling device, an electronic device, a storage medium, and a product. Background Art

[0002] A heterogeneous cluster is a cluster system composed of devices with different architectures or operating systems. Specifically, a heterogeneous cluster contains multiple different types of nodes, which may run different operating systems (such as Linux and Windows) or use different hardware configurations. This diversity enables heterogeneous clusters to better adapt to different workloads and requirements, improving overall computing efficiency and flexibility.

[0003] Heterogeneous clusters include various types of accelerators. During task scheduling, the scheduler allocates accelerator resources to each node based on the resource type and quantity explicitly declared in the received node task. However, this task scheduling approach reduces accelerator resource utilization and is detrimental to task execution efficiency. Summary of the Invention

[0004] The present application provides a task scheduling method, a scheduling device, an electronic device, a storage medium and a product to at least solve the problem of low utilization of accelerator resources in the related art.

[0005] The present application provides a task scheduling method, which is applied to a heterogeneous cluster, wherein the heterogeneous cluster includes multiple accelerators. The method includes: receiving at least one task to be executed, wherein each task includes multiple operators and data dependencies between the multiple operators; determining a first mapping relationship between the multiple operators in each task and the multiple accelerators, and establishing a global computation graph based on the first mapping relationship corresponding to each task; sending each operator to the corresponding accelerator according to the global computation graph, and executing the at least one task through one or more of the multiple accelerators in combination with the data dependencies between the multiple operators.

[0006] The present application also provides a task scheduling device for use in a heterogeneous cluster, the device comprising: a receiving module for receiving at least one task to be executed, wherein each task includes multiple operators and data dependencies between the multiple operators; an establishing module for determining a first mapping relationship between the multiple operators in each task and the multiple accelerators, and establishing a global computation graph based on the first mapping relationship corresponding to each task; a scheduling module for sending each operator to the corresponding accelerator according to the global computation graph, and executing at least one task through one or more of the multiple accelerators in combination with the data dependencies between the multiple operators.

[0007] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any of the above-mentioned task scheduling methods when executing the computer program.

[0008] The present application also provides a non-volatile computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned task scheduling methods are implemented.

[0009] The present application also provides a computer program product, including a computer program, which implements the steps of any of the above-mentioned task scheduling methods when executed by a processor.

[0010] Through this application, by establishing a first mapping relationship between each operator in the task and the accelerator, a global computation graph is determined, a scheduling strategy is uniformly generated, and the operator is sent to the corresponding accelerator during the task execution process. The assigned operator is completed based on each accelerator's own operator library, thereby executing the task. This breaks the type matching restriction between tasks and accelerator resources, and can solve the technical problems of low accelerator resource utilization and poor task execution efficiency in related technologies, thereby achieving the technical effect of improving accelerator resource utilization and scheduling flexibility, and improving task execution efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0012] Figure 1 A flowchart of a task scheduling method provided in an embodiment of the present application;

[0013] Figure 2 This is a schematic diagram of the architecture of a task scheduling method according to a specific embodiment of the present application;

[0014] Figure 3 This is a schematic diagram of operator-to-accelerator mapping for a specific implementation of this application;

[0015] Figure 4 This is a schematic diagram of a task scheduling method according to a specific embodiment of the present application;

[0016] Figure 5 A schematic diagram of a scheduling strategy generation for a specific embodiment of the present application;

[0017] Figure 6 This is a schematic diagram of fault tolerance processing according to a specific embodiment of the present application;

[0018] Figure 7 This is a flowchart of a task scheduling method according to a specific embodiment of the present application;

[0019] Figure 8 This is a schematic diagram of the execution of a global computation graph according to a specific embodiment of the present application;

[0020] Figure 9 A connection diagram of a task scheduling device provided in an embodiment of the present application;

[0021] Figure 10 A block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0022] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0023] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.

[0024] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0025] Multi-tenant clusters are often built in scenarios such as large companies for shared use by multiple users. This aims to centralize computing resources and improve resource utilization. In a multi-tenant cluster, each computing node may use a different operating system and hardware configuration to meet different workloads and requirements, resulting in a heterogeneous system.

[0026] Furthermore, advancements in machine learning (ML) technologies, such as mixture of experts (MoE) and multimodal models, have led to significant asymmetry in the storage and computing resource requirements of ML applications, making multi-tenant cluster systems even more heterogeneous. This asymmetry manifests itself in multiple ways. For example, due to the computational sparsity of the MoE model structure, each inference may involve different control flows, causing different accelerators to be busy. In multimodal models, some inferences involve text, while others involve video, resulting in significant variations in the granularity of inference. Furthermore, due to the inherent nature of the business, load fluctuates significantly over time.

[0027] In addition, with the emergence of new accelerators such as NPU (Neural Network Processing Unit) and TPU (Tensor Processing Unit), the introduction of interconnection technologies such as RDMA (Remote Direct Memory Access) and NVLink (NVIDIA NVLink, a bus and communication protocol developed and launched by NVIDIA), and operational factors such as hardware upgrades, multi-tenant clusters now include multiple types of accelerators, making cluster systems more heterogeneous.

[0028] In the related art, the job scheduler allocates accelerator resources based on the received task declaration, and executes the task through the accelerator resources until the task is completed, and the accelerator resources are released for the execution of the next task. However, due to the asymmetry of ML applications, allocating a fixed number of accelerator resources to ML applications will result in low accelerator utilization in the entire cluster. For example, a single task may not be able to fully utilize the accelerator resources allocated to it, so multiple tasks need to share the accelerator to improve resource utilization; for example, when the business load intensity decreases, the computing resources allocated to the current task cannot be reallocated to other tasks, resulting in idle resources and waste. Therefore, the task scheduling method used in the related art cannot meet the asymmetry and ever-changing computing needs of ML applications.

[0029] During the job scheduler's operation, users submit containerized applications to a container orchestration tool (such as Kubernetes, a portable, extensible, open-source platform for managing containerized workloads and services, supporting declarative configuration and automation) for deployment. Consequently, job schedulers typically have three major limitations: First, they typically use a gang scheduling strategy, meaning that resources declared by a task are allocated at once only when all of those resources are free. These resources are then occupied by the task until it completes, at which point they are released for use by other tasks. Second, the job scheduler assigns accelerators to tasks, meaning that how the accelerator cluster resources are utilized depends on the task's implementation. Third, the scheduler is unaware of the internal details of the task.

[0030] Specifically, the job scheduler allocates accelerator resources based on the resource types and quantities explicitly declared by the received job. This means that the accelerator resources allocated to a job cannot be easily changed during the task scheduling process. Most current parallelization strategies for ML applications are based on 3D parallelism, which only supports homogeneous resources. This means that there are no highly heterogeneous scheduling strategies within the scheduling space of a single job. Therefore, allocating heterogeneous accelerator resources to tasks is not feasible in current practice.

[0031] Furthermore, the job scheduler lacks internal understanding of tasks, making it difficult to schedule tasks to heterogeneous accelerator resources. For example, when multiple tasks share a GPU, the scheduler, lacking detailed understanding of the tasks, cannot manage computing and network interference between tasks. This can affect high-priority tasks and even lead to OOM (Out of Memory) errors.

[0032] Based on the above analysis, it can be seen that the task scheduling method in the related art has the following problems:

[0033] (a) It is impossible to make highly heterogeneous accelerators perform tasks collaboratively.

[0034] (b) It is impossible to achieve rapid elasticity, that is, to quickly adjust the accelerator resources allocated to a single task.

[0035] (c) It is impossible to estimate the feasibility and superiority of scheduling strategies on heterogeneous resources.

[0036] (d) Unable to estimate network,computational interference between tasks.

[0037] To address at least one of the aforementioned technical issues, this application proposes a task scheduling method for a heterogeneous cluster comprising multiple accelerators. Upon receiving at least one task to be executed, a first mapping relationship between operators and accelerators in each task is established, thereby constructing a global computation graph. The global computation graph graphically displays the mapping relationship and generates a unified scheduling strategy. During task processing, each operator is sent to the corresponding accelerator based on the global computation graph, so that the corresponding operator calculation is performed by the corresponding accelerator. Data dependencies between multiple operators are combined to complete data transmission between accelerators and complete task execution. Thus, this method uses a global computational graph (GCG) to comprehensively depict the global state of resource usage of all tasks and accelerators in the heterogeneous cluster. Operators in all user-submitted tasks are integrated into a unified, large-scale directed computation graph, enabling global scheduling and resource management across tasks. This improves the resource utilization efficiency of accelerators in multi-tenant clusters, reduces scheduling latency, and enhances the operational flexibility and system performance of tasks such as large-scale graph neural networks (GNNs) and large language models (LLMs).

[0038] The task scheduling method of the embodiment of the present application is described in detail below with reference to the accompanying drawings. The task scheduling method is executed by a scheduler in a heterogeneous cluster.

[0039] like Figure 1 As shown, the task scheduling method of the embodiment of the present application, applied to a heterogeneous cluster, may include:

[0040] S1, receiving at least one task to be executed, wherein each task includes multiple operators and data dependencies between the multiple operators;

[0041] Specifically, combined Figure 2 As shown, the cluster gatekeeper obtains the online / offline status of each accelerator in the heterogeneous cluster and sends it to the scheduler. The scheduler identifies multiple accelerators in the online state based on the status of each accelerator in the system, and performs subsequent task scheduling applications based on these multiple accelerators. At the same time, the scheduler receives at least one task to be executed (such as Figure 2 Each task is represented by a data stream and includes multiple operators and the data dependencies between them. Operators in each task can be divided according to the operator symbol, and the data dependencies between operators can be determined based on the computational logic of the task.

[0042] S2, determining a first mapping relationship between multiple operators in each task and multiple accelerators, and establishing a global computation graph based on the first mapping relationship corresponding to each task;

[0043] Specifically, each accelerator is equipped with its own operator library, which can meet the computing needs of any operator. In the process of establishing the first mapping relationship, the operators can be evenly distributed according to the number of operators and the number of accelerators of each task, so as to establish the first mapping relationship corresponding to each task; or the preset matching relationship between the operator and the accelerator can be determined in advance, and then after receiving the task, the mapping relationship between each operator and the accelerator in the task is determined by looking up the table to obtain the corresponding first mapping relationship; or the allocation strategy can be set in advance, and the operator and the accelerator are matched based on the preset allocation strategy to establish the corresponding first mapping relationship, for example, according to the computing logic of the operator in each computing subtask, the accelerators are matched from front to back, so as to establish the mapping relationship between each operator and the accelerator. In Figure 3 In the figure, the dotted arrows represent the mapping relationship between each operator and the corresponding accelerator.

[0044] After determining the first mapping relationship between multiple operators and multiple accelerators in each task, a global computation graph is established based on the first mapping relationship corresponding to each task. The mapping of operators to accelerators of all tasks is displayed through the global computation graph to describe the mapping of all tasks. Thus, the computing subtasks in all tasks are merged into a large computation graph through the global computation graph, such as Figure 3 As shown, the scheduling strategy is generated uniformly.

[0045] Furthermore, each task can be divided into multiple computing subtasks, and the first mapping relationship is constructed based on the computing subtasks. By dividing the task modules, the efficiency of constructing the global computing graph is improved. Figure 3 As shown, Task 1 includes two computing subtasks, and a data dependency exists between the two computing subtasks. That is, the computation result of one computing subtask is used for the computation of the other computing subtask. Therefore, upon receiving Task 1, the scheduler can identify the data dependency between the two computing subtasks in Task 1 and between the two computing subtasks, as well as the data dependency between the multiple operators in each computing subtask and between the multiple operators. It can then determine the first mapping relationships between the multiple operators in the two computing subtasks and the multiple accelerators, respectively, thereby speeding up the construction of the global computation graph. Task 2 includes only one computing subtask. Upon receiving Task 2, the scheduler can identify the one computing subtask in Task 2 and the data dependency between the multiple operators in the computing subtask and between the multiple operators. It can then establish the first mapping relationships between the multiple operators in Task 2 and the multiple accelerators.

[0046] S3: Send each operator to the corresponding accelerator according to the global computation graph, and execute at least one task through one or more of the multiple accelerators based on the data dependencies between the multiple operators.

[0047] That is to say, based on the first mapping relationship corresponding to each task in the global computation graph, each operator is sent to the corresponding accelerator respectively, and thus added to the execution queue of the corresponding accelerator, so that the accelerator can execute the received operator calculation tasks in sequence according to the queue order or preset priority through its own operator library, and obtain the operands of each operator in combination with the data dependency relationship, and execute the corresponding operator calculation when the operands of the operator are all calculated.

[0048] Combine Figure 4 As shown, a task includes seven operators, The first mapping relationship between the computing subtask and the multiple accelerators is shown in the figure, that is, the operator Assigned to accelerator a, operator 、 and Assigned to accelerator b, operator and Assigned to accelerator c, operator Assigned to accelerator d. During task execution, operators are assigned to accelerator d based on the global computation graph. Sent to the queue of accelerator a, operator 、 and Sent to the queue of accelerator b, operator and Queue sent to accelerator c, operator The queues sent to accelerator d perform operator calculations based on the corresponding queues in turn. Based on the data dependency, we can see that the operator The calculation result of the operator Therefore, accelerator a completes the operator based on its own operator library. After the calculation of The calculation results are sent to the accelerator c for operator Calculation, another operator The operands of the operator In addition to the calculation results of The calculation result of Upon receiving the operator The calculation results and operators The calculation will start after the calculation result of . And so on until all operators are calculated.

[0049] This embodiment establishes a mapping between operators and accelerators, meaning that accelerators in a heterogeneous cluster expose a unified operator execution interface to the global computation graph, enabling collaborative task execution among heterogeneous accelerators. This embodiment eliminates the need for kernel functions to be compiled into binary code that can be run by all accelerators. Instead, each accelerator's own operator library completes the assigned operator computation. That is, when an operator is dispatched to a corresponding accelerator, that accelerator executes its own operator library to complete the operator computation.

[0050] Therefore, this embodiment supports heterogeneous accelerators (such as GPUs, FPGAs (Field Programmable Gate Arrays), and TPUs) by establishing a unified mapping mechanism from operators to accelerators, exposing a unified operator execution interface. This method does not force all operators to adapt to universal kernel code. This method allows heterogeneous accelerators to use their own native operator libraries to execute tasks, eliminating the performance overhead of cross-platform kernel conversion. This enables multiple heterogeneous devices to collaboratively participate in the parallel computing of tasks, effectively improving overall resource utilization and scheduling flexibility. At the same time, when receiving multiple tasks, the type and number of accelerators assigned to each task can be quickly adjusted through operator matching, thereby improving task scheduling flexibility.

[0051] In some embodiments of the present application, receiving at least one task to be executed includes: receiving a directed acyclic graph of at least one task, wherein the nodes of the directed acyclic graph are used to represent corresponding operators, and the edges of the directed acyclic graph are used to represent data dependencies between operators.

[0052] Specifically, a directed acyclic graph (DAG) is a data structure defined in graph theory, consisting of vertices and edges. Each edge has a clear direction, and the entire graph is acyclic, that is, there is no path in the graph that can start from a point and return to the same point after passing through a series of edges. Figure 2 and Figure 3 The reception of the directed acyclic graph of tasks improves the efficiency of constructing the first mapping relationship and the accuracy of task scheduling.

[0053] In addition to the mapping relationship between operators and accelerators, the global computation graph can also include a directed acyclic graph of each received task, such as Figure 3 As shown, the directed acyclic graph of each task and the mapping of each operator to the accelerator are displayed through the global computation graph to describe the mapping of all tasks. The computing subtasks in all tasks are merged into a large computation graph through the global computation graph, and a unified scheduling strategy is generated for subsequent task scheduling applications.

[0054] In some embodiments of the present application, determining a first mapping relationship between multiple operators in each task and multiple accelerators includes: determining multiple task scheduling strategies corresponding to the task based on a preset scheduling generation method; determining the scheduling performance of each task scheduling strategy based on a preset computing time of each operator in the task in the corresponding accelerator and a preset communication time between multiple accelerators; determining a target task scheduling strategy based on the scheduling performance of each task scheduling strategy, and generating a first mapping relationship between multiple operators in the task and multiple accelerators based on the target task scheduling strategy.

[0055] Specifically, after sending at least one task to the scheduler, several task scheduling strategies using different types and quantities of accelerator resources are first generated based on a preset scheduling generation method. Each task scheduling strategy includes the accelerator allocation corresponding to each operator and the data transmission scheduling between accelerators. Figure 4 For example, this task includes seven operators, To express, Figure 4 The corresponding task scheduling strategy is: computing operators through accelerator a , and the operator The calculation result is sent to accelerator d; the operator is calculated by accelerator b , operator Sum Operator , first of all, the operator Perform calculations and convert the operator The calculation results are sent to the accelerator d, and according to the operator The result calculation operator , and the operator The calculation results are sent to the accelerator c, and the operator The result is saved; through the accelerator d according to the operator Sum Operator The calculation result calculation operator , and the operator The calculation result is sent to accelerator b; accelerator b calculates the value of and The result calculation operator ;Accelerator c according to operator The calculation results of the operator are calculated separately and , thus completing the calculation of the task.

[0056] Then, the total execution time of each task scheduling strategy is determined based on the preset computing time of the operator in the corresponding accelerator and the preset communication time between multiple accelerators in each task scheduling strategy, so as to determine the scheduling performance of each task scheduling strategy, which is used to evaluate the feasibility and execution stability of each task scheduling strategy.

[0057] The preset computing time of the operator in the corresponding accelerator can be determined based on historical computing data or can be a preset time, and there is no specific limitation. Figure 4 Operators in Assigned to accelerator a, for example, you can search for operators through historical execution data The historical computing time in accelerator a is calculated, and the average of the historical computing time is used as the operator The preset calculation duration in accelerator a; if there is no accelerator a calculation operator in the historical data In this case, the operator can be matched with the same type of accelerator according to the type of accelerator a. The historical calculation time of The preset calculation time in accelerator a; a mapping table between operators, accelerator types, and calculation times can also be established in advance, and the corresponding preset execution time can be determined by looking up the table during actual application.

[0058] The preset communication duration between multiple accelerators includes the preset communication duration between each group of accelerators, which can be determined by historical data, or can be a preset duration, or can be estimated based on the bandwidth between each group of accelerators, without specific restrictions. Figure 4 The accelerator a in the operator For example, the calculation result of the operator is sent to accelerator d. The preset communication time between the corresponding accelerators includes accelerator a sending the operator The calculation result is sent to the accelerator d. The historical data transmission time between the operator and the accelerator a can be queried through the historical data. The calculation result is sent to the historical communication time of accelerator d, and the corresponding preset communication time is obtained by taking the average value; it can also be based on the bandwidth between accelerator a and accelerator d, operator The estimated data size can be used to predict the corresponding preset communication duration; a communication duration mapping table between each group of accelerators can also be established in advance, and the corresponding preset communication duration can be determined by looking up the table.

[0059] Taking the total execution time as an example of the scheduling performance of a task scheduling strategy, the preset computation time and preset communication time are selected for each task scheduling strategy. The total execution time is then calculated in combination with the scheduling strategy to determine the scheduling performance of each task scheduling strategy. The task scheduling strategy with the shortest total execution time is selected as the target task scheduling strategy for that task. Based on the target task scheduling strategy, the first mapping relationship corresponding to the task is determined and used to construct the global computation graph.

[0060] This embodiment improves task scheduling performance by performing performance evaluation on multiple task scheduling strategies, thereby ensuring the construction quality of the global computation graph and the task scheduling effect.

[0061] Furthermore, in addition to the above-mentioned process of generating multiple task scheduling strategies corresponding to each task and then evaluating each task scheduling strategy to determine the target scheduling strategy, it is also possible to perform synchronous screening during the scheduling strategy generation process to obtain the final target scheduling strategy. Figure 5 As shown in the figure, after receiving a task, the first node (operator) of the DAG is mapped to an accelerator based on the topological order of the task, serving as the root node of the entire scheduling decision tree. Furthermore, the second node of the DAG is mapped to different accelerators, forming the first-level nodes of the scheduling decision tree. During node generation, a predictor is invoked to estimate the quality of the scheduling decision. If, during decision tree generation, it is found that an accelerator cannot execute the operator or performs poorly, all subsequent scheduling decisions for that node are discarded, resulting in pruning. This process continues in this manner, resulting in the final scheduling decision tree, which serves as the target scheduling policy.

[0062] In some embodiments of the present application, when multiple accelerators are in a task execution state, it also includes: determining the current global computation graph that the multiple accelerators are executing; determining the idle start time of each accelerator based on the current execution state of the current global computation graph, the preset computation time of each operator in the current global computation graph in the corresponding accelerator, and the preset communication time between the multiple accelerators; predicting the execution end time of each task scheduling strategy based on the preset computation time of each operator in the task in the corresponding accelerator, the preset communication time between the multiple accelerators, and the idle start time of each accelerator, so as to determine the scheduling performance of each task scheduling strategy.

[0063] Specifically, when multiple accelerators in a heterogeneous cluster are idle, if they receive a task to be executed, multiple accelerators can perform computing operations as soon as they receive the operator. Therefore, the scheduling performance can be evaluated only based on the total execution time of each task scheduling strategy, and the task scheduling strategy with the shortest total execution time can be used as the target task scheduling strategy to construct a global computation graph.

[0064] However, when multiple accelerators in a heterogeneous cluster are in a task execution state, in order to further improve the accuracy of performance evaluation, the idle start time of each accelerator is predicted.

[0065] Combine Figure 2As shown, the execution time of each operator in the current global computation graph on the corresponding accelerator can be predicted based on the prediction function (such as the Kernel function). Specifically, each operator is executed one by one on the corresponding accelerator in a preset order. At the same time, based on the preset cluster transmission management algorithm, the execution order of all operators on the accelerator can be inferred, and the time for each operator to be executed can be predicted. Moreover, the bandwidth between the two accelerators is also predictable, thereby predicting the communication transmission time between the two accelerators. The real-time mapping of operators to accelerators can be reflected in the current global computation graph to infer the ready operators in the out-of-order queue on each accelerator, and which ready operators are being executed can be inferred based on the priority of operator execution. Ready operators are operators that can enter the execution state. For example, in Figure 4 In the operator Sum Operator There are no operands, so in the operator Sum Operator When an operator is added to the queue of the corresponding accelerator, it is a ready operator. Other operators all have operands. When all operands of the corresponding operator are calculated, the corresponding operator switches to the ready state and becomes a ready operator. Because each operator on the accelerator is executed one by one, and because of the predictability of operators and transmission, the execution completion time of the currently running operator on the accelerator can be accurately predicted, thereby determining which operator will complete execution next, and then knowing which operator this completed operator will make ready. By predicting the execution of operators one by one in this way, the execution completion time of all operators can be iteratively predicted, thereby predicting the idle start time of each accelerator.

[0066] In the process of obtaining the scheduling performance of each task scheduling strategy, the calculation end time of each task scheduling strategy is predicted based on the preset calculation time of each operator in the corresponding accelerator, the preset communication time between multiple accelerators, and the idle start time of each accelerator. In this way, the scheduling performance of each task scheduling strategy is evaluated, and then the task scheduling strategy with the earliest calculation end time is selected as the target scheduling strategy to construct the global computation graph.

[0067] The preset computation times of these operators in their respective accelerators and the predicted communication times between multiple accelerators can be predicted using historical data. Because this information is missing during the first execution of a task, cluster predictions are particularly inaccurate. However, as tasks are iteratively executed, prediction accuracy gradually improves, thereby enhancing scheduling quality. Furthermore, when historical data is unavailable, a pre-set table can be used for lookup and determination.

[0068] In this embodiment, when multiple accelerators receive new tasks while executing tasks, each task scheduling strategy is evaluated based on the predicted idle start time of each accelerator, thereby eliminating task scheduling decisions with particularly late operator completion times. As a result, the target task scheduling decision schedules task calculations to the idle computing area in the cluster, that is, to accelerators with earlier idle start times. This not only improves the computing power utilization of the cluster, but also prevents the execution time of task calculations from being too late, further improving the quality of task scheduling.

[0069] The generation process of this target scheduling strategy decouples specific resource types, enabling scheduling decisions that enable collaborative computing across heterogeneous resources. The scheduling decision tree generated by the pre-scheduling algorithm extends in any direction within the accelerator cluster due to the scale of the directed acyclic graph, thus achieving flexibility. Furthermore, the scheduler's prediction unit considers both the execution time of operators within the corresponding accelerators and the transmission time of tensors—the time it takes to transfer computation results between accelerators—during the prediction process. This allows for assessment of the feasibility and quality of scheduling decisions, ultimately improving the quality of task scheduling.

[0070] In some embodiments of the present application, the idle start time of each accelerator is determined based on the current execution state of the current global computation graph, the preset computation time of each operator in the current global computation graph in the corresponding accelerator, and the preset communication time between multiple accelerators, including: determining the execution completion time of each operator in the current global computation graph based on the current execution state of the current global computation graph, the preset communication time of each operator in the current global computation graph in the corresponding accelerator, and the preset communication time between multiple accelerators; predicting the idle start time of each accelerator based on the execution completion time of each operator in the current global computation graph.

[0071] Specifically, suppose that when a new task is received, multiple accelerators are executing Figure 4 The global computation graph shown in Figure 2, where the operator and The execution has been completed. and Executing, operator 、 and Waiting for execution. Determine the operator based on historical data or preset mapping table The preset calculation time T1 and operator in accelerator a The preset calculation time T5 and operator in accelerator c The preset calculation time T2 and operator in accelerator d The preset calculation time T6 and operator in accelerator c The preset calculation time T7 in accelerator b, accelerator b transmits the operator to accelerator d The preset transmission time T of the calculation result bd , accelerator a transmits operators to accelerator d The preset transmission time T of the calculation result ad , accelerator d transmits operators to accelerator b The preset transmission time T of the calculation result bd .

[0072] Because the operator and The execution has been completed, and the operator The calculation results have been transmitted to the accelerator c, so the operator can be determined based on the actual feedback The calculation ends at t3, operator The calculation ends at time t4, and the accelerator c receives the operator The calculation result is at time t50, and the operator is determined at the same time The calculation starts at time t10. Then, the operator The calculation end time t1=t10+T1, thereby determining the idle start time of accelerator a as t1; operator The calculation end time t2=t3+T bd +T2, thus determining the idle start time of accelerator d as t2; operator The calculation ends at t5=t50+T5, the operator The calculation end time t6 = t5 + T6, thereby determining the idle start time of accelerator c as t6; operator The calculation end time t7=t2+T bd +T7, thereby determining that the idle start time of accelerator b is t7.

[0073] This embodiment determines the calculation end time of each operator based on the preset calculation time of the operator in the corresponding accelerator and the preset communication time between multiple accelerators, thereby predicting the idle start time of each accelerator. The preset calculation time and the preset communication time can be predicted through historical data, and as the task is iteratively executed, the prediction accuracy of the preset calculation time and the preset communication time is improved, so as to improve the prediction accuracy of the idle start time of each accelerator, thereby improving the scheduling quality.

[0074] In some embodiments of the present application, the global computation graph includes status information of each operator in at least one task. In the process of sending each operator to the corresponding accelerator according to the global computation graph, and executing at least one task through one or more of the multiple accelerators in combination with the data dependencies between the multiple operators, it also includes: receiving operator status feedback information sent by one or more of the multiple accelerators; and updating the status information of each operator according to the operator status feedback information.

[0075] Specifically, the global computation graph is a data structure that can be defined as a five-tuple, namely the global computation graph .in, yes DAG(Directed Acyclic Graph , directed acyclic graph), that is, the data flow corresponding to the received task, is the node set of the DAG graph, Used to represent a node, i.e. an operator; is the set of edges between nodes, Used to represent nodes and nodes The data dependencies between them. , represents the set of accelerators in a heterogeneous cluster, and is used to characterize all online accelerators in a heterogeneous cluster. It is a mapping from operators (nodes) to accelerators, indicating that an operator is executed on a certain accelerator. Is the current execution status of the operator (node), indicating whether the operator has been executed. For example, When , it means the operator has not been executed; , it indicates that the operator has been executed.

[0076] After sending the operators to the corresponding accelerators according to the global computation graph, the state of each operator is updated to , calculate the operator through the corresponding accelerator, and feedback the calculation results in real time, and when the corresponding operator status feedback information fed back by the accelerator is completed, the status of the corresponding operator is updated to In addition, the states can be distinguished by the grayscale of the operators in the global computation graph, such as Figure 4 shown.

[0077] This embodiment updates the status of each operator in real time according to the operator status feedback information and displays it, so as to facilitate tracking the task execution status in the system.

[0078] In some embodiments of the present application, the status information of each operator is updated according to the operator status feedback information, including: when it is determined that the operator is in the execution completion state according to the operator status feedback information, the operator and the associated information of the operator in the global computation graph are deleted.

[0079] That is to say, when it is determined that the execution of an operator is completed, the operator is deleted from the global computation graph, and the associated information of the operator in the global computation graph, such as the edges in the DAG graph and the mapping relationship between the operator and the accelerator, is deleted at the same time to reduce memory usage.

[0080] In some embodiments of the present application, in the process of sending each operator to the corresponding accelerator according to the global computation graph, and executing at least one task through one or more of the multiple accelerators in combination with the data dependency relationship between multiple operators, the task scheduling method also includes: in response to the operator insertion instruction, identifying the target operator corresponding to the operator insertion instruction and the data dependency relationship between the target operator and the operators in the global computation graph; establishing a second mapping relationship between the target operator and the multiple accelerators based on the data dependency relationship between the target operator and the operators in the global computation graph, and updating the global computation graph based on the second mapping relationship.

[0081] Specifically, the operator insertion instruction may be triggered when a new task is received or other computing subtasks in a task are completed.

[0082] Assume that when it is determined that a target operator needs to be added to the global computation graph based on the operator insertion instruction, first insert the instruction based on the operator, and the node set in the global computation graph Add a node corresponding to the target operator ,Right now At the same time, determine the data dependency between the target operator and the operators in the global computation graph, and determine its predecessor node as , and to Add an edge about the node in A second mapping relationship between the target operator and the multiple accelerators is established based on the data dependency relationship between the target operator and the operators in the global computation graph, thereby updating the global computation graph so as to complete the calculation of the newly added target operator based on the updated global computation graph.

[0083] This embodiment responds to the operator addition instruction to update the global computation graph, thereby performing computation on the newly added operator based on the updated global computation graph to meet computational requirements.

[0084] In some embodiments of the present application, the global computation graph includes an accelerator set corresponding to multiple accelerators. In the process of sending each operator to the corresponding accelerator according to the global computation graph, and executing at least one task through one or more of the multiple accelerators in combination with the data dependencies between the multiple operators, it also includes: obtaining status information of each accelerator in the heterogeneous cluster; and updating the accelerator set according to the status information of each accelerator.

[0085] Specifically, combined Figure 2 As shown, the cluster watchdog monitors the accelerator status in the cluster system in real time, so as to (Signature_Accelerator_Online) function, (Signature_Accelerator_Offline) informs the scheduler of the accelerator status information in the entire system so that the accelerator set in the global computation graph can be updated in real time based on the accelerator status information, so that subsequent mapping relationships can be accurately established based on the updated accelerator set.

[0086] In some embodiments of the present application, updating the accelerator set based on the status information of each accelerator includes: when it is determined based on the status information of each accelerator that a new online accelerator exists in the heterogeneous cluster, adding the new online accelerator to the accelerator set; and / or when it is determined based on the status information of each accelerator that an offline accelerator exists in the accelerator set, re-establishing a third mapping relationship between unexecuted operators corresponding to the offline accelerator and the online accelerators in the accelerator set, and then deleting the offline accelerator from the accelerator set.

[0087] That is, when the cluster gatekeeper informs the scheduler that a new accelerator is online through the (signature_accelerator_online) function, the newly online accelerator is directly added to the accelerator set for the establishment of a mapping relationship for subsequent task scheduling. , making .

[0088] In cluster, the door dog is opened by The (Signature_Accelerator_Offline) function informs the scheduler that when an accelerator in the global computation graph goes offline, the operator in the offline accelerator is first canceled, and the mapping relationship between the unfinished operators corresponding to the offline accelerator and other online accelerators in the accelerator set is re-established. The offline accelerator is then deleted from the accelerator set, and the unfinished operators corresponding to the offline accelerator are sent to the corresponding online accelerator based on the reconstructed mapping relationship to continue the operator calculation. That is, after receiving When you first connect to the offline accelerator Related operators Cancel, that is (The practical meaning is to reschedule and Related operators, build mapping relationships with other online accelerators); then make .

[0089] This embodiment directly adds newly launched accelerators to the accelerator set of the global computation graph to establish mapping relationships for subsequent tasks. When offline accelerators appear, a fault-tolerant task computing strategy is adopted. The operators corresponding to the offline accelerators are reconstructed and then deleted. Task scheduling accuracy is ensured only by adjusting local mapping and scheduling relationships.

[0090] In some embodiments of the present application, when it is determined based on the current state of each accelerator that there is an offline accelerator in the accelerator set, the method further includes: obtaining intermediate operator calculation results stored in the offline accelerator, and sending the intermediate operator calculation results to the corresponding online accelerator based on the data dependency relationship between multiple operators.

[0091] Specifically, each accelerator is equipped with an out-of-order execution queue. When the scheduler launches an operator to an accelerator, the operator is added to the out-of-order execution queue of the corresponding accelerator. Specifically, an enqueue_op interface can be defined on each accelerator, and the scheduler can launch operators to the accelerator by calling the enqueue_op interface.

[0092] During the accelerator operation, each accelerator monitors its own queue at all times and executes ready operators one by one. When an operator is first added to the queue, all its operands are All the operation data of an operator are calculated, that is, the number of operations is , then the operator is Status. When a When the operator is executed, its calculation result will be broadcast to the operand of the corresponding operator in the entire queue that needs the value, and then the operand becomes At the same time, the calculation results of the corresponding operators are stored in the accelerator's HBM (High Bandwidth Memory).

[0093] When it is determined that the accelerator is offline, the scheduler reads the intermediate operator calculation results stored in the HBM of the offline accelerator from the offline accelerator and forwards them to the corresponding accelerator for subsequent operator calculations. Figure 4 For example, if accelerator b goes offline, the scheduler can store the operators in accelerator b. The calculation results are read and sent to the accelerator d for operator Calculation, while the operator The calculation results are forwarded to the accelerator c for operator and Calculation.

[0094] This embodiment defines a fault tolerance mechanism, such as Figure 6 As shown, the intermediate results stored in the accelerator's HBM are used to restore the calculation progress, without the need to load the intermediate calculation results from remote storage, thereby speeding up the fault-tolerant recovery speed. At the same time, the fault-tolerant recovery mechanism only withdraws the affected operator calculations and accelerators, which will only affect the local accelerator and will not affect the entire cluster, thereby improving the stability of the cluster's computing power. The accelerator will not be stagnant for too long due to fault-tolerant recovery, thereby improving the computing power utilization.

[0095] In some embodiments of the present application, in the process of sending each operator to the corresponding accelerator according to the global computation graph, and executing at least one task through one or more of the multiple accelerators in combination with the data dependency relationship between the multiple operators, it also includes: in the event of an operator calculation error, the operator with the calculation error is removed from the original accelerator, and a fourth mapping relationship is established between the operator with the calculation error and other accelerators in the multiple accelerators, and the global computation graph is updated based on the fourth mapping relationship.

[0096] Specifically, combined Figure 6 As shown, during the operator calculation process, the calculation results of the operator are always retained in the HBM of the corresponding accelerator. When it is determined that an operator calculation error occurs, the scheduler withdraws the operator with the calculation error from the original accelerator and re-establishes the fourth mapping relationship between the operator with the calculation error and other accelerators in the multiple accelerators to update the global calculation graph, thereby reallocating the operator with the calculation error for recalculation of the operator with the calculation error.

[0097] When a calculation error occurs, this embodiment triggers fault tolerance processing and withdraws the affected operators, which will only affect the local cluster and not the entire cluster, thereby improving the stability of the cluster's computing power and improving computing power utilization.

[0098] In some embodiments of the present application, after updating the global computation graph based on the fourth mapping relationship, it also includes: obtaining the operands of the operator with calculation errors, wherein the operands are stored in the corresponding accelerator; and sending the operator with calculation errors and the operands to the corresponding accelerator based on the updated global computation graph.

[0099] That is to say, after the operator with the calculation error is moved out of the original accelerator, the fourth mapping relationship between the operator and other accelerators and the calculation scheduling path related to the operator are re-established, so as to update the global calculation graph and perform task calculation according to the global calculation graph.

[0100] During the operator calculation process, the accelerator will always retain the operator's calculation results in the HBM of the corresponding accelerator. When it is determined that an operator calculation error occurs, the cluster watchdog notifies the scheduler through the sign_acc_out function. The scheduler withdraws the operator with calculation errors from the original accelerator, and reads the operator's operands from the original accelerator or the accelerator used to calculate the operator with calculation errors, and sends the operator with calculation errors and operands to the corresponding accelerator for recalculation of the operator.

[0101] This embodiment only withdraws the affected operators and establishes a mapping relationship between the operators and other accelerators, which will only have a local impact on the cluster. In complex clusters with low mean failure time, this fault tolerance mechanism can improve the stability of the cluster's computing power. At the same time, calculation recovery is performed based on the operands stored in the corresponding accelerator, so that the accelerator will not be stagnant for too long due to fault tolerance recovery, thereby improving computing power utilization.

[0102] In some embodiments of the present application, the task scheduling method further includes: obtaining the actual calculation time of the operator, and determining that a calculation error has occurred in the operator when the actual calculation time of the operator exceeds a preset time; or determining that a calculation error has occurred in the operator when an abnormality occurs in the accelerator where the operator is located; or determining that a calculation error has occurred in the operator corresponding to the operator abnormal instruction in response to an operator abnormal instruction.

[0103] That is to say, when the actual calculation time of the operator is determined based on the operator status information fed back by the accelerator, if the actual calculation time of the operator exceeds the preset time, the accelerator is considered to be unsuitable for the calculation of the operator and a calculation error occurs. Or when it is determined based on the accelerator's status information that the accelerator is offline or has fault feedback, it is considered that the accelerator cannot complete the calculation of the internal operator, and it is determined that all operators corresponding to the accelerator have calculation errors, and all of them are taken out for remapping calculations. Or when an operator exception instruction is received that is triggered externally, it is considered that the operator corresponding to the operator exception instruction has a calculation error, and the calculation is reallocated, thereby ensuring the recognition effect of calculation errors and improving the quality of task calculations.

[0104] In some embodiments of the present application, the task scheduling method further includes: identifying communication willingness between at least one group of accelerators; and generating corresponding communication permission based on the communication willingness between at least one group of accelerators, so that at least one group of accelerators communicate based on the communication permission.

[0105] When all tasks execute complex scheduling strategies in a cluster, allowing communication operators to trigger freely may lead to an uncontrollable cluster structure. Therefore, it is necessary to handle tensor transmission between accelerators in the management of operator granularity.

[0106] To this end, in this embodiment, after a tensor (the result of an operator's calculation) is calculated, the accelerator first notifies the scheduler that the tensor is ready to be transmitted. The scheduler recognizes the accelerators' willingness to communicate, then generates a corresponding communication permission based on the communication willingness and sends it to the corresponding accelerator. The accelerator will not begin transmitting the tensor until it receives the communication permission, which means that the scheduler agrees to transmit the tensor. As a specific embodiment, three special operators can be defined for communication between accelerators: the want operator, the recv operator, and the send operator. The want operator is used to notify the master that the tensor is ready for transmission; the recv and send operators actually call communication library functions; these two operators must be executed in pairs. Because the transmission requires the scheduler's approval, the execution of recv and send requires the scheduler to issue a permission instruction, and the execution of these communication operators is subject to conditional restrictions.

[0107] That is to say, the execution of some operators must meet certain conditions. For example, the execution condition of the pair of send and receive communication operators given above is to receive the communication permission instruction from the scheduler, at which time the communication operator will enter In addition, the execution conditions of some operators can be triggered by operands. For example, when the operands of an operator are all calculated, the operator enters the ready state and starts execution.

[0108] In some embodiments of the present application, when the communication willingness between at least one group of accelerators is identified, it also includes: constructing a communication willingness graph based on the communication willingness between at least one group of accelerators; performing a communication scheduling control strategy on at least one group of accelerators based on the communication willingness graph, so as to generate communication permissions corresponding to each group of accelerators based on the communication scheduling control strategy.

[0109] Specifically, if accelerator a wants to transfer a tensor to accelerator b, the scheduler will receive a communication intent from a to b. When multiple tasks are stacked in a cluster, communication intent becomes quite complex. Under certain constraints (for example, the send and recv operators on a single accelerator cannot execute simultaneously), it is possible to activate as many communication intents as possible. A communication intent graph is used to represent the communication intent within the cluster. Communication is then managed based on a communication scheduling control strategy (such as maximum bipartite matching). In the communication intent graph, nodes represent accelerators, and edges represent the communication intent between a group of accelerators.

[0110] For example, the communication intention map is .in, With the above in The same is true for the nodes of the communication intention graph, namely the accelerator; is the edge of the communication intention graph, Operators in The calculation results are transmitted to , and the transmission has not yet ended; refer to to the transfer being executed.

[0111] The communication intention graph within the scheduler is constructed through the IO interface, and the communication intention within the accelerator cluster is tracked in real time. Each time the communication intention graph is updated, the maximum bipartite matching problem is incrementally solved, and the corresponding communication operator is triggered, thus achieving real-time cluster transmission management.

[0112] For example, , through this function, the transmission event is added to the communication intention graph, ; Then incrementally solve the maximum bipartite matching problem, if the transmission event should be started, then , which triggers communication permission. It means that the ongoing transmission event has been completed, that is, , ; If the completion of the transmission event will change the solution of the maximum bipartite matching, then start the corresponding transmission event, that is, .

[0113] This embodiment makes the data flow in the accelerator cluster controllable, thereby maximizing the utilization of network transmission capacity.

[0114] As an embodiment of the present application, Figure 2 As shown in Figure 1, a heterogeneous cluster system includes a scheduler and multiple accelerators. Both the scheduler and the accelerator expose a set of interfaces. The scheduler and the accelerator can call each other's interfaces through RPC (remote procedure call, calling a local function from a remote network). The cluster watchdog notifies the scheduler of changes in the scale of the entire cluster in real time. Figure 7 As shown, the task scheduling method executed by the scheduler may include the following steps:

[0115] S101: Receive an accelerator cluster structure fed back by a cluster watchdog to determine a plurality of accelerators available for task scheduling.

[0116] S102: Receive at least one task.

[0117] Specifically, it is a directed acyclic graph of received tasks. The computing process of all businesses in the heterogeneous cluster is described in the form of a directed acyclic graph and then sent to the scheduler.

[0118] S103: After receiving the given task execution instruction, start the task.

[0119] S104, based on the received task, first generate several scheduling strategies using different types and quantities of resources, and then use cluster prediction to evaluate the superiority of each scheduling strategy one by one, and select the best target task scheduling strategy.

[0120] S105: Generate a global computation graph based on the target task scheduling strategy.

[0121] S106 , sending the operator to the corresponding accelerator based on the global computation graph, so as to complete the operator computation by calling the corresponding accelerator.

[0122] S107 : When it is recognized that there is a communication intention between the accelerators, triggering communication permission so that the accelerators perform data transmission based on the communication permission.

[0123] S108: Receive feedback information from the accelerator indicating that operator calculation and / or transmission is complete, and update the global computation graph based on the feedback information, for example, marking the operator as completed.

[0124] Further integration Figure 8 As shown in the figure, the task is assigned to accelerator 1 and accelerator 2. After the task execution is started, all related time is tracked, including changes in the global computation graph, changes in the out-of-order queues of accelerator 1 and accelerator 2, and the process of interface calls in the entire system. The details are as follows:

[0125] , call the run_task function, add operators to the global computation graph and generate corresponding mapping relationships.

[0126] and Based on the global computation graph, the scheduler adds operators to the out-of-order queues of accelerator 1 and accelerator 2. The newly added operators are non-ready.

[0127] ,Accelerator 1 calls to notify the scheduler that the operator has been executed.

[0128] , accelerator 1 triggers a communication intention to the scheduler, which is to transfer tensor a (operand) from accelerator 1 to accelerator 2.

[0129] , the scheduler allows tensor transfer and changes the recv operator (receive operator) on accelerator 2 to the ready state.

[0130] , the recv operator on accelerator 2 starts executing, and accelerator 2 calls the left_trace interface of accelerator 1, making the send operator on accelerator 1 become ready.

[0131] , the send operator on accelerator 1 starts executing, docks with the recv operator being executed on accelerator 2, and starts transmission; after the transmission is completed, the recv operator on accelerator 2 calls the report_recv_done function to tell the scheduler that the transmission is complete.

[0132] , accelerator 2 calls the report_operator_done function to tell the scheduler that operators y1 and y2 have completed execution.

[0133] ,Accelerator 2 calls the report_operator_done function to tell the scheduler that operator w has been executed.

[0134] This embodiment merges the computing tasks of all jobs into a large computing graph based on the global computing graph, uniformly generates scheduling strategies, and launches operators to the accelerator cluster, realizing a unified mapping mechanism from operators to accelerators. It supports heterogeneous accelerators to expose a unified operator execution interface instead of forcing all operators to adapt to common kernel codes, eliminating the performance overhead of cross-platform kernel conversion, thereby enabling multiple heterogeneous devices to collaboratively participate in the parallel computing of tasks, effectively improving overall resource utilization and scheduling flexibility.

[0135] At the same time, the global computation graph tracks the working status of all accelerators in the cluster in real time. Based on the current global computation graph state, the cluster's resource distribution and load trends in future periods can be predicted, thereby evaluating the cost and performance impact of different scheduling strategies. It supports generating multiple resource utilization plans when a task is submitted, and uses prediction models to select the optimal plan, achieving fast and flexible scheduling response capabilities.

[0136] Based on the above analysis, this application has the following technical advantages:

[0137] (1) Global operator-level scheduling granularity, breaking through the limitations of existing task-level scheduling;

[0138] (2) Supporting native operator execution on heterogeneous accelerators, eliminating cross-platform execution bottlenecks;

[0139] (3) The cluster structure is predictable, scheduling decisions can be optimized, and it has flexible scheduling capabilities;

[0140] (4) Support cross-task joint scheduling to effectively alleviate the problem of resource fragmentation.

[0141] This task scheduling method is suitable for complex computing scenarios that require high parallelism and heterogeneous resource scheduling, such as large-scale graph neural networks, large language models, and machine learning training and reasoning. Examples are as follows:

[0142] 1. Training: The model training computational process is extracted into a computational graph and then iteratively submitted to the compute nodes. The compute nodes can automatically and efficiently find idle computing power in the cluster for training, automatically handle varying capacity granularity during training, and automatically handle occasional computational errors.

[0143] 2. Inference: Describe the model inference process in the form of a computational graph and submit it to the compute nodes. The compute nodes can automatically adapt to fluctuations in business pressure caused by time periods and can automatically implement pipeline parallelization to fully utilize the cluster's computing power.

[0144] 3. General computing: Describe the task load in each field in the form of a computational graph, and then submit it to the computing nodes to achieve general computing of the cluster.

[0145] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0146] The embodiment of the present application also provides a task scheduling device, which is applied to heterogeneous clusters, such as Figure 9 As shown, the task scheduling device of the embodiment of the present application includes: a receiving module 10, a establishing module 20 and a scheduling module 30.

[0147] Among them, the receiving module 10 is used to receive at least one task to be executed, wherein each task includes multiple operators and data dependencies between the multiple operators; the establishing module 20 is used to determine the first mapping relationship between the multiple operators in each task and the multiple accelerators, and establish a global computation graph based on the first mapping relationship corresponding to each task; the scheduling module 30 is used to send each operator to the corresponding accelerator according to the global computation graph, and execute at least one task through one or more of the multiple accelerators in combination with the data dependencies between the multiple operators.

[0148] In some embodiments of the present application, an establishment module 20 determines a first mapping relationship between multiple operators in each task and multiple accelerators, and is specifically used to: determine multiple task scheduling strategies corresponding to the task based on a preset scheduling generation method; determine the scheduling performance of each task scheduling strategy based on a preset calculation time of each operator in the task in the corresponding accelerator and a preset communication time between multiple accelerators; determine a target task scheduling strategy based on the scheduling performance of each task scheduling strategy, and generate a first mapping relationship between multiple operators in the task and multiple accelerators based on the target task scheduling strategy.

[0149] In some embodiments of the present application, when multiple accelerators are in a task execution state, the establishment module 20 is also used to: determine the current global computation graph being executed by the multiple accelerators; determine the idle start time of each accelerator based on the current execution state of the current global computation graph, the preset computation time of each operator in the current global computation graph in the corresponding accelerator, and the preset communication time between the multiple accelerators; predict the execution end time of each task scheduling strategy based on the preset computation time of each operator in the task in the corresponding accelerator, the preset communication time between the multiple accelerators, and the idle start time of each accelerator, so as to determine the scheduling performance of each task scheduling strategy.

[0150] In some embodiments of the present application, the establishment module 20 determines the idle start time of each accelerator based on the current execution state of the current global computation graph, the preset computation time of each operator in the current global computation graph in the corresponding accelerator, and the preset communication time between multiple accelerators, and is specifically used to: determine the execution completion time of each operator in the current global computation graph based on the current execution state of the current global computation graph, the preset communication time of each operator in the current global computation graph in the corresponding accelerator, and the preset communication time between multiple accelerators; predict the idle start time of each accelerator based on the execution completion time of each operator in the current global computation graph.

[0151] In some embodiments of the present application, the global computation graph includes status information of each operator in at least one task. In the process of sending each operator to the corresponding accelerator according to the global computation graph, and executing at least one task through one or more of the multiple accelerators in combination with the data dependency relationship between the multiple operators, the establishment module 20 is also used to: receive operator status feedback information sent by one or more of the multiple accelerators; and update the status information of each operator according to the operator status feedback information.

[0152] In some embodiments of the present application, the establishment module 20 updates the status information of each operator according to the operator status feedback information, and is specifically used to: when it is determined that the operator is in the execution completion state according to the operator status feedback information, the operator and the associated information of the operator in the global computation graph are deleted.

[0153] In some embodiments of the present application, in the process of sending each operator to the corresponding accelerator according to the global computation graph, and executing at least one task through one or more of the multiple accelerators in combination with the data dependency relationship between multiple operators, the establishment module 20 is also used to: respond to the operator insertion instruction, identify the target operator corresponding to the operator insertion instruction and the data dependency relationship between the target operator and the operator in the global computation graph; establish a second mapping relationship between the target operator and the multiple accelerators based on the data dependency relationship between the target operator and the operators in the global computation graph, and update the global computation graph based on the second mapping relationship.

[0154] In some embodiments of the present application, the global computation graph includes an accelerator set corresponding to multiple accelerators. In the process of sending each operator to the corresponding accelerator according to the global computation graph, and executing at least one task through one or more of the multiple accelerators in combination with the data dependencies between the multiple operators, the establishment module 20 is also used to: obtain status information of each accelerator in the heterogeneous cluster; and update the accelerator set according to the status information of each accelerator.

[0155] In some embodiments of the present application, the establishment module 20 updates the accelerator set based on the status information of each accelerator. Specifically, the establishment module 20 is configured to: when it is determined based on the status information of each accelerator that a new online accelerator exists in the heterogeneous cluster, add the new online accelerator to the accelerator set; and / or when it is determined based on the status information of each accelerator that an offline accelerator exists in the accelerator set, re-establish a third mapping relationship between unexecuted operators corresponding to the offline accelerator and the online accelerators in the accelerator set, and then delete the offline accelerator from the accelerator set.

[0156] In some embodiments of the present application, when it is determined that there is an offline accelerator in the accelerator set based on the current state of each accelerator, the scheduling module 30 is further used to: obtain the intermediate operator calculation results stored in the offline accelerator, and send the intermediate operator calculation results to the corresponding online accelerator based on the data dependency relationship between multiple operators.

[0157] In some embodiments of the present application, in the process of sending each operator to the corresponding accelerator according to the global computation graph, and executing at least one task through one or more of the multiple accelerators in combination with the data dependency relationship between the multiple operators, the scheduling module 30 is also used to: in the event of an operator calculation error, remove the operator with the calculation error from the original accelerator, establish a fourth mapping relationship between the operator with the calculation error and other accelerators in the multiple accelerators, and update the global computation graph based on the fourth mapping relationship.

[0158] In some embodiments of the present application, after updating the global computation graph based on the fourth mapping relationship, the scheduling module 30 is further used to: obtain the operands of the operator with calculation errors, where the operands are stored in the corresponding accelerator; and send the operator with calculation errors and the operands to the corresponding accelerator based on the updated global computation graph.

[0159] In some embodiments of the present application, the scheduling module 30 is also used to: obtain the actual calculation time of the operator, and determine that a calculation error has occurred in the operator when the actual calculation time of the operator exceeds the preset time; or determine that a calculation error has occurred in the operator when an abnormality occurs in the accelerator where the operator is located; or in response to an operator abnormal instruction, determine that a calculation error has occurred in the operator corresponding to the operator abnormal instruction.

[0160] In some embodiments of the present application, the scheduling module 30 is further configured to: identify communication willingness between at least one group of accelerators; and generate corresponding communication permission based on the communication willingness between at least one group of accelerators, so that at least one group of accelerators communicate based on the communication permission.

[0161] In some embodiments of the present application, when the scheduling module 30 identifies the communication willingness between at least one group of accelerators, it is further used to: construct a communication willingness graph according to the communication willingness between at least one group of accelerators; perform a communication scheduling control strategy on at least one group of accelerators based on the communication willingness graph, so as to generate communication permission corresponding to each group of accelerators based on the communication scheduling control strategy.

[0162] In some embodiments of the present application, the receiving module 10 receives at least one task to be executed, and is specifically used to: receive a directed acyclic graph of at least one task, wherein the nodes of the directed acyclic graph are used to represent the corresponding operators, and the edges of the directed acyclic graph are used to represent the data dependency relationship between the operators.

[0163] For the description of the features in the embodiment corresponding to the task scheduling device, please refer to the relevant description of the embodiment corresponding to the task scheduling method, and no further details will be given here.

[0164] The embodiment of the present application also provides an electronic device, such as Figure 10 As shown, the electronic device 100 includes a memory 110 and a processor 120. The memory 110 stores a computer program, and the processor 120 is configured to run the computer program to execute the steps in any of the above-mentioned task scheduling method embodiments.

[0165] An embodiment of the present application further provides a non-volatile computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above-mentioned task scheduling method embodiments when running.

[0166] In an exemplary embodiment, the non-volatile computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0167] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any of the above-mentioned task scheduling method embodiments are implemented.

[0168] An embodiment of the present application also provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above-mentioned task scheduling method embodiments are implemented.

[0169] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0170] The above is a detailed introduction to a task scheduling method, scheduling device, electronic device, storage medium and product provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A task scheduling method, characterized in that: Applied to a heterogeneous cluster including multiple accelerators, the method includes: receiving at least one task to be executed, wherein each task includes a plurality of operators and data dependencies between the plurality of operators; Determine a first mapping relationship between a plurality of operators in each task and the plurality of accelerators, and establish a global computation graph based on the first mapping relationship corresponding to each task; Sending each operator to a corresponding accelerator according to the global computation graph, and executing the at least one task through one or more of the multiple accelerators in combination with data dependencies between the multiple operators; The global computation graph includes an accelerator set corresponding to the multiple accelerators, and in the process of sending each operator to the corresponding accelerator according to the global computation graph, and executing the at least one task through one or more of the multiple accelerators in combination with the data dependencies between the multiple operators, further comprising: Obtaining status information of each accelerator in the heterogeneous cluster; The accelerator set is updated according to the status information of each accelerator.

2. The task scheduling method according to claim 1, characterized in that: The determining of a first mapping relationship between the multiple operators in each task and the multiple accelerators includes: Determine multiple task scheduling strategies corresponding to tasks based on a preset scheduling generation method; Determining the scheduling performance of each task scheduling strategy according to a preset computing time of each operator in the task in the corresponding accelerator and a preset communication time between the plurality of accelerators; A target task scheduling strategy is determined according to the scheduling performance of each task scheduling strategy, and a first mapping relationship between the multiple operators in the task and the multiple accelerators is generated based on the target task scheduling strategy.

3. The task scheduling method according to claim 2, characterized in that: When the plurality of accelerators are in a task execution state, the method further includes: determining a current global computation graph being executed by the plurality of accelerators; Determining an idle start time of each accelerator according to a current execution state of the current global computation graph, a preset computation duration of each operator in the current global computation graph in a corresponding accelerator, and a preset communication duration between the plurality of accelerators; The execution end time of each task scheduling strategy is predicted based on the preset calculation time of each operator in the task in the corresponding accelerator, the preset communication time between the multiple accelerators, and the idle start time of each accelerator to determine the scheduling performance of each task scheduling strategy.

4. The task scheduling method according to claim 3, characterized in that: The determining the idle start time of each accelerator according to the current execution state of the current global computation graph, the preset computation duration of each operator in the current global computation graph in the corresponding accelerator, and the preset communication duration between the multiple accelerators includes: Determining an execution completion time of each operator in the current global computation graph according to a current execution state of the current global computation graph, a preset communication duration of each operator in the current global computation graph in a corresponding accelerator, and a preset communication duration between the multiple accelerators; The idle start time of each accelerator is predicted according to the execution completion time of each operator in the current global computation graph.

5. The task scheduling method according to claim 1, wherein: The global computation graph includes state information of each operator in the at least one task. In the process of sending each operator to a corresponding accelerator according to the global computation graph, and executing the at least one task through one or more of the multiple accelerators in combination with data dependencies between the multiple operators, the process further includes: receiving operator state feedback information sent by one or more of the plurality of accelerators; The state information of each operator is updated according to the operator state feedback information.

6. The task scheduling method according to claim 5, characterized in that: Updating the state information of each operator according to the operator state feedback information includes: When it is determined according to the operator state feedback information that the operator is in an execution completion state, the operator and associated information of the operator in the global computation graph are deleted.

7. The task scheduling method according to claim 1, characterized in that: In the process of sending each operator to a corresponding accelerator according to the global computation graph, and executing the at least one task through one or more of the multiple accelerators in combination with the data dependencies between the multiple operators, the method further includes: In response to an operator insertion instruction, identifying a target operator corresponding to the operator insertion instruction and a data dependency relationship between the target operator and operators in the global computation graph; A second mapping relationship between the target operator and the multiple accelerators is established based on a data dependency relationship between the target operator and operators in the global computation graph, and the global computation graph is updated based on the second mapping relationship.

8. The task scheduling method according to claim 1, characterized in that: Updating the accelerator set according to the status information of each accelerator includes: If it is determined according to the status information of each accelerator that a new online accelerator exists in the heterogeneous cluster, adding the new online accelerator to the accelerator set; and / or When it is determined according to the status information of each accelerator that an offline accelerator exists in the accelerator set, a third mapping relationship between the unexecuted operators corresponding to the offline accelerator and the online accelerators in the accelerator set is re-established, and then the offline accelerator in the accelerator set is deleted.

9. The task scheduling method according to claim 8, characterized in that: In the case where it is determined according to the current state of each accelerator that there is an offline accelerator in the accelerator set, the method further includes: The intermediate operator calculation results stored in the offline accelerator are obtained, and the intermediate operator calculation results are sent to the corresponding online accelerator based on the data dependency relationship between the multiple operators.

10. The task scheduling method according to claim 1, wherein: In the process of sending each operator to a corresponding accelerator according to the global computation graph, and executing the at least one task through one or more of the multiple accelerators in combination with the data dependencies between the multiple operators, the method further includes: In the event of an operator calculation error, the operator with the calculation error is removed from the original accelerator, and a fourth mapping relationship is established between the operator with the calculation error and other accelerators in the multiple accelerators, and the global computation graph is updated based on the fourth mapping relationship.

11. The task scheduling method according to claim 10, characterized in that: After updating the global computation graph based on the fourth mapping relationship, the method further includes: Obtaining an operand of the operator with the calculation error, wherein the operand is stored in a corresponding accelerator; The operator with the calculation error and the operand are sent to the corresponding accelerator based on the updated global computation graph.

12. The task scheduling method according to claim 10, characterized in that: Also includes: Obtaining an actual calculation duration of the operator, and determining that a calculation error occurs in the operator if the actual calculation duration of the operator exceeds a preset duration; or When an accelerator in which an operator is located has an abnormality, determining that a calculation error occurs in the operator; or In response to an operator exception instruction, it is determined that a calculation error occurs in an operator corresponding to the operator exception instruction.

13. The task scheduling method according to claim 1, wherein: Also includes: identifying a willingness to communicate between at least one set of accelerators; A corresponding communication permission is generated based on the communication willingness between the at least one group of accelerators, so that the at least one group of accelerators communicate based on the communication permission.

14. The task scheduling method according to claim 13, wherein: In the case where a communication intention between at least one group of accelerators is identified, the following further includes: constructing a communication willingness graph according to the communication willingness among the at least one group of accelerators; A communication scheduling control strategy is performed on the at least one group of accelerators based on the communication willingness graph, so as to generate communication permissions corresponding to each group of accelerators based on the communication scheduling control strategy.

15. The task scheduling method according to claim 1, wherein: The receiving at least one task to be performed includes: A directed acyclic graph of at least one task is received, wherein nodes of the directed acyclic graph are used to represent corresponding operators, and edges of the directed acyclic graph are used to represent data dependencies between operators.

16. A task scheduling device, characterized in that: Applied to a heterogeneous cluster including multiple accelerators, the device includes: a receiving module, configured to receive at least one task to be executed, wherein each task includes a plurality of operators and data dependencies between the plurality of operators; An establishment module is used to determine a first mapping relationship between a plurality of operators in each task and the plurality of accelerators, and establish a global computation graph based on the first mapping relationship corresponding to each task; a scheduling module, configured to send each operator to a corresponding accelerator according to the global computation graph, and execute the at least one task through one or more of the multiple accelerators in combination with data dependencies between the multiple operators; The global computation graph includes an accelerator set corresponding to the multiple accelerators. The establishment module is further used to: obtain status information of each accelerator in the heterogeneous cluster; and update the accelerator set according to the status information of each accelerator, while sending each operator to the corresponding accelerator according to the global computation graph and executing the at least one task through one or more of the multiple accelerators in combination with the data dependencies between the multiple operators.

17. An electronic device, characterized in that: include: memory for storing computer programs; A processor, configured to implement the steps of the task scheduling method according to any one of claims 1 to 15 when executing the computer program.

18. A non-volatile computer-readable storage medium, characterized in that: The non-volatile computer-readable storage medium stores a computer program, wherein the computer program implements the steps of the task scheduling method according to any one of claims 1 to 15 when executed by a processor.

19. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the task scheduling method according to any one of claims 1 to 15 are implemented.

Images