An operator scheduling method and related device

Abstract

An operator scheduling method includes: obtaining computation operation descriptions of computation operators and communication operation descriptions of communication operators; obtaining a unified scheduling template for computation operators and communication operators based on the computation operation descriptions and communication operation descriptions; and generating a fusion kernel for computation operators and communication operators based on the unified scheduling template. This method uses unified scheduling primitives for computation and communication operations to generate a unified scheduling template corresponding to the operation descriptions. Then, based on the unified scheduling template, it automatically generates a computation-communication fusion kernel with unified basic blocks for computation and communication operations as scheduling units, realizing parallel computation and communication operations. This solves the problems of low masking degree, low computational efficiency, and poor performance encountered in coarse-grained automatic masking optimization of computation and communication, while also avoiding the problems of high development difficulty and low portability when manually developing a computation-communication fusion kernel.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence (AI) technology, and in particular to an operator scheduling method, an operator scheduling device, a computing card, a computing cluster, a computer-readable storage medium, and a computer program product. Background Technology

[0002] With the rise of new computing paradigms such as artificial intelligence (AI) computing and high-performance computing (HPC), the demand for computing power is showing an exponential growth trend. Traditional single-machine, single-card computing architectures can no longer meet the requirements of large-scale computing. For example, in AI computing scenarios, single-machine, single-card architectures cannot meet the requirements of large-scale dataset storage and complex model inference / training. As a result, computing architectures are gradually evolving towards distributed computing.

[0003] Distributed computing distributes computational tasks across multiple computing cards. This necessitates communication between these cards to facilitate data exchange. This communication can be achieved through communication operators, such as aggregate communication operators like AllReduce and ReduceScatter. The execution time of these communication operators can range from 10% to 50% of the total execution time.

[0004] To shorten the overall execution time and improve efficiency, operators used for complex or large-scale operations can be manually broken down into fine-grained pipelining parallel instructions, achieving computational and communication integration. However, this approach is difficult to develop, time-consuming, inefficient, and hard to migrate to other distributed systems, thus failing to meet business requirements. Summary of the Invention

[0005] This application provides an operator scheduling method. This method uses unified scheduling primitives for computational and communication operations to generate a unified scheduling template corresponding to the operation description. Then, based on the unified scheduling template, it automatically generates a fusion kernel with basic blocks (unified for both computational and communication operations) as scheduling units, enabling parallel computational and communication operations. The fusion kernel executes computational and communication operations at the basic block level, achieving fine-grained computation-communication fusion, significantly improving masking, shortening the total execution time, and thus improving overall computational efficiency and performance. Furthermore, this method eliminates the need for manual operator splitting and can be quickly migrated to other distributed systems automatically, exhibiting high availability. This application also provides an operator scheduling device, computing card, computing cluster, computer-readable storage medium, and computer program product corresponding to the above method.

[0006] Firstly, this application provides an operator scheduling method. This operator scheduling method can be executed by an operator scheduling device. The operator scheduling device is used to automatically fuse computational operators and communication operators in a computation and communication subgraph obtained by partitioning a computation graph of a distributed computing task, thereby achieving unified scheduling of computational and communication operators. A computation graph is a graph structure used to represent mathematical computations or program execution. Nodes are used to represent operators in a computation graph. The computation and communication subgraph includes at least one computational operator and at least one communication operator. The operator scheduling device can be software that performs fine-grained fusion of computational and communication operators to achieve unified scheduling, such as an artificial intelligence (AI) compiler or AI executor. The AI ​​compiler may include, but is not limited to, a graph compiler or operator compiler, where the operator compiler is also called a tensor compiler or tensor boost engine (TBE). The AI ​​executor may include, but is not limited to, a graph executor or operator executor. In some examples, the operator scheduling device can be the hardware on which the software described above is deployed, such as a computing card like a GPU, NPU, or TPU, or a computing device that includes the aforementioned computing card.

[0007] In practical implementation, the operator scheduling device can obtain the computational operation descriptions of computational operators and the communication operation descriptions of communication operators. The computational and communication operators are operators in the computational and communication subgraphs obtained by partitioning the computation graph. Based on the computational and communication operation descriptions, the operator scheduling device can obtain a unified scheduling template for the computational and communication operators. This unified scheduling template uses a unified scheduling primitive for both computational and communication operations. This primitive controls or optimizes the execution of computational tasks and defines how to efficiently execute computational tasks on target hardware. The target hardware can be a computing card or computing device in a distributed system. Based on the unified scheduling template, the operator scheduling device can generate a fused kernel (also called a computation-communication fusion kernel or fusion kernel) for the computational and communication operators. The fusion kernel includes kernel functions that implement parallel computational and communication operations, using basic blocks that unify the computational and communication operations as scheduling units.

[0008] This method employs a unified scheduling primitive to generate a unified scheduling template corresponding to the operation descriptions of computational and communication operations. Then, based on this unified scheduling template, it automatically generates a fused kernel that uses basic blocks—which unify the scheduling of computational and communication operations—as scheduling units. This enables parallel computational and communication operations. The fused kernel executes computational and communication operations at the basic block level, achieving fine-grained computational-communication fusion, significantly improving masking, shortening total execution time, and thus enhancing overall computational efficiency and performance. This solves the problems of low masking, low computational efficiency, and poor performance faced by coarse-grained kernel-based fusion. Furthermore, the fused kernel implements computational and communication operations using a single kernel function, resolving the issue of significant kernel launch overhead in coarse-grained kernel-based fusion. Simultaneously, by introducing a unified scheduling template and automatically generating the fused kernel based on it, this method avoids the high development difficulty and low portability associated with manually developing computational-communication fusion kernels.

[0009] In some possible implementations, the operator scheduling device can update the parameters in the unified scheduling template to obtain an updated unified scheduling template. For example, the operator scheduling device can optimize the parameters in the unified scheduling template using tuning algorithms or tools to obtain the updated unified scheduling template. Accordingly, the operator scheduling device can generate a fusion kernel of computational and communication operators based on the updated unified scheduling template.

[0010] This method optimizes the relevant parameters in the unified scheduling template to select a suitable unified scheduling template for fusion kernel generation. This allows for more parallel opportunities, improves the coverage of computation and communication in the fusion kernel, and shortens the total execution time.

[0011] In some possible implementations, the unified scheduling template for computation and communication operators includes multiple templates. The operator scheduling device can update the parameters in multiple templates to obtain multiple updated templates. The operator scheduling device can determine the updated unified scheduling template based on the evaluation metrics of the multiple updated templates.

[0012] This method employs a multi-layered screening mechanism. First, it optimizes the parameters of multiple candidate templates to obtain multiple updated templates. Then, it filters these updated templates based on evaluation metrics to select the best among the best, thereby further improving the coverage of computation and communication in the fusion kernel and shortening the total execution time.

[0013] In some possible implementations, the parameters in the unified scheduling template include at least one of the following: the number of computation blocks for merged communication, the matrix partitioning method, or the shape of the computation blocks for merged communication. The matrix partitioning method can be either regular or irregular. Parameters such as the matrix partitioning method and the number of computation blocks for merged communication can be flexibly configured, which helps enhance data locality, reduce the number of main memory accesses, and improve the performance of the fused kernel.

[0014] In some possible implementations, the operator scheduling device can obtain a unified scheduling template for computation and communication operators written by the user based on computation and communication operation descriptions. Alternatively, the operator scheduling device can obtain a unified scheduling template for computation and communication operators from built-in scheduling templates based on computation and communication operation descriptions. Or, the operator scheduling device can generate a unified scheduling template for computation and communication operators using machine learning based on computation and communication operation descriptions. It should be noted that the operator scheduling device can also combine the above methods to obtain a unified scheduling template. This method supports obtaining unified scheduling templates for computation and communication operators through multiple methods, allowing users to flexibly choose according to business needs, and has high availability.

[0015] In some possible implementations, the operator scheduling device can automatically select the scheduling primitives for constructing a unified scheduling template and the parameters of the unified scheduling template based on the computation operation description and the communication operation description, thereby obtaining a unified scheduling template for both computation and communication operators. This allows for the one-time determination of the unified scheduling template and its parameters, improving template acquisition efficiency.

[0016] In some possible implementations, the unified scheduling template includes multiple scheduling statements. Correspondingly, the operator scheduling device can generate a first code block for the first scheduling statement among the multiple scheduling statements, based on the syntax rules corresponding to the scheduling primitives in the first scheduling statement. The first code block is used to implement the functionality of the scheduling primitives in the first scheduling statement. Then, the operator scheduling device can modify the first code block for the second scheduling statement among the multiple scheduling statements, based on the syntax rules corresponding to the scheduling primitives in the second scheduling statement, to obtain a second code block. The fusion kernel of the computation operator and the communication operator includes the second code block.

[0017] This method supports the automatic generation or modification of code blocks based on the syntax rules corresponding to the scheduling primitives in the scheduling statement, laying the foundation for the automatic generation of fusion kernels based on a unified scheduling template.

[0018] In some possible implementations, the first code block includes code that implements computational and communication operations using basic blocks as scheduling units. The operator scheduling device can modify lines of code in the first code block related to computational order, single communication data size, data locality, or linearized storage, according to the syntax rules corresponding to the scheduling primitives in the second scheduling statement, to obtain the second code block.

[0019] This method first generates code that implements computation and communication operations using basic blocks that unify computation and communication operations as scheduling units. Then, based on the syntax rules corresponding to the scheduling primitives, it modifies the lines of code related to the computation order, the size of data in a single communication, the localization of data, or the linearization of storage, thereby achieving fine-grained integration of computation and communication.

[0020] In some possible implementations, the operator scheduler can construct prompts based on the first scheduling statement and learn to generate a first code block based on these prompts. The first code block conforms to the syntax rules corresponding to the scheduling primitives in the first scheduling statement. This method, by leveraging the code generation capabilities of prompt-learning, generates code blocks that conform to the syntax rules corresponding to the scheduling primitives, thereby enabling the generation of a high-quality fusion kernel.

[0021] In some possible implementations, the operator scheduling device can transform the first scheduling statement according to the syntax rules corresponding to the scheduling primitives in the first scheduling statement to obtain the first code block. The process of generating code through rule transformation is simple, easy to implement, and highly available.

[0022] In some possible implementations, the operator scheduling device can obtain the input description, computation method description, and output description of the computational operation of the computational operator through a first interface, and obtain the operation type of the communication operation of the communication operator through a second interface. The first interface and the second interface can be the same interface or different interfaces.

[0023] This method obtains computation operation descriptions and communication operation descriptions through an interface, which can provide relatively standardized operation description data, thus providing high-quality operation description data for subsequent unified scheduling template acquisition, thereby facilitating the acquisition of high-quality unified scheduling templates.

[0024] In some possible implementations, the scheduling primitive includes at least one of the following: scheduling primitive for multidimensional computation loops, scheduling primitive for linearized storage of intermediate data, scheduling primitive for localization of intermediate data, and scheduling primitive for merging communication of multiple computation blocks.

[0025] This method generates a unified scheduling template corresponding to the operation description by using a unified scheduling primitive for computation and communication operations. Based on the unified scheduling template, it automatically generates a fusion kernel with a unified basic block for computation and communication operations as the scheduling unit, thereby realizing parallel computation and communication operations.

[0026] Secondly, this application provides an operator scheduling device. The operator scheduling device can be software that implements fine-grained fusion of computational and communication operators, such as an AI compiler or an AI executor. The AI ​​compiler may include, but is not limited to, a graph compiler or an operator compiler, and the AI ​​executor may include, but is not limited to, an operator executor or a graph executor. Alternatively, the operator scheduling device can also be hardware that implements fine-grained fusion of computational and communication operators. Specifically, the operator scheduling device may include the following modules:

[0027] The operation description acquisition module is used to acquire the computation operation description of the computation operator and the communication operation description of the communication operator, wherein the computation operator and the communication operator are operators in the computation and communication subgraphs obtained by partitioning the computation graph of the distributed computing task;

[0028] The template acquisition module is used to acquire a unified scheduling template for the computation operator and the communication operator based on the computation operation description and the communication operation description. The unified scheduling template uses a unified scheduling primitive for the computation operation and the communication operation. The scheduling primitive is used to control or optimize the execution of the distributed computing task.

[0029] A fusion kernel generation module is used to generate a fusion kernel for the computation operator and the communication operator based on the unified scheduling template. The fusion kernel includes a kernel function that uses a basic block that unifies the computation operation and the communication operation as a scheduling unit to implement the parallel operation of the computation operation and the communication operation.

[0030] In some possible implementations, the template acquisition module is further used for:

[0031] Update the parameters in the unified scheduling template to obtain the updated unified scheduling template;

[0032] The fusion kernel generation module is used for:

[0033] Based on the updated unified scheduling template, a fusion kernel for the computation operator and the communication operator is generated.

[0034] In some possible implementations, the unified scheduling template for the computation operator and the communication operator includes multiple templates;

[0035] The template acquisition module is specifically used for:

[0036] The parameters in the multiple templates are updated to obtain multiple updated templates;

[0037] Based on the evaluation metrics of the multiple updated templates, the updated unified scheduling template is determined.

[0038] In some possible implementations, the parameters in the unified scheduling template include at least one of the following: the number of computational blocks for merged communication, the matrix partitioning method, or the shape of the computational blocks for merged communication, wherein the matrix partitioning method includes regular partitioning or irregular partitioning.

[0039] In some possible implementations, the template acquisition module is specifically used for:

[0040] Obtain the unified scheduling template for the computation operator and the communication operator written by the user based on the computation operation description and the communication operation description; and / or;

[0041] Based on the computation operation description and the communication operation description, obtain a unified scheduling template for the computation operator and the communication operator from the built-in scheduling template; and / or;

[0042] Based on the computation operation description and the communication operation description, a unified scheduling template for the computation operator and the communication operator is generated through machine learning.

[0043] In some possible implementations, the template acquisition module is specifically used for:

[0044] Based on the computation operation description and the communication operation description, the scheduling primitives for constructing the unified scheduling template and the parameters of the unified scheduling template are automatically selected to obtain the unified scheduling template for the computation operator and the communication operator.

[0045] In some possible implementations, the fusion kernel generation module is specifically used for:

[0046] For the first scheduling statement among the plurality of scheduling statements, a first code block is generated according to the syntax rules corresponding to the scheduling primitive in the first scheduling statement. The first code block is used to implement the function of the scheduling primitive in the first scheduling statement.

[0047] For the second scheduling statement among the plurality of scheduling statements, the first code block is modified according to the syntax rules corresponding to the scheduling primitive in the second scheduling statement to obtain the second code block;

[0048] The fusion kernel of the computation operator and the communication operator includes the second code block.

[0049] In some possible implementations, the first code block includes code that implements the computational operation and the communication operation using a basic block that unifies the computational operation and the communication operation as a scheduling unit;

[0050] The fusion kernel generation module is specifically used for:

[0051] Based on the syntax rules corresponding to the scheduling primitives in the second scheduling statement, modify the lines of code in the first code block that are related to the computation order, the size of data in a single communication, the localization of data, or the linearization of storage to obtain the second code block.

[0052] In some possible implementations, the operation description acquisition module is specifically used for:

[0053] The input description, calculation method description, and output description of the computation operator are obtained through the first interface.

[0054] The operation type of the communication operator is obtained through the second interface.

[0055] In some possible implementations, the scheduling primitive includes at least one of the following: scheduling primitive for multidimensional computation loops, scheduling primitive for linearized storage of intermediate data, scheduling primitive for localization of intermediate data, and scheduling primitive for merging communication of multiple computation blocks.

[0056] Thirdly, this application provides a computing card. The computing card can be a neural processing unit (NPU), a graphics processing unit (GPU), or a tensor processing unit (TPU). The computing card includes computing cores and memory. A computing core is a module in the computing card used to implement computing capabilities. The computing cores of different types of computing cards can have different structures. For example, the computing core of an NPU can be an AI core, which includes matrix computation units, vector computation units, and scalar computation units designed for neural networks. The computing core of a GPU includes multiple stream processors, which are used for parallel processing of data computation tasks. The memory of the computing card is also called device memory or video memory. The computing core is used to execute computer-readable instructions loaded into the memory to perform the operator scheduling method as described in the first aspect of this application or any implementation thereof.

[0057] Fourthly, this application provides a computing cluster. The computing cluster includes multiple computing cards, which can be connected via a bus or network. For example, in a single-machine multi-card (or one-host multi-computing-card) architecture, the multiple computing cards can be connected via a bus. As another example, in a multi-machine multi-card (or multi-host multi-computing-card) architecture, computing cards within the same computing device can be connected via a bus, while computing cards in different computing devices can be connected via a network.

[0058] Fifthly, this application provides a computer-readable storage medium storing instructions that instruct a computing card or computing cluster to execute the operator scheduling method described in the first aspect or any implementation thereof.

[0059] In a sixth aspect, this application provides a computer program product containing instructions that, when run on a computing card or computing cluster, causes the computing card or computing cluster to execute the operator scheduling method described in the first aspect or any implementation thereof.

[0060] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0061] To more clearly illustrate the technical methods of this application, the accompanying drawings used will be briefly described below.

[0062] Figure 1 A schematic diagram illustrating the parallel execution of computational and communication operators provided in this application;

[0063] Figure 2 A schematic diagram of the hardware and software architecture of a computing system provided in this application;

[0064] Figure 3 A flowchart of an operator scheduling method provided in this application;

[0065] Figure 4 This application provides an interactive diagram of computation operators and communication operators in a distributed system.

[0066] Figure 5 This application provides a schematic diagram of a process for determining a unified scheduling template based on an operation description and integrating computation and communication based on the unified scheduling template in a distributed system.

[0067] Figure 6 A schematic diagram illustrating the parameter optimization results of a unified scheduling template provided in this application;

[0068] Figures 7A to 7F A schematic diagram illustrating the generation of a fused kernel provided in this application;

[0069] Figure 8 A schematic diagram of the structure of an operator scheduling device provided in this application;

[0070] Figure 9 A hardware structure diagram of a computing card provided in this application;

[0071] Figure 10 This application provides a schematic diagram of the structure of a computing cluster;

[0072] Figure 11 This is a schematic diagram of another computing cluster structure provided in this application. Detailed Implementation

[0073] The terms "first" and "second" used in the embodiments of this application are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first" and "second" may explicitly or implicitly include one or more of that feature.

[0074] First, some technical terms involved in the embodiments of this application will be introduced.

[0075] An operator (OP) refers to an operation performed on data. Based on the type of operation, operators can be divided into computation operators (such as the 2D convolution operator Conv2D and the max pooling operator MaxPool) and communication operators (such as the collection operator AllGather and the broadcast operator Broadcast). With the rise of new computing paradigms such as artificial intelligence (AI) computing and high-performance computing (HPC), to meet the computing power demands of AI computing or HPC computing tasks, computing tasks can be distributed across multiple computing cards for distributed computing. Multiple computing cards can communicate with each other using aggregated communication operators to improve communication efficiency.

[0076] Collective communication operators are communication operators that implement collective communications. Collective communications is a global communication operation in which all processes in a process group participate. The execution of a collective communication operator depends on the communication domain. The communication domain is the context in which the collective communication operator executes, managing the corresponding communication objects and the resources required for communication. The communication objects can be computing cards, including but not limited to neural processing units (NPUs), graphics processing units (GPUs), tensor processing units (TPUs), and other xPUs. A computing card (such as an NPU) can be a communication object. Each communication object in the communication domain is called a rank, and each rank can be assigned a unique identifier between 0 and n-1 (where n is the number of computing cards in the group).

[0077] Communication operators, such as AllReduce and ReduceScatter, account for a significant proportion of the total execution time. For example, during AI model training or inference, the execution time of AllReduce and ReduceScatter can reach 10% to 50% of the total execution time.

[0078] Parallel execution of computation and communication can achieve computation and communication masking, thus partially overlapping computation and communication times. This masking of some time overhead reduces overall execution time and improves performance. Computation and communication masking can be divided into masking without data dependencies and masking with data dependencies. Masking without data dependencies schedules computation operators and communication operators without data dependencies separately in the computation stream and communication stream, achieving parallel execution of computation and communication operators. This masking mode is relatively simple. However, many computation and communication operators have data dependencies and cannot be directly parallelized.

[0079] Taking model training as an example, to achieve large-scale parallel training of AI models, tensor parallelism (TP) can be used to partition the layers of the AI ​​model, distributing the computation of a certain layer to different computing cards, and then obtaining the complete computation result through ensemble communication. For example, the tensor parallel process may involve AllGather and General Matrix to Matrix Multiplication (GEMM), or it may involve GEMM and ReduceScatter. Among them, AllGather and GEMM have data dependencies, and GEMM and ReduceScatter have data dependencies, so they cannot be directly parallelized.

[0080] By breaking down large computational and communication operations into smaller tasks, more opportunities for parallel computation and communication can be created. Figure 1 As shown, a typical approach is to coarsely decompose large operations of computation and communication operators into multiple kernels. A kernel, also called an operator kernel, is the base class or kernel function for the operator implementation, typically a block of code that executes a specific computational task on the underlying hardware. Kernels executing on different hardware can often be optimized for the execution of operators on that hardware based on its characteristics. Taking the kernels executing computational operators on GPUs and CPUs (also called GPU kernels and CPU kernels) as examples, GPU kernels typically fully utilize their large number of threads and parallel computing architecture, employing multi-threaded parallelism to implement operator computation, improving computational efficiency through proper thread allocation and shared memory utilization. CPU kernels, on the other hand, focus more on utilizing multi-core features and caching mechanisms, accelerating computation through optimized loop structures and data prefetching. Input, output, workspace, and tilling parameters can be defined within a kernel. Computational kernels and communication kernels without data dependencies can execute in parallel, achieving coarse-grained computation-communication masking.

[0081] It's important to note that dividing a kernel into multiple kernels introduces significant kernel launch overhead. Furthermore, kernel-based splitting schemes can only break down larger kernels into smaller ones along one dimension, which can compromise data locality and impact computational efficiency. Additionally, operator splitting at the kernel level cannot perform fine-grained masking, resulting in low masking accuracy.

[0082] To improve coverage, shorten overall execution time, and increase execution efficiency, large operations can be manually broken down into fine-grained pipelining parallel instructions, achieving computation-communication fusion. This computation-communication fusion kernel can be simply referred to as the fusion kernel. The fusion kernel offers high coverage and further reduces overall execution time. However, manually implementing a fusion kernel requires significant human effort, has a high development threshold and low efficiency, and is difficult to migrate to other distributed systems.

[0083] To address the limitation of related technologies in achieving automatic fine-grained fusion of computation and communication operators, this application provides an operator scheduling method. This method can be executed by an operator scheduling device. The operator scheduling device is used to automatically fuse computation and communication operators in the computation and communication subgraphs obtained by partitioning the computation graph of a distributed computing task, thereby achieving unified scheduling of computation and communication operators. It should be noted that the operator scheduling device can be applied to distributed computing scenarios such as model training or model inference, or to HPC scenarios; this application does not impose any limitations on this application.

[0084] The computation and communication subgraph can be a subgraph that partitions a computation graph to distribute execution across multiple computation cards. A computation graph is a graph structure used to represent mathematical computations or program execution. Operators are represented by nodes in a computation graph, and dependencies between operators are represented by directed edges between nodes. The computation and communication subgraph includes at least one computation operator and at least one communication operator.

[0085] In some examples, the operator scheduler can be software that performs fine-grained fusion of computational and communication operators to achieve unified scheduling. For instance, the operator scheduler can be an AI compiler or an AI executor. The AI ​​compiler can include, but is not limited to, a graph compiler or operator compiler (also known as a tensor compiler or tensor boost engine, TBE). The AI ​​executor can include, but is not limited to, a graph executor or an operator executor. In other examples, the operator scheduler can be hardware that deploys the aforementioned software, such as a GPU, NPU, or TPU computing card, or a computing device.

[0086] In practical implementation, the operator scheduling device can obtain the computational operation descriptions of computational operators and the communication operation descriptions of communication operators. The computational and communication operators are operators in the computational and communication subgraphs obtained by partitioning the computation graph. Based on the computational and communication operation descriptions, the operator scheduling device can obtain a unified scheduling template for the computational and communication operators. This unified scheduling template uses a unified scheduling primitive for both computational and communication operations. This primitive controls or optimizes the execution of computational tasks and defines how to efficiently execute computational tasks on target hardware. The target hardware can be a computing card or computing device in a distributed system. Based on the unified scheduling template, the operator scheduling device can generate a fused kernel for computational and communication operators (also called a computational-communication fused kernel or fused kernel).

[0087] A converged kernel comprises kernel functions that use basic blocks, which unify computational and communication operations, as scheduling units to enable parallel computation and communication operations. The granularity of these basic blocks can vary depending on the hardware type; for example, GPU basic blocks can be at the thread block level, while NPU basic blocks can be at the tensor / AI core block level.

[0088] This method employs a unified scheduling primitive to generate a unified scheduling template corresponding to the operation descriptions of computational and communication operations. Then, based on this unified scheduling template, it automatically generates a fused kernel that uses basic blocks—which unify the scheduling of computational and communication operations—as scheduling units. This enables parallel computational and communication operations. The fused kernel executes computational and communication operations at the basic block level, achieving fine-grained computational-communication fusion, significantly improving masking, shortening total execution time, and thus improving overall computational efficiency and performance. This solves the problems of low masking, low computational efficiency, and poor performance faced by coarse-grained kernel-based fusion. Furthermore, the fused kernel implements computational and communication operations using a single kernel function, resolving the issue of significant kernel launch overhead in coarse-grained kernel-based fusion. Simultaneously, by introducing a unified scheduling template and generating the fused kernel based on it, this method avoids the high development difficulty and low portability associated with manually developing computational-communication fusion kernels.

[0089] To make the technical solution of this application clearer and easier to understand, the system architecture of this application will be described below with reference to the accompanying drawings.

[0090] See Figure 2The diagram shows a hardware and software architecture of a computing system. The operator scheduling device can be software in the software layer of the computing system 20, such as a graph compiler, tensor acceleration engine or graph executor, operator executor, or a computing card that deploys the graph compiler, tensor acceleration engine or graph executor, operator executor. The layers of the computing system 20 are described in detail below.

[0091] The hardware layer of computing system 20 includes a computing card 202, which can be an NPU, GPU, or TPU, to provide computing resources. The software layer of computing system 20 includes a heterogeneous computing architecture 204 adapted to the hardware layer, such as a Compute Architecture for Neural Networks (CANN). CANN is an architecture specifically designed and optimized for high-performance neural network computing needs. CANN includes full-stack software support, aiming to provide a powerful software stack for managing AI models, computational flows, and data flows to support the execution of AI models (such as neural network models) on computing card 202.

[0092] For ease of understanding, Figure 2 The heterogeneous computing architecture 204 is used as an example to illustrate CANN. The heterogeneous computing architecture 204 includes the following layers from top to bottom: computing language layer 2042, computing service layer 2044, computing compilation layer 2046, computing execution layer 2048, and computing infrastructure layer 2049.

[0093] The Computing Language Layer 2042 provides operator development interfaces. These interfaces are open programming frameworks that encapsulate the underlying computing service interfaces. The Computing Language Layer 2042 offers application programming interface (API) libraries for device management, context management, stream management, memory management, model loading and execution, operator loading and execution, media data processing, and graph management, enabling users to develop AI applications.

[0094] The Computing Service Layer 2044 provides operator libraries, such as a general neural network (NN) library and a basic linear algebra subprograms (BLAS) library, enabling accelerated computation based on high-performance operators within these libraries. The Computing Service Layer 2044 also provides an Optimization Engine (OE) to improve end-to-end model execution speed through operator autotuning (OPAT), subgraph autotuning (SGAT), gradient autotuning (GDAT), and a model compression toolkit (MCT). It also provides an AI framework adapter for compatibility with different AI frameworks.

[0095] The Computing Compilation Layer 2046 includes a graph compiler, which is specifically used to compile the intermediate representation (IR) obtained from the user-input AI model through model transformation into a hardware-executable model. IR is an intermediate form used to represent AI models (such as deep learning models), typically using a computation graph to represent the structure and computational process of the AI ​​model. The Computing Compilation Layer 2046 also includes a Tensor Boost Engine (TBE), which, through its automatic scheduling mechanism, can efficiently compile operators.

[0096] The Computing Execution Layer 2048 is used to execute AI models or operators, providing functional units such as a runtime library, a graph executor, Digital Vision Pre-Processing (DVPP), Artificial Intelligence Pre-Processing (AIPP), and a Collective Communication Library. It should be noted that... Figure 2 Only some functional units of the computation execution layer 2048 are shown. The computation execution layer 2048 may also include other functional units. For example, the computation execution layer 2048 may also include an operator executor. This application does not limit this.

[0097] The Computing Base Layer 2049 provides basic services for the layers above it, such as Shared Virtual Memory (SVM), Virtual Machine (VM), and Host Device Communication (HDC).

[0098] Furthermore, the software layer of the computing system 20 may also include an AI application layer, which includes AI applications 206 developed based on the heterogeneous computing architecture 204. These AI applications can be AI model training applications or AI model inference applications. When the AI ​​model training application or AI model inference application runs, the graph compiler, tensor acceleration engine, or graph executor in the heterogeneous computing architecture 204 can execute the operator scheduling method of this application to finely fuse the computational operators and communication operators in the computational and communication subgraphs obtained by partitioning the computational graph, thus obtaining a fused kernel. By executing this fused kernel, operator execution efficiency can be improved, kernel launch overhead can be reduced, and performance can be enhanced.

[0099] based on Figure 2 The computing system 20 shown in this application also provides an operator scheduling method. The operator scheduling method of this application will be described in detail below with reference to the accompanying drawings.

[0100] See Figure 3 The flowchart shown illustrates an operator scheduling method, which can be executed by an operator scheduling device. The operator scheduling device can be... Figure 2 The graph compiler, tensor acceleration engine, or graph executor, or operator scheduling device in the computing system 20 shown can be integrated into the aforementioned graph compiler, tensor acceleration engine, or graph executor. The method specifically includes the following steps:

[0101] S302, The operator scheduling device obtains the computation operation description of the computation operator and the communication operation description of the communication operator.

[0102] Computational and communication operators are operators within the computational and communication subgraphs obtained by partitioning the computation graph of a distributed computing task. A distributed computing task is a computational task that can be decomposed into multiple subtasks, executed in parallel on multiple computing nodes (e.g., computing cards or computing devices), and then the execution results from multiple computing nodes are aggregated to obtain the final result. Distributed computing tasks include, but are not limited to, AI computing tasks or HPC computing tasks. AI computing tasks can include model training tasks or model inference tasks.

[0103] A computation graph is a graph structure used to represent mathematical computations or program execution. Nodes in a computation graph represent operators, and directed edges between nodes represent dependencies between operators. For ease of understanding, an AI computation task example is used to illustrate computation graphs. For model training tasks, the computation graph can be the computation graph of the AI ​​model used for training; for model inference tasks, the computation graph can be the computation graph of the AI ​​model used for inference in an AI application. AI models can be models built based on machine learning or deep learning algorithms. AI models can also be categorized into large models and small models based on their parameter scale. Large models can have hundreds of billions or even trillions of parameters; therefore, the training or inference of large models can typically be performed in a distributed system. A distributed system can include multiple computing cards. These computing cards usually need to communicate with each other. Therefore, the computation graph of an AI model can be decomposed into computation and communication subgraphs, thereby enabling distributed computation and communication across multiple computing cards.

[0104] The computation and communication subgraph includes at least one computation operator and at least one communication operator. For example, the computation and communication subgraph may include the matrix multiplication operator Matmul and the global reduction operator AllReduce. The following example uses an AI model to illustrate the transformer model. The transformer model includes a Multilayer Perceptron (MLP) layer, an attention layer, and a mapping layer. The MLP layer is used to increase and then decrease the dimensionality of the input tensor to enhance the expressive power of the transformer model. The attention layer employs a multi-head attention mechanism, splitting the input tensor into multiple sub-tensors. Each sub-tensor serves as the input to an attention head, and each attention head independently performs self-attention computation, obtaining its own output. The outputs of the attention heads can be concatenated and transformed linearly to obtain the final output. This effectively captures various relevant information from the input tensor. The mapping layer is used to convert the input text sequence into vectors that the AI ​​model can process by looking up a vocabulary, and then maps the processed vectors back to the vocabulary to obtain the word at each position. The MLP layer, attention layer, and mapping layer all involve forward computation, backward computation, and ensemble communication between different computation cards. Based on this, the computation graph of the transformer model can be divided into computation and communication subgraphs corresponding to the MLP layer, attention layer, and mapping layer, respectively. Taking the computation and communication subgraph corresponding to the MLP layer as an example, this subgraph can include matrix multiplication operators for forward computation, matrix multiplication operators for backward computation, and two AllReduce operations. Specifically, an AllReduce operation is performed after each forward computation and after each backward computation.

[0105] In practical implementation, the operator scheduling device can provide an interface to obtain computational operation descriptions of computational operators and communication operation descriptions of communication operators. Here, computational operations and communication operations can be collectively referred to as operations. The operation description can describe the input and output of the operation; in other words, the operation description can include both input and output descriptions. Furthermore, for computational operations, the operation description can also include a description of the computation method. For communication operations, the operation description can include the operation type of the communication operation.

[0106] Specifically, the operator scheduling device can obtain the input description, calculation method description, and output description of the computational operation of the computational operator through the first interface, and obtain the operation type of the communication operation of the communication operator through the second interface. The first interface and the second interface can be the same interface or different interfaces.

[0107] For ease of understanding, the descriptions of the Matmul and AllReduce operators in the computation and communication subgraphs are illustrated with examples.

[0108] In this example, the distributed system includes N computing cards. When describing it, the distributed system can be abstracted as a one-dimensional device mesh or a multi-dimensional device mesh, such as a 1-dimensional mesh. <n>Or a 2D mesh<N1×N2> (N1×N2=N). For example... Figure 4 As shown, computational operations in a distributed system can be described as follows: the first dimension (e.g., a column of input matrix A) of Matmul is processed along the Mesh. <n>The 0th dimension of the distribution, the 0th dimension of the input matrix B (e.g., the rows of the input matrix B) along the Mesh <n>In a distributed system with a zero-dimensional distribution, each computing card uses its local data to perform matrix multiplication in parallel. After the computation, each computing card displays a partial sum of the matrix multiplication results. For example, the result on computing card rank 0 is Cpsums0, the result on computing card rank 1 is Cpsums1, the result on computing card rank 2 is Cpsums2, and the result on computing card rank 3 is Cpsums3. Communication operations (data exchange operations) in a distributed system can be described as reducing the partial sums and results of all participating computing cards to obtain the complete result through a global reduction (AllReduce).

[0109] See Figure 5 The computational operation description of the computation operator and the communication operation description of the communication operator can be:

[0110] Mesh <4> mesh / / Represents a distributed system abstracted as a one-dimensional device mesh

[0111] Tensor<? x? xf32,mesh,[[],[0]]>A / / indicates that the first dimension of the input matrix A is along the Mesh <4> 0th dimension distribution

[0112] Tensor<? x256xf32,mesh,[[0],[]]>B / / indicates that the 0th dimension of the input matrix B follows the Mesh <4> 0th dimension distribution

[0113] / / Cpsums is of type Tensor<? x256xf32,mesh,[[],[]],partial=sum[0]>

[0114] Tensor Cpsums = Matmul{__producer}(A,B) / / Matmul is the producer; the output of Matmul is provided to allReduce.

[0115] / / The type of C is Tensor<?x256xf32,mesh,[[],[]]>

[0116] Tensor C=allReduce("sum",Cpsums).

[0117] It should be noted that the content after " / / " in the above operation description is a comment on the operation description.

[0118] S304. The operator scheduling device obtains a unified scheduling template for computation operators and communication operators based on the computation operation description and the communication operation description.

[0119] The unified scheduling template uses a unified scheduling primitive for both computational and communication operations. This unified scheduling primitive is also called the unified computation-communication scheduling primitive, or simply scheduling primitive. The unified computation-communication scheduling primitive includes those used to control or optimize the execution of distributed computing tasks.

[0120] In some possible implementations, the unified scheduling primitive for computation and communication includes at least one of the following: scheduling primitives for multidimensional computation loops, scheduling primitives for linearizing intermediate data storage, scheduling primitives for localizing intermediate data, and scheduling primitives for merging communication among multiple computation blocks. The scheduling primitives for multidimensional computation loops can be scheduling primitives from traditional single-card scheduling, such as loop tiling, loop reordering, splitting, and fuse.

[0121] Loop tiling, also known as loop slicing or nested loop tiling, is a scheduling primitive used to optimize nested loop structures. It divides a large loop iteration space into multiple smaller, relatively independent computational tiles, each containing a contiguous subset of iterations, thereby altering the loop's execution method and improving performance. Reorder is a scheduling primitive that changes the order of nested loops, adjusting the execution order of loops within a set of nested loops to optimize performance. Splitting is used to divide the iteration space of a loop, for example, splitting a complete loop axis (representing one dimension of the loop) into multiple sub-loop axes according to specific rules or factors, thus changing the loop structure for subsequent parallelization, vectorization, and other performance optimization operations. It's worth noting that splitting loop axes can change data access patterns, helping to enhance the spatial and temporal locality of data. For example, appropriately splitting a loop traversing a two-dimensional array makes access to array elements within each sub-loop more concentrated, better aligning with caching mechanisms, improving cache hit rate, reducing the number of data reads from main memory, and improving performance. The `fuse` function, the opposite of the `split` function, is used to merge multiple loop axes (e.g., adjacent loop axes) into a new loop axis. `fuse` can reduce the nesting level of loops, simplify the loop structure, and may also improve data access patterns and enhance data locality, thereby increasing cache hit rate. For example, for two highly related loops that operate on consecutive data blocks, merging their loop axes makes data access more compact, helping to retain related data in the cache and reducing the number of main memory accesses.

[0122] Scheduling primitives for merging communication between multiple computational blocks can also be scheduling primitives for merging communication between multiple computational tiles. In some examples, scheduling primitives for merging communication between multiple computational blocks may include `ipc_chunk`, which allows for fine-grained adjustment of the data size in a single communication session, ensuring communication efficiency. Scheduling primitives for localizing intermediate data may include `ipc_write`, which reduces the storage overhead and data transfer overhead of intermediate data between computation and communication. Scheduling primitives for linearizing intermediate data storage may include `ipc_linear`, which allows for linearized storage of intermediate data, such as intermediate computation results, laying the foundation for parallel computation and communication.

[0123] A unified scheduling template can be generated by filling in scheduling primitives corresponding to computation operation descriptions and scheduling primitives corresponding to communication operation descriptions. (Still using...) Figure 5 For example, the unified scheduling template for the Matmul and AllReduce operators can be:

[0124]

[0125] The unified scheduling template described above uses scheduling primitives such as createTiledSchedule, split, reorder, fuse, ipc_write, ipc_chunk, and ipc_doublebuffer. Here, ipc stands for Inter-Process Communication.

[0126] The scheduling logic of the above unified scheduling template can be: createTiledSchedule(C, tile = Tile) <m0xn0>) Tile the C tensor (of shape m×n)<m0×n0> The system splits the loop into two layers (m1, n1) to form a Tile-level scheduler S. Within the scheduler, S.split(m1, s) further splits the m1 loop dimension into two layers, m1_o and m1_i, where the loop boundary of m1_i is s, resulting in a three-layer loop (m1_o, m1_i, n1). Then, S.reorder reorders the loops to (m1_o, n1, m1_i), and S.loop_fuse merges these three loops into a single loop. S.ipc_write( `Cpsums` writes the intermediate data `Cpsums` calculated in each loop to linear, local storage as input for communication operations; `S.ipc_chunk(Cipc, commChunk)` expands the local storage space into `commChunk` computational chunks (tiles), enabling communication by merging these `commChunk` computational chunks (tiles); `S.ipc_doublebuffer(Cipc)` doubles the local storage space, implementing a double buffer, allowing for pipelined parallelism between computation and communication. Figure 6 As shown, computation and communication can be implemented in ping-pong pipelined parallelism using a double buffer. Specifically, ping-pong pipelined parallelism uses two parallel buffers: one for reading data and the other for writing data. When one buffer is full, the system switches to the other buffer for write operations, while the full buffer can be used for read operations.

[0127] It should be noted that the operator scheduling device can also update the parameters in the unified scheduling template to obtain an updated unified scheduling template. Specifically, the operator scheduling device can perform parameter tuning or parameter optimization on the unified scheduling template, thereby updating the parameters in the unified scheduling template. The parameters in the unified scheduling template may include, but are not limited to, at least one of the following: the number of computational blocks for merged communication, the matrix partitioning method, or the shape of the computational blocks for merged communication. The matrix partitioning method includes regular partitioning or irregular partitioning. Regular partitioning and irregular partitioning can be further subdivided into different subcategories. For example, regular partitioning may include partitioning by row or partitioning by column.

[0128] In this application, the operator scheduling device can obtain a unified scheduling template for computational and communication operators through various methods, and perform parameter tuning on the unified scheduling template. These methods are described below.

[0129] In one possible implementation, the operator scheduling device can obtain a unified scheduling template written by the user using scheduling primitives based on the operation description. In this case, the unified scheduling template is a handwritten scheduling template. Alternatively, the operator scheduling device can obtain unified scheduling templates for computation operators and communication operators from built-in scheduling templates based on the computation operation description and the communication operation description.

[0130] The operator scheduling device can obtain a unified scheduling template for computation and communication operators from a built-in scheduling template through semantic retrieval. Specifically, the operator scheduling device can perform semantic analysis on the descriptions of computational and communication operations respectively, obtaining semantic vectors for the computational and communication operation descriptions. Furthermore, the operator scheduling device can perform semantic analysis on the built-in scheduling template, obtaining the semantic vector of the scheduling template. The operator scheduling device determines the similarity between the semantic vectors of the computational and communication operation descriptions and the semantic vector of the scheduling template. Similarity can be represented by vector distance, such as Euclidean distance or cosine distance. Specifically, the operator scheduling device can calculate the similarity between the semantic vector of the computational operation description and the semantic vector of the scheduling template for each operation description, and can also assign weights to the computational and communication operation descriptions, obtaining the similarity between the semantic vector of the complete operation description and the semantic vector of the scheduling template through weighted calculation. Then, based on the similarity of the semantic vectors, the operator scheduling device can obtain the unified scheduling template for computation and communication operators from the built-in scheduling template. For example, the unified scheduling template for computation operators and communication operators can be the scheduling template with the highest similarity to the semantic vector of the operation description among the built-in scheduling templates, or the scheduling template with a similarity to the semantic vector of the operation description that is greater than the similarity threshold among the built-in scheduling templates.

[0131] It should be noted that the operator scheduling device can also combine handwritten scheduling templates with built-in scheduling templates to obtain a unified scheduling template for computational and communication operators. For example, the operator scheduling device can obtain an initial template from the built-in scheduling templates based on the computational and communication operation descriptions, and then receive feedback from the user on the initial template based on the computational or communication operation descriptions, such as modifications to the initial template, to obtain a unified scheduling template for computational and communication operators.

[0132] In other possible implementations, the operator scheduler can generate a unified scheduling template for computation and communication operators based on the computation operation description and the communication operation description, using machine learning. Machine learning can include, but is not limited to, genetic algorithms (GA) or reinforcement learning. The following example illustrates how the operator scheduler can automatically generate a unified scheduling template using reinforcement learning.

[0133] Specifically, the operator scheduling device can obtain a cost model through offline learning, and then perform a genetic search based on this cost model using a genetic algorithm to find a unified scheduling template for the fusion of computing and communication. Here, the genetic algorithm is part of evolutionary computation, a computational model that simulates Darwin's genetic selection and natural elimination process of biological evolution, and can search for the optimal solution by simulating the natural evolutionary process.

[0134] The unified scheduling template for computational and communication operators generated by the above method can include multiple templates. Based on this, the operator scheduling device can further optimize among the multiple templates to obtain the optimal template. The operator scheduling device can use the optimal template as the unified scheduling template for computational and communication operators.

[0135] Furthermore, the operator scheduler can update the parameters in multiple templates, for example, by fine-tuning the parameters in each template to obtain multiple updated templates. The operator scheduler can then determine the updated unified scheduling template based on the evaluation metrics of these updated templates. The evaluation metrics can be the performance of the fusion kernel generated based on the updated templates on the computing card. Performance can be characterized by metrics such as execution latency. In practice, the operator scheduler can simulate the execution of the fusion kernel to obtain evaluation metrics such as execution latency.

[0136] To facilitate understanding, an example will be used to illustrate this further. The operator scheduling device can optimize the number of computational blocks for merging communication across multiple templates and the matrix partitioning method, thereby determining the optimal template and optimal parameters. For example... Figure 6 As shown, the optimal parameters can be: the number of computational blocks for merging communication, i.e., the number of blocks for merging multiple computational blocks for communication is 4; and the matrix partitioning method is irregular partitioning. Irregular partitioning can involve dividing the matrix and then scanning it row-wise, partitioning it into 4 computational blocks at a time. Based on this, when performing matrix multiplication, the computational blocks in rows 1-3 and 2-1 can be computed in parallel; the computational blocks in rows 2-3 and 3-1 and 4-1-3 can be computed in parallel; and the computational blocks in rows 3-1 and 4-1-3 can be computed in parallel. This allows for fine-grained scheduling of computation and communication. Furthermore, the partitioning of the multidimensional matrix can be irregular, rather than dividing a large multidimensional matrix into smaller multidimensional matrices according to a specific dimension (such as by row or column). This allows for greater utilization of data locality, thus maintaining the efficiency of large multidimensional matrices when performing matrix multiplication and other computational operations.

[0137] It should be noted that the operator scheduling device can also determine the optimal template from multiple templates, and then perform parameter tuning on the optimal template to obtain an updated template. Parameter tuning methods can include, but are not limited to, grid search, iterative local search in the parameter configuration space, and Bayesian optimization.

[0138] The above examples illustrate the separate determination of the unified scheduling template and parameters. In some possible implementations, the operator scheduling device can simultaneously determine both the unified scheduling template and its parameters. Specifically, the operator scheduling device can automatically select the scheduling primitives for constructing the unified scheduling template and its parameters based on the computation operation description and the communication operation description, thereby obtaining the unified scheduling template for both the computation and communication operators. In this method, the scheduling primitives used to construct the unified scheduling template can also be used as parameters. The operator scheduling device performs parameter tuning on both this parameter and the parameters of the unified scheduling template to obtain the unified scheduling template for both the computation and communication operators.

[0139] S306. The operator scheduling device generates a fusion kernel of computational and communication operators based on a unified scheduling template.

[0140] A converged kernel, also known as a converged kernel, includes kernel functions that use basic blocks that unify computational and communication operations as scheduling units to achieve parallel computation and communication operations. These kernel functions can be low-level tensor intermediate representations or intermediate code; "low-level" refers to their level relative to high-level programming languages. Using an operator scheduler as an example of an operator compiler, the compiler can generate low-dimensional tensor intermediate representations or intermediate code for operators in different high-level programming languages, exhibiting good compatibility and usability. Kernel functions can be converted into code that executes computational and communication operations in parallel on the target hardware, achieving instruction-level computation and communication parallelism. For example, the operator scheduler can select an appropriate instruction set based on the characteristics of the target hardware, converting the tensor intermediate representation or intermediate code into machine code or assembly code. This machine code or assembly code can then be executed on the target hardware. It should be noted that during the target hardware compilation phase, the operator scheduler can also apply additional optimization techniques, such as register allocation and instruction scheduling, to improve the performance of the generated code.

[0141] Typically, computational and communication operators each have independent kernels. The fused kernel of this application essentially merges multiple independent computational and communication kernels into a single kernel. On one hand, the fused kernel hides latency and improves the utilization of computing resources by executing computational and communication operations in parallel. Figure 6 As the example illustrates, a fused kernel can achieve ping-pong pipelined parallelism of Matmul computation and AllReduce communication within a single kernel. Without fusion, AllReduce communication typically begins after Matmul computation.

[0142] Next, the process of generating a fusion kernel for computation and communication operators based on a unified scheduling template will be described in detail.

[0143] Specifically, the unified scheduling template can include multiple scheduling statements, each capable of implementing different functions. For the first scheduling statement among these multiple statements, the operator scheduling device generates a first code block based on the syntax rules corresponding to the scheduling primitives in the first scheduling statement. This first code block implements the functionality of the scheduling primitives in the first scheduling statement. Then, for the second scheduling statement among these multiple statements, the operator scheduling device modifies the first code block according to the syntax rules corresponding to the scheduling primitives in the second scheduling statement, obtaining a second code block. The fusion kernel for computation and communication operators can include this second code block.

[0144] In related operator scheduling methods, computation and communication scheduling are separated, which limits the segmentation of data blocks and makes it difficult to flexibly optimize the amount of computational and communication data. At the same time, the computation order is restricted, making it impossible to flexibly improve data locality. In the operator scheduling method of this application, computation and communication are scheduled, and scheduling is performed using basic blocks that unify computational and communication operations as scheduling units. Therefore, it can flexibly adjust the amount of computational data and the computation order. Moreover, communication can capture changes in the computation order, thus supporting fine-grained optimization of communication data volume (such as chunksize).

[0145] Based on this, the first code block may include code that implements computation and communication operations using basic blocks as scheduling units. These basic blocks may include, but are not limited to, thread basic blocks and tensor kernel basic blocks. Correspondingly, the operator scheduling device can modify the lines of code in the first code block related to computation order, the size of data in a single communication, data locality, or linearized storage, according to the syntax rules corresponding to the scheduling primitives in the second scheduling statement, to obtain the second code block.

[0146] The generation of the first code block can be achieved in several ways. These will be explained in detail below.

[0147] In some possible implementations, the operator scheduler can construct a prompt based on a first scheduling statement and generate a first code block corresponding to the first scheduling statement through prompt learning. The first code block conforms to the syntax rules corresponding to the scheduling primitives in the first scheduling statement. Specifically, for different scheduling primitives, the operator scheduler can provide prompt templates corresponding to those primitives. The operator scheduler can fill in the prompt templates based on the first scheduling statement to construct a prompt. Then, the operator scheduler can concatenate the first scheduling statement and the prompt, and input the concatenation result into a code generation model for prediction, thereby generating a first code block that conforms to the syntax rules corresponding to the scheduling primitives in the first scheduling statement.

[0148] In other possible implementations, the operator scheduling device can transform the first scheduling statement according to the syntax rules corresponding to the scheduling primitives in the first scheduling statement to obtain the first code block. For example, the operator scheduling device can search for a target rule template that matches the scheduling primitives in the first scheduling statement from multiple rule templates, and then fill the target rule template according to the first scheduling statement to obtain the first code block.

[0149] The above provides a detailed explanation of the generation of the first code block. For the specific implementation of modifying the first code block to obtain the second code block according to the syntax rules corresponding to the scheduling primitives in the second scheduling statement, please refer to the relevant content description of generating the first code block, which will not be repeated here.

[0150] To facilitate understanding, the following will be combined with... Figure 5 Taking the unified scheduling template in the example, the process of generating the fusion kernel is explained in detail.

[0151] See Figures 7A to 7F The diagram illustrates the generation of a fused kernel. The operator scheduling device can obtain multiple scheduling statements from a unified scheduling template, and then, based on the first scheduling statement "TiledSchedule(S,m1,n1)=createTiledSchedule(C,tile=Tile <m0xn0>)”, the first code block shown as follows is generated through rule conversion:

[0152]

[0153] As shown in Figure 7A , the first code block is used for scheduling in the basic block as a scheduling unit, realizing parallel computing and communication operations.

[0154] Then, the operator scheduling device can modify the code line related to the computing order in the first code block according to the second scheduling statement "m1_o,m1_i=S.split(m1,s)m1_o,n1,m1_i=S.reorder(m1_o,m1_i,n1)" through rule conversion, and the modified code block is shown as follows:

[0155]

[0156] As shown in Figure 7B , the modified code block is Swizzle optimized by split and reorder scheduling primitives, so that the communication can simultaneously perceive the change of the computing order, ensuring the correctness of the order after the communication, and the fine-grained scheduling of the computing can be realized, reducing the negative impact on the computing efficiency.

[0157] As shown in Figure 7C , the operator scheduling device can also fuse multiple loops in the modified code block according to the second scheduling statement "idx=S.loop_fuse(m1_o,n1,m1_i)", and the fused code block is shown as follows:

[0158]

[0159] Then, the operator scheduling device modifies the code line related to the single communication data size in the fused code according to the second scheduling statement "i,j=S.split(idx,commChunk)Cipctmp=S.ipc_chunk(Cpsums,j,commChunk)", and the modified code block is shown as follows:

[0160]

[0161] As shown in Figure 7D , the single communication data size can be adjusted in fine granularity by ipc chunk, for example, the chunksize is adjusted to 4, ensuring the communication efficiency.

[0162] The operator scheduler can also modify the lines of code related to data localization through rule transformation based on the second scheduling statement "AL = S.cache_read(A)BL = S.cache_read(B)Cipc = S.ipc_write(Cpsums)", to obtain the following code block:

[0163]

[0164] like Figure 7E As shown, data localization through ipc_write can reduce the storage overhead of intermediate data between computation and communication, as well as the data transfer overhead.

[0165] The operator scheduler can also modify the lines of code related to pipelining parallelism based on the second scheduling statement "Cipc = S.double_buffer(Cipc)" to obtain the second code block, as shown below:

[0166]

[0167] like Figure 7F As shown, ping-pong pipelined parallelism of computation and communication can be achieved through double_buffer, thereby enabling fine-grained computation-communication pipeline masking.

[0168] It should be noted that the code block in the above example is illustrated using pseudocode. In actual applications, the fusion kernel generated by the operator scheduler can be compiled and executed on the target hardware.

[0169] Furthermore, the operator scheduling device can also measure the performance of the fusion kernel generated based on the unified scheduling template on an actual computing card and obtain measurement results. These measurement results can be fed back to the unified scheduling template generation stage, for example, by reusing it to learn the cost model, improving the accuracy of the cost model, and thus improving the quality of the unified scheduling template searched based on the cost model.

[0170] Considering that the network topology of a distributed system can change dynamically, the operator scheduling device can also detect the network topology. Specifically, the operator scheduling device can detect whether the network topology of the distributed system has changed based on the heartbeat messages between the computing cards in the distributed system. For example, if a computing card does not receive a heartbeat message from a neighboring computing card for m consecutive cycles, it indicates that a link failure (including endpoint failure) has occurred between that computing card and its neighboring computing cards, and the network topology of the distributed system has changed.

[0171] Accordingly, the operator scheduling device can re-execute S302 to S306 to regenerate the fused kernel. Specifically, the operator scheduling device can obtain the computation operation descriptions of the computation operators in the updated distributed system and the communication operation descriptions of the communication operators in the updated distributed system, and then obtain a unified scheduling template for the computation operators and communication operators based on the computation operation descriptions and communication operation descriptions. Next, the operator scheduling device can generate a fused kernel for the computation operators and communication operators based on the unified scheduling template.

[0172] As described above, the operator scheduling method of this application uses unified scheduling primitives for computational and communication operations to generate unified scheduling templates corresponding to the computational operation descriptions of computational operators and the communication operation descriptions of communication operators. Then, based on these unified scheduling templates, it automatically generates a computational-communication fusion kernel with unified basic blocks for computational and communication operations as the scheduling unit, enabling parallel computational and communication operations. This method can automatically achieve fine-grained computational-communication masking with high masking accuracy and without impacting computational performance. Furthermore, this method supports automatic generation of the fusion kernel, eliminating the need for significant manual labor. When the network topology of the distributed system changes, the fusion kernel can be quickly regenerated by updating the operation descriptions, resulting in high portability. In addition, the fusion kernel only needs to be started once, thus resolving the overhead of multiple kernel launches caused by kernel fusion.

[0173] Based on the aforementioned operator scheduling method, this application also provides an operator scheduling device. Next, the operator scheduling device will be described from the perspective of functional modularity.

[0174] See Figure 8 The diagram shows the structure of an operator scheduling device 800, which includes:

[0175] The operation description acquisition module 802 is used to acquire the computation operation description of the computation operator and the communication operation description of the communication operator, wherein the computation operator and the communication operator are operators in the computation and communication subgraphs obtained by partitioning the computation graph of the distributed computing task;

[0176] The template acquisition module 804 is used to acquire a unified scheduling template for the computation operator and the communication operator based on the computation operation description and the communication operation description. The unified scheduling template uses a unified scheduling primitive for the computation operation and the communication operation. The scheduling primitive is used to control or optimize the execution of the distributed computing task.

[0177] The fusion kernel generation module 806 is used to generate a fusion kernel for the computation operator and the communication operator according to the unified scheduling template. The fusion kernel includes a kernel function that uses a basic block that unifies the computation operation and the communication operation as a scheduling unit to implement the parallel operation of the computation operation and the communication operation.

[0178] The operation description acquisition module 802, template acquisition module 804, or fusion kernel generation module 806 can be either software modules or hardware modules. For a detailed implementation of the operation description acquisition module 802, please refer to [reference needed]. Figure 3 The description of S302 in the illustrated embodiment and the specific implementation of the template acquisition module 804 can be found in [reference needed]. Figure 3 The description of S304 in the illustrated embodiment, and the specific implementation of the fusion kernel generation module 806 can be found in [reference needed]. Figure 3 Description of S306 in the illustrated embodiment.

[0179] In some possible implementations, the template acquisition module 804 is further configured to:

[0180] Update the parameters in the unified scheduling template to obtain the updated unified scheduling template;

[0181] The fusion kernel generation module 806 is used for:

[0182] Based on the updated unified scheduling template, a fusion kernel for the computation operator and the communication operator is generated.

[0183] In some possible implementations, the unified scheduling template for the computation operator and the communication operator includes multiple templates;

[0184] The template acquisition module 804 is specifically used for:

[0185] The parameters in the multiple templates are updated to obtain multiple updated templates;

[0186] Based on the evaluation metrics of the multiple updated templates, the updated unified scheduling template is determined.

[0187] In some possible implementations, the parameters in the unified scheduling template include at least one of the following: the number of computational blocks for merged communication, the matrix partitioning method, or the shape of the computational blocks for merged communication, wherein the matrix partitioning method includes regular partitioning or irregular partitioning.

[0188] In some possible implementations, the template acquisition module 804 is specifically used for:

[0189] Obtain the unified scheduling template for the computation operator and the communication operator written by the user based on the computation operation description and the communication operation description; and / or;

[0190] Based on the computation operation description and the communication operation description, obtain a unified scheduling template for the computation operator and the communication operator from the built-in scheduling template; and / or;

[0191] Based on the computation operation description and the communication operation description, a unified scheduling template for the computation operator and the communication operator is generated through machine learning.

[0192] In some possible implementations, the template acquisition module 804 is specifically used for:

[0193] Based on the computation operation description and the communication operation description, the scheduling primitives for constructing the unified scheduling template and the parameters of the unified scheduling template are automatically selected to obtain the unified scheduling template for the computation operator and the communication operator.

[0194] In some possible implementations, the fusion kernel generation module 806 is specifically used for:

[0195] For the first scheduling statement among the plurality of scheduling statements, a first code block is generated according to the syntax rules corresponding to the scheduling primitive in the first scheduling statement. The first code block is used to implement the function of the scheduling primitive in the first scheduling statement.

[0196] For the second scheduling statement among the plurality of scheduling statements, the first code block is modified according to the syntax rules corresponding to the scheduling primitive in the second scheduling statement to obtain the second code block;

[0197] The fusion kernel of the computation operator and the communication operator includes the second code block.

[0198] In some possible implementations, the first code block includes code that implements the computational operation and the communication operation using a basic block that unifies the computational operation and the communication operation as a scheduling unit;

[0199] The fusion kernel generation module 806 is specifically used for:

[0200] Based on the syntax rules corresponding to the scheduling primitives in the second scheduling statement, modify the lines of code in the first code block that are related to the computation order, the size of data in a single communication, the localization of data, or the linearization of storage to obtain the second code block.

[0201] In some possible implementations, the operation description acquisition module 802 is specifically used for:

[0202] The input description, calculation method description, and output description of the computation operator are obtained through the first interface.

[0203] The operation type of the communication operator is obtained through the second interface.

[0204] In some possible implementations, the scheduling primitive includes at least one of the following: scheduling primitive for multidimensional computation loops, scheduling primitive for linearized storage of intermediate data, scheduling primitive for localization of intermediate data, and scheduling primitive for merging communication of multiple computation blocks.

[0205] This application also provides a computing card 202. For example, the computing card 202 can be an NPU, GPU, or TPU. The computing card 202 includes computing cores and memory. The computing core is a module in the computing card 202 used to implement computing capabilities. The computing cores of different types of computing cards 202 can be structurally different. For example, the computing core of an NPU can be an AI core, which includes matrix calculation units, vector calculation units, and scalar calculation units designed for neural networks. The computing core of a GPU includes multiple stream processors, which are used to process data operation tasks in parallel. The memory of the computing card 202 is also called device memory or video memory, and can typically include global memory. The computing core is used to execute computer-readable instructions loaded into the memory to execute the operator scheduling method of the foregoing embodiments.

[0206] To facilitate understanding, the hardware architecture of the computing card 202 will be illustrated below using the NPU as an example, along with accompanying diagrams. The NPU can be a single-core or multi-core architecture. For ease of description, a single-core architecture will be used as an example.

[0207] See Figure 9 The diagram shows a hardware architecture of a computing card 202, which includes an AI core 2022 and global memory 2024. The AI ​​core 2022, also known as the AI ​​Core, is the core of the computing card 202 and typically employs a Domain Specific Architecture (DSA) to adapt to common applications and algorithms in a specific domain. Global memory 2024 is used to store the input, intermediate, or output data during AI core computation.

[0208] The AI ​​Core 2022 is responsible for executing computationally intensive operators related to scalars, vectors, and tensors. The AI ​​Core 2022 includes several basic computational units: matrix (Cube) computation units, vector (Vector) computation units, and scalar (Scalar) computation units. These units perform different types of data computations. It should be noted that these different types of computational units form multiple independent execution pipelines; through unified scheduling and mutual cooperation, computational efficiency can be optimized.

[0209] Hardware architectures are categorized into coupled and separated architectures based on whether matrix computation units and vector computation units are deployed on the same core. This application uses a separated architecture as an example. In the separated architecture, the AI ​​core 2022 is split into a matrix computation core 2022A and a vector computation core 2022B. Matrix computation core 2022A is also called AI Cube (AIC), and vector computation core 2022B is also called AI Vector (AIV). Matrix computation core 2022A and vector computation core 2022B are independent of each other, each having its own scalar computation unit and capable of independently loading its own code, thus achieving decoupling between matrix and vector computations. Figure 9 As shown, data can be transferred between matrix computation core 2022A and vector computation core 2022B through global memory 2024.

[0210] The AI ​​Core 2022 also includes storage units (such as hardware storage and data transfer units) and control units. The AI ​​Core 2022 includes internal and external storage. Global memory 2024 can serve as external storage for the AI ​​Core 2022, also known as off-core storage. Memory storage can be buffers, including but not limited to L0 buffers, L1 buffers, and Unified Buffers (UB). The L0 buffer can be further divided into L0A, L0B, and L0C. The AI ​​Core 2022 can load data from external storage into internal storage to complete corresponding computational tasks. It should be noted that in the separate architecture, the matrix computation core 2022A adds a bias table buffer (BT buffer) and a fixed pipe buffer (FP buffer) to the existing L0 and L1 buffers. The BT buffer stores the bias of the AI ​​model, and the FP buffer stores quantization parameters and activation parameters (such as ReLU parameters).

[0211] To support data transmission and handling within the AI ​​Core 2022, the AI ​​Core 2022 also includes a Bus Interface Unit (BIU), Memory Transfer Engine 1 (MTE1), Memory Transfer Engine 2 (MTE2), and Memory Transfer Engine 3 (MTE3). The BIU serves as the interface between the AI ​​Core and the bus; the MTEs are data transfer units that handle data transfer between different buffers. Figure 9 (Not shown in the image) is the interface between the AI ​​core 2022 and the bus. MTE is for data transfer, which completes the data transfer between different buffers.

[0212] In the discrete architecture, the matrix computation core 2022A can include 5 parallel execution units (transfer units and computation units) and 7 memory units. The 5 parallel execution units can be MTE1, MTE2, MTE3, and the matrix computation unit. The 7 memory units include the off-core global memory 2024 (off-core memory) and the L1 buffer, L0A, L0B, L0C, BT buffer, and FP buffer. The vector computation core 2022B can include 4 parallel execution units and 2 memory units. The 4 parallel execution units can be MTE2, MTE3, the vector computation unit, and the scalar computation unit. The 2 memory units include the global memory 2024 and the unified buffer.

[0213] The data flow for vector computation can be represented as follows: data is moved from global memory 2024 to a unified buffer; the vector computation unit reads data from the unified buffer to perform vector computation; and the computation result is then moved back to global memory 2024. Therefore, the data flow for vector computation can be represented as GM-UB-[Vector]-UB-GM. Similarly, the data flow for matrix computation can be represented as follows: data is moved from global memory 2024 to the L1 buffer, then from the L1 buffer to L0A / L0B; the matrix computation unit reads data from L0A / L0B to perform matrix computation; the computation result is then moved to L0C, and finally, via a fixed pipe, to either global memory 2024 or the L1 buffer. Therefore, the data flow for matrix computation can be represented as GM-L1-L0A / L0B-[Cube]-L0C-FixPipe-GM, or GM-L1-L0A / L0B-[Cube]-L0C-FixPipe-L1.

[0214] It should be noted that the AI ​​Core 2022 may also include a control unit ( Figure 9 (Not shown in the diagram). The control unit includes at least one of the following: System Control, Instruction Dispatch, Cube Queue, Vector Queue, and Memory Transformation Queue. The System Control module is responsible for directing and coordinating the overall operation mode of the AI ​​Core 2022, configuring parameters, and implementing power consumption control. When instructions are sequentially dispatched through the Instruction Dispatch module, they are sent to the Cube Queue, Vector Queue, and Memory Transformation Queue, respectively, depending on their type. Thus, the Cube computation unit, Vector computation unit, and Memory Transformation Engine can execute corresponding tasks based on the instructions in their respective queues.

[0215] based on Figure 9 The present application also provides a computing cluster, in addition to the computing card 202 shown. The computing cluster includes multiple computing cards, which can be connected via a bus or network. The architecture of the computing cluster is described below with reference to the accompanying drawings.

[0216] First, see Figure 10 The diagram illustrates a computing cluster architecture, which can be a single-machine, multi-GPU architecture. In this architecture, the computing cluster includes a CPU 11 on the host side and multiple computing cards 202 on the device side. Further, the host side may also include host memory 12. The CPU 11, host memory 12, and multiple computing cards 202 can be connected via a bus. In some examples, the CPU 11 can partition the computing graph of distributed computing tasks loaded into the host memory 12 into computing and communication subgraphs, and then schedule these subgraphs to the computing cards 202. The computing cards 202 can run an operator scheduling device, such as a graph compiler, to automatically and finely fuse the computing and communication operators in the operator and communication subgraphs.

[0217] Specifically, the computing card 202 can obtain the computational operation descriptions of the computing operators and the communication operation descriptions of the communication operators. Then, based on these descriptions, it obtains a unified scheduling template for the computing and communication operators. This template uses unified scheduling primitives for both computational and communication operations, which are used to control or optimize the execution of distributed computing tasks. Next, the computing card can generate a fusion kernel for both computational and communication operators based on the unified scheduling template. This fusion kernel includes kernel functions that implement parallel computational and communication operations, using basic blocks that unify the computational and communication operations as scheduling units.

[0218] Secondly, see Figure 11 The diagram illustrates another computing cluster architecture. This computing cluster can be a multi-machine, multi-GPU architecture, where it can include multiple computing devices. Each computing device includes a CPU 11 on the host side and multiple computing GPUs 202 on the device side. The host side may also include host memory 12. Computing GPUs 202 within the same computing device can be connected via a bus. Computing GPUs 202 in different computing devices can be connected via a network. The CPU 11 on the host side can partition the computation graph of distributed computing tasks into computation and communication subgraphs, and then schedule these subgraphs to the computing GPUs 202. The computing GPUs 202 can automatically and finely fuse the computation and communication operators in the operator and communication subgraphs.

[0219] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct a computing card or computing cluster to execute the aforementioned operator scheduling method.

[0220] This application also provides a computer program product containing instructions. The computer program product may be software or program products containing instructions, capable of running on a computing card or stored on any usable medium. When the computer program product runs on at least one computing card, it causes the at least one computing card to execute the above-described operator scheduling method.

[0221] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention. < / n> < / n> < / n>

Claims

1. An operator scheduling method, characterized in that, The method includes: Obtain the computation operation description of the computation operator and the communication operation description of the communication operator, wherein the computation operator and the communication operator are operators in the computation and communication subgraphs obtained by partitioning the computation graph of the distributed computing task; Based on the description of the computation operation and the description of the communication operation, a unified scheduling template for the computation operator and the communication operator is obtained. The unified scheduling template uses a unified scheduling primitive for the computation operation and the communication operation. The scheduling primitive is used to control or optimize the execution of the distributed computing task. Based on the unified scheduling template, a fusion kernel for the computation operator and the communication operator is generated. The fusion kernel includes a kernel function that uses a basic block that unifies the computation operation and the communication operation as a scheduling unit to implement the parallel operation of the computation operation and the communication operation.

2. The method according to claim 1, characterized in that, The method further includes: Update the parameters in the unified scheduling template to obtain the updated unified scheduling template; The step of generating the kernel for the computation operator and the communication operator based on the unified scheduling template includes: Based on the updated unified scheduling template, a fusion kernel for the computation operator and the communication operator is generated.

3. The method according to claim 2, characterized in that, The unified scheduling template for the computation operator and the communication operator includes multiple templates; Updating the parameters in the unified scheduling template to obtain the updated unified scheduling template includes: The parameters in the multiple templates are updated to obtain multiple updated templates; Based on the evaluation metrics of the multiple updated templates, the updated unified scheduling template is determined.

4. The method according to claim 2 or 3, characterized in that, The parameters in the unified scheduling template include at least one of the following: the number of computational blocks for merged communication, the matrix partitioning method, or the shape of the computational blocks for merged communication. The matrix partitioning method includes regular partitioning or irregular partitioning.

5. The method according to any one of claims 1 to 4, characterized in that, The step of obtaining a unified scheduling template for the computation operator and the communication operator based on the computation operation description and the communication operation description includes: Obtain the unified scheduling template for the computation operator and the communication operator written by the user based on the computation operation description and the communication operation description; and / or; Based on the computation operation description and the communication operation description, obtain a unified scheduling template for the computation operator and the communication operator from the built-in scheduling template; and / or; Based on the computation operation description and the communication operation description, a unified scheduling template for the computation operator and the communication operator is generated through machine learning.

6. The method according to claim 1, characterized in that, The step of obtaining a unified scheduling template for the computation operator and the communication operator based on the computation operation description and the communication operation description includes: Based on the computation operation description and the communication operation description, the scheduling primitives for constructing the unified scheduling template and the parameters of the unified scheduling template are automatically selected to obtain the unified scheduling template for the computation operator and the communication operator.

7. The method according to any one of claims 1 to 6, characterized in that, The unified scheduling template includes multiple scheduling statements. The step of generating the fusion kernel of the computation operator and the communication operator based on the unified scheduling template includes: For the first scheduling statement among the plurality of scheduling statements, a first code block is generated according to the syntax rules corresponding to the scheduling primitive in the first scheduling statement. The first code block is used to implement the function of the scheduling primitive in the first scheduling statement. For the second scheduling statement among the plurality of scheduling statements, the first code block is modified according to the syntax rules corresponding to the scheduling primitive in the second scheduling statement to obtain the second code block; The fusion kernel of the computation operator and the communication operator includes the second code block.

8. The method according to claim 7, characterized in that, The first code block includes code that implements the computational operation and the communication operation using a unified basic block as the scheduling unit; The step of modifying the first code block to obtain the second code block according to the syntax rules corresponding to the scheduling primitives in the second scheduling statement includes: Based on the syntax rules corresponding to the scheduling primitives in the second scheduling statement, modify the lines of code in the first code block that are related to the computation order, the size of data in a single communication, the localization of data, or the linearization of storage to obtain the second code block.

9. The method according to any one of claims 1 to 8, characterized in that, The acquisition of computational operation descriptions of computational operators and communication operation descriptions of communication operators includes: The input description, calculation method description, and output description of the computation operator are obtained through the first interface. The operation type of the communication operator is obtained through the second interface.

10. The method according to any one of claims 1 to 9, characterized in that, The scheduling primitives include at least one of the following: scheduling primitives for multidimensional computation cycles, scheduling primitives for linearized storage of intermediate data, scheduling primitives for localization of intermediate data, and scheduling primitives for merging and communicating multiple computation blocks.

11. An operator scheduling device, characterized in that, The device includes: The operation description acquisition module is used to acquire the computation operation description of the computation operator and the communication operation description of the communication operator, wherein the computation operator and the communication operator are operators in the computation and communication subgraphs obtained by partitioning the computation graph of the distributed computing task; The template acquisition module is used to acquire a unified scheduling template for the computation operator and the communication operator based on the computation operation description and the communication operation description. The unified scheduling template uses a unified scheduling primitive for the computation operation and the communication operation. The scheduling primitive is used to control or optimize the execution of the distributed computing task. A fusion kernel generation module is used to generate a fusion kernel for the computation operator and the communication operator based on the unified scheduling template. The fusion kernel includes a kernel function that uses a basic block that unifies the computation operation and the communication operation as a scheduling unit to implement the parallel operation of the computation operation and the communication operation.

12. The apparatus according to claim 11, characterized in that, The template acquisition module is also used for: Update the parameters in the unified scheduling template to obtain the updated unified scheduling template; The fusion kernel generation module is used for: Based on the updated unified scheduling template, a fusion kernel for the computation operator and the communication operator is generated.

13. The apparatus according to claim 12, characterized in that, The unified scheduling template for the computation operator and the communication operator includes multiple templates; The template acquisition module is specifically used for: The parameters in the multiple templates are updated to obtain multiple updated templates; Based on the evaluation metrics of the multiple updated templates, the updated unified scheduling template is determined.

14. The apparatus according to claim 12 or 13, characterized in that, The parameters in the unified scheduling template include at least one of the following: the number of computational blocks for merged communication, the matrix partitioning method, or the shape of the computational blocks for merged communication. The matrix partitioning method includes regular partitioning or irregular partitioning.

15. The apparatus according to any one of claims 11 to 14, characterized in that, The template acquisition module is specifically used for: Obtain the unified scheduling template for the computation operator and the communication operator written by the user based on the computation operation description and the communication operation description; and / or; Based on the computation operation description and the communication operation description, obtain the unified scheduling template for the computation operator and the communication operator from the built-in scheduling template; and / or; Based on the computation operation description and the communication operation description, a unified scheduling template for the computation operator and the communication operator is generated through machine learning.

16. The apparatus according to claim 11, characterized in that, The template acquisition module is specifically used for: Based on the computation operation description and the communication operation description, the scheduling primitives for constructing the unified scheduling template and the parameters of the unified scheduling template are automatically selected to obtain the unified scheduling template for the computation operator and the communication operator.

17. The apparatus according to any one of claims 11 to 16, characterized in that, The fusion kernel generation module is specifically used for: For the first scheduling statement among the plurality of scheduling statements, a first code block is generated according to the syntax rules corresponding to the scheduling primitive in the first scheduling statement. The first code block is used to implement the function of the scheduling primitive in the first scheduling statement. For the second scheduling statement among the plurality of scheduling statements, the first code block is modified according to the syntax rules corresponding to the scheduling primitive in the second scheduling statement to obtain the second code block; The fusion kernel of the computation operator and the communication operator includes the second code block.

18. The apparatus according to claim 17, characterized in that, The first code block includes code that implements the computational operation and the communication operation using a unified basic block as the scheduling unit; The fusion kernel generation module is specifically used for: Based on the syntax rules corresponding to the scheduling primitives in the second scheduling statement, modify the lines of code in the first code block that are related to the computation order, the size of data in a single communication, the localization of data, or the linearization of storage to obtain the second code block.

19. The apparatus according to any one of claims 11 to 18, characterized in that, The operation description acquisition module is specifically used for: The input description, calculation method description, and output description of the computation operator are obtained through the first interface. The operation type of the communication operator is obtained through the second interface.

20. The apparatus according to any one of claims 11 to 19, characterized in that, The scheduling primitives include at least one of the following: scheduling primitives for multidimensional computation cycles, scheduling primitives for linearized storage of intermediate data, scheduling primitives for localization of intermediate data, and scheduling primitives for merging and communicating multiple computation blocks.

21. A computing card, characterized in that, The computing card includes a computing core and memory, the computing core being used to execute computer-readable instructions loaded into the memory to perform the operator scheduling method as described in any one of claims 1 to 10.

22. A computing cluster, characterized in that, The computing cluster includes multiple computing cards as described in claim 18, which are connected to each other via a bus or network.

23. A computer-readable storage medium, characterized in that, Includes computer-readable instructions; the computer-readable instructions are used to implement the operator scheduling method according to any one of claims 1 to 10.

24. A computer program product, characterized in that, Includes computer-readable instructions; the computer-readable instructions are used to implement the operator scheduling method according to any one of claims 1 to 10.

Images