Ai task processing method, apparatus, and computing device cluster

Abstract

An AI task processing method, comprising: acquiring cluster topological information of an AI accelerator cluster, model attribute parameters of a first model, and an AI task distributed parallel policy, the first model being used for executing a first AI task, and the cluster topological information comprising topological structures of the AI accelerator cluster and the states of AI accelerators in the AI accelerator cluster (S801); on the basis of the cluster topological information, the model attribute parameters and the AI task distributed parallel policy, searching the AI accelerator cluster for a first topological structure, the first topological structure being a topological structure of an AI accelerator network which is capable of executing the first AI task and involves communication time shorter than that of at least one second topological structure, and the second topological structure being an available topological structure other than the first topological structure in the AI accelerator cluster (S802); and, on the basis of the AI task distributed parallel policy, deploying the first model on the first topological structure so as to execute the first AI task (S803).

Description

An AI task processing method, device and computing device cluster

[0001] The present application claims priority to the Chinese patent application No. 2024113524808, filed on September 26, 2024, and entitled "An AI task processing method, device and computing device cluster", the entire contents of which are incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the field of artificial intelligence (AI) technology, and in particular to an AI task processing method, device and computing device cluster. BACKGROUND

[0003] An AI accelerator cluster is a high-performance computing infrastructure, which mainly consists of multiple AI accelerators (such as graphics processing units (GPUs), tensor processing units (TPUs), neural processing units (NPUs), etc.). Different AI accelerators can be connected through high-speed networks to achieve efficient parallel computing and data transmission. Such a cluster is designed specifically for processing large-scale AI tasks (such as large language model training and inference tasks, etc.). In AI task processing, high efficiency is crucial. Therefore, how to improve the efficiency of AI accelerator clusters in processing AI tasks is a technical problem that needs to be solved at present. SUMMARY

[0004] The present application provides an AI task processing method, device, computing device cluster, computer storage medium and computer product, which can improve the efficiency of AI accelerator clusters in processing AI tasks.

[0005] In a first aspect, the present application provides an AI task processing method, comprising: obtaining cluster topology information of an AI accelerator cluster, model attribute parameters of a first model, and an AI task distributed parallel strategy, the first model being used to execute a first AI task, the cluster topology information comprising a topology structure of the AI accelerator cluster and a state of an AI accelerator in the AI accelerator cluster; searching, based on the cluster topology information, the model attribute parameters and the AI task distributed parallel strategy, to obtain a first topology structure from the AI accelerator cluster, wherein the first topology structure is an AI accelerator network topology structure that can execute the first AI task and has a communication time lower than at least one second topology structure, the second topology structure being an available topology structure in the AI accelerator cluster other than the first topology structure; and deploying, based on the AI task distributed parallel strategy, the first model on the first topology structure to execute the first AI task.

[0006] Thus, since the searched first topology structure is a topology structure of an AI accelerator network with short communication time and capable of executing the first AI task, after the model is deployed on the first topology structure, the task related to the model can be executed with small communication overhead, thereby improving the processing efficiency of the AI task. In addition, the first topology structure is a topology of a network composed of AI accelerators available in the AI accelerator cluster, and thus this AI task processing manner does not need to change the structure of the cluster topology information of the AI accelerator cluster, thereby reducing the hardware cost (for example, a configurable optical switch).

[0007] In a possible implementation, the first topology structure is searched from the AI accelerator cluster based on the cluster topology information, the model attribute parameter, and the AI task distributed parallel strategy, including: based on the AI task distributed parallel strategy and the first quantity in the model attribute parameter, combining the AI accelerators in the AI accelerator cluster to obtain N topology structures satisfying the AI task distributed parallel strategy and the first quantity, N≥1, wherein the first quantity is the number of AI accelerators required to execute the first AI task; based on the cluster topology information, eliminating topology structures not existing in the cluster topology information from the N topology structures to obtain M topology structures; and screening the first topology structure from the M topology structures. In this way, by combining the AI accelerators in the AI accelerator cluster, topology structures satisfying the requirements can be obtained, and by eliminating the unavailable topology structures, the topology structures satisfying the requirements and available can be screened, and then the first topology structure can be screened from the topology structures.

[0008] In a possible implementation, the first topology structure is screened from the M topology structures, including: calculating the communication time corresponding to each of the M topology structures to obtain M communication times; and taking the topology structure corresponding to the shortest communication time in the M communication times as the first topology structure. In this way, the topology structure with the shortest communication time can be screened, and thus the first AI task can be executed with the highest efficiency.

[0009] In a possible implementation, the transmission path of the data in the third topology structure of the M topology structures includes single-hop and multi-hop, and the third topology structure is any one of the M topology structures. At this time, the communication time corresponding to the third topology structure is calculated by: calculating the transmission time of the data in the third topology structure from the source node to the destination node through different transmission paths to obtain K transmission times; and taking the first transmission time in the K transmission times as the communication time corresponding to the third topology structure, where the first transmission time is lower than at least one transmission time in the K transmission times except the first transmission time. In this way, all possible transmission paths in a topology structure can be considered, so that not only the case of AI accelerator direct connection can be considered, but also the case of AI accelerator detour can be considered, so that the finally calculated communication time is consistent with the actual communication time of the topology structure.

[0010] In a possible implementation, before the first topology structure is selected from the M topology structures, the method further includes: calculating, based on the model attribute parameter and the AI task distributed parallel strategy, a workload Workload of a network layer in the first model in the AI accelerator cluster, where the Workload of the network layer is a collective communication data size corresponding to a collective communication operator included in the network layer, and the Workload of the network layer is used to calculate the communication time corresponding to the topology structure of the AI accelerator network in the AI accelerator cluster. Since the workload of the network layer in the model can represent the collective communication data size corresponding to each collective communication operator included in the network layer, the communication time calculated based on the workload can better reflect the actual time required for executing the first AI task.

[0011] In a possible implementation, the Workload of the network layer in the first model in the AI accelerator cluster is calculated based on the model attribute parameter and the AI task distributed parallel strategy, including: in a case where the AI task distributed parallel strategy is data parallel, the Workload of the weight gradient of the first network layer is calculated based on the model attribute parameter to obtain the Workload of the first network layer, where the first network layer is any one of the network layers in the first model.

[0012] In a possible implementation, the Workload of the network layer in the first model in the AI accelerator cluster is calculated based on the model attribute parameter and the AI task distributed parallel strategy, including: in a case where the AI task distributed parallel strategy is tensor parallel, the Workload of the forward propagation part of the first network layer and the Workload of the input gradient are calculated based on the model attribute parameter to obtain the Workload of the first network layer, where the first network layer is any one of the network layers in the first model.

[0013] In a possible implementation, based on the model attribute parameters and the AI task distributed parallel strategy, the Workload of the network layer in the first model in the AI accelerator cluster is calculated, including: in the case that the AI task distributed parallel strategy is tensor and data mixed parallel, based on the model attribute parameters, the Workload of the forward propagation part of the first network layer, the Workload of the weight gradient and the Workload of the input gradient are calculated to obtain the Workload of the first network layer, wherein the first network layer is any one of the network layers in the first model.

[0014] In a possible implementation, the transmission path of the data in the first topology structure includes: single-hop.

[0015] In a possible implementation, the transmission path of the data in the first topology structure further includes: multi-hop. In this way, both the case of AI accelerator direct connection and the case of AI accelerator detour can be considered.

[0016] In a possible implementation, the communication time corresponding to the first topology structure is calculated based on the transmission time of the data in the first topology structure from the source node to the destination node via different transmission paths. For example, the shortest or shorter transmission time can be taken as the communication time corresponding to the first topology structure.

[0017] In a possible implementation, the model attribute parameters include: parameters that determine the model structure and size, parameters related to the training data volume, and parameters related to the number of AI accelerators required for model training or inference.

[0018] In a second aspect, the present application provides an AI task processing apparatus, including: an obtaining module and a communication module. The obtaining module is configured to obtain the cluster topology information of the AI accelerator cluster, the model attribute parameters of the first model and the AI task distributed parallel strategy, the first model being used to execute a first AI task, and the cluster topology information including the topology structure of the AI accelerator cluster and the state of the AI accelerators in the AI accelerator cluster. The processing module is configured to search for a first topology structure from the AI accelerator cluster based on the cluster topology information, the model attribute parameters and the AI task distributed parallel strategy, wherein the first topology structure is capable of executing the first AI task, and the communication time of the AI accelerator network of the first topology structure is lower than that of at least one second topology structure, and the second topology structure is an available topology structure in the AI accelerator cluster except the first topology structure. The processing module is further configured to deploy the first model on the first topology structure based on the AI task distributed parallel strategy to execute the first AI task.

[0019] In a possible implementation, when the processing module searches for the first topology structure from the AI accelerator cluster based on the cluster topology information, the model attribute parameters and the AI task distributed parallel strategy, the processing module is specifically configured to: based on the AI task distributed parallel strategy and a first quantity in the model attribute parameters, combine the AI accelerators in the AI accelerator cluster to obtain N topology structures that satisfy the AI task distributed parallel strategy and the first quantity, N > 1, where the first quantity is the number of AI accelerators required to execute the first AI task; based on the cluster topology information, eliminate topology structures that do not exist in the cluster topology information from the N topology structures to obtain M topology structures; and select the first topology structure from the M topology structures.

[0020] In a possible implementation, when the processing module selects the first topology structure from the M topology structures, the processing module is specifically configured to: calculate a communication time corresponding to each of the M topology structures to obtain M communication times; and take a topology structure corresponding to a shortest communication time in the M communication times as the first topology structure.

[0021] In a possible implementation, a transmission path of data in a third topology structure in the M topology structures includes single-hop and multi-hop, and the third topology structure is any one of the M topology structures. At this time, when the processing module calculates the communication time corresponding to the third topology structure, the processing module is specifically configured to: calculate transmission times of data from a source node to a destination node in the third topology structure via different transmission paths to obtain K transmission times; and take a first transmission time in the K transmission times as the communication time corresponding to the third topology structure, where the first transmission time is lower than at least one transmission time other than the first transmission time in the K transmission times.

[0022] In a possible implementation, before the processing module selects the first topology structure from the M topology structures, the processing module is further configured to: based on the model attribute parameters and the AI task distributed parallel strategy, calculate a workload Workload of a network layer in the first model in the AI accelerator cluster, where the Workload of the network layer is a collective communication data size corresponding to a collective communication operator included in the network layer, and the Workload of the network layer is used to calculate a communication time corresponding to a topology structure of an AI accelerator network in the AI accelerator cluster.

[0023] In a possible implementation, when the processing module calculates the Workload of the network layer in the first model in the AI accelerator cluster based on the model attribute parameter and the AI task distributed parallel strategy, the processing module is specifically configured to: in a case where the AI task distributed parallel strategy is data parallel, calculate the Workload of the weight gradient of the first network layer based on the model attribute parameter to obtain the Workload of the first network layer, where the first network layer is any network layer in the first model.

[0024] In a possible implementation, when the processing module calculates the Workload of the network layer in the first model in the AI accelerator cluster based on the model attribute parameter and the AI task distributed parallel strategy, the processing module is specifically configured to: in a case where the AI task distributed parallel strategy is tensor parallel, calculate the Workload of the forward propagation part of the first network layer and the Workload of the input gradient based on the model attribute parameter to obtain the Workload of the first network layer, where the first network layer is any network layer in the first model.

[0025] In a possible implementation, when the processing module calculates the Workload of the network layer in the first model in the AI accelerator cluster based on the model attribute parameter and the AI task distributed parallel strategy, the processing module is specifically configured to: in a case where the AI task distributed parallel strategy is tensor and data mixed parallel, calculate the Workload of the forward propagation part of the first network layer, the Workload of the weight gradient, and the Workload of the input gradient based on the model attribute parameter to obtain the Workload of the first network layer, where the first network layer is any network layer in the first model.

[0026] In a possible implementation, the transmission path of the data in the first topology structure includes: single hop.

[0027] In a possible implementation, the transmission path of the data in the first topology structure further includes: multi-hop.

[0028] In a possible implementation, the communication time corresponding to the first topology structure is calculated based on the transmission time of the data in the first topology structure from the source node to the destination node via different transmission paths.

[0029] In a possible implementation, the model attribute parameter includes: a parameter that determines the model structure and size, a parameter related to the training data volume, and a parameter related to the number of AI accelerators required for model training or inference.

[0030] In a third aspect, the present application provides a computing device cluster, comprising at least one computing device, each computing device comprising a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster performs the method described in the first aspect or any possible implementation manner of the first aspect.

[0031] In a fourth aspect, the present application provides a computer readable storage medium, comprising computer program instructions, when the computer program instructions are executed by a computing device, the computing device performs the method described in the first aspect or any possible implementation manner of the first aspect; or when the computer program instructions are executed by a computing device cluster, the computing device cluster performs the method described in the first aspect or any possible implementation manner of the first aspect. Exemplarily, the computing device cluster can comprise one or more computing devices.

[0032] In a fifth aspect, the present application provides a computer program product comprising instructions, when the instructions are executed by a computing device, the computing device performs the method described in the first aspect or any possible implementation manner of the first aspect, or when the instructions are executed by a computing device cluster, the computing device cluster performs the method described in the first aspect or any possible implementation manner of the first aspect. Exemplarily, the computing device cluster can comprise one or more computing devices.

[0033] It can be understood that the beneficial effects of the above-mentioned second aspect to fifth aspect can refer to the related description in the first aspect, which will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0034] FIG. 1 is a schematic diagram of an application scenario provided by an embodiment of the present application;

[0035] FIG. 2 is a schematic diagram of an architecture of an AI task processing system provided by an embodiment of the present application;

[0036] FIG. 3 is a schematic diagram of a process of Workload of each network layer in a computing neural network model provided by an embodiment of the present application;

[0037] FIG. 4 is a schematic diagram of a process of searching a first topology provided by an embodiment of the present application;

[0038] FIG. 5 is a schematic diagram of direct connection and detour between AI accelerators provided by an embodiment of the present application;

[0039] FIG. 6 is a schematic diagram of core running steps of a topology searcher provided by an embodiment of the present application;

[0040] FIG. 7 is a schematic diagram of interaction between a tenant and a cloud computing platform provided by an embodiment of the present application;

[0041] FIG. 8 is a flow diagram of an AI task processing method according to an embodiment of the present application;

[0042] FIG. 9 is a structural diagram of an AI task processing apparatus according to an embodiment of the present application;

[0043] FIG. 10 is a structural diagram of a computing device according to an embodiment of the present application;

[0044] FIG. 11 is a structural diagram of a computing device cluster according to an embodiment of the present application;

[0045] FIG. 12 is a structural diagram of another computing device cluster according to an embodiment of the present application. DETAILED DESCRIPTION

[0046] The term "and / or" used herein is a description of an association relationship between associated objects, which means that there can be three relationships, for example, A and / or B can mean that there are three cases of A alone, A and B together, and B alone. The symbol " / " used herein represents an or relationship of associated objects, for example, A / B represents A or B.

[0047] The terms "first" and "second" and the like in the description and claims herein are used to distinguish different objects, and are not used to describe a specific order of the objects. For example, the first response message and the second response message are used to distinguish different response messages, and are not used to describe a specific order of the response messages.

[0048] In the embodiments of the present application, the words "exemplary" or "for example" are used to mean serving as an example, instance, or illustration. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of the present application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Rather, the use of the words "exemplary" or "for example" is intended to present relevant concepts in a concrete manner.

[0049] In the description of the embodiments of the present application, unless otherwise specified, "a plurality of" means two or more, for example, a plurality of processing units means two or more processing units, and the like; a plurality of elements means two or more elements, and the like.

[0050] First, the related technical terms involved in the technical solutions provided by the present application are introduced.

[0051] (1) Neural network model

[0052] A neural network model is a computational model inspired by the structure of the human brain, which processes information by mimicking the way neurons in the human brain connect and interact. In a neural network model, network layers are the basic units that make up the model, which is a hierarchical structure for processing data. Each network layer can be composed of multiple neurons, which are responsible for receiving input, performing calculations, and passing the results to the next layer. In addition, each network layer can be responsible for performing specific computational tasks. The network layers in a neural network model can include one or more of the following: input layer, hidden layer, output layer, fully connected layer, convolutional layer, pooling layer, self-attention layer, normalization layer, multilayer perceptron (MLP), etc. For example, a neural network model can be, but is not limited to, a deep neural network (DNN) model, a convolutional neural network (CNN) model, or a transformer model, etc.

[0053] (2) AI task distributed parallel strategy

[0054] The AI task distributed parallel strategy is a method for improving the efficiency of neural network model training and inference in AI training and inference tasks. The AI task distributed parallel strategy can include data parallelism, tensor parallelism, or hybrid parallelism. Among them, data parallelism is to replicate the neural network model to multiple AI accelerators and split the large-scale data set into multiple small batches, then perform model training or inference on multiple AI accelerators in parallel. Tensor parallelism is to distribute different network layers or tensors of a single neural network model to different AI accelerators, so that different parts of the neural network model can be calculated in parallel. Hybrid parallelism refers to combining data parallelism and tensor parallelism, splitting and distributing in two dimensions of data and model, to achieve more fine-grained parallelization.

[0055] (3) Model attribute parameters

[0056] Model attribute parameters are a series of parameters used to describe the characteristics, configurations and behaviors of a neural network model. Model attribute parameters include, but are not limited to, parameters that determine the structure and size of the model, parameters related to the amount of training data, and parameters related to the number of AI accelerators required for model training or inference. For example, when the architecture of a neural network model is a transformer architecture, the model attribute parameters can include: the number of attention layers in the neural network model, the number of heads per attention layer, the number of neurons (such as: hidden neurons, neurons in feedforward neural networks (FFN), etc.), the data type of the weights, the batch size, the sequence length, the splitting method when tensor parallelism, the number of AI accelerators required, the number of AI accelerators used for data parallelism, the number of AI accelerators used for tensor parallelism, etc. When the neural network model is a multi-layer perceptron, the model attribute parameters can include: the number of neurons, the data type of the weights, the batch size, the splitting method when tensor parallelism, the number of AI accelerators required, the number of AI accelerators used for data parallelism, the number of AI accelerators used for tensor parallelism, etc.

[0057] (4) Collective communication mode

[0058] Collective communication mode refers to the way in which data is exchanged between multiple processes or threads in parallel computing and network communication, in which all processes or threads participate in communication. This communication mode can be applied in a large-scale AI accelerator cluster of a computer.

[0059] (5) Collective communication operator

[0060] A collective communication operator is a specific operation or function that implements collective communication, such as: all-reduce operator, all-to-all operator, reduce-scatter operator, etc.

[0061] (6) Workload

[0062] The workload of a network layer refers to the collective communication data size corresponding to each collective communication operator included in the network layer. The collective communication data size refers to the amount of data sent or received by participating processes in a collective communication operation.

[0063] (7) Forwarding part

[0064] Forward propagation refers to the process of data passing through a neural network model from the input layer to the output layer through a series of hidden layers. The forward propagation of a certain network layer refers to the process of data processing in the specific layer during the forward propagation of the neural network model. For example, the workload of the forward propagation of a network layer can refer to the amount of work required to process or transmit the output data of the layer in order to prepare the input of the next layer after the forward propagation calculation of the network layer is completed.

[0065] (8) Input gradient

[0066] Input gradient refers to the gradient value calculated for the input data of the model during the backpropagation process of the neural network model. These gradient values can be used to measure the sensitivity of the loss function with respect to the input data. The input gradient of a certain network layer refers to the gradient value calculated for the input value of the specific layer during the backpropagation process of the neural network model. These gradient values reflect the sensitivity of the loss function with respect to the input of the specific layer. For example, the workload of the input gradient of a network layer can refer to the amount of work required to process or transmit the gradient data in order to pass the gradient information to the previous layer (i.e., the "next layer" with respect to the current layer) after the backpropagation calculation of the network layer is completed.

[0067] (9) Weight gradient

[0068] Weight gradient refers to the gradient of the loss function with respect to the weights and bias parameters of the model during the backpropagation process of the neural network model. The weight gradient of a certain network layer refers to the gradient calculated for the weights and bias parameters of the specific layer during the backpropagation process of the neural network model. For example, in a distributed training process, the workload of the weight gradient of a network layer refers to the amount of data transmitted between computing nodes to realize the update of the model weights of the network layer.

[0069] (10) Bypass

[0070] Bypass refers to the process of an AI accelerator forwarding data to another AI accelerator through an intermediate AI accelerator to pass the data to the other AI accelerator.

[0071] (11) Hierarchical communication

[0072] In an AI accelerator cluster, hierarchical communication refers to a communication architecture that allows nodes or components of different dimensions (such as NPUs or switches in the AI accelerator cluster) to effectively exchange information in the cluster (such as collective communication between AI accelerators). This architecture can effectively utilize bandwidth resources of different dimensions, improve communication efficiency, reduce data transmission delay, and allow the cluster to be better expanded and managed.

[0073] (12) AI accelerator cluster

[0074] AI accelerator cluster refers to a large-scale computing cluster composed of a large number of AI accelerator resources (hundreds, thousands, or even tens of thousands of AI accelerators) combined through a network. The cluster can be composed of multiple nodes. The multiple nodes in the cluster can be interconnected, and the interconnection technology is not limited. For example, multiple nodes can be combined into a powerful computing platform through high-speed interconnection technology to significantly improve the communication performance between nodes. Since a super node is a computing platform that integrates multiple high-performance computing units and is tightly connected through high-speed interconnection technology, and the nodes in the cluster can also be tightly connected through high-speed interconnection technology, a super node can also be understood as a cluster in this application.

[0075] The technical solutions provided in this application are described below.

[0076] For example, FIG. 1 shows a schematic diagram of an application scenario provided by an embodiment of the present application. As shown in FIG. 1, in this scenario, there is at least one cabinet 11, at least one AI accelerator mainboard 12 is configured in the cabinet 11, and at least one AI accelerator 13 is configured on the AI accelerator mainboard 12. The different cabinets 11 can be connected through switches 14, but are not limited to this. The AI accelerators 13 in these cabinets 11 constitute an AI accelerator cluster. The AI accelerators in the AI accelerator cluster can communicate in a collective communication mode. In addition, a control device 15 can also be configured in this scenario. The AI accelerator cluster can be controlled through the control device 15, such as allocating AI accelerators for AI tasks, etc. In the control device 15, an independent number can be set for each cabinet 11, an independent number can also be set for each AI accelerator mainboard 12, and an independent number can also be set for each AI accelerator 13. For example, in a cabinet 11, the AI accelerators 13 on the same AI accelerator mainboard 12 can be considered to be in the same dimension, and the AI accelerators 13 on different AI accelerator mainboards 12 can be considered to be in different dimensions. Different cabinets 11 can be understood as different dimensions. It should be understood that the application scenario shown in FIG. 1 is only a schematic illustration and does not constitute a limitation on the present application.

[0077] Generally, in the application scenario shown in FIG. 1, different topologies of AI accelerator networking for an AI task will bring different communication overheads, and different communication overheads will affect the processing efficiency of the AI task. Therefore, if a topology (such as an optimal or better topology) that matches the AI task can be found for the AI task from the AI accelerator cluster, the efficiency of the AI accelerator cluster in processing the AI task will be greatly improved. Based on this idea, an AI task processing system is provided in the embodiments of the present application. The AI task processing system will be introduced below in combination with the application scenario shown in FIG. 1.

[0078] For example, FIG. 2 shows an architecture schematic diagram of an AI task processing system provided by the embodiments of the present application. As shown in FIG. 2, the AI task processing system mainly consists of an NPU supernode 100 and an AI service platform 200.

[0079] The NPU supernode 100 belongs to the hardware part. The NPU supernode can use a product of one generation or use products of multiple different generations, which is not limited here. For example, the NPU supernode can mainly consist of a third NPU supernode 110 and a fourth NPU supernode 120. The third NPU supernode 110 is a supernode composed of a third NPU chip, and the fourth NPU supernode 120 is a supernode composed of a fourth NPU chip. Since there is a generation difference between the third NPU supernode 110 and the fourth NPU supernode 120, there is also a difference in software between the two. Because of the difference in software, the fourth NPU supernode 220 can be directly controlled by a supernode network controller 240, while the third NPU supernode 110 needs to be indirectly controlled by the supernode network controller 240. It should be noted that the NPU supernode 100 can be understood as an AI accelerator cluster composed of the AI accelerator 13 shown in FIG. 1. The NPU supernode 100 can also be replaced by other AI accelerator supernodes, such as a TPU supernode, a GPU supernode, etc., and the replaced scheme is still within the scope of the present application.

[0080] The AI service platform 200 belongs to a software part, which can be but is not limited to configured in the control device 15 shown in FIG. 1. It can include a resource manager 210, an AI framework 220, a collective communication module 230, and a supernode network controller 240. Among them, the resource manager 210 is mainly used to provide high-level scheduling strategies and resource management functions to optimize the execution efficiency and resource utilization of large-scale AI workloads. The AI framework 220 is an efficient, easy-to-use, and flexible deep learning framework, mainly used to build deep learning models. Exemplarily, the AI framework 220 can be a framework such as pytorch, tensorflow, or mindspore. The collective communication module 230 is mainly used to provide efficient data exchange capabilities for large-scale parallel computing tasks, support multiple collective communication primitives and algorithms, optimize collective communication efficiency, and improve system performance to accelerate the execution of AI and high-performance computing applications. Exemplarily, the collective communication module 230 can be but is not limited to the Huawei Collective Communication Library (HCCL). The supernode network controller 240 is mainly used to centrally manage and optimize network communication to ensure high speed, low latency, and high reliability of data transmission. The supernode network controller 240 can include an AI cluster communication control engine 241. The AI cluster communication control engine 241 is mainly used to control the collective communication part in the entire AI task. The AI cluster communication control engine 241 can include a communication domain orchestration module 2411, a hierarchical communication module 2412, and a network topology search module 2413. Among them, the communication domain orchestration module 2411 is mainly used to orchestrate the collective communication part of the AI task, such as decomposition, ordering, scheduling, etc. of the collective communication operator. The hierarchical communication module 2412 is mainly used for hierarchical collective communication according to the topology, such as when the optimal topology structure searched by the network topology search module 2413 is multi-dimensional, the hierarchical communication module 2412 can orchestrate the collective communication operator in different dimensions to realize efficient utilization of the topology bandwidth in different dimensions. The network topology search module 2413 is mainly used to search the network topology to search for a topology structure (such as the optimal or better topology structure, etc.) matching the AI task from the NPU networking. Among them, the topology structure matching the AI task refers to the network topology of the AI accelerator in the AI accelerator cluster that has a communication time lower than at least one other available topology structure under the condition of meeting the AI task (including model structure, parallel strategy, etc. information). For example, when the AI accelerator cluster contains 3 available topology structures, if the communication time corresponding to the topology structure 1 is t1, the communication time corresponding to the topology structure 2 is t2, and the communication time corresponding to the topology structure 3 is t3, and t1 < t2 < t3, then the topology structure matching the AI task is the topology structure 1 or 2.

[0081] In this embodiment, the network topology search module 2413 can include a Workload generator 24131 and a topology searcher 24132. The Workload generator 24131 is mainly used to calculate the Workload of each network layer in the neural network model based on the model attribute parameters of the neural network model and the distributed parallel strategy of the AI task. For example, the Workload of a network layer can include one or more of the Workload of the forward propagation part of the network layer, the Workload of the input gradient, and the Workload of the weight gradient. For example, as shown in FIG. 3, the execution steps of the Workload generator 24131 can include: in S301, reading the model attribute parameters of the neural network model and the distributed parallel strategy of the AI task, which can be uploaded to the AI service platform 200 by the user. In S302, the model attribute parameters of the neural network model and the distributed parallel strategy of the AI task are parsed, the neural network model architecture (such as the transformer layer, and the attention layer and MLP layer contained therein) is initialized, and the distributed parallel strategy of the AI task (such as data / tensor / mixed parallel, and the number of AI accelerators required) is initialized. In S303, it is determined whether the distributed parallel strategy of the AI task is mixed parallel, if yes, S304 is executed, otherwise, S305 is executed. In S304, if the distributed parallel strategy of the AI task is mixed parallel, the Workload of the forward propagation part, the input gradient, and the weight gradient of each layer of the neural network model in the AI accelerator cluster is calculated. The Workload of the weight gradient of a network layer can be calculated by multiplying the parameter quantity (i.e., the number of weights) of the network layer by the data type of each weight. The Workload of the forward propagation part and the input gradient of a network layer can be calculated by using the megatron algorithm to calculate the parameter quantity (such as the number of neurons, the number of attention layer heads, etc.) of the network layer. For example, the parameter quantity of a network layer can be included in the model attribute parameters. In S305, it is determined whether the distributed parallel strategy of the AI task is data parallel, if yes, S306 is executed, otherwise, S307 is executed. In S306, if the distributed parallel strategy of the AI task is data parallel, the Workload of the weight gradient of each layer of the neural network model in the AI accelerator cluster is calculated. The collection communication data size in the Workload of the weight gradient of a network layer is proportional to the number of weights and the data type of the weights of the layer. In S307, it is determined whether the distributed parallel strategy of the AI task is tensor parallel, if yes, S308 is executed, otherwise, an error is reported. In S308, if the distributed parallel strategy of the AI task is tensor parallel, the Workload of the forward propagation part and the input gradient of each layer of the neural network model in the AI accelerator cluster is calculated.It should be understood that, since the AI task distributed parallel strategy is data parallel, the models deployed on different AI accelerators are the same, so only the Workload of the weight gradient of each layer of the neural network model needs to be calculated. When the AI task distributed parallel strategy is tensor parallel, the models deployed on different AI accelerators are different, so after the forward propagation and the backward propagation are completed, data aggregation needs to be performed, and therefore, the Workload of the forward propagation part and the input gradient of each layer of the neural network model needs to be calculated. In S309, after all the calculations are completed, all the Workload calculation results are collected and output. For example, the output of the Workload generator 24131 can include the collective communication operator and the collective communication data size involved in the forward propagation part of each network layer in the neural network, the collective communication operator and the collective communication data size involved in the input gradient, and the collective communication operator and the collective communication data size involved in the weight gradient. One network layer can have one Workload, and the two can be in a one-to-one correspondence; and the Workload of one network layer can be the sum of the Workload of the forward propagation part, the Workload of the input gradient, and the Workload of the weight gradient of the network layer. For example, as shown in Table 1, when the AI task distributed parallel strategy is hybrid parallel, in the output of the Workload generator 24131, the forward propagation part, the input gradient, and the weight gradient of the MLP layer are all related to the collective communication operator Allreduce, but the collective communication data sizes involved in different parts can be different; the input gradient and the weight gradient of the Query layer are both related to the collective communication operator Allreduce, but the collective communication data sizes involved in different parts are different, and the Query layer does not involve the Workload of the forward propagation part; the weight gradient of the Key layer and the Value layer are both related to the collective communication operator Allreduce, and the collective communication data sizes involved in the two are different, and both of them do not involve the Workload of the forward propagation part and the input gradient; the forward propagation part and the input gradient of the Layernorm layer are both related to the collective communication operator Allgather, but the collective communication data sizes involved in different parts are different, and the weight gradient of the Layernorm layer is related to the collective communication operator Allreduce.

[0082] Table 1

[0083] The topology searcher 24132 mainly searches the topology of the AI accelerator required by the AI task based on the model attribute parameters of the neural network model, the AI task distributed parallel strategy, and the cluster topology information of the AI accelerator, and finally selects the topology structure (such as the topology structure with the shortest or shorter communication time) matched with the AI task. For example, the cluster topology information of the AI accelerator cluster can include the topology structure of the AI accelerator cluster and the state of the AI accelerator. For example, as shown in FIG. 4, the execution steps of the topology searcher 24132 can include: in S401, reading the model attribute parameters of the neural network model, the AI task distributed parallel strategy, and the cluster topology information of the AI accelerator cluster. The cluster topology information of the AI accelerator cluster can be provided by the supplier of the AI accelerator cluster, but is not limited thereto. In S402, according to the AI task distributed parallel strategy and the required number of AI accelerators, the AI accelerators in the AI accelerator cluster are arranged and combined to obtain N topology structures that meet the AI task distributed parallel strategy and the required number of AI accelerators. For example, if the required number of AI accelerators is 4 and the AI task distributed parallel strategy is data parallel, then the topology structures that meet the data parallel and the AI accelerator number of 4 can have a topology structure of 2 rows x 2 columns and a topology structure of 1 row x 4 columns. In S403, based on the state (whether available) of the AI accelerator in the cluster topology information, the topology structures that do not exist in the topology structure of the AI accelerator cluster are removed from the N topology structures to obtain M topology structures, M ≤ N. Since all the topology structures that meet the requirements are obtained in S402, but the AI accelerators in these topology structures can be in an unavailable state, and when there is an unavailable AI accelerator in a certain topology structure, the topology structure will be invalid, therefore, the topology structures that do not exist in the topology structure of the AI accelerator cluster need to be removed from the N topology structures to obtain M topology structures. The states of the AI accelerators in the M topology structures are all available. In S404, the topology structure matched with the AI task is selected from the M topology structures. For example, the M topology structures can be input into a pre-trained neural network model to obtain the topology structure matched with the AI task. In this way, the topology structure matched with the AI task is selected. The topology structure matched with the AI task is the AI accelerator topology structure that can make the current AI task most efficient or relatively efficient. In addition, in S404, the transmission time of data in the M topology structures can also be calculated respectively to obtain the communication time corresponding to each of the M topology structures. Then, the communication time corresponding to different topology structures in the M topology structures is compared, and the topology corresponding to the shortest communication time is taken as the topology structure matched with the AI task.Of course, the topology structure corresponding to the shortest communication time can not be selected, and other topology structures corresponding to other communication times can be selected, as long as the topology structure corresponding to the longest communication time is not selected, and the specific selection can be determined according to actual conditions, which is not limited here. In some embodiments, when calculating the communication time corresponding to the topology structure, the communication time corresponding to each of the M topology structures can be calculated based on the workload generated by the Workload generator 24131 and each of the M topology structures. Wherein, for any one topology structure, the workload and the topology structure can be input into a preset formula to obtain the communication time corresponding to the topology structure. It should be understood that different set communication operators have different set communication modes; further, according to the preset formula, the communication time can be calculated according to the corresponding (same or different) set communication mode, the size of the communication amount, and the topology structure, bandwidth and other information of the network. It should be understood that when the AI task distributed parallel strategy is tensor parallel or data parallel, each of the M topology structures is one dimension, either tensor dimension or data dimension, so the communication time corresponding to each of the topology structures in these topology structures can be directly calculated. Therefore, when the AI task distributed parallel strategy is tensor parallel or data parallel, for any one topology structure, the communication time between each two AI accelerators can be directly calculated to calculate the communication time corresponding to different network layers and different operators, and then the sum of the communication times corresponding to different network layers and different operators is taken as the communication time corresponding to the topology structure. When the AI task distributed parallel strategy is hybrid parallel, each of the M topology structures is two dimensions, one dimension is tensor dimension and the other dimension is data dimension, so each of the topology structures in these topology structures includes two-dimensional sub-topology structures. Therefore, at this time, for any one of the M topology structures, the communication time corresponding to the two-dimensional sub-topology structures in the two dimensions of the topology structure needs to be calculated respectively; then, the sum of the communication times corresponding to the two-dimensional sub-topology structures is taken as the communication time corresponding to the topology structure. In some embodiments, the transmission path of data in at least one of the M topology structures can include single-hop and multi-hop. At this time, in the process of calculating the communication time corresponding to the topology structure, not only the direct connection between the AI accelerators can be considered, but also the possibility of AI accelerator detour can be considered, that is, all transmission paths can be considered, so that the calculated communication time can more truly reflect the transmission time consumption of data in the corresponding topology structure.For example, as shown in FIG. 5, taking 4 AI accelerators as an example, when AI accelerator 1 is a source node and AI accelerator 2 is a destination node, if the AI accelerator has the capability of data forwarding, the data transmission between AI accelerator 1 and 2 can not only be directly transmitted from AI accelerator 1 to AI accelerator 2, but also can be realized by detouring (i.e. through AI accelerator 3 or 4 for data forwarding). Therefore, there are multiple transmission paths for transmitting data from AI accelerator 1 to AI accelerator 2, which are AI accelerator 1->2 (direct connection (i.e. single hop)), AI accelerator 1->3->2 (detour (i.e. multi-hop)), AI accelerator 1->4->3->2 (detour (i.e. multi-hop)), and combinations of multiple paths in the above three transmission paths, etc. When calculating the communication time of transmitting data from AI accelerator 1 to AI accelerator 2, the communication time of data transmission through different transmission paths can be calculated respectively, and the appropriate communication time (e.g. the shortest or shorter communication time, etc.) is selected as the communication time of AI accelerator 1 to 2, so that the calculated communication time can more truly reflect the transmission time consumption of data in the corresponding topology. Referring to FIG. 5, assuming that the amount of data sent is s, the connection bandwidth is b, and the delay is d, and assuming that s / b>>d, the influence of delay on communication time can be ignored, then the communication time of data from AI accelerator 1 to 2 through different transmission paths can be:

[0084] 1) When directly connected, the communication time is t=s / b;

[0085] 2) When directly connected + detour (2 hops), 2s / 3 data is sent through direct connection, and s / 3 data is sent through detour (2 hops), then the communication time is t=max(2s / (3b),2*s / (3b))=2s / (3b), which is 2 / 3 of only the direct connection case;

[0086] 3) When directly connected + detour (2 hops) + detour (3 hops), 6s / 11 data is sent through direct connection, 3s / 11 data is sent through detour (2 hops), and 2s / 11 data is sent through detour (3 hops), then the communication time is t=max(6s / (11b),2*3s / (11b),3*2s / (11b))=6s / (11b), which is 6 / 11 of only the direct connection case.

[0087] From the communication times of the three transmission paths calculated above, it can be seen that when the detour forwarding strategy is adopted, there are more optional paths for data transmission. When the detour is combined with direct connection, it is more efficient and shorter in communication time than direct connection alone. When there are more detour paths, the communication efficiency can be further improved and the communication time can be reduced. Therefore, by considering different connection modes including AI accelerator direct connection and detour in the process of calculating the communication time corresponding to the topology structure, the calculated communication time can more truly reflect the transmission time consumption of data in the corresponding topology structure, so that the topology structure matched with the AI task can be more accurately screened out. It should be understood that when the topology structure is of multiple dimensions, the calculation of the communication time can also take into account the layered communication algorithm. In addition, each of the M topology structures can include single-hop and multi-hop, or a part of the topology structures can include single-hop, while another part of the topology structures can include single-hop and multi-hop, or all the topology structures can include single-hop, etc., which can be determined according to actual conditions, and is not limited herein.

[0088] For example, when the topology structure of the AI accelerator cluster is a three-dimensional topology structure, and the AI accelerator is an NPU, the output of the topology searcher 24132 can include the following information: 1) Rack_id: the id of the cabinet where the NPU is located; 2) Board_id: the id of the AI accelerator mainboard where the NPU is located; 3) NPU_id: the id of the NPU on the specific AI accelerator mainboard; 4) DP_group: data parallel group, which indicates that the NPU in the same data parallel group performs data parallel; 5) TP_group: tensor parallel group, which indicates that the NPU in the same tensor parallel group performs tensor parallel. The output example at this time can be as shown in the following "Table 2". In Table 2, it is indicated that the AI task distributed parallel strategy is hybrid parallel, and the selected NPU is the 0th NPU and the 1st NPU on the 0th AI accelerator mainboard in the 0th cabinet, and the 0th NPU and the 1st NPU on the 1st AI accelerator mainboard in the 0th cabinet. Among them, the 0th NPU and the 1st NPU on the 0th AI accelerator mainboard in the 0th cabinet are data parallel; the 0th NPU and the 1st NPU on the 1st AI accelerator mainboard in the 0th cabinet are data parallel. The 0th NPU on the 0th AI accelerator mainboard in the 0th cabinet and the 0th NPU on the 1st AI accelerator mainboard are tensor parallel; the 1st NPU on the 0th AI accelerator mainboard in the 0th cabinet and the 1st NPU on the 1st AI accelerator mainboard in the 0th cabinet are tensor parallel.

[0089] Table 2

[0090] For the convenience of understanding the workflow of the topology searcher 24132, the core running steps thereof are described below by taking the AI accelerator as an NPU and the search for an optimal topology structure as an example. Exemplarily, when the AI task distributed parallel strategy is tensor and data hybrid parallel, as shown in (A) of FIG. 6, the core running steps of the topology searcher 24132 are as follows:

[0091] (1) At the beginning, Workload, model attribute parameters and the AI task distributed parallel strategy (i.e., model parallel strategy) are read first, and the cluster topology information of the NPU cluster is read, mainly including the number of NPU in each dimension in the NPU cluster and the distribution of available NPU.

[0092] (2) According to the required number of NPU in the model attribute parameters and the AI task distributed parallel strategy, the NPU is arranged and combined to obtain all topology combinations that satisfy the number of NPU and the parallel strategy.

[0093] (3) The possible topologies obtained in step (2) are preliminarily searched in the topology structure of the NPU cluster.

[0094] (4) The topologies that do not exist in the topology structure of the NPU cluster are eliminated, and the searchable topologies are left.

[0095] (5) The remaining searchable topologies are cycled, and the sub-topologies of the tensor dimension or the data dimension are cycled.

[0096] (6) In each cycle, the communication time corresponding to the topology is calculated.

[0097] (7) After the cycle is completed, the communication times calculated by all topologies are collected.

[0098] (8) The communication times of different topologies are compared, and the optimal topology (i.e., the optimal topology structure) corresponding to the shortest communication time is found.

[0099] When the AI task distributed parallel strategy is tensor parallel or data parallel, as shown in (B) of FIG. 6, the core running steps of the topology searcher 24132 are as follows:

[0100] (1) At the beginning, Workload, model attribute parameters and the AI task distributed parallel strategy are read first, and the cluster topology information of the NPU cluster is read, mainly including the number of NPU in each dimension in the NPU cluster and the distribution of available NPU.

[0101] (2) According to the required number of NPU in the model attribute parameters and the AI task distributed parallel strategy, the NPU is arranged and combined to obtain all topology combinations that satisfy the number of NPU and the parallel strategy.

[0102] (Three) In the topology structure of the NPU cluster, a preliminary search is performed on the possible topologies obtained in "Step Two".

[0103] (Four) Topologies that do not exist in the topology structure of the NPU cluster are eliminated, and searchable topologies are left.

[0104] (Five) The remaining searchable topologies are cycled.

[0105] (Six) In each cycle, the communication time corresponding to the topology is calculated.

[0106] (Seven) After the cycle is completed, the communication times calculated for all topologies are collected.

[0107] (Eight) The communication times of different topologies are compared, and the optimal topology corresponding to the shortest communication time (i.e., the optimal topology structure) is found.

[0108] The above is the introduction of the AI task processing system provided by the embodiments of the present application. In order to facilitate the understanding of the technical solutions provided by the embodiments of the present application, the following will take the training of a certain large language model in the NPU networking based on full connection topology, and the mixed parallel strategy of data parallelism and tensor parallelism, and search for the optimal topology structure as an example to introduce the technical solutions provided by the embodiments of the present application.

[0109] For example, part of the model attribute parameters are as follows:

[0110] Table 3

[0111] The mixed parallel strategy of data parallelism and tensor parallelism is adopted. Among them, the current mainstream Megatron splitting method is adopted for tensor parallelism. For the specific parallel strategy and the corresponding NPU required quantity parameters, please refer to Table 4 as follows:

[0112] Table 4

[0113] The NPU networking cluster based on full connection topology used contains 128 cabinets in total, each cabinet contains 8 NPU boards (i.e., AI accelerator mainboards), and each board has 8 NPUs. It performs full connection networking according to the horizontal, vertical, and inter-cabinet three dimensions to form a 3D-full connection topology structure. The total number of NPUs in this NPU networking is 128*8*8=8192.

[0114] Considering the actual situation, some NPUs will be occupied and not available. Therefore, a part of the NPUs in the NPU cluster are in the non-available state, and the remaining NPUs are in the available state. For the NPU available state statistics, please refer to Table 5 as follows:

[0115] Table 5

[0116] Under the model attribute parameters and the AI task distributed parallel strategy described above, the process of searching for the optimal topology structure is as follows:

[0117] (I) According to the input model attribute parameters and AI task distributed parallel strategy information, Workload is generated in the Workload generation module 24131, including collective communication operators and collective communication data sizes.

[0118] At the beginning, the input parameters of the model attribute parameters and the AI task distributed parallel strategy are read. Then, it is judged which parallel strategy is used. If data parallel or hybrid parallel is used, the collective communication operators and the collective communication data sizes of the data parallel part are calculated, and the corresponding specific communication part is the weight gradient. If it is judged that tensor parallel or hybrid parallel is used, the collective communication operators and the collective communication data sizes of the tensor parallel part are calculated, and the corresponding specific communication part includes forward and input gradients. It should be understood that when hybrid parallel is used, both data parallel and tensor parallel parts need to be calculated. After all the calculations are completed, the calculation results are summarized and the workload calculation results are output.

[0119] (II) The generated Workload results and the cluster topology information of the NPU cluster are input. In the topology searcher 24132, the topology of the NPU required by the AI task is searched and the communication time is calculated, and finally the optimal topology with the shortest communication time is selected.

[0120] At the beginning, the topology searcher 24132 reads the Workload file and the cluster topology information. According to the NPU number and the AI task distributed parallel strategy information, all possible topology shapes are obtained by permutation and combination. According to the topology structure in the cluster topology information, all topology shapes that cannot be searched in the topology structure of the NPU cluster are removed, which means that the remaining topology shapes can be searched in the topology structure of the NPU cluster. In the remaining topologies, their corresponding communication times are calculated. Finally, the topology scheme corresponding to the minimum communication time is output.

[0121] After the topology search, the optimal topology scheme can be compared with the topology obtained by the baseline algorithm. The baseline scheme selected here is the greedy algorithm, which preferentially selects empty or NPU available cabinets. The comparison results of the optimal topology and the baseline algorithm are as follows in Table 6:

[0122] Table 6

[0123] Table 6 is the comparison of the optimal topology and the baseline algorithm results. Among them, the definition of the topology shape (i.e. the optimal topology structure) is [[data parallel cabinet number, data parallel NPU board number per cabinet, NPU number on each NPU board of data parallel], [tensor parallel cabinet number, tensor parallel NPU board number per cabinet, NPU number on each NPU board of tensor parallel]].

[0124] It should be understood that under the system shown in FIG. 2, when an AI task (such as a training task or an inference task, etc.) is started to run, the supernode network control platform 240 can be started to run first and activate the AI cluster communication control engine 241. Inside the AI cluster communication control engine 241, the network topology search module 2411 can search for the optimal network topology structure matched with the AI task. The network topology structure searched by the network topology search module 2411 will act on the NPU supernode 100 hardware to determine and configure the NPU network topology. For example, to configure which NPU participates in the AI task. Then, the optimal topology structure searched can be used on the NPU supernode 100 to execute the AI task, such as to do large language model training or inference, etc.

[0125] As can be seen from the above description, through the above AI task processing system, a topology structure (such as the optimal or better topology structure, etc.) matched with an AI task can be found for an AI task from an AI accelerator cluster, and thus the efficiency of the AI accelerator cluster in processing the AI task can be greatly improved. In order to facilitate understanding of the process of calculating Workload and communication time by the AI task processing system, examples are given below for introduction.

[0126] For example, assuming that an AI model is composed of two layers of MLP, the input vector and the output vector of the two MLP layers are both 1024 in length, the model weight data type is float32, and each weight occupies 4 Bytes. The model is trained on 4 NPUs through data parallel, and it is assumed that the topology structure of the NPU cluster is 2D-torus, i.e. the two dimensions are ring topology.

[0127] First, calculate the Workload of the model training. Because the AI distributed parallel strategy is data parallel, the Workload only contains the weight gradient. The communication type can be Allreduce. For each MLP layer, the communication volume = 2 * MLP input vector length * MLP output vector length * the number of bytes occupied by the model weight type. There is a coefficient of 2 because Allreduce needs to do Reducescatter and Allgather two steps, and the communication of these two steps is the same, so for Allreduce, multiply by 2. Therefore, the communication volume of one layer of MLP is: 2 * 1024 * 1024 * 4 = 8388608 (B) = 8 (MB). Similarly, the communication volume of another MLP is also 8 MB.

[0128] Then, because 4 NPUs are sampled for data parallel training, the topology shape of the NPU required to train the model has three types, which are: 1 * 4 (i.e. 1 row and 4 columns), 2 * 2 and 4 * 1. Because the two MLPs in the model are exactly the same, the Workload is also the same, so the communication size can be combined, that is, for the entire model, the communication type is Allreduce, and the communication size is 8 + 8 = 16 (MB). Assuming that the x-axis bandwidth of the network is 2 MB / s and the y-axis bandwidth is 1 MB / s. Because the network of the NPU cluster is torus, the collective communication adopts the ring algorithm.

[0129] When the topology is 1 * 4, it is a complete ring, and the communication time = ((NPU number - 1) / NPU number) * communication volume / x-axis bandwidth = (3 / 4) * 16 / 2 = 6 (s).

[0130] When the topology is 2 * 2, hierarchical communication can be done: first, the x-axis does reducescatter communication time = ((x-axis NPU number - 1) / x-axis npu number) * communication volume / 2 / x-axis bandwidth = ((2 - 1) / 2) * 16 / 2 / 2 = 2 (s);

[0131] Second, the y-axis does Allreduce communication time = ((y-axis NPU number - 1) / y-axis npu number) * communication volume / x-axis NPU number / y-axis bandwidth = ((2 - 1) / 2) * 16 / 2 / 1 = 4 (s);

[0132] Third, the x-axis does allgather communication time = ((x-axis NPU number - 1) / x-axis npu number) * communication volume / 2 / x-axis bandwidth = ((2 - 1) / 2) * 16 / 2 / 2 = 2 (s). The communication time calculated by the above three steps can be used to calculate the total communication time = 2 + 4 + 2 = 8 (s).

[0133] When the topology is 4*1, it is also a complete ring, and the communication time = ((NPU number-1) / NPU number)*communication volume / y-axis bandwidth = (3 / 4)*16 / 1 = 12(s).

[0134] Finally, from the communication times corresponding to the above three topologies, it can be seen that when the topology is 1*4, the communication time is the shortest, so the searched topology is the 1*4 topology.

[0135] It should be understood that the AI task processing system described above can be configured on a cloud computing platform, for example, deployed on at least one instance of a virtual machine or a container, etc., so that the cloud computing platform can provide an AI task processing service. Of course, the AI task processing system can also be configured on nodes other than the cloud computing platform, for example, can be deployed in at least one data center, or deployed on at least one server, which can be determined according to actual conditions, and is not limited here. Among them, the cloud computing platform can provide a page related to a public cloud service for a tenant to remotely access the public cloud service. In this embodiment, the tenant (also referred to as "user") can purchase an AI task processing service provided by the AI task processing system in advance on the cloud computing platform. For ease of understanding, the interaction mode between the tenant and the cloud computing platform is described below. As shown in FIG. 7, the interaction between the tenant and the cloud computing platform mainly includes: the tenant logs in to the cloud computing platform 700 through a client web page, selects and purchases a cloud service related to the AI task processing system (i.e., an AI task processing service) in the cloud computing platform 700, and after the purchase, the tenant can process the AI task on the cloud computing platform 700 based on the functions provided by the AI task processing service. Among them, the cloud computing platform 700 is mainly used to manage the infrastructure running the AI task processing service. For example, the infrastructure running the AI task processing service can include multiple data centers set in different regions, and each data center includes multiple servers. The data center can provide basic resources such as computing resources and storage resources for the AI task processing service. Therefore, when the tenant purchases and uses the AI task processing service, the tenant mainly pays for the resources used. When the tenant uses the AI task processing service, the tenant can input data related to the AI task (such as model attribute parameters, AI task distributed parallel strategy, cluster topology information, etc.) through the configuration interface, application program interface (application program interface, API) or interface provided by the cloud computing platform 700 for interaction with the tenant, and the cloud computing platform 700 can perform AI task processing according to the input of the tenant.

[0136] Next, based on the above content, an AI task processing method provided by the embodiment of the present application is introduced.

[0137] Exemplarily, FIG. 8 shows a flowchart of an AI task processing method provided by an embodiment of the present application. It can be understood that the method can be executed by any device, equipment, platform, or cluster of equipment having computing and processing capabilities. Exemplarily, the method can be executed by an AI task processing device, which can be implemented by software and / or hardware, and can be configured in an electronic device or a server, typically, a cloud computing platform. For ease of description, the cloud computing platform will be taken as an execution subject in the following description. As shown in FIG. 8, the AI task processing method can include the following steps:

[0138] S801, obtaining cluster topology information of an AI accelerator cluster, model attribute parameters of a first model, and an AI task distributed parallel strategy, the first model being used to execute a first AI task, and the cluster topology information including a topology structure of the AI accelerator cluster and states of AI accelerators in the AI accelerator cluster.

[0139] In this embodiment, the cloud computing platform can obtain the model attribute parameters of the first model and the AI task distributed parallel strategy from a client associated with the cloud computing platform. The first model is used to execute a first AI task (such as a model training task or an inference task, etc.). The model attribute parameters and the AI task distributed parallel strategy can be uploaded to the client by a user. Exemplarily, the client associated with the cloud computing platform can be a desktop application, a mobile application, a Web application, or a Web-based application, etc.

[0140] In addition, the cloud computing platform can obtain the cluster topology information of the AI accelerator cluster from a provider providing an AI task processing service. For example, the provider providing the AI task processing service can upload the cluster topology information to the cloud computing platform. Exemplarily, the AI accelerator cluster can include a plurality of AI accelerators, such as NPUs, TPUs, etc. The cluster topology information of the AI accelerator cluster can include a topology structure of the AI accelerator cluster and states of AI accelerators in the cluster (such as whether the AI accelerators are occupied).

[0141] S802, searching for a first topology structure from the AI accelerator cluster based on the cluster topology information, the model attribute parameters, and the AI task distributed parallel strategy, wherein the first topology structure is executable for the first AI task, and a communication time of the first topology structure is lower than that of an AI accelerator network of at least one second topology structure, and the second topology structure is an available topology structure in the AI accelerator cluster except the first topology structure.

[0142] In this embodiment, the cloud computing platform can search in the AI accelerator cluster based on the cluster topology information, the model attribute parameters and the AI task distributed parallel strategy, so as to search the topology structure of the AI accelerator network that can execute the first AI task. In order to guarantee the execution efficiency of the first AI task, the first topology structure can be selected from the searched topology structure. The first topology structure is the topology structure of the AI accelerator network that can execute the first AI task and has a communication time lower than at least one second topology structure. The second topology structure is the available topology structure in the AI accelerator cluster except the first topology structure. For example, the cluster topology information, the model attribute parameters and the AI task distributed parallel strategy can be processed by the neural network model to obtain the first topology structure. For example, the transmission path of data in the first topology structure includes single-hop and / or multi-hop. When the transmission path of data in the first topology structure includes multi-hop, the AI accelerator bypassing situation can be considered, not only the AI accelerator direct connection situation.

[0143] As a possible implementation, the AI accelerators in the AI accelerator cluster can be combined based on the AI task distributed parallel strategy and the first number in the model attribute parameters to obtain N topologies satisfying the AI task distributed parallel strategy and the first number, N≥1. For example, when the AI task distributed parallel strategy is data parallel or tensor parallel, and the first number is 4, the topologies that can be filtered from the AI accelerator cluster are at least one topology of 1*4, at least one topology of 2*2, and at least one topology of 4*1. The first number is the number of AI accelerators required to execute the first AI task. Then, based on the cluster topology information, topologies that do not exist in the cluster topology information are removed from the N topologies to obtain M topologies. For example, topologies containing AI accelerators in the unavailable state are removed. Finally, a first topology can be filtered from the M topologies. For example, the first topology can be filtered by the communication time corresponding to the M topologies. In some embodiments, the communication time corresponding to each of the M topologies can be calculated to obtain M communication times. For example, the M topologies can be processed by a neural network model. For example, when the transmission path of data in a certain topology (hereinafter referred to as a “third topology”) includes single-hop and multi-hop, the transmission time of data from a source node to a destination node in the third topology via different transmission paths can be calculated to obtain K transmission times. Then, the first transmission time in the K transmission times is taken as the communication time corresponding to the third topology. The first transmission time is lower than at least one transmission time other than the first transmission time in the K transmission times, so as to calculate a more optimal communication time corresponding to the third topology. After calculating the communication time corresponding to each topology, the topology corresponding to the shortest or relatively shorter communication time in the M communication times can be taken as the first topology. S803, the first model is deployed on the first topology based on the AI task distributed parallel strategy to execute the first AI task.

[0144] In this embodiment, after the first topology is filtered, the cloud computing platform can deploy the first model on the first topology based on the AI task distributed parallel strategy, so that the first AI task can be executed. For example, when the AI task distributed parallel strategy is data parallel, if the first topology contains 4 AI accelerators, the cloud computing platform can copy the first model 4 times and configure one first model on each AI accelerator.

[0145] Thus, since the searched first topology structure is a topology structure of an AI accelerator network that can perform the first AI task in the AI accelerator cluster and has a relatively small communication time, after the model is deployed on the first topology structure, the task related to the model can be performed with a relatively small communication overhead, thereby improving the processing efficiency of the AI task. In addition, the first topology structure is a topology of a network composed of AI accelerators available in the AI accelerator cluster, and thus this AI task processing manner does not need to change the structure of the cluster topology information of the AI accelerator cluster, thereby reducing the hardware cost (for example, a configurable optical switch). Meanwhile, in searching for the first topology structure, all possible topology structures can be considered, so that not only the direct connection of the AI accelerators can be considered, but also the detour of the AI accelerators can be considered, and when the topology structure has multiple dimensions, the calculation of the communication time can also take into account the hierarchical communication algorithm, so that the first topology structure can be searched under any cluster topology information of the AI accelerator cluster, and thus the communication overhead can be maximally reduced for any AI accelerator cluster.

[0146] In some embodiments, in S802, before the communication time corresponding to the topology structure is calculated, the workload of a network layer in the first model in the AI accelerator cluster can be calculated based on the model attribute parameters and the AI task distributed parallel strategy. The workload of a network layer is the collective communication data size corresponding to the collective communication operator included in the network layer. The workload of a network layer can be used to calculate the communication time corresponding to the topology structure of the AI accelerator network in the AI accelerator cluster. The process of calculating the workload can be referred to the related description in the foregoing FIG. 3, which will not be repeated here.

[0147] For example, when the AI task distributed parallel strategy is data parallel, the workload of the weight gradient of the first network layer can be calculated based on the model attribute parameters to obtain the workload of the first network layer. The first network layer is any network layer in the first model. When the AI task distributed parallel strategy is tensor parallel, the workload of the forward propagation part of the first network layer and the workload of the input gradient can be calculated based on the model attribute parameters to obtain the workload of the first network layer. The first network layer is any network layer in the first model. When the AI task distributed parallel strategy is mixed parallel of tensor and data, the workload of the forward propagation part of the first network layer, the workload of the weight gradient, and the workload of the input gradient can be calculated based on the model attribute parameters to obtain the workload of the first network layer. The first network layer is any network layer in the first model.

[0148] It can be understood that the size of the serial number of each step in the above embodiments does not mean the order of execution, and the execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. In addition, the various embodiments described above or the technical features involved in the embodiments can be combined according to the actual situation, and the combined scheme is still within the protection scope of the present application.

[0149] Based on the method in the above embodiments, the embodiments of the present application also provide an AI task processing device.

[0150] For example, FIG. 9 shows a structural schematic diagram of an AI task processing device provided by an embodiment of the present application. As shown in FIG. 9, the AI task processing device 900 includes an acquisition module 901 and a processing module 902. The acquisition module 901 is configured to acquire cluster topology information of an AI accelerator cluster, model attribute parameters of a first model, and an AI task distributed parallel strategy, the first model being used to execute a first AI task, and the cluster topology information including a topology structure of the AI accelerator cluster and a state of an AI accelerator in the AI accelerator cluster. The processing module 902 is configured to search for a first topology structure from the AI accelerator cluster based on the cluster topology information, the model attribute parameters, and the AI task distributed parallel strategy, wherein the first topology structure is an AI accelerator network topology structure of which the communication time is lower than that of at least one second topology structure, and the second topology structure is an available topology structure in the AI accelerator cluster except the first topology structure. The processing module 902 is further configured to deploy the first model on the first topology structure based on the AI task distributed parallel strategy, so as to execute the first AI task.

[0151] In some embodiments, when the processing module 902 searches for the first topology structure from the AI accelerator cluster based on the cluster topology information, the model attribute parameters, and the AI task distributed parallel strategy, it is specifically configured to: combine the AI accelerators in the AI accelerator cluster based on the AI task distributed parallel strategy and a first quantity in the model attribute parameters, to obtain N topology structures that satisfy the AI task distributed parallel strategy and the first quantity, N≥1, wherein the first quantity is the number of AI accelerators required to execute the first AI task; eliminate topology structures that do not exist in the cluster topology information from the N topology structures based on the cluster topology information, to obtain M topology structures; and screen the first topology structure from the M topology structures.

[0152] In some embodiments, when the processing module 902 screens the first topology structure from the M topology structures, it is specifically used for: calculating the communication time corresponding to each of the M topology structures to obtain M communication times; and taking the topology structure corresponding to the shortest communication time in the M communication times as the first topology structure.

[0153] In some embodiments, the transmission path of data in the third topology structure in the M topology structures includes single-hop and multi-hop, and the third topology structure is any one of the M topology structures; when calculating the communication time corresponding to the third topology structure, the processing module 902 is specifically used for: calculating the transmission time of data from the source node to the destination node in the third topology structure through different transmission paths to obtain K transmission times; and taking the first transmission time in the K transmission times as the communication time corresponding to the third topology structure, wherein the first transmission time is lower than at least one transmission time in the K transmission times except the first transmission time.

[0154] In some embodiments, before the processing module 902 screens the first topology structure from the M topology structures, it is further used for: based on the model attribute parameter and the AI task distributed parallel strategy, calculating the Workload of the network layer in the first model in the AI accelerator cluster, wherein the Workload of the network layer is the collective communication data size corresponding to the collective communication operator contained in the network layer, and the Workload of the network layer is used to calculate the communication time corresponding to the topology structure of the AI accelerator network in the AI accelerator cluster.

[0155] In some embodiments, when the processing module 902 calculates the Workload of the network layer in the first model in the AI accelerator cluster based on the model attribute parameter and the AI task distributed parallel strategy, it is specifically used for: in the case that the AI task distributed parallel strategy is data parallel, based on the model attribute parameter, calculating the Workload of the weight gradient of the first network layer to obtain the Workload of the first network layer, wherein the first network layer is any one of the network layers in the first model.

[0156] In some embodiments, when the processing module 902 calculates the Workload of the network layer in the first model in the AI accelerator cluster based on the model attribute parameter and the AI task distributed parallel strategy, it is specifically used for: in the case that the AI task distributed parallel strategy is tensor parallel, based on the model attribute parameter, calculating the Workload of the forward part of the first network layer and the Workload of the input gradient to obtain the Workload of the first network layer, wherein the first network layer is any one of the network layers in the first model.

[0157] In some embodiments, the processing module 902, when calculating the Workload of the network layer in the first model in the AI accelerator cluster based on the model attribute parameters and the AI task distributed parallel strategy, specifically calculates the Workload of the forward propagation part, the Workload of the weight gradient, and the Workload of the input gradient of the first network layer based on the model attribute parameters to obtain the Workload of the first network layer in the case that the AI task distributed parallel strategy is tensor and data mixed parallel.

[0158] In some embodiments, the transmission path of the data in the first topology structure includes single-hop.

[0159] In some embodiments, the transmission path of the data in the first topology structure further includes multi-hop.

[0160] In some embodiments, the communication time corresponding to the first topology structure is calculated based on the transmission time of the data in the first topology structure from the source node to the destination node via different transmission paths.

[0161] In some embodiments, the model attribute parameters include parameters that determine the model structure and size, parameters related to the amount of training data, and parameters related to the number of AI accelerators required for model training or inference.

[0162] In some embodiments, the obtaining module 901 and the processing module 902 shown in FIG. 9 can be implemented by software or by hardware. For example, the implementation of the obtaining module 901 is described below. Similarly, the implementation of the processing module 902 can refer to the implementation of the obtaining module 901.

[0163] As an example of a software functional unit, the obtaining module 901 can include code running on a computing instance. The computing instance can include at least one of a physical host (computing device), a virtual machine, and a container. Further, the computing instance can be one or more. For example, the obtaining module 901 can include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code can be distributed in the same region (region) or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code can be distributed in the same availability zone (AZ) or in different AZs, and each AZ includes one data center or multiple data centers with similar geographical locations. Generally, one region can include multiple AZs.

[0164] Similarly, the plurality of hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Among them, usually one VPC is set in one region, and communication between two VPCs in the same region and between VPCs in different regions needs to set a communication gateway in each VPC to realize the interconnection between VPCs through the communication gateway.

[0165] As an example of a hardware functional unit, the obtaining module 901 can include at least one computing device, such as a server, etc. Alternatively, the obtaining module 901 can also be a device implemented by an application-specific integrated circuit (ASIC) or a programmable logic device (PLD), etc. Among them, the above-mentioned PLD can be implemented by a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0166] The plurality of computing devices included in the obtaining module 901 can be distributed in the same region or in different regions. The plurality of computing devices included in the obtaining module 901 can be distributed in the same AZ or in different AZs. Similarly, the plurality of computing devices included in the obtaining module 901 can be distributed in the same VPC or in multiple VPCs. Among them, the plurality of computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, and GALs, etc.

[0167] It should be noted that in other embodiments, the obtaining module 901 can be used to perform any step of the AI task processing method described in the above embodiments, and the processing module 902 can be used to perform any step of the AI task processing method described in the above embodiments. In addition, the steps responsible for the obtaining module 901 and the processing module 902 can be specified as needed, and the entire function of the AI task processing apparatus 900 shown in FIG. 9 can be realized by the obtaining module 901 and the processing module 902 respectively realizing different steps of the AI task processing method described in the above embodiments.

[0168] The application also provides a computing device 1000. As shown in FIG. 10, the computing device 1000 includes a bus 1002, a processor 1004, a memory 1006, and a communication interface 1008. The processor 1004, the memory 1006, and the communication interface 1008 communicate through the bus 1002. The computing device 1000 can be a server or an electronic device. It should be understood that the number of processors and memories in the computing device 1000 is not limited.

[0169] The bus 1002 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one line is shown in FIG. 10, but it does not mean that there is only one bus or only one type of bus. The bus 1004 can include a path for transmitting information between various components (e.g., the memory 1006, the processor 1004, the communication interface 1008) of the computing device 1000.

[0170] The processor 1004 can include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP), etc.

[0171] The memory 1006 can include a volatile memory, such as a random access memory (RAM). The processor 1004 can also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0172] The memory 1006 stores executable program code, and the processor 1004 executes the executable program code to respectively implement the functions of the acquisition module 901 and the processing module 902 shown in FIG. 9, thereby implementing the AI task processing method described in the above embodiments. That is, the memory 1006 stores instructions for executing the AI task processing method described in the above embodiments.

[0173] Alternatively, the memory 1006 stores executable code that the processor 1004 executes to respectively implement the functions of the AI task processing apparatus 900 shown in FIG. 9 described above, thereby implementing the AI task processing method described in the above embodiments. That is, the memory 1006 stores instructions for executing the AI task processing method described in the above embodiments.

[0174] The communication interface 1003 uses a transceiving module such as, but not limited to, a network interface card, a transceiver, to implement communication between the computing device 1000 and other devices or communication networks.

[0175] The embodiments of the present disclosure also provide a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center.

[0176] As shown in FIG. 11, the computing device cluster includes at least one computing device 1000. The memory 1006 in one or more computing devices 1000 in the computing device cluster can store the same instructions for executing the AI task processing method described in the above embodiments.

[0177] In some possible implementations, the memory 1006 in one or more computing devices 1000 in the computing device cluster can also respectively store partial instructions for executing the AI task processing method described in the above embodiments. In other words, the combination of one or more computing devices 1000 can collectively execute the instructions for executing the AI task processing method described in the above embodiments.

[0178] It should be noted that the memory 1006 in different computing devices 1000 in the computing device cluster can store different instructions, respectively used to execute part of the functions of the AI task processing apparatus 900 shown in FIG. 9 described above. That is, the instructions stored in the memory 1006 in different computing devices 1000 can implement the functions of one or more of the obtaining module 901 and the processing module 902.

[0179] In some possible implementation, one or more of the computing devices in the cluster of computing devices can be connected through a network. In some possible implementation, the network can be a wide area network or a local area network, etc. FIG. 12 illustrates one possible implementation. As shown in FIG. 12, two computing devices 1000A and 1000B are connected through a network. Specifically, the computing devices are connected to the network through a communication interface in each of the computing devices. In this type of possible implementation, the memory 1006 in the computing device 1000A stores instructions for performing the functions of the obtaining module 901. Meanwhile, the memory 1006 in the computing device 1000B stores instructions for performing the functions of the processing module 902.

[0180] It should be understood that the functions of the computing device 1000A shown in FIG. 12 can also be performed by multiple computing devices 1000. Similarly, the functions of the computing device 1000B can also be performed by multiple computing devices 1000.

[0181] The embodiments of the present application also provide another cluster of computing devices. The connection relationship between the computing devices in the cluster of computing devices can be similar to the connection relationship between the computing devices in the cluster of computing devices described with reference to FIG. 11 and FIG. 12. The difference is that the memory 1006 in one or more of the computing devices 1000 in the cluster of computing devices can store the same instructions for performing the method in the above-described embodiments.

[0182] In some possible implementation, the memory 1006 in one or more of the computing devices 1000 in the cluster of computing devices can also respectively store partial instructions for performing the data processing method described above. In other words, the combination of one or more of the computing devices 1000 can collectively execute the instructions for performing the data processing method described above.

[0183] Based on the method in the above-described embodiments, the embodiments of the present application provide a computer readable storage medium including computer program instructions, when the computer program instructions are executed by a computing device, the computing device performs the method in the above-described embodiments; or when the computer program instructions are executed by a cluster of computing devices, the cluster of computing devices performs the method in the above-described embodiments. For example, the computer readable storage medium can be any available medium or a data storage device including one or more available media that a computing device is capable of storing. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk), etc.

[0184] Based on the method in the above embodiments, the embodiments of the present application provide a computer program product containing instructions, which, when executed by a computing device, cause the computing device to perform the method in the above embodiments, or, when executed by a computing device cluster, cause the computing device cluster to perform the method in the above embodiments.

[0185] It can be understood that the processor in the embodiments of the present application can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor can be a microprocessor or any conventional processor.

[0186] The method steps in the embodiments of the present application can be realized in the form of hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in a random access memory (RAM), a flash memory, a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically EPROM (EEPROM), a register, a hard disk, a mobile hard disk, a CD-ROM or any other form of storage medium well known in the art. An exemplary storage medium is coupled to the processor, so that the processor can read information from and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and the storage medium can be located in an ASIC.

[0187] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in or transmitted by a computer readable storage medium. The computer instructions can be transmitted from one website, computer, cloud computing platform or data center to another website, computer, cloud computing platform or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) manner. The computer readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a cloud computing platform, data center, etc. integrated with one or more available media sets. The available media can be magnetic media (such as floppy disk, hard disk, magnetic tape), optical media (such as DVD), or semiconductor media (such as solid state disk (SSD)) and the like.

[0188] It can be understood that various numerical numbers involved in the embodiments of the present application are only distinguished for convenience of description, and are not used to limit the scope of the embodiments of the present application.

[0189] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. An AI task processing method, characterized by, The method comprises: obtaining cluster topology information of an AI accelerator cluster, model attribute parameters of a first model, and an AI task distributed parallel strategy, the first model being used to execute a first AI task, and the cluster topology information comprising a topology structure of the AI accelerator cluster and states of AI accelerators in the AI accelerator cluster; based on the cluster topology information, the model attribute parameters, and the AI task distributed parallel strategy, searching for a first topology structure from the AI accelerator cluster, wherein the first topology structure is an AI accelerator network topology structure that can execute the first AI task and has a lower communication time than at least one second topology structure, and the second topology structure is an available topology structure in the AI accelerator cluster other than the first topology structure; based on the AI task distributed parallel strategy, deploying the first model on the first topology structure to execute the first AI task.

2. The method of claim 1, wherein, The searching for the first topology structure from the AI accelerator cluster based on the cluster topology information, the model attribute parameters, and the AI task distributed parallel strategy comprises: based on the AI task distributed parallel strategy and a first quantity in the model attribute parameters, combining AI accelerators in the AI accelerator cluster to obtain N topology structures that satisfy the AI task distributed parallel strategy and the first quantity, N≥1, wherein the first quantity is a required number of AI accelerators for executing the first AI task; based on the cluster topology information, eliminating topology structures that do not exist in the cluster topology information from the N topology structures to obtain M topology structures; screening the first topology structure from the M topology structures.

3. The method of claim 2, wherein, The screening of the first topology structure from the M topology structures comprises: calculating a communication time corresponding to each of the M topology structures to obtain M communication times; taking a topology structure corresponding to a shortest communication time in the M communication times as the first topology structure.

4. The method of claim 3, wherein, A transmission path of data in a third topology structure in the M topology structures comprises single-hop and multi-hop, and the third topology structure is any one of the M topology structures; The calculation of the communication time corresponding to the third topology structure comprises: calculating transmission times of data from a source node to a destination node via different transmission paths in the third topology structure to obtain K transmission times; taking a first transmission time in the K transmission times as the communication time corresponding to the third topology structure, wherein the first transmission time is lower than at least one transmission time other than the first transmission time in the K transmission times.

5. The method according to any of claims 2-4, characterized by, Before the screening of the first topology structure from the M topology structures, the method further comprises: based on the model attribute parameters and the AI task distributed parallel strategy, calculating a workload of a network layer in the first model in the AI accelerator cluster, wherein the workload of the network layer is a collective communication data size corresponding to a collective communication operator included in the network layer, and the workload of the network layer is used to calculate a communication time corresponding to a topology of an AI accelerator network in the AI accelerator cluster.

6. The method of claim 5, wherein, The calculation of the workload of the network layer in the first model in the AI accelerator cluster based on the model attribute parameters and the AI task distributed parallel strategy includes: In a case where the AI task distributed parallel strategy is data parallel, based on the model attribute parameters, calculating a workload of a weight gradient of a first network layer to obtain the workload of the first network layer, wherein the first network layer is any one of the network layers in the first model.

7. The method of claim 5, wherein, The calculation of the workload of the network layer in the first model in the AI accelerator cluster based on the model attribute parameters and the AI task distributed parallel strategy includes: In a case where the AI task distributed parallel strategy is tensor parallel, based on the model attribute parameters, calculating a workload of a forward propagation part of a first network layer and a workload of an input gradient to obtain the workload of the first network layer, wherein the first network layer is any one of the network layers in the first model.

8. The method of claim 5, wherein, The calculation of the workload of the network layer in the first model in the AI accelerator cluster based on the model attribute parameters and the AI task distributed parallel strategy includes: In a case where the AI task distributed parallel strategy is tensor and data hybrid parallel, based on the model attribute parameters, calculating a workload of a forward propagation part of a first network layer, a workload of a weight gradient, and a workload of an input gradient to obtain the workload of the first network layer, wherein the first network layer is any one of the network layers in the first model.

9. The method according to any of claims 1 to 8, characterized in that, The transmission path of data in the first topology structure includes single-hop.

10. The method of claim 9, wherein, The transmission path of data in the first topology structure further includes multi-hop.

11. The method of claim 10, wherein, The communication time corresponding to the first topology structure is calculated based on transmission times of data in the first topology structure from a source node to a destination node via different transmission paths.

12. The method of any one of claims 1-11, wherein, The model attribute parameters include parameters that determine model structure and size, parameters related to training data volume, and parameters related to the number of AI accelerators required for model training or inference.

13. An AI task processing apparatus, characterized by comprising: The method includes: an obtaining module, configured to obtain cluster topology information of an AI accelerator cluster, model attribute parameters of a first model, and an AI task distributed parallel strategy, wherein the first model is used to perform a first AI task, the cluster topology information includes a topology structure of the AI accelerator cluster and states of AI accelerators in the AI accelerator cluster; The processing module is configured to search for a first topology structure from the AI accelerator cluster based on the cluster topology information, the model attribute parameter, and the AI task distributed parallel strategy, wherein the first topology structure is capable of executing the first AI task, and a communication time of the first topology structure is lower than that of at least one second topology structure of an AI accelerator network of a topology structure of the AI accelerator cluster. The processing module is further configured to deploy the first model on the first topology structure based on the AI task distributed parallel strategy, so as to execute the first AI task.

14. The apparatus of claim 13, wherein, When the processing module searches for the first topology structure from the AI accelerator cluster based on the cluster topology information, the model attribute parameter, and the AI task distributed parallel strategy, the processing module is specifically configured to: combine AI accelerators in the AI accelerator cluster based on the AI task distributed parallel strategy and a first quantity in the model attribute parameter, to obtain N topology structures satisfying the AI task distributed parallel strategy and the first quantity, N≥1, wherein the first quantity is a required number of AI accelerators for executing the first AI task; eliminate, based on the cluster topology information, a topology structure that does not exist in the cluster topology information from the N topology structures, to obtain M topology structures; select the first topology structure from the M topology structures.

15. The apparatus of claim 14, wherein, When the processing module selects the first topology structure from the M topology structures, the processing module is specifically configured to: calculate a communication time corresponding to each of the M topology structures, to obtain M communication times; take a topology structure corresponding to a shortest communication time in the M communication times as the first topology structure.

16. The apparatus of claim 15, wherein, A transmission path of data in a third topology structure in the M topology structures includes single-hop and multi-hop, and the third topology structure is any one of the M topology structures. When the processing module calculates the communication time corresponding to the third topology structure, the processing module is specifically configured to: calculate transmission times of data from a source node to a destination node in the third topology structure via different transmission paths, to obtain K transmission times; take a first transmission time in the K transmission times as the communication time corresponding to the third topology structure, wherein the first transmission time is lower than at least one transmission time other than the first transmission time in the K transmission times.

17. The apparatus of any of claims 14-16, wherein, Before the processing module selects the first topology structure from the M topology structures, the processing module is further configured to: calculate, based on the model attribute parameter and the AI task distributed parallel strategy, a workload Workload of a network layer in the first model in the AI accelerator cluster, wherein the Workload of the network layer is a collective communication data size corresponding to a collective communication operator included in the network layer, and the Workload of the network layer is used to calculate a communication time corresponding to a topology structure of an AI accelerator network in the AI accelerator cluster.

18. The apparatus of claim 17, wherein, The processing module is specifically configured to, when calculating the Workload of the network layer in the first model in the AI accelerator cluster based on the model attribute parameter and the AI task distributed parallel strategy, specifically for: In the case that the AI task distributed parallel strategy is data parallel, the processing module is configured to calculate the Workload of the weight gradient of the first network layer based on the model attribute parameter, to obtain the Workload of the first network layer, wherein the first network layer is any one of the network layers in the first model.

19. The apparatus of claim 17, wherein, The processing module is specifically configured to, when calculating the Workload of the network layer in the first model in the AI accelerator cluster based on the model attribute parameter and the AI task distributed parallel strategy, specifically for: In the case that the AI task distributed parallel strategy is tensor parallel, the processing module is configured to calculate the Workload of the forward propagation part of the first network layer and the Workload of the input gradient based on the model attribute parameter, to obtain the Workload of the first network layer, wherein the first network layer is any one of the network layers in the first model.

20. The apparatus of claim 17, wherein, The processing module is specifically configured to, when calculating the Workload of the network layer in the first model in the AI accelerator cluster based on the model attribute parameter and the AI task distributed parallel strategy, specifically for: In the case that the AI task distributed parallel strategy is tensor and data hybrid parallel, the processing module is configured to calculate the Workload of the forward propagation part of the first network layer, the Workload of the weight gradient, and the Workload of the input gradient based on the model attribute parameter, to obtain the Workload of the first network layer, wherein the first network layer is any one of the network layers in the first model.

21. The apparatus of any of claims 13-20, wherein, The transmission path of data in the first topology structure includes single-hop.

22. The apparatus of claim 21, wherein, The transmission path of data in the first topology structure further includes multi-hop.

23. The apparatus of claim 22, wherein, The communication time corresponding to the first topology structure is calculated based on the transmission time of data in the first topology structure from the source node to the destination node via different transmission paths.

24. The apparatus of any of claims 13-23, wherein, The model attribute parameter includes a parameter that determines the model structure and size, a parameter related to the training data volume, and a parameter related to the number of AI accelerators required for model training or inference.

25. A cluster of computing devices, characterized in that, The at least one computing device each includes a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method of any one of claims 1-12.

26. A computer-readable storage medium, characterized in that, The computer program instructions, when executed by the computing device cluster, cause the computing device cluster to execute the method of any one of claims 1-12, wherein the computing device cluster includes at least one computing device.

27. A computer program product comprising instructions, wherein: The computer program instructions, when executed by the computing device cluster, cause the computing device cluster to execute the method of any one of claims 1-12, wherein the computing device cluster includes at least one computing device.

Images