Communication orchestration method and apparatus, and computer program product

Abstract

The present application relates to the technical field of distributed computing, and discloses a communication orchestration method and apparatus, and a computer program product. A computational model is distributed and runs on a plurality of computing nodes, and the method comprises: obtaining model information and runtime strategy information of the computational model, and network topology information and network resource information of the plurality of computing nodes; and determining a communication orchestration result of the plurality of computing nodes on the basis of the model information, the runtime strategy information, the network topology information, and the network resource information. The model information and the runtime strategy information implicitly indicate communication requirements of the plurality of computing nodes during execution of the computational model. The network topology information and the network resource information represent links and communication resources available for communication among the plurality of computing nodes. By comprehensively analyzing the above information, an improved communication orchestration result that meets the above requirements can be obtained, thereby reducing communication conflicts, lowering communication latency, and improving overall communication performance.

Description

Communication orchestration method, device and computer program product

[0001] The present application claims priority to the Chinese patent application No. 202411245660.6, filed on September 5, 2024, and entitled "Communication orchestration method, device and computer program product", the entire content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the technical field of distributed computing, and in particular to a communication orchestration method, device and computer program product. BACKGROUND

[0003] With the deepening and improvement of research on computing models such as artificial intelligence (AI) models, the parameter quantity of computing models has also rapidly increased to the level of billions, hundreds of billions or even trillions, and the computing quantity has also rapidly increased. However, the iteration speed of computing hardware is far behind the update speed of software, and how to support large parameter computing models under limited hardware resources is a problem that the industry focuses on.

[0004] Currently, the industry mainly solves this problem through some parallel strategies, such as data parallelism, tensor parallelism, pipeline parallelism, expert parallelism, model parallelism and sequence parallelism, and distributes a computing model to multiple computing nodes for running through these parallel strategies. However, the running related data of the computing model usually has global correlation, so in the case of distributing a computing model to multiple computing nodes, communication needs to be performed between the multiple computing nodes to achieve the purpose of data interaction between the computing nodes. How to improve the communication performance in this case to improve the overall running performance of the computing model is the top priority. SUMMARY

[0005] The present application provides a communication orchestration method, device and computer program product, which can reduce communication conflicts between multiple computing nodes, reduce communication latency and improve overall communication performance in the case of distributing a computing model to run on multiple computing nodes. The technical solution is as follows:

[0006] In a first aspect, a communication orchestration method is provided, in which a computing model is distributed to run on multiple computing nodes, and the method comprises:

[0007] Obtaining model information and running strategy information of the computing model, and network topology information and network resource information of the multiple computing nodes; determining a communication orchestration result of the multiple computing nodes based on the model information, the running strategy information, the network topology information and the network resource information.

[0008] The model information and the running strategy information imply communication requirements of a plurality of computing nodes in a running process of a computing model, and the network topology information and the network resource information represent links and communication resources that can be provided for communication between the plurality of computing nodes. Through comprehensive analysis of the model information and the running strategy information of the computing model and the network topology information and the network resource information of the plurality of computing nodes, the plurality of computing nodes are more reasonably arranged for communication, so as to obtain a more optimal communication arrangement result that meets the communication requirements, thereby reducing communication conflicts between the plurality of computing nodes, reducing communication latency, and improving overall communication performance.

[0009] In a possible implementation, based on the model information, the running strategy information, the network topology information, and the network resource information, the communication arrangement result of the plurality of computing nodes is determined, including: determining each two computing nodes that need to communicate based on the running strategy information; determining communication requirements of each two computing nodes that need to communicate based on the model information and the running strategy information; and determining the communication arrangement result based on the communication requirements of each two computing nodes that need to communicate, the network topology information, and the network resource information. That is, the communication requirements of each two computing nodes that need to communicate are analyzed first, and then the optimal communication arrangement result that meets the communication requirements is determined under the constraints of the network topology and the network resources.

[0010] In a possible implementation, based on the model information and the running strategy information, the communication requirements of each two computing nodes that need to communicate are determined, including: determining running related information of each two computing nodes that need to communicate based on the model information and the running strategy information; and determining the communication requirements of each two computing nodes that need to communicate based on the running related information of each two computing nodes that need to communicate.

[0011] In a possible implementation, the model information includes model structure and model parameter quantity; the running strategy information includes at least one of running configuration parameters and parallel strategy information; the network topology information includes node identification and node-to-node topology information of a plurality of communication nodes, the plurality of communication nodes being used to implement communication between the plurality of computing nodes; and the network resource information includes node resource information of the plurality of communication nodes.

[0012] In a possible implementation, the running configuration parameters include at least one of a global batch size, a micro batch size, and an input sequence length of the computing model in a training case.

[0013] In a possible implementation, the communication orchestration result of the plurality of computing nodes is determined based on the model information, the running strategy information, the network topology information and the network resource information, including: determining traffic demand information based on the model information and the running strategy information, the traffic demand information representing communication demand of the plurality of computing nodes; determining available resource information based on the network topology information and the network resource information, the available resource information representing network resources available to the plurality of computing nodes; and determining the communication orchestration result based on the traffic demand information and the available resource information. That is, the communication demand of the plurality of computing nodes is analyzed first, and then the available resources are analyzed, and finally the optimal communication orchestration result meeting the communication demand is determined under the constraint of the available resources.

[0014] In the method, the communication orchestration result is determined based on the traffic demand information and the available resource information, including: determining the communication orchestration result that minimizes a target parameter of an optimization algorithm based on the traffic demand information and the available resource information, the target parameter including at least one of end-to-end training time, end-to-end inference time, end-to-end communication latency, end-to-end communication link number and end-to-end communication hop number. That is, the optimization algorithm is used to determine the communication orchestration result with the optimization of the end-to-end communication performance as the target.

[0015] In a possible implementation, the traffic demand information includes at least one of communication traffic size and expected communication time between each two computing nodes that need to communicate.

[0016] In the method, the communication traffic size includes communication traffic size when communication is performed in a target parallel strategy, and the target parallel strategy refers to a parallel strategy used between the two corresponding computing nodes.

[0017] In a possible implementation, the communication orchestration result includes at least one of traffic orchestration information and device configuration information; the traffic orchestration information includes at least one of origin-destination (OD) matrix, traffic packet sending time and packet sending path; and the device configuration information includes at least one of communication queue, traffic priority and communication operator implementation.

[0018] In a possible implementation, the communication orchestration result includes operator execution optimization information; the operator execution optimization information includes at least one of an optimized computation graph, operator fusion information, and a masking configuration between communication and computation; the optimized computation graph indicates changes in an operator graph of the computation model after communication orchestration relative to an operator graph before communication orchestration; the operator fusion information indicates fusion between computation operators in the computation model; and the masking configuration between communication and computation indicates a masking condition between a communication task and a computation task of the computation model. That is, the scheme can also optimize the computation graph, computation operators, and communication and computation masking configuration of the computation model based on the communication orchestration result, to improve communication efficiency and improve the running efficiency of the computation model as much as possible.

[0019] In a second aspect, a communication orchestration apparatus is provided, which has functions to implement behaviors of the communication orchestration method in the first aspect. The communication orchestration apparatus includes one or more modules for implementing the communication orchestration method provided in the first aspect.

[0020] That is, a communication orchestration apparatus is provided, which includes an obtaining module and a determining module.

[0021] The obtaining module is configured to obtain model information and running strategy information of a computation model, and network topology information and network resource information of a plurality of computing nodes, the computation model being distributed and running on the plurality of computing nodes.

[0022] The determining module is configured to determine a communication orchestration result of the plurality of computing nodes based on the model information, the running strategy information, the network topology information, and the network resource information.

[0023] In a possible implementation, the determining module includes:

[0024] A first determining sub-module is configured to determine, based on the running strategy information, each two computing nodes that need to communicate.

[0025] A second determining sub-module is configured to determine, based on the model information and the running strategy information, communication requirements of each two computing nodes that need to communicate.

[0026] A third determining sub-module is configured to determine the communication orchestration result based on the communication requirements of each two computing nodes that need to communicate, the network topology information, and the network resource information.

[0027] In a possible implementation, the second determining sub-module is specifically configured to:

[0028] Determine, based on the model information and the running strategy information, running-related information of each two computing nodes that need to communicate.

[0029] determine, based on the running related information of each two computing nodes that need to communicate, communication requirements of each two computing nodes that need to communicate.

[0030] In a possible implementation, the model information includes a model structure and a model parameter quantity.

[0031] The running strategy information includes at least one of a running configuration parameter and parallel strategy information.

[0032] The network topology information includes node identification and inter-node topology information of a plurality of communication nodes, and the plurality of communication nodes are used to implement communication between the plurality of computing nodes.

[0033] The network resource information includes node resource information of the plurality of communication nodes.

[0034] In a possible implementation, the running configuration parameter includes at least one of a global batch size, a micro batch size and an input sequence length of the computing model in a training case.

[0035] In a possible implementation, the determining module includes:

[0036] The fourth determining sub-module is configured to determine, based on the model information and the running strategy information, traffic requirement information, the traffic requirement information representing communication requirements of the plurality of computing nodes.

[0037] The fifth determining sub-module is configured to determine, based on the network topology information and the network resource information, available resource information, the available resource information representing network resources available to the plurality of computing nodes.

[0038] The sixth determining sub-module is configured to determine, based on the traffic requirement information and the available resource information, a communication orchestration result.

[0039] In a possible implementation, the sixth determining sub-module is specifically configured to:

[0040] determine, based on the traffic requirement information and the available resource information, the communication orchestration result by using an optimization algorithm, the communication orchestration result minimizing at least one of an end-to-end training time, an end-to-end inference time, an end-to-end communication latency, an end-to-end communication link quantity and an end-to-end communication hop quantity.

[0041] In a possible implementation, the traffic requirement information includes at least one of a communication traffic size and an expected communication time between each two computing nodes that need to communicate.

[0042] In a possible implementation, the communication traffic size includes a communication traffic size when communication is performed in a target parallel strategy, the target parallel strategy being a parallel strategy adopted between the two corresponding computing nodes.

[0043] In a possible implementation, the communication orchestration result includes at least one of traffic orchestration information and device configuration information.

[0044] The traffic orchestration information includes at least one of a traffic OD matrix, traffic packet sending time, and packet sending path.

[0045] The device configuration information includes at least one of a communication queue, traffic priority, and a communication operator implementation.

[0046] In a possible implementation, the communication orchestration result includes operator execution optimization information.

[0047] The operator execution optimization information includes at least one of an optimized computing graph, operator fusion information, and a masking configuration between communication and computation.

[0048] The optimized computing graph indicates a change in an operator graph of the computing model after communication orchestration relative to an operator graph before communication orchestration.

[0049] The operator fusion information indicates fusion between computing operators in the computing model.

[0050] The masking configuration between communication and computation indicates a masking condition between a communication task and a computing task of the computing model.

[0051] In a third aspect, a computer device is provided, including a processor and a memory, the memory being configured to store a program for executing the communication orchestration method in the first aspect and store data related to the communication orchestration method in the first aspect. The processor is configured to execute the program stored in the memory.

[0052] In a possible implementation, the computer device can further include a communication bus configured to establish a connection between the processor and the memory.

[0053] In a fourth aspect, a computer readable storage medium is provided, in which instructions are stored, and when the instructions are run on a computer, the computer is caused to perform the steps of the communication orchestration method in the first aspect.

[0054] In a fifth aspect, a computer program product containing instructions which, when executed on a computer, cause the computer to perform the steps of the communication orchestration method of the first aspect is provided. Alternatively, the present application provides a computer program which, when executed on a computer, causes the computer to perform the steps of the communication orchestration method of the first aspect.

[0055] The technical effects obtained by the second aspect, the third aspect, the fourth aspect and the fifth aspect are similar to the technical effects obtained by the corresponding technical means in the first aspect, and thus are not described here again. BRIEF DESCRIPTION OF DRAWINGS

[0056] Fig. 1 is a running system architecture diagram of a computing model provided by an embodiment of the present application;

[0057] Fig. 2 is a structural schematic diagram of a computer device provided by an embodiment of the present application;

[0058] Fig. 3 is a flowchart of a communication orchestration method provided by an embodiment of the present application;

[0059] Fig. 4 is a flowchart of another communication orchestration method provided by an embodiment of the present application;

[0060] Fig. 5 is a principle schematic diagram of a linear programming algorithm provided by an embodiment of the present application;

[0061] Fig. 6 is a logic schematic diagram of a communication orchestration method provided by an embodiment of the present application;

[0062] Fig. 7 is a structural schematic diagram of a communication orchestration apparatus provided by an embodiment of the present application. DETAILED DESCRIPTION

[0063] The embodiments of the present application will be further described in detail below with reference to the accompanying drawings.

[0064] Firstly, some terms related to the embodiments of the present application are introduced.

[0065] Distributed training: When training a computing model, since the parameter quantity of the current computing model is usually large, a single machine (such as a server) and a single card (such as a neural processing unit (NPU) or a graphics processing unit (GPU)) cannot accommodate it, and thus the computing model and data need to be split to different machines or cards for collaborative training, in which parallel splitting strategies (simply referred to as parallel strategies or parallel modes) involved include data parallelism (DP), tensor parallelism (TP), pipeline parallelism (PP), sequence parallelism (SP), expert parallelism (EP), and the like.

[0066] Collective communication: During the running of a computing model, the inter-card communication behaviors generated by different parallel strategies are inconsistent, and the communication primitives and communication operators used are also inconsistent. For example, TP communication usually uses an AllReduce communication operator, which can complete inter-card data communication and reduction operations. In addition, different collective communication algorithms can be used for different architectures.

[0067] Next, the background knowledge related to the embodiments of the present application is introduced.

[0068] With the continuous growth of the parameter quantity of a computing model, computing models with trillion and hundred trillion parameters are emerging, and at the same time, the input sequence length of multi-modal models such as text-to-video and text-to-image is starting to move towards 10 million, and single card and single machine cannot meet the training requirements, and thousands of cards or even tens of thousands of cards are needed to complete the running of models with such a large parameter quantity, and DP, TP, PP, SP, EP and other parallel strategies are used to realize distributed running to reduce the single-card memory overhead. However, parallel communication will inevitably introduce additional communication overhead, and the communication volume and communication requirements of different parallel splitting strategies are different, and the complexity of their configuration leads to the following problems and challenges in the distributed running of current computing models:

[0069] 1. Different traffic allocation leads to differences in communication performance: Full mesh / Torus network topology can achieve multi-link communication, but it will lead to uneven distribution of communication bandwidth; point-to-point communication has multiple-hop communication paths, leading to differences in communication latency.

[0070] 2. Multi-factor nonlinear coupling: Different parallel strategies and collective communication optimization algorithms have large differences in their impact on communication performance, and have high nonlinear and strong coupling on communication efficiency and latency performance.

[0071] 3. Traffic conflict leads to end-to-end performance degradation: Due to the synchronization characteristics of parallel communication, the network instantaneous traffic is large, and the probability of traffic conflict is high, which leads to end-to-end performance degradation.

[0072] To solve the above problems, the industry proposes an optimization scheme for different parallel strategies and different collective communication algorithms. That is, different communication modes are designed for different splitting strategies and collective communication algorithms. For example, for TP communication using AllReduce collective communication, the Ring AllReduce algorithm is used, which can split the TP communication data according to the card number in the communication domain, and in a single communication, the data split according to the ring communication, so as to ensure the conflict-free point-to-point communication in the TP communication domain, and at the same time, the theoretical transmission bandwidth can be achieved when sending data. For SP communication, the Ring P2P Attention method is used, which can be similar to Ring AllReduce, and the data blocks are communicated in turn according to the ring, which not only avoids conflict, but also reduces the exposure time of communication through the mutual masking of communication and calculation.

[0073] This scheme can only reduce the communication delay of a single communication operator by optimizing the communication operator under different parallel strategies, but since most computing models often need to use multiple parallel strategies in combination, the conflict probability of communication traffic under different behaviors in multiple communication domains is greatly increased. For example, in distributed training across racks or data centers, PP / DP traffic may conflict, resulting in increased end-to-end training latency, performance degradation, and even training interruption due to packet loss.

[0074] Moreover, the communication volume and communication delay requirements of different splitting strategies differ, and cannot adapt to the end-to-end performance requirements under different interconnection topologies. Specifically, TP, SP, EP, and other communication volumes are relatively large, and the communication delay requirement is high, generally using large bandwidth direct connection, but PP, DP, and other communication volumes are relatively small, generally cross-machine communication, and different topologies obtain different communication performance. Therefore, the configuration and traffic allocation of the communication strategy must consider the specific configuration of the actual physical environment and network topology.

[0075] To solve the above problems, the industry also proposes another scheme, that is, for switches, routers, and other network devices, the quality of service (QoS) strategy, such as priority, round-robin scheduling, etc., can be used to avoid traffic conflict.

[0076] For example, by marking priority, isolate background traffic interference. Specifically, for burst traffic caused by parallel strategy, by setting high priority, isolate irrelevant traffic influence. But due to the characteristics of synchronous burst of parallel training traffic, a large amount of traffic communication will be generated at the same time, all of which have high priority, which is difficult to plan.

[0077] For another example, resource reservation. Specifically, configure network devices such as switches to reserve bandwidth for point-to-point services to avoid traffic conflicts. However, the concurrent traffic in distributed training is large, and it is impossible to configure reservations one by one, and the configuration is complex, and the user threshold is high. Moreover, the scale of the computing cluster is large, usually in the order of thousands, ten thousands or even hundreds of thousands of cards, and the communication domain and point-to-point communication volume are strongly related to model characteristics and training parameter configuration. By manually configuring network devices to ensure the conflict-free communication of traffic between any two points on the cluster, not only is the workload complex, but also the network devices of the aggregation node need to support flexible packet identification functions and support massive table item configurations, and current devices basically cannot meet the configuration requirements or the configuration cost is extremely high.

[0078] To sum up, the related technology still has problems such as point-to-point communication imbalance and conflict probability, thereby causing the increase of end-to-end communication time overhead, even causing packet loss, congestion and other problems, degrading the running performance of the computing model, and even causing running interruption. In order to solve these problems, the embodiments of the present application provide a more optimal communication arrangement method, which aims to realize conflict-free arrangement of traffic by perceiving and modeling the model structure, parameter configuration, network topology, network resource information, etc., and optimize the end-to-end performance.

[0079] Next, the implementation environment related to the embodiments of the present application is introduced.

[0080] FIG. 1 is a system architecture diagram of a computing model running system provided by an embodiment of the present application. Referring to FIG. 1, the running system includes a plurality of computing nodes and a plurality of communication nodes, and the plurality of communication nodes are used to realize the communication between the plurality of computing nodes, that is, to connect the plurality of computing nodes.

[0081] Among them, one computing node can be an NPU, GPU or CPU, and the types of different computing nodes can be the same or different, which is not limited by the embodiments of the present application. In addition, the configurations of different computing nodes can be the same or different, which is also not limited by the embodiments of the present application.

[0082] In a possible implementation manner, the plurality of computing nodes can be contained in one computing device, such as a server, or can be distributed on a plurality of computing devices, such as a plurality of servers.

[0083] In the case that the plurality of computing nodes are contained in one computing device, the plurality of communication nodes can include a bus or other communication means connecting the plurality of computing nodes on the computing device, etc. In the case that the plurality of computing nodes are distributed on a plurality of computing devices, one computing device includes at least one computing node, and the plurality of communication nodes can include communication means connecting the plurality of computing devices and communication means connecting all computing nodes in the same computing device. Among them, the communication means connecting the plurality of computing devices can include switches, routers, network cards, bridges and other devices.

[0084] In an embodiment, the running system includes N servers, each server includes 8 NPUs, and the 8 NPUs of each server can communicate with the NPUs of other servers and also communicate with each other.

[0085] It should be understood that although the computing model in the embodiments of the present application runs on a plurality of computing nodes, the communication arrangement method provided by the embodiments of the present application can run on any one or more computer devices, which can be devices not including the above computing nodes, such as personal desktop computers, notebook computers or other computing devices, and can also be servers or other devices including processors, such as switches, routers and the like. Of course, the computer device can also be a device including one or more of the above computing nodes, such as a server or a processor in the running system. That is, the embodiments of the present application do not limit the execution subject of the above communication arrangement method.

[0086] FIG. 2 is a structural schematic diagram of a computer device provided by an embodiment of the present application. The computer device is used to execute the steps of the communication arrangement method provided by the embodiments of the present application. The computer device includes one or more processors 201, a communication bus 202, a memory 203 and one or more communication interfaces 204.

[0087] The processor 201 is a general central processing unit (CPU), a natural network processor (NP), a microprocessor, or one or more integrated circuits used to implement the schemes of the present application, such as an application-specific integrated circuit (ASIC), a programmable logic device (PLD) or a combination thereof. In a possible implementation, the PLD is a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.

[0088] The communication bus 202 is used to transmit information between the above components. In a possible implementation, the communication bus 202 is divided into an address bus, a data bus, a control bus and the like. For the convenience of representation, only one thick line is used in FIG. 2, but it does not mean that there is only one bus or only one type of bus.

[0089] In a possible implementation, the memory 203 is a read-only memory (ROM), a random access memory (RAM), an electrically erasable programmable read-only memory (EEPROM), an optical disk (including a compact disc read-only memory (CD-ROM), a compact disk, a laser disk, a digital versatile disk, a Blu-ray disk and the like), a magnetic disk storage medium or other magnetic storage device, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and capable of being accessed by a computer, but is not limited thereto. The memory 203 exists independently and is connected to the processor 201 through the communication bus 202, or the memory 203 is integrated with the processor 201.

[0090] The communication interface 204 uses any transceiver-like mechanism for communicating with other devices or communication networks. The communication interface 204 includes a wired communication interface, and in one possible implementation, a wireless communication interface. The wired communication interface is, for example, an Ethernet interface. In one possible implementation, the Ethernet interface is an optical interface, an electrical interface, or a combination thereof. The wireless communication interface is, for example, a wireless local area networks (WLAN) interface, a cellular network communication interface, or a combination thereof.

[0091] In some embodiments, the computer device includes multiple processors, such as processor 201 and processor 205 as shown in FIG. 2. Each of these processors is a single-core processor or a multi-core processor. In one possible implementation, the processor herein refers to one or more devices, circuits, and / or processing cores for processing data (such as computer program instructions).

[0092] In a specific implementation, as an example, the computer device further includes an output device 206 and an input device 207. The output device 206 is in communication with the processor 201 and is capable of displaying information in various ways. For example, the output device 206 is a liquid crystal display (LCD), a light emitting diode (LED) display device, a cathode ray tube (CRT) display device, or a projector. The input device 207 is in communication with the processor 201 and is capable of receiving user input in various ways. For example, the input device 207 is a mouse, a keyboard, a touch screen device, or a sensor device.

[0093] In some embodiments, the memory 203 is used to store program code 210 for implementing the solutions of the present application, and the processor 201 is capable of executing the program code 210 stored in the memory 203. The program code includes one or more software modules, and the computer device is capable of implementing the communication orchestration method provided in the embodiment of FIG. 3 below by means of the processor 201 and the program code 210 in the memory 203.

[0094] It should be noted that the computing model in the embodiments of the present application can be a machine learning model, such as a large language model (a kind of deep learning model), or other models. In the case where the computing model is a machine learning model, the machine learning model can be a model for inference or a model for training, that is, the communication orchestration method provided in the embodiments of the present application can be applied to the scenario of model inference or the scenario of model training.

[0095] It should also be understood that the system architecture and business scenarios described in the embodiments of the present application are for more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of system architecture and the appearance of new business scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0096] FIG. 3 is a flowchart of a communication orchestration method provided by an embodiment of the present application. The method is applied to a computer device and is used for pre-orchestrating communication of a computing model running on multiple computing nodes. Please refer to FIG. 3. The method includes the following steps.

[0097] Step 301: Obtain model information and running strategy information of the computing model, and network topology information and network resource information of the multiple computing nodes.

[0098] In order to be able to perform more comprehensive and reasonable communication orchestration, the embodiments of the present application need to obtain information in two aspects, one aspect is computing model related information, and the other aspect is network communication related information. Among them, the computing model related information mainly includes model information and running strategy information, and the network communication related information mainly includes network topology information and network resource information (also referred to as physical environment information, communication resource information or physical resource information, etc.). Based on the information in the two aspects, comprehensive and reasonable communication orchestration is performed.

[0099] Among them, the model information includes information of the computing model itself; the running strategy information represents a strategy related to running of the computing model; the network topology information represents a network topology for communication of the multiple computing nodes; and the network resource information represents communication resources provided for the multiple computing nodes.

[0100] In the embodiments of the present application, the model information includes model structure and model parameter quantity. Different model structures can have different influences on communication traffic size and / or communication timing between the computing nodes. The size of the model parameter quantity also has a great influence on the communication traffic size and / or communication timing between the computing nodes.

[0101] Among them, the model structure is the structure of the computing model. Taking a computing model as a large language model for example, the structure of the computing model can be a Transformer structure based on attention mechanism, a convolutional neural network, a recurrent neural network or a generative adversarial network, etc. Among them, the Transformer structure can include an embedding layer, an attention layer, a multi-layer perceptron (MLP) layer and an activation layer, etc., and the convolutional neural network can include an input layer, a hidden layer, a convolutional layer, an activation layer, a pooling layer and a fully connected layer, etc. For the structure of the computing model, please refer to related technologies, and the embodiments of the present application do not make a detailed introduction.

[0102] The model parameter quantity represents the size of a parameter of the computing model. In the embodiments of the present application, all parameter quantities of the computing model can be obtained. In other words, the model parameter quantity includes the parameter quantities of all parameters of the computing model. In some other embodiments, part of the parameter quantities of the computing model can also be obtained. For example, the model parameter quantity includes some parameter quantities that have an impact on the communication traffic size and / or communication timing between the plurality of computing nodes.

[0103] In a possible implementation, the model parameter quantity includes one or more of the following: the number of layers, the size of a hidden layer, the number of attention heads, the size of an attention head, the number of experts, the capacity factor (cap), top_k, and the like. It should be understood that if the computing model includes a hidden layer, the model parameter quantity can include the size of the hidden layer. If the computing model includes an attention module, the model parameter quantity can include at least one of the number of attention heads and the size of an attention head. If the computing model is a mixtrue of experts (MoE) model, the model parameter quantity can include at least one of the number of experts, the cap, and top_k, and the like. The top_k indicates that the first k experts of the MoE model process the token, and the cap indicates the upper limit of the token that each expert can process. That is, what the model parameter quantity specifically includes is related to the specific structure of the computing model, and the embodiments of the present application do not specifically limit what the model parameter quantity includes.

[0104] The running strategy information includes at least one of a running configuration parameter and parallel strategy information. Different running configuration parameters can have different impacts on the communication traffic size and / or communication timing between the computing nodes. Different parallel strategies also have a great impact on the communication traffic size and / or communication timing between the computing nodes.

[0105] The running configuration parameter includes a configuration parameter related to the running of the computing model, and in the embodiments of the present application, includes some configuration parameters that have an impact on the communication traffic size and / or communication timing between the plurality of computing nodes. For example, the running configuration parameter includes at least one of the following: the global batch size (GBS) of the computing model in the training case, the micro batch size (MBS), the input sequence length of the computing model, whether to enable activation value, whether to enable optimizer offloading, whether to enable communication masking, whether to enable operator fusion, whether to enable PP interleaving algorithm, whether to enable re-computation, and the corresponding re-computation type (including attention head re-computation, selection re-computation, full re-computation, communication re-computation, and the like).

[0106] Wherein, the global batch size refers to the total amount of data required to be processed by the model in a training step during the model training process. The micro batch size refers to the amount of data required to be processed by each data parallelism single time after starting data parallelism, that is, the size of the amount of training data divided into each data parallelism. There is the following formula: GBS=MBS*DP*MBN. Wherein, MBN is the full name of micro batch number. Taking GBS as 128 and DP as 64 (that is, 64 data parallelisms) as an example, the formula indicates that the amount of data required to be trained on each data parallelism is 128 / 64=2, and assuming that MBS is set to 1, the amount of data required to be processed by each data parallelism single time is 1, and a total of two times (MBN) are required to process two data.

[0107] It should be understood that the running configuration parameter can include a parameter configured by a user using the computing model and / or a default configuration parameter of the computing model (such as a parameter configured by a provider of the computing model), and embodiments of the present application do not limit what the running configuration parameter specifically includes.

[0108] The parallel strategy information represents a parallel strategy adopted by the parallel running of the computing model. In embodiments of the present application, the parallel strategy information includes a parallel degree of each parallel strategy in one or more parallel strategies, and the parallel degree represents a maximum number of parallel executions of a parallel object under the corresponding parallel strategy. Wherein, the one or more parallel strategies include at least one strategy in data parallelism (DP), tensor parallelism (TP), pipeline parallelism (PP), sequence parallelism (SP), and expert parallelism (EP). Accordingly, the parallel degree in the parallel strategy information includes at least one of a data parallel degree, a tensor parallel degree, a pipeline parallel degree, a sequence parallel degree, and an expert parallel degree. The data parallel degree represents a maximum number of data parallel executions under the data parallel strategy; the tensor parallel degree represents a maximum number of tensor parallel executions under the tensor parallel strategy; the pipeline parallel degree represents a maximum number of pipeline parallel executions under the pipeline parallel strategy; the sequence parallel degree represents a maximum number of input sequence parallel executions under the sequence parallel strategy; and the expert parallel degree represents a maximum number of expert parallel executions under the expert parallel strategy.

[0109] In another aspect, the tensor parallel degree represents the size of the model parameter splitting; the pipeline parallel degree represents the size of the number of layers of the computing model divided into different computing cards according to the number of layers (such as a 128-layer computing model, and the PP parallel degree is 2, which indicates that there are 64 layers on each computing card); the data parallel degree represents the number of parallel training paths when data parallelism is performed; and the expert parallel degree represents the size of parallel computation when expert parallelism is performed in the MoE model.

[0110] In a possible implementation, the running strategy information can include, in addition to the information introduced above, other strategy information that has an impact on the size of the communication traffic and / or the timing of the communication between the computing nodes. For example, the running strategy information further includes at least one of zero redundancy optimizer (ZeRO) configuration information and offloading strategy configuration information. The ZeRO configuration information includes whether to enable optimizer sharding, weight sharding, weight gradient sharding, or shard space, and the like. The offloading strategy configuration information includes whether to enable weight offloading or activation value offloading, and the like. For the related description of ZeRO and offloading strategy, refer to related technologies, and the embodiments of the present application are not limited in this regard.

[0111] The network topology information represents the network topology relationship between the plurality of communication nodes, and the network topology relationship has an impact on the communication path, the communication hop count, or the communication delay, and the like between the plurality of computing nodes. The plurality of communication nodes are used to implement the communication between the plurality of computing nodes, that is, to connect the plurality of computing nodes. In the embodiments of the present application, the network topology information includes node identifiers of the plurality of communication nodes and inter-node topology information, and the inter-node topology information includes a topology form and node information of two communication nodes having a connection relationship.

[0112] The network resource information includes node resource information of the plurality of communication nodes. The node resource information includes node bandwidth information, and in a possible implementation, further includes delay and other parameter information, and in another possible implementation, further includes one or more of communication interfaces, network types, physical locations, and the like of the nodes. Different network resources also have different impacts on the communication path, the communication hop count, or the communication delay, and the like between the plurality of computing nodes.

[0113] In a possible implementation, the network resource information further includes at least one of inter-node link information and inter-node connection mode information of the plurality of communication nodes. The inter-node link information includes node information at both ends of a fixed link, and the fixed link refers to a link that has been determined in advance. If the communication arrangement method provided by the embodiments of the present application is not used, the plurality of communication nodes will build a link according to the fixed link. If the communication arrangement method provided by the embodiments of the present application is used, the fixed link can be optimized through pruning, and the plurality of communication nodes will build a link according to the pruned link. The inter-node connection mode information indicates whether the nodes are electrically connected, optically connected, or connected in other types.

[0114] In a possible implementation, in addition to obtaining the network topology information and the network resource information, communication policy information and the like can also be obtained, the communication policy information representing a communication policy of the plurality of communication nodes, such as a congestion control policy, a load balancing policy, a routing policy, a buffer size, and the like adopted by the communication nodes.

[0115] In the embodiment of the present application, there are many ways to obtain the model information and the running policy information of the computing model, such as obtaining a part of the information by obtaining a Rank table file and obtaining another part of the information by using some interfaces of the computing model. The embodiment of the present application does not limit the specific way of obtaining the information. There are also many ways to obtain the network topology information and the network resource information, such as obtaining the information by using the controllers of the plurality of communication nodes. The embodiment of the present application also does not limit the specific way of obtaining the information.

[0116] Step 302: determining a communication orchestration result of the plurality of computing nodes based on the model information, the running policy information, the network topology information, and the network resource information.

[0117] Referring to FIG. 4, in the embodiment of the present application, determining the communication orchestration result of the plurality of computing nodes based on the model information, the running policy information, the network topology information, and the network resource information includes the following steps 3021 to 3023.

[0118] Step 3021: determining traffic demand information based on the model information and the running policy information, the traffic demand information representing a communication demand of the plurality of computing nodes.

[0119] In the embodiment of the present application, one specific implementation of determining the traffic demand information based on the model information and the running policy information is as follows: determining each pair of computing nodes that need to communicate based on the running policy information; determining running related information of each pair of computing nodes that need to communicate based on the model information and the running policy information; and determining a communication demand of each pair of computing nodes that need to communicate based on the running related information of each pair of computing nodes that need to communicate.

[0120] The each pair of computing nodes that need to communicate is a pair of computing nodes that need to communicate in the plurality of computing nodes. That is, based on the running policy information, at least one pair of computing nodes can be determined from the plurality of computing nodes, and each pair of computing nodes includes two computing nodes that need to communicate. In other words, the each pair of computing nodes is two computing nodes included in any one of the at least one pair of computing nodes.

[0121] Each two computing nodes that need to communicate are determined, that is, each two computing nodes that need to communicate are represented by a communication domain. For example, the communication domain is [0, 3] and [2, 4], which indicates that the computing node 0 needs to communicate with the computing node 3, and the computing node 2 needs to communicate with the computing node 4. Of course, the communication domain can be represented in other ways in addition to the [0, 3] manner, and the embodiments of the present application do not limit this. In the embodiments of the present application, the division of the communication domain is related to the configuration of the computing cluster (that is, the cluster formed by multiple computing nodes) and the specific implementation of the training framework, and the determination method of the communication domain can refer to related technologies, and the embodiments of the present application do not make too much introduction.

[0122] The running related information of each two computing nodes that need to communicate includes information related to the two computing nodes in the model information and the running strategy information, such as including a part of the structure related to the running of the two computing nodes in the model structure, the parameter amount related to the running of the two computing nodes in the model parameter amount, the parallel strategy information related to the communication between the two computing nodes in the running strategy information, and the running configuration parameter.

[0123] In a possible implementation, the traffic demand information includes at least one of the communication traffic size and the predicted communication time between each two computing nodes that need to communicate.

[0124] The communication traffic size includes the communication traffic size when the target parallel strategy is used for communication. The target parallel strategy refers to the parallel strategy used between the corresponding two computing nodes, that is, the parallel strategy used between the two computing nodes that need to communicate.

[0125] Next, the specific implementation of determining the traffic demand information based on the model information and the running strategy information in the embodiments of the present application is exemplarily introduced.

[0126] Taking the model parameter amount including the size of the hidden layer, the running configuration parameter including the micro-batch size and the input sequence length, and the parallel strategy information including the parallel degree as an example, assuming that the size of the hidden layer is denoted as h, the micro-batch size is denoted as mbs, and the input sequence length is denoted as seqlen, and the parallel degree is represented by the parallel name, such as TP representing the tensor parallel degree, TP = 8 representing the tensor parallel degree being 8. In addition, FP is used to represent the training precision, for example, when the training precision is Float16, FP = 2, and when the training precision is Float32, FP = 4. For a sparse mixture expert (MoE) model, E is used to represent the number of experts that the MoE model has, EP represents the expert parallel degree, top_k represents that the token is processed by the top k experts, and cap represents the upper limit of the token that each expert can process.

[0127] Table 1 shows the calculation formula of the communication traffic size of a single card (i.e., one computing node) under different parallel strategies (also referred to as parallel modes).

[0128] Table 1

[0129] In a possible implementation, after determining each two computing nodes that need to communicate based on the running strategy information, and determining the running related information of each two computing nodes that need to communicate based on the model information and the running strategy information, the communication traffic size between each two computing nodes that need to communicate can be calculated according to the formula in Table 2.

[0130] In a possible implementation, the expected communication time between each two computing nodes that need to communicate can also be determined based on the running related information of each two computing nodes that need to communicate. There are many specific implementations, which are not limited by the embodiments of the present application.

[0131] In the embodiments of the present application, the traffic demand information can be represented by a traffic communication matrix. Table 2 is a schematic diagram of a traffic communication matrix provided by the embodiments of the present application. Table 2 takes eight computing nodes as an example, and the numbers of the eight computing nodes are N_0, N_1, N_2, N_3, N_4, N_5, N_6, and N_7 in sequence. It is assumed that TP=4, PP=2, and the training precision is Float16. For TP, the single card communication traffic size is mbsxseqlenxhx2 (denoted as X), and the corresponding communication domain is [0~3] and [4~7]. For PP, the single card communication traffic size is mbsxseqlenxhx2 / TP (denoted as Y), and the corresponding communication domain is [0, 4], [1, 5], [2, 6], and [3, 7]. The traffic communication matrix corresponding to the eight computing nodes is shown in Table 2. Wherein, "0" indicates that communication is not needed between the corresponding two computing nodes, and "X" and "Y" indicate that communication is needed between the corresponding two computing nodes. Wherein, "X" indicates the communication traffic size when TP parallel and Ring AllReduce communication operator are used for communication, and "Y" indicates the communication traffic size when PP parallel and Send / Receive mode are used for communication.

[0132] Table 2

[0133] In addition, the traffic communication matrix can also include timing information, for example, a TP communication traffic cycle is P (such as a time length), indicating that communication traffic with a size of X occurs every P time length, which can be represented by replacing "X" in Table 2 with "X, P" to represent the communication traffic size and timing information. The timing information can represent the expected communication time to some extent.

[0134] Step 3022: Based on the network topology information and the network resource information, determine the available resource information, which represents the network resources available to the plurality of computing nodes.

[0135] In the embodiments of the present application, the available resource information includes available bandwidth information. The available bandwidth can also be referred to as effective bandwidth. In some other embodiments, the available resource information also includes available port information, etc. Hereinafter, the available resource information including the available bandwidth information is mainly taken as an example for introduction.

[0136] For scenarios where there are many-to-one (i.e., traffic from multiple ingress ports to one egress port), large-to-small (i.e., traffic from high-speed ports to low-speed ports, such as traffic from 100 Gbps ports to 80 Gbps ports), and physical links actually have different efficiencies, traffic queuing and even congestion often occur in these scenarios, resulting in increased traffic latency and decreased port efficiency. For these scenarios, the embodiments of the present application provide two ways to model the effective bandwidth of the link to determine the available bandwidth information.

[0137] The first way is to use actual measurement. That is, by testing the actual traffic bandwidth in the network, the effective bandwidth of the network is modeled. For example, assuming that the nominal value of the link bandwidth is 100 Gbps, but through network measurement for a period of time, it is found that the actual rate is only 80 Gbps, then the bandwidth efficiency is determined to be 0.8, that is, the available bandwidth information is 80%. Communication nodes such as switches and routers usually support related measurement methods.

[0138] The second way is a mathematical theory modeling method. Queuing theory, network calculus, and other mathematical theories can be used for modeling and analysis to determine the available bandwidth information. For example, assuming that the modeling function is f(x), x includes the current network topology, node nominal bandwidth, and other related physical environment parameters, then the actual bandwidth (i.e., available bandwidth) is f(x).

[0139] In addition to the above two ways to determine the available resource information, other ways can also be used to determine the available resource information, which is not limited in the embodiments of the present application.

[0140] Step 3023: Based on the traffic demand information and the available resource information, determine the communication orchestration result.

[0141] In a possible implementation, the communication orchestration result comprises at least one of traffic orchestration information and device configuration information. The traffic orchestration information comprises at least one of a traffic OD matrix, traffic packet sending time, and packet sending path, and the device configuration information comprises at least one of a communication queue, traffic priority, and a communication operator implementation. In addition, the specific content included in the communication orchestration result can also be determined according to requirements.

[0142] In a possible implementation, the communication orchestration result further comprises operator execution optimization information of the computing model, which indicates at least one of a change of an operator graph of the computing model relative to an operator graph before communication orchestration, fusion between computing operators in the computing model, and a masking configuration between communication and computation.

[0143] For example, the operator execution optimization information comprises at least one of an optimized computing graph, operator fusion information, and a masking configuration between communication and computation. The optimized computing graph indicates a change of an operator graph of the computing model after communication orchestration relative to an operator graph before communication orchestration. The operator fusion information indicates fusion between computing operators in the computing model, that is, which computing operators are fused and into what computing operator. The masking configuration between communication and computation indicates a masking condition between communication tasks and computing tasks of the computing model, for example, an execution order between multiple communication tasks and multiple computing tasks of the computing model, and whether at least one communication task can be masked (or hidden, or parallel) in at least one computing task, for example, whether an execution process of a communication task can be orchestrated in a computing process of a computing task, thereby masking the communication task.

[0144] It should be understood that, by optimizing the computing graph, the computing time can be reduced. By fusing the operators, the computing time can also be reduced. By masking the communication task in the computing task, the time occupied by the communication task can be reduced, and the running efficiency of the computing model can be improved.

[0145] In the embodiments of the present application, based on the traffic demand information and the available resource information, one implementation of determining the communication orchestration result is: determining the communication orchestration result by an optimization algorithm based on the traffic demand information and the available resource information. That is, by abstracting the communication orchestration problem into a mathematical optimization problem, an optimal solution of the communication orchestration is obtained, thereby providing a theoretical support for conflict-free traffic orchestration.

[0146] In a possible implementation, the target parameter of the optimization algorithm includes at least one of an end-to-end training time, an end-to-end inference time, an end-to-end communication latency, an end-to-end communication link quantity, and an end-to-end communication hop quantity. The end-to-end training time refers to a time from when a computing model starts processing a training data to when the computing model completes gradient updating based on the training data in a model training scenario, and includes a forward calculation time, a reverse gradient calculation time, a communication time, a gradient updating time, and the like. The end-to-end inference time refers to an inference time of the computing model in a model inference scenario. The end-to-end communication latency refers to a sum of communication latencies of computing nodes that need to communicate during running of the computing model. The end-to-end communication link quantity refers to a sum of communication link quantities of the computing nodes that need to communicate during running of the computing model. The end-to-end communication hop quantity refers to a sum of communication hop quantities of the computing nodes that need to communicate during running of the computing model.

[0147] Based on this, based on the traffic demand information and the available resource information, the implementation process of determining the communication orchestration result by the optimization algorithm includes: determining, based on the traffic demand information and the available resource information, the communication orchestration result that minimizes the target parameter by the optimization algorithm.

[0148] The optimization algorithm includes at least one of a linear programming algorithm, a heuristic algorithm, and a meta-heuristic algorithm.

[0149] Next, an implementation of determining the communication orchestration result by the linear programming algorithm is exemplarily introduced.

[0150] After the traffic demand information and the available resource information are determined, the input of the linear programming algorithm is the traffic demand information and the available resource information, the traffic demand information is represented by an n*n traffic communication matrix, and the available resource information is represented by an n*n available resource matrix. ij The traffic communication matrix represents a traffic size that needs to be transmitted between the computing node i and the computing node j (the computing node i and the computing node j can be directly connected or multi-hop connected); and the communication orchestration output includes a packet sending path between each two computing nodes that need to communicate and a traffic size on each path. The constraint condition of the linear programming algorithm is that the total communication time is minimized and the link quantity is as few as possible, that is, the target parameter includes the end-to-end training / inference time and the end-to-end communication link quantity.

[0151] FIG. 5 is a schematic diagram of linear programming provided by an embodiment of the present application. As shown in FIG. 5, taking three computing nodes as an example, the traffic communication matrix is represented as follows: ij The traffic communication matrix represents a traffic size that needs to be transmitted between the computing node i and the computing node j (the computing node i and the computing node j can be directly connected or multi-hop connected); and the communication orchestration output includes a packet sending path between each two computing nodes that need to communicate and a traffic size on each path. The constraint condition of the linear programming algorithm is that the total communication time is minimized and the link quantity is as few as possible, that is, the target parameter includes the end-to-end training / inference time and the end-to-end communication link quantity. ij The traffic communication matrix represents a traffic size that needs to be transmitted between the computing node i and the computing node j (the computing node i and the computing node j can be directly connected or multi-hop connected); and the communication orchestration output includes a packet sending path between each two computing nodes that need to communicate and a traffic size on each path. The constraint condition of the linear programming algorithm is that the total communication time is minimized and the link quantity is as few as possible, that is, the target parameter includes the end-to-end training / inference time and the end-to-end communication link quantity. ij The traffic communication matrix represents a traffic size that needs to be transmitted between the computing node i and the computing node j (the computing node i and the computing node j can be directly connected or multi-hop connected); and the communication orchestration output includes a packet sending path between each two computing nodes that need to communicate and a traffic size on each path. The constraint condition of the linear programming algorithm is that the total communication time is minimized and the link quantity is as few as possible, that is, the target parameter includes the end-to-end training / inference time and the end-to-end communication link quantity.ij = 1 / η ij In a specific embodiment, the linear programming function of capacity-constrained unbalanced optimal transport (CUOT) can be used to solve the linear programming problem shown in FIG. 5 based on parameters such as η ij , C ij , and the like.

[0152] The related introduction of the linear programming algorithm can also refer to the related technology, and the embodiments of the present application will not be explained too much.

[0153] In addition to the linear programming algorithm, heuristic algorithms or meta-heuristic algorithms such as tabu-search algorithm, ant colony algorithm, genetic algorithm, etc. can also be used to determine the communication arrangement result. Taking the tabu-search algorithm as an example, the implementation of the tabu-search algorithm includes the following steps:

[0154] 1. Construct the objective function, assuming that the end-to-end communication delay is the target parameter.

[0155] 2. Initialize the search space, including randomly generating the communication flow and communication time between two points.

[0156] 3. For the parameters in the search space, solve the objective function, select the local optimal solution in the search space, and refresh the tabu table.

[0157] 4. For the parameters of the current search space, generate neighbor solutions by randomly changing some variables, and refresh to a new search space.

[0158] 5. Repeat steps 3-5 for the new search space.

[0159] 6. When the optimal solution no longer changes after several iterations, or the set number of iterations is reached, output the final solution.

[0160] In addition, for heuristic algorithms such as ant colony algorithm and genetic algorithm, the specific method of solving may be different according to the specific framework they adopt, but the general idea is similar, and the embodiments of the present application will not be introduced in detail.

[0161] From the above, in the embodiments of the present application, each two computing nodes that need to communicate can be determined based on the running strategy information, and then the communication demand of each two computing nodes that need to communicate can be determined based on the model information and the running strategy information. After that, the communication arrangement result is determined based on the communication demand of each two computing nodes that need to communicate, the network topology information and the network resource information.

[0162] In some embodiments, the information obtained further includes communication strategy information. In this case, in one possible implementation, step 302 can include determining the communication orchestration result of the plurality of computing nodes based on the model information, the running strategy information, the network topology information, the network resource information, and the communication strategy information. Correspondingly, step 3023 can include determining the communication orchestration result based on the traffic demand information, the available resource information, and the communication strategy information.

[0163] As can be seen from the above, in some embodiments, the information obtained further includes communication strategy information. In this case, in one possible implementation, step 302 can include determining the communication orchestration result of the plurality of computing nodes based on the model information, the running strategy information, the network topology information, the network resource information, and the communication strategy information. Correspondingly, step 3023 can include determining the communication orchestration result based on the traffic demand information, the available resource information, and the communication strategy information.

[0164] After obtaining the communication orchestration result, the communication orchestration result can be distributed or configured to the corresponding communication nodes. For example, device configuration information such as communication queue, traffic priority, and communication operator implementation mode is configured to the corresponding communication nodes, and traffic OD matrix, traffic packet sending time, and traffic packet sending path are distributed to the corresponding communication nodes.

[0165] In a specific embodiment, the communication orchestration result can be compiled and combined with the ensemble communication library interface to complete the configuration and distribution of the communication orchestration result. Taking a communication node as an example, the switch can be configured through Restful. The above configuration and distribution operations can also be completed by calling existing interfaces in Pytorch, MindSpore, etc. (to reuse existing software and hardware configuration interfaces as much as possible) or adding new interfaces.

[0166] Next, the communication orchestration method provided by the embodiments of the present application will be exemplarily explained again in combination with FIG. 6.

[0167] FIG. 6 is a logic diagram of a communication orchestration method according to an embodiment of the present application. Referring to FIG. 6, model information is obtained through a model information interface (not shown), running strategy information is obtained through a running strategy interface (not shown) (for example, parallel strategy information is obtained through a parallel strategy interface (not shown)), and network topology structure and network resource information are obtained through a topology structure and physical environment information interface (not shown). Then, traffic demand information is obtained by analyzing the model information and the running strategy information, and available resource information is obtained by analyzing the network topology structure and the network resource information. After that, the optimal communication orchestration result is determined by an optimization algorithm with the traffic demand information and the available resource information as constraint conditions.

[0168] The communication arrangement method provided by the embodiments of the present application can be integrated into a training framework such as Pytorch or MindSpore, specific software such as a switch or a router, or a product combining software and hardware, or can also be implemented as an independent software and hardware entity.

[0169] One of the scenarios to which the embodiments of the present application can be applied is a distributed model training scenario under a given cluster configuration, where the given cluster configuration is, for example, a fixed link. The embodiments of the present application can realize conflict-free traffic arrangement between different communication domains and traffic communication matrices under various parallel strategies by analyzing and modeling the model structure and the cluster interconnection architecture, thereby reducing the end-to-end training communication time overhead and optimizing the end-to-end training performance.

[0170] Another applicable scenario of the embodiments of the present application is a scenario of optimizing the training strategy configuration and the network topology. The embodiments of the present application can determine the best interconnection bandwidth and communication path in the cluster by conflict-free arrangement of traffic, so as to prune and converge the network topology, avoid bandwidth redundancy and waste, and reduce the deployment cost of the cluster network.

[0171] The pruning of the network topology includes removing a link in the network topology that is not on the packet sending path in the communication arrangement result. For example, removing a link with no traffic or a link with a high cost.

[0172] Exemplarily, taking four communication nodes as an example, before the communication arrangement, the network topology indicates that there is a link between each two of the four communication nodes. However, after the communication arrangement, if the packet sending path in the communication arrangement result does not include the packet sending path between communication node 1 and communication node 2, it indicates that communication node 1 and communication node 2 will not communicate, and then the link between communication node 1 and communication node 2 can be removed.

[0173] The communication arrangement method provided by the embodiments of the present application can support distributed models of a cluster pullout (i.e., the distance between the computing nodes in the pullout computing cluster) or even cross-domain clusters, that is, the communication arrangement method provided by the embodiments of the present application is applicable regardless of whether the distance between the computing nodes in the computing cluster is far or near or whether the computing nodes are cross-domain.

[0174] In the embodiment of the present application, the model information and the running strategy information imply the communication requirements of the plurality of computing nodes in the running process of the computing model, the network topology information and the network resource information represent the links and communication resources that can be provided for the communication between the plurality of computing nodes, and the plurality of computing nodes are more reasonably arranged for communication by comprehensively analyzing the model information and the running strategy information of the computing model and the network topology information and the network resource information of the plurality of computing nodes, so as to obtain a more optimal communication arrangement result that meets the above communication requirements, thereby reducing communication conflicts, reducing communication latency, and improving overall communication performance.

[0175] FIG. 7 is a structural schematic diagram of a communication arrangement device provided by an embodiment of the present application. The communication arrangement device 700 can be realized by software, hardware, or a combination of both as part or all of a computer device. The computer device can be the computer device shown in FIG. 2. In the embodiment of the present application, the computing model is distributed and runs on a plurality of computing nodes. Referring to FIG. 7, the device 700 includes an acquisition module 701 and a determination module 702.

[0176] The acquisition module 701 is configured to acquire model information and running strategy information of a computing model, and network topology information and network resource information of a plurality of computing nodes.

[0177] The determination module 702 is configured to determine a communication arrangement result of the plurality of computing nodes based on the model information, the running strategy information, the network topology information, and the network resource information.

[0178] In a possible implementation, the determination module 702 includes:

[0179] A first determination sub-module is configured to determine, based on the running strategy information, each two computing nodes that need to communicate.

[0180] A second determination sub-module is configured to determine, based on the model information and the running strategy information, communication requirements of each two computing nodes that need to communicate.

[0181] A third determination sub-module is configured to determine, based on the communication requirements of each two computing nodes that need to communicate, the network topology information, and the network resource information, the communication arrangement result.

[0182] In a possible implementation, the second determination sub-module is specifically configured to:

[0183] Determine, based on the model information and the running strategy information, running related information of each two computing nodes that need to communicate.

[0184] Determine, based on the running related information of each two computing nodes that need to communicate, the communication requirements of each two computing nodes that need to communicate.

[0185] In a possible implementation, the model information comprises a model structure and a model parameter quantity.

[0186] The running strategy information comprises at least one of a running configuration parameter and parallel strategy information.

[0187] The network topology information comprises node identification of a plurality of communication nodes and inter-node topology information, the plurality of communication nodes being configured to implement communication between the plurality of computing nodes.

[0188] The network resource information comprises node resource information of the plurality of communication nodes.

[0189] In a possible implementation, the running configuration parameter comprises at least one of a global batch size, a micro batch size and an input sequence length of the computing model in a training case.

[0190] In a possible implementation, the determining module 702 comprises:

[0191] A fourth determining sub-module, configured to determine, based on the model information and the running strategy information, traffic demand information, the traffic demand information representing communication demand of the plurality of computing nodes.

[0192] A fifth determining sub-module, configured to determine, based on the network topology information and the network resource information, available resource information, the available resource information representing network resources available to the plurality of computing nodes.

[0193] A sixth determining sub-module, configured to determine, based on the traffic demand information and the available resource information, the communication orchestration result.

[0194] In a possible implementation, the sixth determining sub-module is specifically configured to:

[0195] determine, based on the traffic demand information and the available resource information, the communication orchestration result by an optimization algorithm, the communication orchestration result minimizing at least one of an end-to-end training time, an end-to-end inference time, an end-to-end communication latency, an end-to-end communication link quantity and an end-to-end communication hop quantity.

[0196] In a possible implementation, the traffic demand information comprises at least one of a communication traffic size and an expected communication time between each two computing nodes that need to communicate.

[0197] In a possible implementation, the communication traffic size comprises a communication traffic size when communication is performed in a target parallel strategy, the target parallel strategy being a parallel strategy adopted between the two corresponding computing nodes.

[0198] In a possible implementation, the communication orchestration result comprises at least one of traffic orchestration information and device configuration information.

[0199] The traffic orchestration information comprises at least one of a traffic OD matrix, traffic packet sending time, and packet sending path.

[0200] The device configuration information comprises at least one of a communication queue, traffic priority, and a communication operator implementation.

[0201] In a possible implementation, the communication orchestration result comprises operator execution optimization information.

[0202] The operator execution optimization information comprises at least one of an optimized computation graph, operator fusion information, and a masking configuration between communication and computation.

[0203] The optimized computation graph indicates a change of an operator graph of the computation model after communication orchestration relative to an operator graph before communication orchestration.

[0204] The operator fusion information indicates fusion between computation operators in the computation model.

[0205] The masking configuration between communication and computation indicates a masking condition between a communication task and a computation task of the computation model.

[0206] In the embodiments of the present application, the model information and the running strategy information imply communication requirements of multiple computing nodes in the running process of the computation model, and the network topology information and the network resource information represent links and communication resources that can be provided for communication between the multiple computing nodes. Through comprehensive analysis of the model information and the running strategy information of the computation model and the network topology information and the network resource information of the multiple computing nodes, the multiple computing nodes are more reasonably orchestrated, so as to obtain a more optimal communication orchestration result that meets the above communication requirements, thereby reducing communication conflicts, reducing communication latency, and improving overall communication performance.

[0207] It should be noted that the communication orchestration apparatus provided in the above embodiments is only exemplified by the division of the above functional modules in the communication orchestration process. In actual applications, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the apparatus is divided into different functional modules to complete all or part of the above described functions. In addition, the communication orchestration apparatus and the communication orchestration method embodiments provided in the above embodiments belong to the same concept, that is, the above communication orchestration apparatus can be used to realize the functions of communication orchestration in the above method embodiments, and thus can also realize the beneficial effects possessed by the above method embodiments. For specific implementation process, refer to the method embodiments, which will not be repeated here.

[0208] The embodiment of the present application further provides a computer readable storage medium, which stores instructions, and when the instructions are executed on a computer, the computer is caused to perform the steps of the communication arrangement method shown in the method embodiment.

[0209] The embodiment of the present application further provides a computer program product containing instructions, and when the instructions are executed on a computer, the computer is caused to perform the steps of the communication arrangement method shown in the method embodiment. Alternatively, the embodiment of the present application provides a computer program, and when the computer program is executed on a computer, the computer is caused to perform the steps of the communication arrangement method shown in the method embodiment.

[0210] In the above embodiment, the implementation can be achieved by software, hardware, firmware or any combination thereof, entirely or partially. When the implementation is achieved by software, the implementation can be achieved in the form of a computer program product, entirely or partially. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is entirely or partially generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer readable storage medium or transferred from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be transferred from one website, computer, server or data center to another website, computer, server or data center through wired (for example: coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (for example: infrared, wireless, microwave, etc.) mode. The computer readable storage medium can be any available medium that can be accessed by a computer, or a data storage device such as a server, data center, etc. integrated with one or more available media. The available medium can be a magnetic medium (for example: floppy disk, hard disk, magnetic tape), an optical medium (for example: digital versatile disc (DVD)) or a semiconductor medium (for example: solid state disk (SSD)) and the like. It is worth noting that the computer readable storage medium mentioned in the embodiment of the present application can be a non-volatile storage medium, in other words, it can be a non-transitory storage medium.

[0211] It should be understood that "at least one" mentioned herein refers to one or more, and "multiple" refers to two or more. In the description of the embodiments of the present application, " / " represents the meaning of or, for example, A / B can represent A or B; "and / or" herein only describes the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can represent: A exists alone, A and B exist together, and B exists alone. In addition, in order to clearly describe the technical solutions of the embodiments of the present application, in the embodiments of the present application, "first", "second" and the like are used to distinguish the same items or similar items with basically the same function and effect. The skilled in the art can understand that "first", "second" and the like do not limit the quantity and execution order, and "first", "second" and the like do not necessarily mean different.

[0212] It should be noted that the information (including but not limited to user equipment information, user personal information, etc.), data (including but not limited to data for analysis, stored data, displayed data, etc.) and signals involved in the embodiments of the present application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0213] The above describes the embodiments provided by the present application, and does not limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A communication orchestration method, characterized by, The computing model is distributed to run on a plurality of computing nodes, and the method comprises: obtaining model information and running strategy information of the computing model, and network topology information and network resource information of the plurality of computing nodes; determining a communication arrangement result of the plurality of computing nodes based on the model information, the running strategy information, the network topology information and the network resource information.

2. The method of claim 1, wherein, The determination of the communication arrangement result of the plurality of computing nodes based on the model information, the running strategy information, the network topology information and the network resource information comprises: determining each two computing nodes that need to communicate based on the running strategy information; determining communication requirements of each two computing nodes that need to communicate based on the model information and the running strategy information; determining the communication arrangement result based on the communication requirements of each two computing nodes that need to communicate, the network topology information and the network resource information.

3. The method of claim 2, wherein, The determination of the communication requirements of each two computing nodes that need to communicate based on the model information and the running strategy information comprises: determining running-related information of each two computing nodes that need to communicate based on the model information and the running strategy information; determining the communication requirements of each two computing nodes that need to communicate based on the running-related information of each two computing nodes that need to communicate.

4. The method of any one of claims 1-3, wherein: the model information comprises model structure and model parameter quantity; the running strategy information comprises at least one of running configuration parameter and parallel strategy information; the network topology information comprises node identification and inter-node topology information of a plurality of communication nodes, the plurality of communication nodes being used to implement communication between the plurality of computing nodes; the network resource information comprises node resource information of the plurality of communication nodes.

5. The method of claim 4, wherein, The running configuration parameter comprises at least one of global batch size, micro batch size and input sequence length of the computing model in a training case.

6. The method according to any one of claims 1 to 5, wherein, The determination of the communication arrangement result of the plurality of computing nodes based on the model information, the running strategy information, the network topology information and the network resource information comprises: determining traffic demand information representing communication requirements of the plurality of computing nodes based on the model information and the running strategy information; determining available resource information representing network resources available for use by the plurality of computing nodes based on the network topology information and the network resource information; determining the communication arrangement result based on the traffic demand information and the available resource information.

7. The method of claim 6, wherein, The determination of the communication arrangement result based on the traffic demand information and the available resource information comprises: determine, based on the traffic demand information and the available resource information, the communication orchestration result that minimizes an objective parameter of an optimization algorithm, the objective parameter comprising at least one of end-to-end training time, end-to-end inference time, end-to-end communication latency, end-to-end communication link number, and end-to-end communication hop number.

8. The method of claim 6 or 7, wherein, The traffic demand information comprises at least one of communication traffic size and expected communication time between each two computing nodes that need to communicate.

9. The method of claim 8, wherein, The communication traffic size comprises a communication traffic size when communication is performed using a target parallel strategy, the target parallel strategy being a parallel strategy used between the corresponding two computing nodes.

10. The method of any one of claims 1-9, wherein, The communication orchestration result comprises at least one of traffic orchestration information and device configuration information. The traffic orchestration information comprises at least one of traffic origin-destination (OD) matrix, traffic packet sending time, and packet sending path. The device configuration information comprises at least one of communication queue, traffic priority, and communication operator implementation.

11. The method of any one of claims 1-10, wherein, The communication orchestration result comprises operator execution optimization information. The operator execution optimization information comprises at least one of an optimized computation graph, operator fusion information, and a mask configuration between communication and computation. The optimized computation graph indicates changes in an operator graph of the computing model after communication orchestration relative to the operator graph before communication orchestration. The operator fusion information indicates fusion between computing operators in the computing model. The mask configuration between communication and computation indicates a mask condition between communication tasks and computing tasks of the computing model.

12. A communication orchestration apparatus, characterized by comprising: A computing model is distributed and run on a plurality of computing nodes, and the apparatus comprises: an obtaining module configured to obtain model information and running strategy information of the computing model, and network topology information and network resource information of the plurality of computing nodes; a determining module configured to determine, based on the model information, the running strategy information, the network topology information, and the network resource information, a communication orchestration result of the plurality of computing nodes.

13. The apparatus of claim 12, wherein, The determining module comprises: a first determining sub-module configured to determine, based on the running strategy information, each two computing nodes that need to communicate; a second determining sub-module configured to determine, based on the model information and the running strategy information, communication demand of each two computing nodes that need to communicate; a third determining sub-module configured to determine, based on the communication demand of each two computing nodes that need to communicate, the network topology information, and the network resource information, the communication orchestration result.

14. The apparatus of claim 12 or 13, wherein the model information comprises model structure and model parameter quantity; the running strategy information comprises at least one of running configuration parameter and parallel strategy information; the network topology information comprises node identification and inter-node topology information of a plurality of communication nodes, the plurality of communication nodes being used to implement communication between the plurality of computing nodes; the network resource information comprises node resource information of the plurality of communication nodes.

15. The apparatus of any one of claims 12-14, wherein, The determining module comprises: a fourth determining sub-module, configured to determine traffic demand information based on the model information and the running strategy information, the traffic demand information representing communication demands of the plurality of computing nodes; a fifth determining sub-module, configured to determine available resource information based on the network topology information and the network resource information, the available resource information representing network resources available to the plurality of computing nodes; a sixth determining sub-module, configured to determine the communication orchestration result based on the traffic demand information and the available resource information.

16. The apparatus of claim 15, wherein, The sixth determining sub-module is specifically configured to: determine, based on the traffic demand information and the available resource information, the communication orchestration result that minimizes a target parameter of an optimization algorithm, the target parameter including at least one of end-to-end training time, end-to-end inference time, end-to-end communication latency, end-to-end communication link quantity, and end-to-end communication hop quantity.

17. The apparatus of claim 15 or 16, wherein, The traffic demand information includes at least one of communication traffic size and predicted communication time between each two computing nodes that need to communicate.

18. The apparatus of any one of claims 12-17, wherein, The communication orchestration result includes at least one of traffic orchestration information and device configuration information. The traffic orchestration information includes at least one of traffic origin-destination (OD) matrix, traffic packet sending time, and packet sending path. The device configuration information includes at least one of communication queue, traffic priority, and communication operator implementation.

19. The apparatus of any one of claims 12-18, wherein, The communication orchestration result includes operator execution optimization information. The operator execution optimization information includes at least one of an optimized computation graph, operator fusion information, and communication and computation masking configuration. The optimized computation graph indicates changes in an operator graph of the computing model after communication orchestration relative to the operator graph before communication orchestration. The operator fusion information indicates fusion between computing operators in the computing model. The communication and computation masking configuration indicates masking between communication tasks and computing tasks of the computing model.

20. A computer program product, characterised in that, The computer program product stores computer instructions, and the computer instructions are executed by a processor to implement the method in any one of claims 1-11.

Images