Data communication method, distributed system and related equipment

Abstract

The invention provides a data communication method, a distributed system and related equipment, and the method comprises the steps: before the distributed training of an AI model is started, the distributed system determines the number M of cards matched with cross-regional communication in advance based on the communication bandwidth converged between regions, so that after the system starts the distributed training, if set communication is needed, the number M of cards matched with the cross-regional communication is determined; the number of the processors in each time of cross-regional communication is M, and the communication traffic of the M processors in the cross-regional communication is ensured not to exceed the cross-regional bandwidth, so that the cross-regional communication can be ensured not to be congested, the problems of a back pressure mechanism caused by the congestion and further reduction of the cross-regional communication bandwidth are avoided, and the communication efficiency is improved. Therefore, the overall communication efficiency of integrated communication is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence (AI), and more particularly to a data communication method, a distributed system, and related devices. Background Technology

[0002] With the rapid development of AI models, the parameter scale of large models and the training datasets are becoming increasingly larger. This necessitates larger distributed systems for training these models, leading to a significant increase in hardware costs. To reduce these costs, bandwidth-converging networking architectures have emerged. Bandwidth convergence refers to a distributed system where multiple regions exist, each with high bandwidth, while the bandwidth between spine nodes connecting different regions is relatively low. This reduces the number of switches and optimizes the use of fiber optic cables, thereby lowering hardware costs.

[0003] However, due to bandwidth convergence between spine nodes, cross-regional communication can become congested when data traffic is too high. To cope with congestion, network protocols usually trigger backpressure mechanisms, where the sender reduces the data transmission rate, further reducing the effective bandwidth of cross-regional interconnection, which in turn affects the overall efficiency of ensemble communication and the overall training efficiency of the model. Summary of the Invention

[0004] This application provides a data communication method, a distributed system, and related equipment to solve the problem of low efficiency in cross-regional communication in distributed systems, which in turn affects the overall efficiency of aggregated communication.

[0005] Firstly, a data communication method is provided, which is applied to a distributed system. The distributed system includes a first region and a second region, each region including multiple processors. The distributed system is used to execute a distributed training task of an AI model, and each processor is used to execute a subtask of the distributed training task. The method includes the following steps: a first processor in the first region acquires first data to be communicated; the first processor communicates with other processors in the first region within the region, sends the first data to be communicated to other processors in the first region, receives data sent by other processors in the first region, and obtains a communication result within the region; the first processor communicates with a second processor in the second region across regions, sends the communication result within the region to the second processor, and receives data sent by the second processor to obtain a communication result across regions. When the first processor communicates with the second processor across regions, M-1 processors in the first region also communicate across regions. The total communication bandwidth of the M processors formed by the first processor and the M-1 processors is adapted to the total communication bandwidth between the first region and the second region; the first processor communicates with other processors in the first region within the region, sends the communication result across regions to other processors in the first region, and receives data sent by other processors in the first region to obtain a communication result across regions.

[0006] Implementing the method described in the first aspect, before the distributed training of the AI ​​model begins, the distributed system pre-determines the number of concurrent cards M adapted for cross-regional communication based on the communication bandwidth of inter-regional convergence. This ensures that after the system starts distributed training, if aggregate communication is required, the number of processors for each cross-regional communication is M, ensuring that the communication volume of M processors during cross-regional communication does not exceed the cross-regional bandwidth. This ensures that cross-regional communication will not experience congestion, avoids backpressure mechanisms caused by congestion and further reduction of cross-regional communication bandwidth, thereby improving the overall communication efficiency of aggregated communication.

[0007] In one possible implementation, each region includes M communication groups, and each of the M communication groups includes at least one processor. The M processors participating in cross-region communication form different communication groups.

[0008] In the above implementation, during cross-regional communication, each communication group has only one processor communicating externally. With M communication groups, there are M processors simultaneously conducting cross-regional communication, ensuring that the amount of communication during each cross-regional communication does not exceed the cross-regional bandwidth, thereby ensuring that cross-regional communication will not experience congestion.

[0009] In one possible implementation, the cross-regional communication type includes sequential communication, whereby after the first processor communicates with the second processor in the second region, the first processor sends a notification message to the third processor, which instructs the third processor to communicate with the fourth processor in the second region. The first and third processors belong to a first communication group, and the second and fourth processors belong to a second communication group.

[0010] In practice, processors within the same communication group can take turns performing cross-region communication. After each processor completes its cross-region communication, it sends a notification message to the next processor in the same communication group according to the communication order. The next processor can then respond to the notification message and begin cross-region communication. This process continues until the last processor finishes its communication and sends a notification message to the first processor. The first processor then has no more data to communicate, and the cross-region communication process terminates. In this approach, if the number of processors in the communication group is N, N cross-region communications will be performed.

[0011] Specifically, the first intra-regional communication can be achieved using `reducescatter`. Each processor sends its local data to be communicated to other processors within the region, and then receives the data to be communicated from other processors within the region, obtaining the intra-regional communication result. Next, cross-regional communication can be achieved using `allreduce`. Each processor sends its local intra-regional communication result to processors in other regions and receives the intra-regional communication results from processors in other regions, obtaining the cross-regional communication result. Then, it notifies the next processor to start cross-regional communication, and so on, until all processors have obtained the corresponding cross-regional communication results. Finally, the second intra-regional communication can be achieved using `allgather`. Each processor sends the obtained cross-regional communication result to other nodes within the region and receives the cross-regional communication results from other processors within the region, obtaining the final aggregated communication result.

[0012] The above implementation method, by having processors within the same communication group take turns performing cross-regional communication, not only ensures that the communication volume during each cross-regional communication is adapted to the bandwidth between regions, avoiding congestion in the convergence bandwidth between the first and second regions, thus avoiding backpressure mechanisms caused by congestion and further reduction in cross-regional communication bandwidth, but also improves the overall communication efficiency of the aggregated communication. Simultaneously, the processing load is distributed more evenly among each processor, preventing any single processor from being overloaded and optimizing resource utilization.

[0013] In one possible implementation, the cross-regional communication type includes proxy communication, in which the first processor and the second processor in the second region conduct multiple cross-regional communications. The first processor is a proxy processor of the first communication group, and the first processor includes the communication data to be communicated by all processors in the first communication group. Each time cross-regional communication occurs, a total of M proxy processors in the first region participate in the cross-regional communication.

[0014] In practice, during cross-regional communication, only the proxy processor works in each communication group, while other processors do not work. The proxy processor can aggregate the data on other processors in the communication group before cross-regional communication, and then send out all the data in the communication group that needs to be communicated across regions through multiple cross-regional communications.

[0015] Furthermore, the processors within each communication group can aggregate data to the proxy processor before cross-regional communication. Specifically, the first processor performs inter-group communication with processors in other communication groups in the first region, sending data to be communicated to processors in other communication groups in the first region, receiving data sent by processors in other communication groups, and obtaining inter-group communication results. The first processor also performs intra-group communication with other processors in the first communication group, sending inter-group communication results to other processors in the first communication group, receiving data sent by other processors in the first communication group, and obtaining intra-regional communication results.

[0016] In specific implementation, the complete communication process of proxy communication can be as follows: During the first intra-regional communication, inter-group reduce-scatter can be performed within the region. Each processor sends its local data to be communicated to processors in other communication groups, receives data to be communicated from processors in other communication groups, and obtains the intra-group communication result. Then, a gather is performed within the communication group, where each processor sends its intra-group communication result to the proxy processor within the communication group, allowing the proxy processor to obtain the intra-regional communication result. Next, cross-regional communication is achieved through allreduce. Each proxy processor sends its local intra-regional communication result to proxy processors in other regions and receives intra-regional communication results from proxy processors in other regions, obtaining the cross-regional communication result. Finally, during the second intra-regional communication, intra-group scatter can be performed within the region. Each proxy processor sends cross-regional communication results to other processors within the communication group, allowing each processor to obtain the cross-regional communication results from the proxy processors within its group. Then, an inter-group allgather is performed. Each processor sends the cross-regional communication results of its own communication group to processors in other communication groups within the region, and receives cross-regional communication results from processors in other communication groups, thus obtaining the global communication result and the final aggregated communication result.

[0017] The above implementation method, where the proxy processors within each communication group perform cross-regional communication, not only ensures that the communication volume during each cross-regional communication is adapted to the bandwidth between regions, avoiding congestion of the convergence bandwidth between the first and second regions, but also reduces the number of communications. If the number of processors in the communication group is N, the number of cross-regional communications is no more than N. This is because after the data is aggregated to the proxy processor, the data sent in each aggregated communication may be the data to be communicated by two or more processors within the communication group, thus reducing the number of communications. Simultaneously, this method only requires the proxy processor to establish external communication links, which can also reduce link costs. However, the corresponding proxy processors have high performance requirements and need to have large storage capacity. The two communication types supported by this application can be flexibly selected according to actual application scenarios, allowing the technical solution of this application to be flexibly deployed to meet the needs of various users.

[0018] In one possible implementation, each region includes at least one computing node, which includes a network communication library and at least one processor. Before a first processor in a first region acquires the first data to be communicated, the first processor receives a communication strategy sent by the network communication library of the first computing node. The first computing node includes a first processor. The communication strategy is determined by the network communication library based on the number M of communication groups. The number M of communication groups is determined by the network communication library after determining the total communication bandwidth between the first region and the second region based on the network topology information of the distributed system, combined with the communication type and the processing bandwidth of each processor.

[0019] It should be understood that traditional technologies typically report network congestion to the scheduler after it is detected. The scheduler then adjusts the communication method based on some network congestion control strategies. However, this approach of adjusting only after congestion occurs can lead to network traffic jitter and fail to fully utilize network bandwidth. The above implementation determines the communication strategy between processors based on the convergence bandwidth between the first and second regions before the aggregated communication begins, thus avoiding network congestion or jitter and improving the overall aggregated communication efficiency.

[0020] Secondly, a data communication method is provided, which is applied to a distributed system. The distributed system includes a first region and a second region. Each region includes multiple computing nodes. Each computing node includes a network communication library and at least one processor. The distributed system is used to execute a distributed training task of an AI model. Each processor is used to execute a subtask of the distributed training task. The method includes the following steps: the network communication library of the first computing node among the multiple computing nodes obtains the network topology information of the distributed system; the network communication library of the first computing node determines the total communication bandwidth between the first region and the second region and the bandwidth of each processor based on the network topology information; the network communication library of the first computing node generates a communication strategy based on the total communication bandwidth and the bandwidth of each processor. The communication strategy includes: each time the processors of the first region communicate with the processors of the second region, a total of M processors in the first region participate in the cross-region communication, and the total bandwidth of the M processors is adapted to the total communication bandwidth between the first region and the second region; the network communication library of a computing node sends the communication strategy to at least one processor on the first computing node.

[0021] Implementing the method described in the second aspect, before the aggregated communication, the network communication library determines the communication strategy between processors based on the convergence bandwidth between the first and second regions, and distributes it to each processor. This ensures that when processors perform aggregated communication, they can do so according to the communication strategy. This guarantees that only M processors in each region participate in cross-regional communication, and the total bandwidth of the M processors is adapted to the convergence bandwidth between the regions. This ensures that cross-regional communication will not experience congestion, avoids backpressure mechanisms caused by congestion, and prevents further reduction in cross-regional communication bandwidth, thereby improving the overall communication efficiency of aggregated communication.

[0022] In one possible implementation, each region includes M communication groups, and each of the M communication groups includes at least one processor. The communication strategy includes: the M processors participating in cross-region communication form different communication groups.

[0023] In the above implementation, during cross-regional communication, only one processor in each communication group communicates externally. With M communication groups, M processors can simultaneously perform cross-regional communication, ensuring that the communication volume during each cross-regional communication does not exceed the cross-regional bandwidth, thus preventing congestion. Furthermore, traditional technologies typically report network congestion to the scheduler after detection, and the scheduler adjusts the communication method based on network congestion control strategies. However, this post-congestion adjustment leads to network traffic fluctuations and fails to fully utilize network bandwidth. The method provided in this application, where the controller determines the communication strategy between processors based on the convergence bandwidth between the first and second regions before aggregated communication, avoids network congestion or fluctuations, and improves the overall efficiency of aggregated communication.

[0024] Thirdly, a distributed system is provided, which includes a first region and a second region, each region including multiple processors. The distributed system is used to execute distributed training tasks of AI models, and each processor is used to execute subtasks of the distributed training tasks. The distributed system includes a first processor in the first region and a second processor in the second region.

[0025] The first processor is used to acquire the first data to be communicated, and the second processor is used to acquire the second data to be communicated.

[0026] The first processor is used to communicate with other processors in the first region, send first data to be communicated to other processors in the first region, receive data sent by other processors in the first region, and obtain communication results within the first region; the second processor is used to communicate with other processors in the second region, send second data to be communicated to other processors in the second region, receive data sent by other processors in the second region, and obtain communication results within the second region.

[0027] The first processor is used to communicate across regions with the second processor, send communication results within a first region to the second processor, and receive data sent by the second processor to obtain a first cross-regional communication result; the second processor is used to communicate across regions with the first processor, send communication results within a second region to the first processor, and receive data sent by the first processor to obtain a second cross-regional communication result. When the first processor and the second processor are communicating across regions, M-1 processors in the first region are also communicating across regions. The total communication bandwidth of the M processors formed by the first processor and the M-1 processors is adapted to the total communication bandwidth between the first region and the second region.

[0028] The first processor is used to communicate with other processors in the first region within the region, send a first cross-regional communication result to other processors in the first region, and receive data sent by other processors in the first region to obtain a set communication result; the second processor is used to communicate with other processors in the second region within the region, send a second cross-regional communication result to other processors in the second region, and receive data sent by other processors in the second region to obtain a set communication result.

[0029] Implementing the distributed system described in the third aspect, before the distributed training of the AI ​​model begins, the distributed system pre-determines the number of concurrent cards M adapted for cross-regional communication based on the communication bandwidth of inter-regional convergence. This ensures that after the system starts distributed training, if aggregate communication is required, the number of processors for each cross-regional communication is M, ensuring that the communication volume of M processors during cross-regional communication does not exceed the cross-regional bandwidth. This ensures that cross-regional communication will not experience congestion, avoids backpressure mechanisms caused by congestion and further reduction of cross-regional communication bandwidth, thereby improving the overall communication efficiency of aggregated communication.

[0030] In one possible implementation, each region includes M communication groups, and each of the M communication groups includes at least one processor. The M processors participating in cross-region communication form different communication groups.

[0031] In one possible implementation, the cross-regional communication type includes sequential communication, whereby a first processor sends a notification message to a third processor, the notification message instructing the third processor to conduct cross-regional communication with a fourth processor in the second region, wherein the first and third processors belong to a first communication group, and the second and fourth processors belong to a second communication group, and the second processor sends a notification message to the fourth processor, the notification message instructing the fourth processor to conduct cross-regional communication with the third processor in the first region.

[0032] In one possible implementation, the cross-regional communication type includes proxy communication, a first processor for performing multiple cross-regional communications with a second processor, wherein the first processor is a proxy processor of a first communication group, the first processor includes the communication data to be communicated of all processors in the first communication group, and each time cross-regional communication occurs, a total of M proxy processors in the first region participate in the cross-regional communication, and a second processor for performing multiple cross-regional communications with the first processor, wherein the second processor is a proxy processor of a second communication group, the second processor includes the communication data to be communicated of all processors in the second communication group, and each time cross-regional communication occurs, a total of M proxy processors in the second region participate in the cross-regional communication.

[0033] In one possible implementation, a first processor is configured to perform inter-group communication with processors of other communication groups in a first region, send first communication data to processors of other communication groups in the first region, receive data sent by processors of other communication groups, obtain a first inter-group communication result, perform intra-group communication with other processors of the first communication group, send the first inter-group communication result to other processors in the first communication group, receive data sent by other processors of the first communication group, and obtain a communication result within the first region. A second processor is configured to perform inter-group communication with processors of other communication groups in a second region, send second communication data to processors of other communication groups in the second region, receive data sent by processors of other communication groups, obtain a second inter-group communication result, perform intra-group communication with other processors of the second communication group, send the second inter-group communication result to other processors in the second communication group, receive data sent by other processors of the second communication group, and obtain a communication result within the second region.

[0034] In one possible implementation, each region includes at least one computing node, the at least one computing node includes a network communication library and at least one processor, a first processor for receiving a communication strategy sent by the network communication library of the first computing node, the first computing node including a first processor, the communication strategy being determined by the network communication library based on the number M of communication groups, the number M of communication groups being determined by the network topology information of the distributed system after determining the total communication bandwidth between the first region and the second region, combined with the communication type and the processing bandwidth of each processor, a second processor for receiving a communication strategy sent by the network communication library of the second computing node, the second computing node including a second processor, the communication strategy being determined by the network communication library based on the number M of communication groups.

[0035] Fourthly, a network communication library is provided, which is applied to a distributed system. The distributed system includes a first region and a second region, each region including multiple computing nodes. Each computing node includes the network communication library and at least one processor. The distributed system is used to execute distributed training tasks of AI models, and each processor is used to execute sub-tasks of the distributed training tasks. The network communication library includes: an acquisition unit for acquiring network topology information of the distributed system; a determination unit for determining the total communication bandwidth between the first region and the second region and the bandwidth of each processor based on the network topology information; a determination unit for generating a communication strategy based on the total communication bandwidth and the bandwidth of each processor. The communication strategy includes: each time a processor in the first region communicates with a processor in the second region, a total of M processors in the first region participate in the cross-region communication, and the total bandwidth of the M processors is adapted to the total communication bandwidth between the first region and the second region; and a sending unit for sending the communication strategy to at least one processor on the first computing node.

[0036] The network communication library described in the fourth aspect is implemented. Before the aggregated communication, the network communication library determines the communication strategy between processors based on the convergence bandwidth between the first and second regions and distributes it to each processor. This allows the processors to perform aggregated communication according to the communication strategy, ensuring that only M processors in each region participate in cross-regional communication. The total bandwidth of the M processors is adapted to the convergence bandwidth between the regions, thereby ensuring that cross-regional communication will not be congested, avoiding backpressure mechanisms caused by congestion and further reduction of cross-regional communication bandwidth, thus improving the overall communication efficiency of aggregated communication.

[0037] In one possible implementation, each region includes M communication groups, and each of the M communication groups includes at least one processor. The communication strategy includes: the M processors participating in cross-region communication form different communication groups.

[0038] Fifthly, a computing device is provided, the computing device including a processor and a memory, the memory for storing instructions and the processor for executing the instructions, such that the computing device implements the method described in the second aspect.

[0039] In a sixth aspect, a computer-readable storage medium is provided, wherein instructions are stored therein, which are executed by a computing device or a cluster of computing devices to implement the method described in the second or third aspect.

[0040] In a seventh aspect, a computing device cluster is provided, the computing device cluster including at least one computing device, each of the at least one computing device including a processor and a memory, the processor of the at least one computing device being configured to execute instructions stored in the memory of the at least one computing device, such that the computing device cluster implements the method described in the third aspect.

[0041] Eighthly, a computer program product including instructions is provided, the computer program product including instructions that are capable of running on a computing device or stored in any available medium, and when the computer program product is run on a computing device or a cluster of computing devices, cause the computing device or the cluster of computing devices to perform the method described in the first aspect.

[0042] In a ninth aspect, a chip is provided, the chip including a power supply unit and a processing unit, the power supply unit being used to supply power to the processing unit, such that the processing unit implements the method described in the first aspect.

[0043] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0044] Figure 1 This is a schematic diagram of the structure of a distributed system that includes a bandwidth convergence region.

[0045] Figure 2 This is an architecture diagram of a distributed system provided in this application;

[0046] Figure 3 This is an example diagram of a distributed system deployed in a cloud environment, as provided in this application;

[0047] Figure 4 This is a flowchart illustrating the steps of a data communication method provided in this application during the configuration phase.

[0048] Figure 5 This is a flowchart illustrating the steps of the first processor in the communication phase of a data communication method provided in this application.

[0049] Figure 6 This is a flowchart illustrating the steps of a data communication method provided in this application involving multiple processors during the communication phase.

[0050] Figure 7 This is a flowchart illustrating one possible implementation of the data communication method provided in this application;

[0051] Figure 8 This is a flowchart illustrating another possible implementation of the data communication method provided in this application;

[0052] Figure 9 This is a schematic diagram of the structure of a network communication library provided in this application;

[0053] Figure 10 This is a schematic diagram of the structure of a computing device provided in this application;

[0054] Figure 11 This is an example diagram of a computing device cluster provided in this application. Detailed Implementation

[0055] In the training of AI models, a single computing node cannot handle the entire computational load. Therefore, AI model training is usually completed through a distributed system, with multiple computing nodes processing computational tasks in parallel. Common distributed training algorithms include data parallelism, tensor parallelism, and pipelined parallelism. Data parallelism refers to splitting the dataset and distributing it across multiple nodes for training. Tensor parallelism refers to distributing the model parameters across multiple nodes for training. Pipeline parallelism refers to splitting the model according to network layers and distributing it across multiple nodes for training. Distributed training also includes many other algorithms, which will not be listed here.

[0056] Distributed systems require a large number of nodes, leading to a significant increase in hardware costs for model training. To reduce these costs, bandwidth-converging network topologies have emerged. Bandwidth convergence refers to a distributed system where multiple regions exist, each with high bandwidth. The bandwidth between spine nodes connecting different regions is much smaller than the total bandwidth within each region. This reduces the number of switches and optimizes the use of fiber optic cables, thereby lowering hardware costs.

[0057] For example, Figure 1 This is a schematic diagram of the structure of a distributed system that includes a bandwidth convergence region, such as... Figure 1 As shown, the distributed system includes multiple regions, such as region A and region B. Each region includes multiple computing nodes 101 and switching nodes 102. The computing nodes 101 can be servers or processors, and the switching nodes 102 can be switches, routers or switching chips.

[0058] The switching node 102 includes leaf nodes and spine nodes. Leaf nodes are connected to computing nodes 101, and spine nodes are connected to leaf nodes. These switches can construct a multi-layered, large-scale interconnected network. The network topology of this interconnected network may include, but is not limited to, dragonfly (or dragonfly+) networks, torus networks, fat-tree networks, etc., and this application does not impose specific limitations. Figure 1 In the example, swap nodes A2 and B2 can be leaf nodes, and swap nodes A1 and B1 can be spine nodes. In a specific implementation, each region may also include more computing nodes 101, leaf nodes, and spine nodes; this application does not impose specific limitations.

[0059] Furthermore, the bandwidth of region A and region B is not convergent, and regions A and B are interconnected through convergent bandwidth, resulting in higher bandwidth within each region and relatively lower bandwidth between spine nodes connecting different regions. In other words, in... Figure 1 In this example, the interconnection bandwidth between intermediate spine nodes is smaller than the total bandwidth of the 101 network ports of the computing nodes. Multiple leaf nodes can interconnect with the same spine node, which can reduce the number of spine nodes, reduce the number of cables and optical fibers, and thus reduce the hardware cost of the entire distributed system.

[0060] It should be noted that, Figure 1In the example, switching node A1 and switching node B1 are directly connected, and the bandwidth between them is smaller than the bandwidth within the region. In some possible implementations, switching node A1 can also be connected to switching node B1 through a bandwidth convergence region. This bandwidth convergence region may include at least one switching node, and the total communication bandwidth of this region is smaller than the total communication bandwidth within regions A and B. That is to say, regions A and B can be connected as follows: Figure 1 The direct connection shown can also be connected through a single switching node or multiple switching nodes. Regardless of the method, the communication bandwidth between region A and region B is less than the bandwidth within the region. This application does not impose any specific limitations.

[0061] use Figure 1 When training an AI model in the distributed system shown, regardless of the distributed training algorithm used (e.g., data parallelism, tensor parallelism, pipelined parallelism), each computing node 101 needs to synchronize the activation values ​​and weights obtained from local training to other nodes via aggregate communication, ensuring that each node receives the complete model's weights and activation values. Aggregate communication is a common communication method in distributed systems, involving global communication operations with all nodes participating. Examples include AllGather communication, which aggregates data from each node to ensure each node includes data from all nodes; and ReduceScatter communication, which performs the same reduction operation on all nodes and then distributes the results to each node. It should be understood that aggregate communication can include many other types, which will not be listed here.

[0062] Thus, when computing node 101 performs aggregated communication, due to bandwidth convergence between spine nodes, when the data traffic is too large, cross-regional communication will become congested. In order to deal with congestion, the network protocol will usually trigger a backpressure mechanism, and the sender will reduce the data transmission rate, which will further reduce the effective bandwidth of cross-regional interconnection, thereby affecting the efficiency of overall aggregated communication.

[0063] In summary, to save hardware costs, current distributed systems typically use less bandwidth for inter-regional communication than for intra-regional communication. This means that during model training, if multiple computing nodes in a distributed system need to exchange data via aggregate communication, the convergence of cross-regional exchange bandwidth can trigger a backpressure mechanism if congestion occurs, further reducing the convergence bandwidth of cross-regional exchange. This, in turn, affects the overall efficiency of aggregate communication and the training efficiency of the distributed system.

[0064] To address the issue of congestion during cross-regional bandwidth convergence in distributed systems, which can lead to backpressure and further reduce cross-regional convergence bandwidth, thus impacting overall ensemble communication efficiency, this application provides a data communication method. Before the distributed training of the AI ​​model begins, the distributed system pre-determines the number M of concurrent cards adapted for cross-regional communication based on the inter-regional convergence communication bandwidth. This ensures that after the system starts distributed training, if ensemble communication is required, the number of processors involved in each cross-regional communication is M. This guarantees that the communication volume of M processors during cross-regional communication does not exceed the cross-regional bandwidth, thereby preventing congestion and avoiding backpressure mechanisms caused by congestion, as well as further reductions in cross-regional communication bandwidth, ultimately improving the overall communication efficiency of ensemble communication.

[0065] The technical solution provided in this application will be explained below with reference to the accompanying drawings.

[0066] Figure 2 This application provides an architecture diagram of a distributed system, such as... Figure 2 As shown, the architecture may include a distributed system 200 and clients 300, wherein the number of clients 300 that establish connections with the distributed system 200 in this architecture may be one or more. Figure 2 The example given is a single client 300, and this application does not impose any specific limitations. A communication connection exists between the distributed system 200 and the client 300, which can be either a wired or wireless connection. Wired connections include Ethernet, wired lines, cables, etc., while wireless connections include wireless LAN (Wi-Fi), cellular networks, etc. It may also include the Internet, local area network (LAN), etc., which simultaneously support wired and wireless connections, and of course, other connection types may also be included, which this application does not impose any specific limitations on.

[0067] The client 300 is used for human-computer interaction and can be deployed on terminal devices, computing devices, or edge computing devices. Terminal devices include personal computers, smartphones, wearable devices, handheld processors, tablets, mobile laptops, augmented reality (AR) devices, virtual reality (VR) devices, smart conferencing devices, etc., without specific limitations. Computing devices can be bare metal servers (BMS), virtual machines, or containers. A BMS refers to a general-purpose physical server, such as an ARM server or an x86 server; a virtual machine refers to a complete computer system simulated by software, possessing full hardware system functionality and running in a completely isolated environment. Any task that can be performed on a physical computer can also be performed in a virtual machine. When creating a virtual machine on a computing device, a portion of the physical machine's hard drive and memory capacity needs to be used as the virtual machine's hard drive and memory capacity. Each virtual machine has its own independent basic input / output system (BIOS), hard drive, and operating system, allowing it to be operated like a physical machine. A container is a portable software unit that combines an application and all its dependencies into a single software package. This package is not limited by the underlying host operating system, eliminating the need to build complex environments and simplifying the application development and deployment process. Edge computing devices are hardware devices located close to the data source or data user.

[0068] The distributed system 200 may be a computing device or a cluster of computing devices. The description of the computing devices can be found above, and this application does not impose any specific limitations. Furthermore, the distributed system 200 may include an AI platform 103, a cluster management platform 104, a scheduler 105, multiple computing nodes 101, and multiple switching nodes 102.

[0069] AI Platform 103 is a system or service integrating various AI technologies and tools to help users (including but not limited to developers, enterprises, researchers, etc.) quickly build, train, deploy, and apply AI models. AI Platform 103 provides users with a training environment for machine learning and deep learning models, supports various algorithms and frameworks such as TensorFlow and PyTorch, allows users to deploy trained models to production environments, and provides API interfaces to facilitate the integration of AI models into functions such as image recognition, natural language processing, and speech recognition. It also provides automated machine learning (AutoML) tools to recommend optimal model parameter configurations and may include more features. This application does not limit the specific functional types of the AI ​​platform. In its implementation, AI Platform 103 can be deployed on one or more computing devices; this application does not impose specific limitations.

[0070] The cluster management platform 104 is responsible for monitoring, managing, and maintaining the hardware resources of the distributed system 200, including computing resources, storage resources, and network resources. Specific functions include, but are not limited to, hardware configuration, network configuration, fault detection and handling, and monitoring of resource usage for computing nodes 101. The cluster management platform 104 ensures that all hardware resources are maintained in optimal condition to support efficient system operation. Common cluster management platforms 104 include Kubernetes and Apache Mesos.

[0071] Scheduler 105 is responsible for allocating tasks to computing nodes based on the current resource status and task requirements in order to maximize resource utilization. Its main functions include, but are not limited to, task allocation, task queue management, and rescheduling after task failure, in order to optimize the system's task scheduling strategy.

[0072] Computing node 101 is responsible for processing tasks issued by the scheduler. Computing node 101 can be a computing device or a chip. The description of the computing device can be found above, and this application does not impose specific limitations. The chip can be a processor implemented using a central processing unit (CPU), an application-specific integrated circuit (ASIC), or a programmable logic device (PLD). The PLD can be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a system-on-a-chip (SoC), or any combination thereof, and this application does not impose specific limitations. It may also include processors or accelerator cards that have good performance when performing machine learning tasks and can improve the speed and efficiency of neural network training, such as graphics processing units (GPUs), neural network processing units (NPUs), data processing units (DPUs), tensor processing units (TPUs), etc., which are not specifically limited in this application. Computing node 101 may also be a cluster of computing devices composed of multiple computing devices, or a chip system composed of multiple chips, which are not specifically limited in this application.

[0073] Furthermore, the computing node 101 may include a network communication library 1011, at least one processor 1012, and at least one network interface card (NIC) 1013. The processor 1012 can communicate with other processors 1012, computing nodes 101, or switching nodes 102 via the NIC 1013. In some embodiments, the processor 1012 may also serve as the computing node 101, and the NIC 1013 may also serve as the switching node 102; this application does not impose specific limitations.

[0074] The network communication library 1011 is a software component for implementing network communication. The network communication library 1011 may include application programming interfaces (APIs), which can perform specific network operations, such as data sending, receiving, and configuration. Developers can use APIs to configure data synchronization and communication between different computing nodes.

[0075] The processor 1012 can be a CPU, or a GPU or accelerator card with good performance when performing machine learning tasks, such as an NPU, DPU, or TPU. This application does not make any specific limitations.

[0076] The network interface card 1013 is a hardware device used to connect computing nodes to a network. The network interface card 1013 can be a physical network interface card, such as an expansion card inserted into the motherboard or a chipset integrated on the motherboard. It can also be a wireless network interface card, such as a wireless network interface card, or a virtual network interface card. This application does not make any specific limitations.

[0077] Switching node 102 is responsible for enabling aggregated communication between computing nodes 101. Switching node 102 can be a network communication device or chip with the function of transmitting data via electrical signals and / or optical signals, such as a switch, router, or switching chip. It can provide a dedicated electrical or optical signal path for any two nodes connected to switching node 102. The aforementioned any two nodes can be two computing nodes 101, two switching nodes 102, or one computing node 101 and one switching node 102. This application does not make a specific limitation.

[0078] Furthermore, the switching node 102 may store a routing table or a media access control (MAC) address table. The routing table or MAC address table may include multiple entries, each of which may represent a forwarding path. Each entry may include at least a source address, a destination address, and a corresponding next-hop address. After receiving a data packet sent by the computing node 101, the switching node 102 may query the routing table or MAC address table based on the source address and destination address carried in the data packet to obtain the forwarding path of the data packet, determine the next-hop address, and then forward the data packet to the next-hop address. In specific implementations, if the switching node 102 is a Layer 2 switch (link layer switch) or a switching chip, a MAC address table can be generated using switch forwarding algorithms such as Address Resolution Protocol (ARP). The source address and destination address can be MAC addresses. If the switching node 102 is a Layer 3 switch (network layer switch), router, or switching chip, a routing table can be generated using routing algorithms such as Routing Information Protocol (RIP) or Border Gateway Protocol (BGP). The aforementioned source address and destination address can be IP addresses, and this application does not impose specific limitations on them.

[0079] The switching node 102 includes leaf nodes and spine nodes. Leaf nodes are connected to compute nodes 101, and spine nodes are connected to leaf nodes. These switches can construct a multi-layered, large-scale interconnected network. The network topology of this interconnected network may include, but is not limited to, dragonfly (or dragonfly+) networks, torus networks, fat-tree networks, etc., and this application does not impose specific limitations. The interconnected network may include multiple regions, each region may include at least one compute node 101 and at least one switching node 102. The bandwidth within a region is not convergent, but the bandwidth between spine nodes connecting different regions is convergent. The large-scale interconnected network can be a high-performance dedicated network used for synchronizing data between different training cards during distributed training of large AI models, and can perform high-performance data transmission based on the Remote Direct Memory Access (RDMA) protocol.

[0080] For example, in Figure 2 In the example, region A includes at least compute node A and exchange nodes A1 to A3, and may also include other compute nodes and exchange nodes, not listed in the example. Figure 2 The diagram illustrates this. The processor in compute node A can communicate with other processors within the region via network interface cards (NICs) and switching nodes. Switching nodes A2 and A3 are leaf nodes connected to the compute nodes, and switching node A1 is a spine node that enables inter-region communication. Similarly, region B includes at least compute node B and switching nodes B1 through B3, and may also include other compute nodes and switching nodes not shown in the diagram. Figure 2 The diagram shows that processor 1012 in compute node B can communicate with other processors 1012 in the region through network interface cards and switching nodes. Switching nodes B2 and B3 are leaf nodes connected to the compute nodes, and switching node B1 is a spine node that enables inter-region communication.

[0081] Within a given region, the bandwidth between computing and switching nodes does not converge; the bandwidth between regions A and B does not converge; regions A and B are interconnected through convergent bandwidth, resulting in higher bandwidth within each region. However, the bandwidth between spine nodes connecting different regions is relatively lower. In other words, in... Figure 2 In the example, the interconnection bandwidth between switching node A1 and switching node B1 is W1, and the total bandwidth within region A is W2. Therefore, W1 is less than W2.

[0082] For another example, suppose region A includes 16 training cards (which can be on a single compute node or multiple compute nodes), each with a parameterized network interface bandwidth of 400Gbps. Then the total bandwidth of region A is 6.4Tbps, meaning the bandwidth within the region does not converge. Similarly, region B also includes 16 training cards, each with a parameterized network interface bandwidth of 400Gbps. The total bandwidth of region B is also 6.4Tbps, but the bandwidth between region A and region B is 1.6Tbps, indicating that the bandwidth between regions converges. The above examples are for illustrative purposes only and are not intended to be specific.

[0083] It should be understood that Figure 1 When the distributed system 200 performs distributed training on the AI ​​model, the processors 1012 need to communicate collectively. The bandwidth within the region does not converge, while the bandwidth between regions converges, which will cause congestion during inter-regional communication. After congestion occurs, based on the back pressure mechanism of the communication protocol, the sender will reduce the data transmission rate, which will further reduce the convergence bandwidth across regions and affect the overall collective communication efficiency.

[0084] Optionally, the client 300 and the distributed system 200 may be deployed on the same computing device, or the client 300 and the distributed system 200 may be deployed on different computing devices in the same computing device cluster, or the client 300 and the distributed system 200 may be deployed on different computing devices in different computing device clusters. This application does not make any specific limitations.

[0085] Optionally, the AI ​​platform 103, the cluster management platform 104, and the scheduler 105 may be deployed on the same computing device, or partially deployed on the same computing device and partially deployed on different computing devices, or all deployed on different computing devices. This application does not make any specific limitations.

[0086] Optionally, the client 300 can be software or an application running on a user-controlled terminal device or computing device, such as a personal computer (PC) client, a web client accessed via a browser, an application (APP) client running on a mobile terminal, or a cloud platform console; this application does not impose specific limitations. The user holding the client 300 can be a developer of the training model, such as a programmer or AI engineer; this application does not impose specific limitations.

[0087] Optionally, the client 300 may be a model service platform or a machine learning framework client for handling machine learning-related tasks. Such clients provide a series of tools and services for the entire process of defining, training, deploying, inferring, and managing machine learning models. The model service platform may be ModelArts, OpenAI, etc., and the machine learning framework may be TensorFlow, PyTorch, MindSpore, etc. This application does not make any specific limitations.

[0088] Optionally, the client 300 can also be a client of a cloud platform, used for users to purchase and rent various cloud services. The data communication method provided in this application can be one of these cloud services, which users can purchase separately to optimize the efficiency of aggregate communication during model training. Alternatively, the cloud platform provides users with a comprehensive service, and the data communication method provided in this application can be a sub-service of this comprehensive cloud service. For example, if the comprehensive cloud service is an AI cloud service, the data communication method provided in this application can be a sub-service of this AI cloud service to optimize the efficiency of aggregate communication during model training.

[0089] Optionally, the client 300 may also be a client of the cluster management platform 104 or a client of the scheduler 105; this application does not impose any specific limitations.

[0090] The preceding text has described in detail the possible deployment methods of client 300, distributed system 200, and their various components. In actual deployment, flexible deployment can be carried out according to specific application scenarios and business requirements. The following section provides examples of actual deployment methods for client 300, distributed system 200, and their various components in specific scenarios.

[0091] For example, distributed system 200 and client 300 can be deployed on an enterprise's internal network. Distributed system 200 is deployed on a server or server cluster purchased by the enterprise, and client 300 is a remote desktop connection tool or server management console software. Employees of the enterprise can remotely connect to the server or server cluster where distributed system 200 is located through client 300 and use distributed system 200 to optimize the ensemble communication efficiency during the model training process.

[0092] For example, distributed system 200 can be deployed in a cloud environment. Figure 3 This is an example diagram of a distributed system 200 deployed in a cloud environment, as provided in this application. Figure 3As shown, a user can initiate a purchase request for AI cloud services through client 300. These AI cloud services include sub-services corresponding to the data communication method provided in this application. Specifically, after client 300 sends the purchase request to the cloud platform, the cloud platform can grant client 300 access to the distributed system 200 provided in this application, enabling the user to use the distributed system 200 to optimize the ensemble communication efficiency during model training.

[0093] The cloud platform also maintains various basic resources, including computing resources, storage resources, network resources, and security resources, to meet the computing needs of the distributed system 200 under different scales and loads. Furthermore, these computing resources can be dynamically scaled according to the usage needs of the distributed system 200 to ensure the stable operation of the distributed system 200 and provide users with reliable AI cloud services.

[0094] It should be understood that the above application scenarios are for illustrative purposes only. The distributed system 200 and client 300 can be flexibly deployed according to actual business needs. They will not be listed here.

[0095] The possible system architecture of this application has been described above. The following is a brief explanation of how this system architecture implements the data communication method provided in this application.

[0096] In this embodiment, the AI ​​platform 103 receives a model training request from the client 300, generates a corresponding training scheduling task, and then sends the training scheduling task to the scheduler 105. The scheduler 105 can send a network topology acquisition request to the cluster management platform 104. The cluster management platform 104 acquires the network topology information and sends it to the scheduler 105. This network topology information is used to describe various elements in the network (such as node information, processor information, bandwidth, topology type, etc.). The scheduler 105 can send the network topology information to multiple computing nodes 101 in the distributed system 200. The network communication library 1011 of the computing node 101 can determine the communication strategy of the processor 1012 according to the network topology information. The communication strategy includes the number of communication groups in each region of the distributed system. The number of communication groups is determined based on the number of concurrent cards for inter-regional communication adaptation. The number of concurrent cards can be determined according to the network topology information. The network communication library 1011 distributes the communication strategy to each processor. Thus, when the distributed system 200 starts model training, during the aggregate communication process, only one processor 1012 participates in each communication group during each cross-regional communication. Since the number of communication groups is determined based on the number of concurrent cards adapted for inter-regional communication, this ensures that cross-regional communication will not be congested, avoids backpressure mechanisms caused by congestion, and thus improves the overall communication efficiency of aggregate communication.

[0097] In the specific implementation, during the collective communication process, a first intra-regional communication can be performed first, so that each processor 1012 obtains the intra-regional communication result. Then, multiple cross-regional communications are performed. In each cross-regional communication process, the number of processors participating in the communication is the same as the number of communication groups. The processors 1012 participating in the communication come from different communication groups. After at least one processor 1012 in each communication group obtains the cross-regional communication result, a second intra-regional communication can be performed, so that each processor 1012 obtains all the cross-regional communication results, and thus obtains the collective communication result.

[0098] It should be understood that traditional technologies typically report to the scheduler 105 after detecting network congestion. The scheduler 105 then adjusts the communication method based on some network congestion control strategies. However, this method of adjusting after congestion occurs can cause network traffic to fluctuate and fail to fully utilize network bandwidth. Under the technical solution of this application, the number of communication groups is determined before the aggregated communication, which can avoid network congestion or fluctuation and improve the overall aggregated communication efficiency.

[0099] In one possible implementation, during cross-region communication, processors 1012 within the same communication group can take turns communicating with other processors 1012 within the same communication group. After each processor 1012 completes its cross-region communication, it can send a notification message to the next processor 1012 in the same communication group according to the cross-region communication order. The next processor 1012 can respond to the notification message and begin cross-region communication. After communication is completed, it sends a notification message to the next processor 1012, and so on, until all processors 1012 in the group have completed communication. That is, if the number of processors in the communication group is N, then N cross-region communication operations will be performed.

[0100] Specifically, the first intra-regional communication can be achieved through reducescatter. Each processor 1012 sends its local data to be communicated to other processors within the region, and then receives the data to be communicated from other processors 1012 within the region, obtaining the intra-regional communication result. Next, cross-regional communication can be achieved through allreduce. Each processor 1012 sends its local intra-regional communication result to processors 1012 in other regions, and receives the intra-regional communication results sent by processors 1012 in other regions, obtaining the cross-regional communication result. Then, it notifies the next processor 1012 to start cross-regional communication, and so on, until all processors 1012 have obtained the corresponding cross-regional communication results. Finally, the second intra-regional communication can be achieved through allgather. Each processor 1012 sends the obtained cross-regional communication result to other nodes within the region, and receives the cross-regional communication results sent by other processors 1012 within the region, obtaining the final aggregated communication result.

[0101] In another possible implementation, during cross-region communication, each communication group can have a proxy processor. This proxy processor handles the cross-region communication. In this case, during the first intra-region communication, other processors within the group send the data to be communicated to the proxy processor. During the second intra-region communication, the proxy processor distributes the cross-region communication results to the other processors within the communication group.

[0102] Specifically, during the initial intra-regional communication, inter-group reduce-scatter can be performed within the region. Each processor 1012 sends its local data to be communicated to processors 1012 in other communication groups, receives data to be communicated from processors 1012 in other communication groups, and obtains the intra-group communication results. Then, a gather is performed within the communication group, where each processor 1012 sends its intra-group communication results to the proxy processors within the communication group, allowing the proxy processors to obtain the intra-regional communication results. Next, cross-regional communication is achieved through allreduce. Each proxy processor sends its local intra-regional communication results to proxy processors in other regions and receives intra-regional communication results from proxy processors in other regions, thus obtaining the cross-regional communication results. Finally, during the second intra-regional communication, intra-group scatter can be performed first, with each agent processor sending cross-regional communication results to other processors in the communication group, so that each processor can obtain the cross-regional communication results of the agent processors in the group. Then, inter-group allgather can be performed, with each processor sending the cross-regional communication results of its own communication group to the processors 1012 in other communication groups in the region, and receiving the cross-regional communication results of other communication groups sent by the processors 1012 in other communication groups, thereby obtaining the global communication results and the final aggregated communication results.

[0103] Optionally, the communication strategy determined by the network communication library 1011 may include the number of communication groups in each region, the communication group to which each processor belongs, and the communication objects for each processor in the first region communication, inter-region communication, and second region communication. It should be understood that the communication strategy may also include other information required by the processor 1012 for implementing the collective communication process described above, which is not specifically limited in this application.

[0104] It should be noted that, Figure 2 The distributed system 200 shown is an example. In a specific implementation, the distributed system 200 may include more or fewer components. For example, the scheduler 105 and the cluster management platform 104 may be merged into one component. Alternatively, the distributed system 200 may also include components such as a file system and object storage. This application does not make any specific limitations.

[0105] In summary, this application provides a distributed system in which the number of communication groups is determined in advance based on the number of concurrent cards adapted to the inter-regional communication bandwidth before the system starts aggregated communication. This ensures that only one processor in each communication group participates in communication during each cross-regional communication after the system starts aggregated communication, thereby ensuring that cross-regional communication will not be congested and improving the efficiency of aggregated communication.

[0106] The above text combined Figures 1-3 The distributed system provided in this application has been explained and described below, in conjunction with... Figures 4-7 The data communication method provided in this application will be explained and described. This data communication method can be applied to... Figure 2 and Figure 3 In the distributed system 200 shown, among which, Figure 4 The configuration phase of the data communication method has been explained. Figures 5-8 The communication phase in the data communication method is explained.

[0107] Figure 4 This is a flowchart illustrating the steps of a data communication method provided in this application during the configuration phase, as shown below. Figure 4 As shown, the method may include the following steps:

[0108] S410: AI platform 103 sends training scheduling tasks to scheduler 105.

[0109] Optionally, the AI ​​platform 103 can first receive the model training request sent by the client 300, generate a corresponding training scheduling task, and then send the training scheduling task to the scheduler 105. The model training request may include dataset information, model configuration information, etc. The dataset information includes the location of the dataset used to train the model, which may be a file path or a database link. The model configuration information includes the AI ​​model's structural data, hyperparameters, and configuration information. The structural data includes data describing the model's structure, such as the number of layers and activation functions. The hyperparameters include information such as the model's learning rate, batch size, and number of training epochs. The configuration information includes data such as the optimizer type, loss function type, and regularization method. The above examples are for illustrative purposes only and are not intended to be specific limitations in this application.

[0110] Optionally, the AI ​​platform 103 can analyze the dataset information and model configuration information in the model training request to determine that the necessary data for model training is available. Then, it generates a corresponding training scheduling task. This task may include a task description, resource requirements, and execution strategy. The task description includes detailed information required for model training, such as the model type, the required dataset, the expected output, and the specific objective of the task. Resource requirements include the computing resources required for model training, such as the number of processors and memory size. The execution strategy may include information about the multiple subtasks included in the training scheduling task, as well as the execution order between the subtasks, ensuring that model training proceeds steadily. It should be understood that the above examples are for illustrative purposes, and the training scheduling task may include more content; this application does not impose specific limitations.

[0111] S420: Scheduler 105 obtains network topology information from cluster management platform 104.

[0112] In practical implementation, network topology information is used to describe various devices in the network and the connections between them. This network topology information can be used to determine processor bandwidth and convergence bandwidth for cross-region communication. The convergence bandwidth for cross-region communication refers to the convergence bandwidth between switching nodes connecting two regions. For detailed descriptions of regions and convergence bandwidth, please refer to [reference needed]. Figures 1-3 The relevant descriptions of the embodiments will not be repeated here.

[0113] Optionally, the network topology information may directly include the aforementioned processor bandwidth and convergence bandwidth, or it may include relevant data used to calculate the processor bandwidth and convergence bandwidth, such as region information, that is, the number of regions included in the distributed system and related information for each region, such as the number of nodes included in each region, the number of processors included in each node, and the processor bandwidth. The above examples are for illustration purposes. The network topology information may also include more content, such as information about the computing nodes where the processors reside, the connection relationships between computing nodes, information about switching nodes, the connection relationships between switching nodes, the network hierarchy, network service configuration, etc., which are not specifically limited in this application.

[0114] In a specific implementation, the cluster management platform 104 is used to monitor and manage the physical resources in the distributed system 200, and therefore can provide the scheduler 105 with network topology information of the distributed system 200. Optionally, the cluster management platform 104 can automatically identify devices in the network using network scanning technologies such as Simple Network Management Protocol (SNMP) and Internet Control Message Protocol (ICMP); then, it uses network management protocols to collect configuration and performance data of the identified devices, such as SNMP and Link Layer Discovery Protocol (LLDP); it obtains real-time network traffic status based on network traffic analysis protocols, such as netflow and sampled flow (sflow); and it draws a corresponding topology map based on the above information, continuously updating the topology map according to the real-time network traffic status. Thus, when the scheduler 105 sends a request to the cluster management platform 104 for network topology information, the cluster management platform 104 can generate the current network topology information based on the real-time updated topology. The above examples are for illustrative purposes only and are not intended to limit the scope of this application.

[0115] S430: Scheduler 105 sends network topology information to network communication library 1011 of computing node 101.

[0116] It should be understood that Figure 4 The flowchart shown only depicts one computing node 101 for ease of explanation. In actual processing, the scheduler 105 can send network topology information to the network communication library 1011 of each computing node 101 in the distributed system 200.

[0117] S440: Network communication library 1011 determines the processor's communication strategy based on network topology information.

[0118] In a specific implementation, the network communication library 1011 of each computing node 101 can determine the processor's communication strategy based on the network topology information. The communication strategy includes the number of communication groups in each region of the distributed system. The number of communication groups is determined based on the number of concurrent cards for inter-regional communication adaptation, and the number of concurrent cards can be determined based on the network topology information.

[0119] In its implementation, the network communication library 1011 can determine the convergence bandwidth of communication between regions and the bandwidth of the processor based on the network topology information, thereby obtaining the number of concurrent cards adapted for cross-regional communication, and further obtaining the number of communication groups within a region. Specifically, the number of concurrent cards M adapted for cross-regional communication can be determined based on the ratio between the convergence bandwidth and the bandwidth, thereby obtaining the number of communication groups M in each region.

[0120] For example, assuming the convergence bandwidth between regions is 1.6Tbps and the processor bandwidth is 400Gbps, then the number of concurrent cards that can communicate concurrently across regions is 1.6Tbps / 400Gbps = 4. The number of communication groups within each region is 4. The above example is for illustrative purposes only and is not intended to impose specific limitations.

[0121] It should be noted that the network topology information may include the convergence bandwidth of cross-regional communication. In this case, the network communication library 1011 can calculate the number of communication groups based on the convergence bandwidth. If the network topology information does not directly include the convergence bandwidth, the network communication library 1011 can also determine the convergence bandwidth between each region in the distributed system 200 and the spine nodes connecting each region based on the device information in the network topology information and the connection relationship between the devices. This application does not limit this.

[0122] Understandably, the network communication library 1011 determines the number of concurrent cards M that can be adapted for cross-regional communication based on the convergence bandwidth of cross-regional communication, and then determines the number of communication groups as M. In subsequent aggregated communication, each communication group has only one processor communicating with other processors in each cross-regional communication. This ensures that the number of processors communicating at the same time is exactly the same as the number of adapted concurrent cards, thereby avoiding cross-regional communication congestion and improving the efficiency of aggregated communication.

[0123] In specific implementation, the communication strategy may include the number of communication groups in each region, the communication group to which each processor belongs, the communication object of each processor in each communication, and more. The specific details can be determined according to actual business needs, and this application does not impose any specific limitations.

[0124] S450: Network communication library 1011 sends communication policy to processor 1012.

[0125] As can be seen from the foregoing, computing node 101 includes at least one processor 1012. Network communication library 1011 can send the communication strategy to each processor 1012 in computing node 101, so that each processor 1012 in computing node 101 where network communication library 1011 is located can complete collective communication according to the communication strategy.

[0126] It should be understood that the network communication libraries 1011 on different computing nodes 101 are configured with the same algorithm, and can all calculate the corresponding communication strategy based on the network topology information. Since the algorithm is the same, the calculated communication strategy is also the same. In this way, the computational pressure of calculating the communication strategy is distributed among the computing nodes 101, which can reduce the processing pressure of the scheduler 105. In some possible implementations, the step of calculating the communication strategy can also be implemented by the scheduler 105. That is, after the scheduler 105 obtains the network topology information, it calculates the communication strategy and then sends it to the network communication library 1011 of the computing node 101. Then, the network communication library 1011 distributes the communication strategy to the processor 1012. Both of the above methods can realize the technical solution of this application, and this application does not make specific limitations.

[0127] S460: Processor 1012 uses a communication strategy for collective communication.

[0128] As mentioned above, in distributed training, the training task of the AI ​​model is assigned to multiple computing nodes for parallel execution, thereby accelerating the model training process. Collective communication is a crucial part of distributed training, primarily used to achieve data synchronization between different computing nodes. There are many types of collective communication, including broadcast, all-reduce, all-gather, scatter, reduce, and reduce-scatter, as well as other types, which will not be listed here. During collective communication, computing nodes simultaneously transmit data. The transmitted data (or data to be communicated) may include, but is not limited to, model parameters, gradient information, partial calculation results, hyperparameters, or configuration information. The specific communication content of collective communication can be determined according to actual business needs, and this application does not impose specific limitations.

[0129] In this embodiment of the application, when ensemble communication is required during model training, the ensemble communication method of this application can be used to accelerate communication efficiency. Specifically, during ensemble communication, only one processor 1012 participates in communication for each cross-regional communication. Since the number of communication groups is determined based on the number of concurrent cards adapted for inter-regional communication, this ensures that cross-regional communication will not experience congestion, thereby improving the overall communication efficiency of ensemble communication.

[0130] To facilitate a better understanding of this application, the following is a collection of... Figures 5-8 This section explains the steps involved in the S460 processor 1012 performing aggregate communication using a communication strategy. Figure 5 Taking the first processor in the first region as an example, the process of its collective communication using a communication strategy is described in detail. Figures 6-8Taking multiple processors as an example, this paper describes in detail the process of multiple processors using their respective communication strategies to perform collective communication.

[0131] Figure 5 This is a flowchart illustrating the steps of the first processor in the communication phase of a data communication method provided in this application. Figure 5 In the scenario shown, the distributed system 200 includes a first region and a second region. The first region includes a first processor, and the second region includes a second processor, as shown below. Figure 5 As shown, when the first processor uses a communication strategy for collective communication, the specific steps can be as follows:

[0132] S510: The first processor acquires the first data to be communicated.

[0133] In specific implementation, the first data to be communicated refers to the data that the first processor needs to synchronize with other processors through aggregated communication, including but not limited to memory parameters, gradient information, etc. The specific content of the data to be communicated can be determined based on the actual business scenario, and this application does not impose specific limitations. Similarly, the second processor can also obtain the second data to be communicated.

[0134] S520: The first processor communicates with other processors in the first region within the region and obtains the results of the communication within the region.

[0135] In a specific implementation, the first processor can send the first data to be communicated to other processors within the region and receive data sent by other processors within the region to obtain the communication result within the region. The first processor can receive some or all of the data to be communicated from other processors within the region; the specific processors communicating with the first processor can be determined according to the actual algorithm, and this application does not impose specific limitations. Similarly, the second processor in the second region can also communicate with other processors within the second region to obtain the communication result within the region.

[0136] S530: The first processor communicates with the second processor in the second region across regions to obtain the results of the cross-regional communication.

[0137] In practice, the first processor can send the communication results within the same region to the second processor, receive the data sent by the second processor, and obtain the cross-regional communication results. Similarly, the second processor can send the communication results within the same region to the first processor, receive the data sent by the first processor, and obtain the corresponding cross-regional communication results.

[0138] Optionally, during cross-regional communication, when the first processor and the second processor communicate across regions, the M-1 processors in the first region also communicate across regions. The total communication bandwidth of the M processors formed by the first processor and the M-1 processors is adapted to the total communication bandwidth between the first region and the second region, thereby avoiding the problem of cross-regional communication blockage and improving communication efficiency.

[0139] In the specific implementation, each region may include M communication groups, and each communication group may include at least one processor. The M processors participating in cross-region communication come from different communication groups. That is to say, during cross-region communication, only one processor from each communication group is communicating externally. This ensures that the total communication bandwidth of the M processors is compatible with the total communication bandwidth between the first and second regions, thereby avoiding cross-region communication congestion and improving communication efficiency.

[0140] In one possible implementation, the cross-regional communication type includes sequential communication, whereby after the first processor communicates with the second processor in the second region, the first processor sends a notification message to the third processor, which instructs the third processor to communicate with the fourth processor in the second region. The first and third processors belong to a first communication group, and the second and fourth processors belong to a second communication group.

[0141] Similarly, after the second processor communicates with the processors of other communication groups in the second region, it can send the data to be communicated to the processors of other communication groups in the second region, receive the data sent by the processors of other communication groups, obtain the communication results between the two groups, communicate with the processors of other communication groups within the second group, send the communication results between the two groups to the processors of other communication groups in the second group, receive the data sent by the processors of other communication groups in the second group, and obtain the communication results within the second region.

[0142] In simple terms, during cross-region communication, each processor in each communication group performs cross-region communication in turn. After each processor completes cross-region communication, it can notify the next processor in the group to start cross-region communication. This continues until the last processor completes its communication and then notifies the first processor to start cross-region communication. Since the first processor has no more data to send, cross-region communication can be terminated at this point.

[0143] In another possible implementation, the cross-regional communication type includes proxy communication, in which the first processor and the second processor in the second region conduct multiple cross-regional communications. The first processor is a proxy processor of the first communication group, and the first processor includes the communication data to be communicated by all processors in the first communication group. Each time cross-regional communication occurs, a total of M proxy processors in the first region participate in the cross-regional communication.

[0144] Similarly, the second processor can communicate with the first processor multiple times across regions. The second processor is a proxy processor of the second communication group. The second processor includes the communication data of all processors in the second communication group. Each time there is cross-regional communication, there are M proxy processors in the second region participating in the cross-regional communication.

[0145] In simple terms, during cross-regional communication, only the proxy processor within each communication group communicates externally. The proxy processor performs multiple cross-regional communications, sending data from each accelerator card within the communication group to the proxy processor in another region. Therefore, the proxy processor at S520 needs to aggregate the intra-group communication results from other processors within the communication group.

[0146] In a specific implementation, in the scenario of proxy communication, at S520, the first processor communicates with the processors of other communication groups in the first region, sends the data to be communicated to the processors of other communication groups in the first region, receives the data sent by the processors of other communication groups, and obtains the inter-group communication result. The first processor also communicates with other processors in the first communication group, sends the inter-group communication result to the other processors in the first communication group, receives the data sent by the other processors in the first communication group, and obtains the intra-regional communication result.

[0147] Similarly, the second processor communicates with processors in other communication groups in the second region, sends data to be communicated to processors in other communication groups in the second region, receives data sent by processors in other communication groups, obtains the communication results between the two groups, communicates with other processors in the second communication group, sends the communication results between the two groups to other processors in the second communication group, receives data sent by other processors in the second communication group, and obtains the communication results within the second region.

[0148] In this way, before cross-regional communication, multiple processors within each communication group can aggregate the data that needs to be communicated across regions onto a proxy processor, which then executes S530 to complete the cross-regional communication. The proxy processor can perform multiple cross-regional communications until all the aggregated data to be communicated has been communicated.

[0149] S540: The first processor communicates with other processors in the first region within the region to obtain the results of the aggregated communication.

[0150] In specific implementation, the first processor can send the cross-regional communication result obtained in S530 to other processors in the first region, and then receive the data sent by other processors to obtain the collective communication result. Similarly, the second processor can also communicate with other processors within the region to obtain the collective communication result. Through S510 to S540, the first processor not only sends its first data to be communicated to other processors, but also receives the data to be communicated from all processors in the first and second regions, realizing data synchronization among multiple processors in the distributed system and obtaining the corresponding collective communication result.

[0151] Figure 6 This is a flowchart illustrating the steps of multiple processors in the communication phase of a data communication method provided in this application. Figure 6 In the diagram, a small white square represents the processor, and arrows indicate the direction of communication. For example... Figure 6 As shown, in the collective communication process, a first intra-regional communication can be performed first, so that each processor 1012 obtains the intra-regional communication result. Then, multiple cross-regional communications are performed. In each cross-regional communication process, the number of processors participating in the communication is the same as the number of communication groups. The processors 1012 participating in the communication come from different communication groups. After at least one processor 1012 in each communication group obtains the cross-regional communication result, a second intra-regional communication can be performed, so that each processor 1012 obtains all the cross-regional communication results, and thus obtains the collective communication result.

[0152] In simple terms, during each cross-regional communication, only one processor 1012 in each communication group communicates with other processors. This ensures that the number of concurrent cards matches the convergence bandwidth during each cross-regional communication. It should be understood that traditional technologies typically report to the scheduler 105 after detecting network congestion. The scheduler 105 adjusts the communication method based on some network congestion control strategies. However, this method of adjusting after congestion occurs can cause network traffic jitter and fail to fully utilize network bandwidth. Under the technical solution of this application, the number of communication groups is determined before the aggregated communication, which can avoid network congestion or jitter and improve the overall aggregated communication efficiency.

[0153] In one possible implementation, during cross-region communication, processors 1012 within the same communication group can take turns communicating with other processors 1012 within the same communication group. After each processor 1012 completes its cross-region communication, it can send a notification message to the next processor 1012 in the same communication group, following the cross-region communication order. The next processor 1012 can respond to the notification message and begin cross-region communication. After completing communication, it sends a notification message to the next processor 1012, and so on, until all processors 1012 in the group have completed communication. That is, if the number of processors in the communication group is M, then M cross-region communication operations will be performed.

[0154] Figure 7 This is a flowchart illustrating one possible implementation of the data communication method provided in this application, as shown below. Figure 7 As shown, the first intra-regional communication is achieved through reducescatter. Each processor 1012 sends its local data to be communicated to other processors within the region, and then receives the data to be communicated from other processors 1012 within the region, obtaining the intra-regional communication result. Next, cross-regional communication is achieved through allreduce. Each processor 1012 sends its local intra-regional communication result to processors 1012 in other regions, and receives the intra-regional communication results sent by processors 1012 in other regions, obtaining the cross-regional communication result. Then, it notifies the next processor 1012 to start cross-regional communication, and so on, until all processors 1012 have obtained the corresponding cross-regional communication results. Finally, the second intra-regional communication is achieved through allgather. Each processor 1012 sends the obtained cross-regional communication result to other nodes within the region, and receives the cross-regional communication results sent by other processors 1012 within the region, obtaining the final aggregated communication result.

[0155] For example, in Figure 7 In the example shown, during the first cross-regional communication, the first processor in each communication group communicates with processors in other regions, with the number of processors communicating being the same as the number of communication groups. After the first processor finishes communication, it sends a notification message to the second processor in the communication group. This notification message can be a synchronization signal or other type of message, which is not specifically limited in this application. Then, during the second cross-regional communication, the second processor in each communication group communicates with processors in other regions, with the number of processors communicating still being the same as the number of communication groups, ensuring that the total communication bandwidth matches the convergence bandwidth. This process continues until the last processor in the communication group finishes communication. At this point, the last processor sends a notification message to the first processor. The first processor, having no data to communicate, will stop cross-regional communication and begin the next second intra-regional communication.

[0156] In another possible implementation, during cross-region communication, each communication group can have a proxy processor. This proxy processor handles the cross-region communication. In this case, during the first intra-region communication, other processors within the group send the data to be communicated to the proxy processor. During the second intra-region communication, the proxy processor distributes the cross-region communication results to the other processors within the communication group.

[0157] Figure 8 This is a flowchart illustrating another possible implementation of the data communication method provided in this application, such as... Figure 8 As shown, during the first intra-regional communication, inter-group reduce-scatter can be performed within the region. Each processor 1012 sends its local data to be communicated to processors 1012 in other communication groups, receives data to be communicated from processors 1012 in other communication groups, and obtains the intra-group communication results. Then, a gather is performed within the communication group, where each processor 1012 sends its intra-group communication results to the proxy processors within the communication group, allowing the proxy processors to obtain the intra-regional communication results. Next, cross-regional communication is achieved through allreduce. Each proxy processor sends its local intra-regional communication results to proxy processors in other regions and receives intra-regional communication results from proxy processors in other regions, thus obtaining the cross-regional communication results. Finally, during the second intra-regional communication, intra-group scatter can be performed first, with each agent processor sending cross-regional communication results to other processors in the communication group, so that each processor can obtain the cross-regional communication results of the agent processors in the group. Then, inter-group allgather can be performed, with each processor sending the cross-regional communication results of its own communication group to the processors 1012 in other communication groups in the region, and receiving the cross-regional communication results of other communication groups sent by the processors 1012 in other communication groups, thereby obtaining the global communication results and the final aggregated communication results.

[0158] For example, in Figure 8 In the example shown, during the first intra-regional communication, the communication groups first reduce scatter, allowing each processor to obtain data from other communication groups within the same region. Then, the communication groups gather the data and concentrate it on the proxy processor. During cross-regional communication, only the proxy processor of each communication group communicates externally. After multiple inter-regional communications, the proxy accelerator card first distributes data to each processor in the communication group through intra-group scatter, and then the communication groups all gather to obtain the final aggregated communication result.

[0159] It's understandable, it's understandable. Figure 7In the implementation shown, when communicating across regions, each accelerator card communicates with the outside world in sequence. The implementation logic is simple and does not require additional data aggregation steps. Figure 8 In the implementation shown, during cross-regional communication, only the proxy acceleration card communicates externally. The proxy acceleration card aggregates data from other acceleration cards, which can reduce the number of communications, but requires additional data aggregation steps. The specific method of cross-regional communication can be flexibly selected based on specific business needs, and this application does not impose specific limitations.

[0160] In specific implementation, when S410 receives a model training request sent by client 300, the model training request may include the implementation type of cross-regional communication, such as... Figure 7 The implementation type shown is sequential communication. Figure 8 The implementation type shown is proxy communication, which enables the network communication library 1011 to determine the corresponding communication strategy based on the algorithm corresponding to the implementation type selected by the user. This allows the accelerator card to perform aggregate communication according to the implementation type selected by the user when it performs aggregate communication based on the communication strategy.

[0161] Understandably, after obtaining the results of the ensemble communication, the processor can continue to complete the distributed training of the AI ​​model based on the results of the ensemble communication. If ensemble communication is required in the future, the data communication method provided in this application can also be used for ensemble communication. The communication efficiency of each ensemble communication is very high, thereby improving the overall training efficiency of distributed training.

[0162] In summary, this application provides a data communication method. Before the distributed training of the AI ​​model begins, the system pre-determines the number of communication groups based on the number of concurrent cards adapted to the inter-regional communication bandwidth. This ensures that after the system starts distributed training, if aggregated communication is required, only one processor in each communication group participates in communication during each cross-regional communication. This ensures that cross-regional communication will not experience congestion, avoids backpressure mechanisms caused by congestion, and prevents further reduction in cross-regional communication bandwidth, thereby improving the overall communication efficiency of aggregated communication.

[0163] The data communication method and distributed system provided in this application have been described in detail above. The following section will combine... Figure 9 The structure of the processor provided in this application will be described.

[0164] Figure 9This is a schematic diagram of the structure of a network communication library provided in this application. The network communication library can be the network communication library 1011 described above. The network communication library 1011 is deployed on a computing node, which includes the network communication library 1011 and multiple processors. The network communication library 1011 can be a software device or component for implementing network communication. The network communication library 1011 may include a series of APIs for performing specific network operations, such as data sending, receiving, and configuration. Developers can use the APIs to configure the data communication methods provided in this application, enabling the network communication library 1011 to achieve the following: Figure 9 The functions described by the multiple unit modules shown.

[0165] Specifically, such as Figure 9 As shown, the network communication library 1011 includes an acquisition unit 910, a determination unit 920, and a transmission unit 930. It should be understood that... Figure 9 This is an exemplary division method. The network communication library 1011 can be divided into more or fewer unit modules, and this application does not make any specific limitations.

[0166] The acquisition unit 910, the determination unit 920, and the transmission unit 930 can all be implemented in software or in hardware. For example, the implementation of the acquisition unit 910 will be described below. Similarly, the implementation of the determination unit 920 and the transmission unit 930 can refer to the implementation of the acquisition unit 910.

[0167] As an example of a software functional unit, the acquisition unit 910 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the aforementioned computing instance may be one or more. For example, the acquisition unit 910 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0168] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0169] As an example of a hardware functional unit, the acquisition unit 910 may include at least one computing device, such as a server. Alternatively, the acquisition unit 910 may be implemented using a central processing unit (CPU), an application-specific integrated circuit (ASIC), or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a data processing unit (DPU), a neural network processing unit (NPU), a system-on-a-chip (SoC), an offload card, an accelerator card, or any combination thereof.

[0170] The multiple computing devices included in the acquisition unit 910 can be distributed in the same region or in different regions. Similarly, the multiple computing devices included in the acquisition unit 910 can be distributed in the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices included in the acquisition unit 910 can be distributed in the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, GALs, DPUs, NPUs, SoCs, offloading cards, and accelerator cards.

[0171] It should be noted that, in other embodiments, the steps implemented by the acquisition unit 910, the determination unit 920, and the sending unit 930 can be specified as needed. The acquisition unit 910, the determination unit 920, and the sending unit 930 respectively implement different steps in the data communication method to realize all the functions of the network communication library 1011.

[0172] The functions of the acquisition unit 910, the determination unit 920, and the sending unit 930 are explained below.

[0173] Acquisition unit 910 is used to acquire the network topology information of the distributed system, specifically for implementing... Figure 4 The operation steps described in S430 of the embodiment.

[0174] The determining unit is configured to determine the total communication bandwidth between the first region and the second region, as well as the bandwidth of each processor, based on the network topology information, specifically for implementing... Figure 4 The operation steps described in S440 of the embodiment.

[0175] The determining unit 920 is configured to generate a communication strategy based on the total communication bandwidth and the bandwidth of each processor. The communication strategy includes: each time the processors in the first region and the processors in the second region conduct cross-regional communication, a total of M processors in the first region participate in the cross-regional communication, and the total bandwidth of the M processors is adapted to the total communication bandwidth between the first region and the second region. Specifically, this is used to implement... Figure 4 The operation steps described in S440 of the embodiment.

[0176] The sending unit 930 is used to send the communication strategy to at least one processor on the first computing node, specifically for implementing... Figure 4 The operation steps described in S450 of the embodiment.

[0177] In one possible implementation, each region includes M communication groups, each of the M communication groups includes at least one processor, and the communication strategy includes: the M processors participating in cross-region communication form different communication groups.

[0178] In one possible implementation, the communication policy issued by the network communication library is used to instruct the processor to execute... Figure 5 The illustrated steps demonstrate how to implement aggregated communication in a distributed system. Each computing node in the distributed system has a network communication library that can issue communication policies to the processors on that node, enabling the entire distributed system to communicate according to... Figures 6-8 The steps shown complete the collection communication. Further details are omitted here.

[0179] In summary, this application provides a network communication library. This processor is applied to a distributed system. Before the distributed training of the AI ​​model begins, the system pre-determines the number of communication groups based on the number of concurrent cards adapted to the inter-regional communication bandwidth. This ensures that after the system starts distributed training, if aggregated communication is required, only one processor in each communication group participates in communication during each cross-regional communication. This ensures that cross-regional communication will not experience congestion, avoids backpressure mechanisms caused by congestion, and prevents further reduction in cross-regional communication bandwidth, thereby improving the overall communication efficiency of aggregated communication.

[0180] The data communication method, distributed system, and network communication library provided in this application have been described in detail above. The following section will combine... Figure 10 and Figure 11 The computing device provided in this application will be explained.

[0181] Figure 10 This is a schematic diagram of the structure of a computing device provided in this application, such as... Figure 10 As shown, the computing device 900 includes: a bus 902, a processor 904, a memory 906, an accelerator card 907, and a communication interface 908. The processor 904, memory 906, and communication interface 908 communicate with each other via the bus 902. The computing device 900 can be a server. It should be understood that this application does not limit the number of processors, memory, and accelerator cards in the computing device 900. The computing device 900 can be used to deploy the aforementioned components. Figures 1-8 The computing node 101 in this embodiment. Memory 906 is used to store instructions, and processor 904 is used to execute the instructions stored in memory 906. Memory 906 stores program code, and processor 904 can call the program code stored in memory 906 to execute. Figure 4 S410 to S460 and their optional steps are described.

[0182] It should be understood that the memory 906 stores executable program code, and the processor 904 executes this executable program code to achieve the above. Figures 1 to 9 The operational steps of the data communication method described in each embodiment are as follows. The program code may include one or more software units, which are... Figure 9 The embodiment includes an acquisition unit, a determination unit, and a sending unit. The acquisition unit is used to acquire network topology information of the distributed system, specifically for implementing... Figure 4S430 and its optional steps in the embodiment. The determining unit is used to determine the total communication bandwidth between the first region and the second region, as well as the bandwidth of each processor, based on network topology information, thereby generating a communication strategy. This communication strategy includes that each time the first processor and the second processor conduct cross-region communication, a total of M processors in the first region participate in the cross-region communication, and the total bandwidth of these M processors is adapted to the total cross-region communication bandwidth between the first region and the second region. Specifically used to implement... Figure 4 The embodiment includes S440 and its optional steps. The sending unit is used to send the communication strategy to at least one processor on the computing node, specifically for implementing... Figure 4 The S450 and its optional steps in the embodiment are described below. Further details about each unit will not be elaborated here.

[0183] The 902 bus can be a Peripheral Component Interconnect Express (PCIe) bus, an Extended Industry Standard Architecture (EISA) bus, a Unified Bus (Ubus or UB), a Compute Express Link (CXL) bus, a Cache Coherent Interconnect for Accelerators (CCIX) bus, etc. The Unified Bus is also known as the Lingqu bus. Buses can be divided into address buses, data buses, control buses, etc. For ease of representation, Figure 10 The bus 902 is represented by only one line, but this does not mean that there is only one bus or one type of bus. The bus 902 may include a path for transmitting information between various components of the computing device 900 (e.g., memory 906, processor 904, communication interface 908). The unified bus may also be called the Lingqu bus.

[0184] The processor 904 may include any one or more computing devices such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP) or a digital signal processor (DSP), an ASIC, an FPGA, a CPLD, an NPU, a SoC, an offload card, or an accelerator card.

[0185] Memory 906 may include volatile memory, such as random access memory (RAM). Memory 906 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD). Furthermore, memory 906 may also be implemented using storage class memory (SCM), phase change memory (PCM), or other types of storage media.

[0186] It is worth noting that the same type of storage medium can be configured in the same computing device to realize the function of memory 906, or two or more types of storage media can be configured to realize the function of memory 906. This application does not limit this.

[0187] The communication interface 908 uses transceiver modules, such as, but not limited to, network interface cards and transceivers, to enable communication between the computing device 900 and other devices or communication networks.

[0188] Furthermore, the accelerator card 907 can be implemented using computing devices such as DPU, NPU, GPU, XPU, SoC, or AI chips. The accelerator card 907 is used to implement distributed training of AI models. It includes a processing unit and a power supply circuit. The power supply circuit supplies power to the processing unit, enabling it to perform the following steps:

[0189] Obtain the first piece of data to be communicated;

[0190] It performs intra-regional communication with other processors in the first region, sends the first data to be communicated to other processors in the first region, receives data sent by other processors in the first region, and obtains the intra-regional communication results.

[0191] The first processor communicates with the second processor in the second region across regions, sends the communication results within the region to the second processor, and receives the data sent by the second processor to obtain the cross-regional communication results. When the first processor communicates with the second processor across regions, M-1 processors in the first region also communicate across regions. The total communication bandwidth of the M processors formed by the first processor and M-1 processors is adapted to the total communication bandwidth between the first region and the second region.

[0192] It can communicate with other processors in the first region within the region, send cross-regional communication results to other processors in the first region, and receive data sent by other processors in the first region to obtain set communication results.

[0193] It should be understood that the accelerator card 907 can be understood as the processor 1012 mentioned above, that is, the processor that participates in the distributed training of the model, while the processor 904 refers to the processing core in the computing device 900, such as the CPU. The two are different concepts.

[0194] This application also provides a computing device cluster, such as Figure 11 As shown, Figure 11 This is an example diagram of a computing device cluster provided in this application, which may include at least one Figure 10 The computing device of the embodiment is used to deploy the distributed system described above.

[0195] As one possible implementation, the same implementation can be stored in the memory 906 of one or more computing devices 900 in a computing device cluster. Figures 4 to 8 The instructions describe the operational steps of the data communication method. Alternatively, different implementations may be stored separately in the memory 906 of one or more computing devices 900. Figures 4 to 8 The data communication method described refers to the instructions for the operational steps that the computing node 101 needs to implement. In other words, a combination of one or more computing devices 900 can jointly execute the instructions for implementing the operational steps that the computing node 101 needs to implement in the data communication method.

[0196] This application embodiment also provides a chip, which includes a processing unit and a power supply circuit. The power supply circuit supplies power to the processing unit, enabling the processing unit to perform... Figures 4 to 8 The operation steps implemented by the processor 1012 in the data communication method of the embodiment will not be described in detail here for the sake of brevity.

[0197] This application also provides a chip system comprising multiple chips as described above, which is used to deploy the distributed training system described above.

[0198] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product runs on at least one computing device, it causes the at least one computing device to execute the data communication method provided in this application.

[0199] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium capable of being stored by a computing device, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct a computing device to execute the data communication method provided in this application.

[0200] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of this application.

Claims

1. A data communication method, characterized in that, The method is applied to a distributed system, which includes a first region and a second region, each region including multiple processors. The distributed system is used to execute a distributed training task of an AI model, and each processor is used to execute a subtask of the distributed training task. The method includes: The first processor in the first region acquires the first data to be communicated. The first processor communicates with other processors in the first region, sends the first data to be communicated to other processors in the first region, receives data sent by other processors in the first region, and obtains the communication result within the region. The first processor communicates with the second processor in the second region across regions, sends the communication results within the region to the second processor, and receives the data sent by the second processor to obtain the cross-regional communication results. When the first processor communicates with the second processor across regions, M-1 processors in the first region also communicate across regions. The total communication bandwidth of the M processors formed by the first processor and the M-1 processors is adapted to the total communication bandwidth between the first region and the second region. The first processor communicates with other processors in the first region within the region, sends the cross-regional communication results to other processors in the first region, and receives data sent by other processors in the first region to obtain a set communication result.

2. The method according to claim 1, characterized in that, Each region includes M communication groups, and each of the M communication groups includes at least one processor. The M processors participating in cross-regional communication form different communication groups.

3. The method according to claim 2, characterized in that, The cross-regional communication type includes sequential communication. After the first processor and the second processor in the second region perform cross-regional communication, the method further includes: The first processor sends a notification message to the third processor, the notification message being used to instruct the third processor to conduct cross-regional communication with the fourth processor in the second region, wherein the first processor and the third processor belong to a first communication group, and the second processor and the fourth processor belong to a second communication group.

4. The method according to claim 2, characterized in that, The cross-regional communication type includes proxy communication, and the first intra-regional communication between the first processor and other processors in the first region includes: The first processor communicates with the second processor in the second region multiple times across regions. The first processor is a proxy processor of the first communication group. The first processor includes the communication data to be communicated by all processors in the first communication group. Each time there is cross-regional communication, there are M proxy processors in the first region participating in the cross-regional communication.

5. The method according to claim 4, characterized in that, The first processor communicates with other processors in the first region, sending the first data to be communicated to other processors in the first region, receiving data sent by other processors in the first region, and obtaining the intra-regional communication result includes: The first processor performs inter-group communication with processors in other communication groups in the first region, sends the data to be communicated to processors in other communication groups in the first region, receives data sent by processors in other communication groups, and obtains the inter-group communication result. The first processor communicates with other processors in the first communication group, sends the inter-group communication results to other processors in the first communication group, receives data sent by other processors in the first communication group, and obtains the communication results within the region.

6. The method according to any one of claims 3 to 5, characterized in that, Each region includes at least one computing node, the at least one computing node includes a network communication library and at least one processor, and before the first processor in the first region acquires the first data to be communicated, the method further includes: The first processor receives a communication strategy sent by the network communication library of the first computing node, the first computing node including the first processor, and the communication strategy is determined by the network communication library based on the number M of the communication groups. The number M of the communication groups is determined by the network communication library after determining the total communication bandwidth between the first region and the second region based on the network topology information of the distributed system, combined with the communication type and the processing bandwidth of each processor.

7. A data communication method, characterized in that, The method is applied to a distributed system, which includes a first region and a second region. Each region includes multiple computing nodes, and each computing node includes a network communication library and at least one processor. The distributed system is used to execute a distributed training task of an AI model, and each processor is used to execute a subtask of the distributed training task. The method includes: The network communication library of the first computing node among the plurality of computing nodes obtains the network topology information of the distributed system. The network communication library of the first computing node determines the total communication bandwidth between the first region and the second region, as well as the bandwidth of each processor, based on the network topology information; The network communication library of the first computing node generates a communication strategy based on the total communication bandwidth and the bandwidth of each processor. The communication strategy includes: each time the processors of the first region and the processors of the second region conduct cross-region communication, a total of M processors in the first region participate in the cross-region communication, and the total bandwidth of the M processors is adapted to the total communication bandwidth between the first region and the second region. The network communication library of the first computing node sends the communication strategy to at least one processor on the first computing node.

8. The method according to claim 7, characterized in that, Each region includes M communication groups, and each of the M communication groups includes at least one processor. The communication strategy includes: the M processors participating in cross-region communication form different communication groups.

9. A distributed system, characterized in that, The distributed system includes a first region and a second region, each region including multiple processors. The distributed system is used to execute a distributed training task of an AI model, and each processor is used to execute a subtask of the distributed training task. The distributed system includes a first processor in the first region and a second processor in the second region. The first processor is used to acquire the first data to be communicated; The second processor is used to acquire the second data to be communicated; The first processor is configured to communicate with other processors in the first region, send the first data to be communicated to other processors in the first region, receive data sent by other processors in the first region, and obtain the communication result in the first region. The second processor is used to communicate with other processors in the second region, send the second data to be communicated to other processors in the second region, receive data sent by other processors in the second region, and obtain the communication result in the second region. The first processor is configured to communicate across regions with the second processor, send communication results within the first region to the second processor, and receive data sent by the second processor to obtain the first cross-region communication result; The second processor is used to communicate across regions with the first processor, send communication results within the second region to the first processor, and receive data sent by the first processor to obtain a second cross-region communication result. When the first processor and the second processor communicate across regions, M-1 processors in the first region also communicate across regions. The total communication bandwidth of the M processors formed by the first processor and the M-1 processors is adapted to the total communication bandwidth between the first region and the second region. The first processor is configured to communicate with other processors in the first region within the region, send the first cross-regional communication result to other processors in the first region, and receive data sent by other processors in the first region to obtain a set communication result; The second processor is used to communicate with other processors in the second region within the region, send the second cross-regional communication result to other processors in the second region, and receive data sent by other processors in the second region to obtain a set communication result.

10. The distributed system according to claim 9, characterized in that, Each region includes M communication groups, and each of the M communication groups includes at least one processor. The M processors participating in cross-regional communication form different communication groups.

11. The distributed system according to claim 10, characterized in that, The communication types of the cross-regional communication include sequential communication; The first processor is configured to send a notification message to the third processor, the notification message being configured to instruct the third processor to conduct cross-regional communication with the fourth processor in the second region, wherein the first processor and the third processor belong to a first communication group, and the second processor and the fourth processor belong to a second communication group; The second processor is configured to send a notification message to the fourth processor, the notification message being used to instruct the fourth processor to conduct cross-regional communication with the third processor in the first region.

12. The distributed system according to claim 11, characterized in that, The communication types for cross-regional communication include proxy communication; The first processor is used to perform multiple cross-regional communications with the second processor. The first processor is a proxy processor of the first communication group. The first processor includes the communication data to be communicated by all processors in the first communication group. During each cross-regional communication, there are M proxy processors in the first region participating in the cross-regional communication. The second processor is used to perform multiple cross-regional communications with the first processor. The second processor is a proxy processor of the second communication group. The second processor includes the communication data to be communicated by all processors in the second communication group. During each cross-regional communication, a total of M proxy processors in the second region participate in the cross-regional communication.

13. The distributed system according to claim 12, characterized in that, The first processor is configured to perform inter-group communication with processors of other communication groups in the first region, send the first data to be communicated to processors of other communication groups in the first region, receive data sent by processors of other communication groups, obtain a first inter-group communication result, perform intra-group communication with other processors of the first communication group, send the first inter-group communication result to other processors in the first communication group, receive data sent by other processors of the first communication group, and obtain communication results within the first region. The second processor is configured to perform inter-group communication with processors in other communication groups in the second region, send the data to be communicated to processors in other communication groups in the second region, receive data sent by processors in other communication groups, obtain inter-group communication results, perform intra-group communication with other processors in the second communication group, send the inter-group communication results to other processors in the second communication group, receive data sent by other processors in the second communication group, and obtain communication results within the second region.

14. The distributed system according to any one of claims 11 to 13, characterized in that, Each region includes at least one computing node, and the at least one computing node includes a network communication library and at least one processor; The first processor is configured to receive a communication strategy sent by the network communication library of the first computing node. The first computing node includes the first processor. The communication strategy is determined by the network communication library based on the number M of the communication groups. The number M of the communication groups is determined by the network topology information of the distributed system to determine the total communication bandwidth between the first region and the second region, combined with the communication type and the processing bandwidth of each processor. The second processor is configured to receive a communication strategy sent by the network communication library of the second computing node, the second computing node including the second processor, and the communication strategy being determined by the network communication library based on the number M of the communication groups.

15. A network communication library, characterized in that, The network communication library is applied to a distributed system, which includes a first region and a second region. Each region includes multiple computing nodes, and each computing node includes the network communication library and at least one processor. The distributed system is used to execute distributed training tasks for AI models, and each processor is used to execute subtasks of the distributed training task. The network communication library includes: The acquisition unit is used to acquire the network topology information of the distributed system; A determining unit is configured to determine the total communication bandwidth between the first region and the second region, as well as the bandwidth of each processor, based on the network topology information. The determining unit is configured to generate a communication strategy based on the total communication bandwidth and the bandwidth of each processor. The communication strategy includes: each time the processors in the first region and the processors in the second region conduct cross-regional communication, a total of M processors in the first region participate in the cross-regional communication, and the total bandwidth of the M processors is adapted to the total communication bandwidth between the first region and the second region. A sending unit is used to send the communication strategy to at least one processor on the first computing node.

16. The network communication library according to claim 15, characterized in that, Each region includes M communication groups, and each of the M communication groups includes at least one processor. The communication strategy includes: the M processors participating in cross-region communication form different communication groups.

17. A chip, characterized in that, The chip includes a power supply unit and a processing unit. The power supply unit supplies power to the processing unit, enabling the processing unit to implement the method as described in any one of claims 1 to 6.

18. A computer-readable storage medium, characterized in that, It includes computer program instructions, which, when executed by a computing device, cause the computing device to perform the method as described in claim 7 or 8.

19. A computer program product containing instructions, characterized in that, When the instruction is executed by the computing device, the computing device performs the method as described in claim 7 or 8.

Images