Data exchange method and system

Abstract

The invention provides a data exchange method and system, which are applied to a data exchange system comprising a plurality of devices, each of the plurality of devices comprises a plurality of nodes, the plurality of nodes of the same device exchange data through links among the nodes, and the nodes of different devices exchange data through links among the devices, the first node receives multiple copies of first data sequentially sent by the second node through the first link; in the process of receiving the multiple pieces of first data, multiple pieces of second data are sent to the third node through the second link; wherein the first link and the second link are different types of links, and the different types of links comprise the links between the nodes and the links between the devices. Through the method, when data exchange needs to be carried out among a plurality of nodes, bandwidth resources of two links can be utilized at the same time, the utilization rate of the bandwidth of the system is improved, and the communication efficiency of the system is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a data exchange method and system. BACKGROUND

[0002] Multi-die package is a kind of integrated circuit packaging technology, which is used to package multiple dies together to obtain a chip to improve the computing power of a single chip. In a distributed computing system including multiple chips, if each chip includes multiple dies, the distributed computing system will include two communication planes, one is the communication plane between the dies in different chips, and the other is the communication plane between the multiple dies in the same chip. When performing distributed computing through the above-mentioned distributed computing system, for example, training an artificial intelligence (AI) model, each die in each chip will participate in the calculation and obtain the calculation result, and when the dies in multiple chips exchange data through collective communication to synchronize the calculation results, the data exchange will be performed through the above-mentioned two communication planes. However, the bandwidths of the above-mentioned two communication planes are different, therefore, how to fully utilize the bandwidths of the above-mentioned two communication planes to realize efficient data exchange between the multiple dies in multiple chips is a technical problem to be solved. SUMMARY

[0003] The present application provides a data exchange method and system, which realizes data exchange between multiple nodes by simultaneously utilizing the bandwidths of two communication planes in a data exchange system, thereby improving the bandwidth utilization of the system and the data exchange efficiency.

[0004] In a first aspect, the present application provides a data exchange method, which is used in a data exchange system including multiple devices, each device in the multiple devices including multiple nodes, the multiple nodes of the same device exchanging data through a first link between nodes, and the nodes of different devices exchanging data through a second link between devices. For a first node in a device, the first node receives multiple copies of first data sent by a second node of another device through the second link in sequence; the first node sends multiple copies of second data to a third node in the same device through the first link in the process of receiving the multiple copies of first data; the first node also sends multiple copies of third data to the second node through the second link in the process of receiving the multiple copies of first data, and the second node sends multiple copies of fourth data to a fourth node in the same device through the first link in the process of receiving the multiple copies of third data.

[0005] In the aforementioned data exchange system, communication links between nodes in different devices constitute one communication plane, while communication links between nodes within the same device constitute another. When multiple nodes in the data exchange system need to exchange data, the aforementioned data exchange method allows for simultaneous data exchange between nodes on different devices and between nodes within the same device. Therefore, this method can utilize the bandwidth of both communication planes simultaneously, improving system bandwidth utilization, reducing system latency, and increasing system data exchange efficiency. Furthermore, by simultaneously utilizing the bandwidth of both communication planes, the traffic during the entire data exchange process can be evenly distributed across both communication planes, achieving rational utilization of the bandwidth of both planes and improving system bandwidth utilization.

[0006] The above methods can realize allgather communication in aggregation communication, such as allgather communication based on the butterfly algorithm, thereby improving the utilization of system bandwidth and communication efficiency during the implementation of allgather communication through the above data exchange system.

[0007] In one possible implementation, the aforementioned multiple sets of second data are identical to the multiple sets of first data, and the multiple sets of fourth data are identical to the multiple sets of third data. During the process of receiving multiple sets of first data, the first node sends multiple sets of second data to the third node via the second link, including: after receiving one set of first data sent by the second node via the second link each time, the first node sends the received set of first data to the third node via the first link. Similarly, during the process of receiving multiple sets of third data sent by the first node via the second link, the second node sends the received set of third data to the fourth node within the same device via the first link.

[0008] In one possible implementation, in the process of receiving multiple sets of first data, the first node, in addition to sending multiple sets of first data to the third node, also sends multiple sets of fifth data to the third node through the first link. The aforementioned multiple sets of fifth data are data generated by the first node.

[0009] Since the first node and the third node are nodes within the same device, the bandwidth between nodes within the same device is usually greater than the bandwidth between nodes on different devices. Therefore, after sending a first piece of data to the third node, the first node can send a piece of data generated during the calculation process to the third node before receiving or sending the next piece of first data. This can further improve the bandwidth utilization between nodes within the same device and improve data exchange efficiency.

[0010] In a possible implementation, the multiple pieces of second data are data that already exist in the first node before the multiple pieces of first data are received.

[0011] In a possible implementation, the device is a chip, and the nodes are dies, and a plurality of dies are packaged in each chip. The device can also be a server, and the nodes can be acceleration cards. The method can be applied to a data exchange system including chips and dies, and can also be applied to a data exchange system including servers and acceleration cards.

[0012] In a possible implementation, if the device is a chip and the nodes are dies, the links between the nodes are Universal Chiplet Interconnect Express (UCIe) buses or other links that can enable communication between dies; and the links between the devices can be NVLink buses, Peripheral Component Interconnect Express (PCIe) buses, or Unified Buses (UBs). If the device is a server and the nodes are acceleration cards, the first links between the nodes are NVLink buses, PCIe buses, or UB buses, and the second links between the devices are Ethernet links, InfiniBand (IB) links, or fiber links.

[0013] In a possible implementation, the data exchange system is used for training of an artificial intelligence (AI) model, and the data transmitted between the nodes included in the data exchange system is data generated in the training of the AI model.

[0014] The data exchange system is used for training of an AI model, and the data exchange between the nodes in the training of the AI model is implemented by using the data exchange method, for example, allgather communication based on a butterfly algorithm, which can reduce the communication time in the training of the model and improve the efficiency of the training of the model.

[0015] In a second aspect, the present application provides a data exchange method, which is used in a data exchange system comprising a plurality of devices, each of the plurality of devices comprising a plurality of nodes, the plurality of nodes of the same device exchanging data through first links between the nodes, and the nodes of different devices exchanging data through second links between the devices. For a first node in a device, the first node receives a plurality of first data sent by a second node in another device through the second link in turn; the first node sends a plurality of second data to a third node in the same device through the first link in the process of receiving the plurality of first data; the first node also sends a plurality of third data to a fourth node in another device through the second link in the process of receiving the plurality of first data; the fourth node sends a plurality of fourth data to a fifth node in the same device through the first link in the process of receiving the plurality of third data.

[0016] In the above data exchange system, the communication links between the nodes of different devices constitute a communication plane, and the communication links between the nodes of the same device constitute another communication plane. When the plurality of nodes included in the data exchange system need to exchange data with each other, the above data exchange method can be used to exchange data between the nodes of different devices while exchanging data between the nodes of the same device, so that the bandwidth of the above two communication planes can be used at the same time by the above method, the bandwidth utilization of the system is improved, the time delay of the system is reduced, and the data exchange efficiency of the system is improved. By using the bandwidth of the two communication planes at the same time, the traffic in the entire data exchange process can be balanced to the two communication planes, the bandwidth of the two planes can be reasonably utilized, and the bandwidth utilization of the system is improved.

[0017] The above method can be used to realize allgather communication in aggregate communication, such as allgather communication based on a ring algorithm, so as to improve the bandwidth utilization of the system in the allgather communication process realized by the above data exchange system and improve the communication efficiency.

[0018] In a possible implementation, the plurality of second data is the same as the plurality of first data, and the plurality of fourth data is the same as the plurality of third data; in the process of receiving the plurality of first data, the first node sends the plurality of second data to the third node through the second link, including: the first node sends the received first data to the third node through the first link after receiving the first data sent by the second node through the second link each time. In the process of receiving the plurality of third data sent by the first node through the second link each time, the second node sends the received third data to the fourth node in the same device through the first link.

[0019] In a possible implementation, in the process of receiving the multiple pieces of first data, the first node, in addition to sending the multiple pieces of first data to the third node, also sends multiple pieces of fifth data to the third node through the first link, the multiple pieces of fifth data being data generated by the first node.

[0020] Since the first node and the third node are nodes in the same device, the bandwidth between nodes in the same device is usually greater than the bandwidth between nodes of different devices, and therefore, the first node can send data generated by the first node when performing calculation to the third node after sending a piece of first data to the third node, before receiving the next piece of first data or sending the next piece of first data, which can further improve the bandwidth utilization of the bandwidth between nodes in the same device and improve the data exchange efficiency.

[0021] In a possible implementation, the multiple pieces of second data are data already existing in the first node before receiving the multiple pieces of first data.

[0022] In a possible implementation, the device is a chip, and the nodes are dies, and a plurality of dies are packaged in each chip. The device can also be a server, and the nodes can be acceleration cards. The method can be applied to a data exchange system including a chip and a die, and can also be applied to a data exchange system including a server and an acceleration card.

[0023] In a possible implementation, if the device is a chip and the nodes are dies, the links between the nodes are UCIe buses or other links that can realize communication between dies; and the links between the devices can be NVLink buses, PCIe buses, or UB buses. If the device is a server and the nodes are acceleration cards, the first links between the nodes are NVLink buses, PCIe buses, or UB buses, and the second links between the devices are Ethernet links, IB links, or optical fiber links.

[0024] In a possible implementation, the data exchange system is used for training of an artificial intelligence (AI) model, and the data transmitted between the nodes included in the data exchange system is data generated in the training of the AI model.

[0025] The data exchange system is used for training of an AI model, and the data exchange between the nodes in the training of the AI model is realized through the data exchange method, for example, allgather communication based on a ring algorithm, which can reduce the communication time in the model training process and improve the efficiency of the model training.

[0026] In a third aspect, the present application provides a data exchange method applied to a data exchange system comprising a plurality of devices, each of the plurality of devices comprising a plurality of nodes, the plurality of nodes of the same device exchanging data through first links between the nodes, and the nodes of different devices exchanging data through second links between the devices. For a first node in a device, the first node receives a plurality of first data successively sent by a second node in the same device through the first links; after receiving each of the first data sent by the second node, the first node performs a reduction operation on one of the first data and one of third data to obtain one of second data, and then sends one of the second data to a third node in another device through the second links.

[0027] In the above data exchange system, the communication links between the nodes in different devices constitute one communication plane, and the communication links between the nodes in the same device constitute another communication plane. When the plurality of nodes included in the data exchange system need to exchange data with each other, the above data exchange method can be used to exchange data between the nodes of different devices while exchanging data between the nodes in the same device, so that the bandwidth of the above two communication planes can be used simultaneously by the above method, the bandwidth utilization of the system is improved, the time delay of the system is reduced, and the data exchange efficiency of the system is improved. By simultaneously using the bandwidth of the two communication planes, the traffic in the entire data exchange process can be balanced to the two communication planes, the bandwidth of the two planes is reasonably utilized, and the bandwidth utilization of the system is improved.

[0028] The above method can realize reduce_scatter communication in aggregate communication, improve the bandwidth utilization of the system in the process of realizing reduce_scatter communication through the above data exchange system, and improve the communication efficiency.

[0029] In a possible implementation, the above method further comprises: the first node sends a plurality of fourth data to the second node through the first links, and the second node performs a reduction operation on each of the fourth data and one of fifth data to obtain a reduction result, and sends the reduction result to a fourth node in another device through the second links; wherein the third node and the fourth node are located in the same device.

[0030] The first node sends data to the second node in the process of receiving data sent by the second node, which can fully utilize the bidirectional bandwidth between the two nodes, improve the bandwidth utilization, and at the same time, can utilize the bandwidth utilization between the second node and the fourth node.

[0031] In a possible implementation, the method further includes: the first node receiving multiple pieces of sixth data sent by the third node through the second link; each piece of the sixth data is obtained by performing a reduction operation on a piece of seventh data sent by the fourth node and received by the third node; and the first node performing a reduction operation on the multiple pieces of the received sixth data to obtain a reduction result.

[0032] The first node receives data sent by the third node in the process of sending data to the third node, which can fully utilize the bidirectional bandwidth between the two nodes and improve the bandwidth utilization.

[0033] In a possible implementation, the device is a chip, and the nodes are dies, and a plurality of dies are packaged in each chip. The device can also be a server, and the nodes can be acceleration cards. The method can be applied to a data exchange system including chips and dies and can also be applied to a data exchange system including servers and acceleration cards.

[0034] In a possible implementation, if the device is a chip and the nodes are dies, the links between the nodes are UCIe buses or other links that can realize communication between dies; and the links between the devices can be NVLink buses, PCIe buses or UB buses. If the device is a server and the nodes are acceleration cards, the first links between the nodes are NVLink buses, PCIe buses or UB buses, and the second links between the devices are Ethernet links, IB links or fiber links.

[0035] In a possible implementation, the data exchange system is used for training of an artificial intelligence (AI) model, and the data transmitted between the nodes included in the data exchange system is data generated in the training of the AI model.

[0036] The data exchange system is used for training of an AI model, and the data exchange between the nodes in the training of the AI model is realized by using the data exchange method, for example, allgather communication based on a ring algorithm, which can reduce the communication time in the model training process and improve the efficiency of the model training.

[0037] In a fourth aspect, the present application provides a computing node, which includes a plurality of processing circuits and a communication interface, the communication interface is used for inter-node communication, and each processing circuit is used for executing the data exchange method in the first aspect or any possible implementation of the first aspect.

[0038] In a fifth aspect, the present application provides a computing node, comprising a plurality of processing circuits and a communication interface, the communication interface being configured to perform inter-node communication, each of the processing circuits being configured to perform the data exchange method according to the second aspect or any possible implementation of the second aspect.

[0039] In a sixth aspect, the present application provides a computing node, comprising a plurality of processing circuits and a communication interface, the communication interface being configured to perform inter-node communication, each of the processing circuits being configured to perform the data exchange method according to the third aspect or any possible implementation of the third aspect.

[0040] On the basis of the implementation manners of the aspects provided by the present application, further combinations can be made to provide more implementation manners. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 is a schematic diagram of a data exchange system provided by the present application;

[0042] Figure 2 is a method for implementing allgather provided by the present application;

[0043] Figure 3 is a schematic diagram of another data exchange system provided by the present application;

[0044] Figure 4 is a flowchart of a data exchange method provided by the present application;

[0045] Figure 5 is a schematic diagram of a method for implementing reduce_scatter provided by the present application;

[0046] Figure 6 is a schematic diagram of a method for implementing allgather provided by the present application;

[0047] Figure 7 is another schematic diagram of a method for implementing allgather provided by the present application;

[0048] Figure 8 is another schematic diagram of a method for implementing allgather provided by the present application;

[0049] Figure 9 is a schematic diagram of a computing node provided by the present application;

[0050] Figure 10 is a schematic diagram of a computing device provided by the present application. DETAILED DESCRIPTION

[0051] The data exchange method provided by the present application will be described below in combination with the accompanying drawings.

[0052] With the development of computer technology, the scale of data that needs to be processed by applications is also increasing, for example, in the current AI large model training and other scenarios, the scale of data that needs to be processed is getting larger and larger. In order to solve the computing problem of large-scale data, distributed computing application arises at the historic moment, distributed computing performs parallel computing through a distributed computing system to improve computing efficiency. The distributed computing system can be a system including multiple servers and multiple acceleration cards, wherein each server includes multiple acceleration cards; the distributed computing system can also be a system including multiple chips and multiple dies, wherein each chip includes multiple dies. After the distributed computing system completes the computation, it needs to synchronize data between each acceleration card or each die in the system. The above two kinds of distributed computing systems both include two communication planes, and the distributed computing system needs to exchange data between each acceleration card or each die through the two communication planes to synchronize the computing result. For example, in the distributed computing system including multiple servers and multiple acceleration cards, the communication link between multiple acceleration cards in the same server is one communication plane, and the communication link between acceleration cards in different servers is another communication plane; in the distributed computing system including multiple chips and multiple dies, the communication link between multiple dies in the same chip is one communication plane, and the communication link between dies in different chips is another communication plane.

[0053] However, in the process of exchanging data through the above two communication planes, the current communication mode of the distributed computing system has the problem of low bandwidth utilization of the two communication planes. Taking the distributed computing system including multiple chips and multiple dies as an example, the problem of low bandwidth utilization of the distributed computing system is described below.

[0054] Referring to Figure 1 , Figure 1 is a schematic diagram of a data exchange system provided by the present application, which includes multiple chips, each chip including multiple dies, Figure 1 In the example of each chip including four dies, the multiple dies in each chip are fully interconnected, and the dies in different chips are fully interconnected through an interconnection network. The data exchange system can perform distributed computing as a distributed computing system. As shown in Figure 1 The data exchange system includes two communication planes, one of which is a communication plane composed of communication links between multiple dies in the same chip; the other is a communication plane composed of communication links between dies in different chips. For ease of description, the communication link between multiple dies in the same chip is referred to as die2die, and the communication link between dies in different chips is referred to as chip2chip.

[0055] When calculations are performed using the aforementioned data exchange system, and multiple dies within the system (comprising multiple chips) need to exchange data to synchronize the results, these dies can exchange data via aggregated communication. However, the current communication method cannot fully utilize the bandwidth of the two communication planes mentioned above.

[0056] For example, taking the above example where each chip includes two dies, such as... Figure 2 As shown, Figure 2 This application provides a method for implementing allgather. Figure 2 The system consists of four chips: chip1, chip2, chip3, and chip4. Each chip includes two dies: chip1 includes dies D0 and D1, chip2 includes dies D2 and D3, chip3 includes dies D4 and D5, and chip4 includes dies D6 and D7. The data in dies D0 to D7 are M0 to M7, respectively. It should be noted that data Mi is a single data unit, which can be a vector, matrix, or tensor, etc., and the size of each data unit is N. When implementing AllGather using the Butterfly algorithm, AllGather communication is achieved through the following three steps:

[0057] (1) Data is exchanged between two dies within the same chip via die2die, such as Figure 2 As shown, in chip 1, D0 sends data M0 to D1, and D1 sends data M1 to D0. The amount of data sent by D0 and D1 is N. The data exchange in the other three chips is the same as in chip 1, and will not be described again here. During the execution of step (1), the flow rate in one direction of die2die is N. After completing step (1), the amount of data in each die is 2N. The two dies in each chip contain the same data. For example, D0 and D1 both contain data M0 and M1.

[0058] (2) Two dies within different chips form a communication group, and no two communication groups contain the same dies. Two dies within a communication group exchange data via chip-to-chip communication, with each die sending its locally stored 2N data to the other die in the communication group. For example... Figure 2As shown, the eight dies form four communication groups: D0 and D2, D1 and D3, D4 and D6, and D5 and D7. D0 sends data M0 and M1 to D2, and D2 sends data M2 and M3 to D0. The data exchange in other communication groups is similar to that in D0 and D2, and will not be described again here. During the execution of step (2), the flow rate in one direction of chip2chip is 2N. After completing step (2), the amount of data in each die is 4N. The two dies in each chip contain the same data, and the two dies in each communication group also contain the same data. For example, D0 and D1 both contain data M0, M1, M2, and M3, and D0 and D2 both contain data M0, M1, M2, and M3.

[0059] (3) Two dies containing different data form a communication group. Two dies within a communication group exchange data via chip-to-chip communication. Each die sends its locally stored 4N data to another die within the communication group, ultimately ensuring that the data contained in all eight dies is identical. For example... Figure 2 As shown, the eight dies form four communication groups: D0 and D4, D1 and D5, D2 and D6, and D3 and D7. D0 sends data M0, M1, M2, and M3 to D4, and D4 sends data M4, M5, M6, and M7 to D0. Data exchange in other communication groups follows the same pattern as D0 and D4, and will not be described further here. During step (3), the flow rate in one direction of chip2chip is 4N. After completing step (3), the amount of data in each die is 8N, and the data included in each die is the same.

[0060] In the above process of implementing AllGather using the Butterfly algorithm, the bandwidth of only one communication plane is used in each step. For example, in step (1), only the die2die communication link is used, and in steps (2) and (3), only the chip2chip communication link is used. This does not fully utilize the bandwidth of the two communication planes, resulting in low bandwidth utilization of the distributed computing system. In addition, only step (1) uses the die2die communication link, and the amount of data transmitted through die2die is only N, while the amount of data transmitted through chip2chip is 6N, indicating an imbalance in the amount of data transmitted through the two communication planes.

[0061] In distributed computing systems comprising multiple servers and accelerator cards, the servers are analogous to chips and the accelerator cards to dies. Therefore, these systems also suffer from low bandwidth utilization. Consequently, how to efficiently utilize the bandwidth of the chip-to-chip and die-to-die communication planes is a pressing technical problem that needs to be solved.

[0062] See Figure 3 , Figure 3 This is a schematic diagram of another data exchange system provided in this application. The data exchange system includes multiple devices, and each device includes multiple nodes. Figure 3 Taking a device with four nodes as an example, multiple nodes within the same device are fully interconnected via inter-node communication links, while nodes in different devices are fully interconnected via an inter-device network. This data exchange system can function as a distributed computing system for distributed computing. The communication links between multiple nodes within the same device constitute a first communication plane, and the communication links between nodes in different devices constitute a second communication plane. In this application, the links in the first communication plane are referred to as inter-node links, and the links in the second communication plane are referred to as inter-device links.

[0063] Figure 1 In this context, "die" is equivalent to... Figure 3 The nodes in Figure 1 The chip in the middle is equivalent to Figure 3 The equipment in, therefore, is related to the above. Figure 1 The data exchange system shown is the same. Figure 3 In the distributed computing process shown in the data exchange system, if nodes in the system need to exchange data to synchronize computing results, the current communication method cannot fully utilize the available resources. Figure 3 The bandwidth of the two communication planes.

[0064] This application provides a data exchange method, applicable to... Figure 3 The data exchange system shown improves bandwidth utilization and efficiency by using two communication planes simultaneously during data exchange. It also enhances the transmission of data through communication links between nodes within the same device, further improving data exchange efficiency.

[0065] for Figure 3The data exchange system shown can be a chip, a die, or a high-speed interconnect between dies, such as the Universal Chip Interconnect Express (UCIe). The links between devices can be NVLink, PCIe, or Unified Bus (UB). Alternatively, the devices can be servers, nodes can be accelerators, and the links between nodes can be NVLink, PCIe, or UB. The links between devices can also be Ethernet, infiniteband (IB), or fiber optic links. The accelerators can be any of the following processors: Graphics Processing Unit (GPU), Neural-Network Processing Unit (NPU), Tensor Processing Unit (TPU), or Deep Learning Processing Unit (DPU).

[0066] The data exchange method provided in this application will be described in detail below with reference to the accompanying drawings. Figure 4 This is a flowchart illustrating a data exchange method provided in this application, which includes steps S401 to S402.

[0067] S401. The second node sends multiple copies of the first data to the first node in sequence.

[0068] After each node in the data exchange system completes the calculation and obtains the calculation result, it needs to exchange data through aggregate communication in order to synchronize the calculation result among the nodes. For any node, when it needs to send the calculation result to other nodes, the node can slice the calculation result to obtain multiple data, and then send these multiple data to other nodes in sequence.

[0069] In this application, if a second node needs to send a calculation result to a first node, the second node divides the calculation result into multiple first data portions and then sends these multiple first data portions to the first node sequentially through the communication link between the second node and the first node. The second node and the first node may be nodes within the same device or nodes within different devices. If the second node and the first node are nodes within the same device, then the communication link between the second node and the first node is a communication link between nodes in the first communication plane; if the second node and the first node are nodes in different devices, then the communication link between the second node and the first node is a communication link between devices in the second communication plane.

[0070] S402. The first node receives multiple sets of first data sent by the second node, and in the process of receiving multiple sets of first data, sends multiple sets of second data to the third node.

[0071] If the first node and the second node are two nodes within the same device, then the first node and the third node are two nodes in different devices; if the first node and the second node are two nodes in different devices, then the first node and the third node are two nodes within the same device.

[0072] The first node receives multiple sets of first data sent by the second node, and while receiving these sets of first data, it simultaneously sends multiple sets of second data to the third node. That is, the first node does not wait until it has received all the first data sent by the second node before sending the second data to the third node; rather, the first node sends the second data to the third node while receiving the first data. For example, if the multiple sets of second data sent by the first node to the third node are the same as the multiple sets of second data received by the first node, then the second node sends one set of first data to the first node, and then sends the next set of first data to the first node. The first node, after receiving one set of first data, sends the received set of second data to the third node, and after receiving the next set of first data, sends another set of second data to the third node. If the multiple sets of second data are different from the multiple sets of first data, then the second node sends the aforementioned multiple sets of first data to the first node sequentially, and the first node simultaneously sends the aforementioned multiple sets of second data to the third node sequentially.

[0073] The first node, second node, and third node mentioned above constitute a communication group. In this application, a communication group includes at least two nodes within the same device and one node from another device. For example, when the first node and second node are two nodes in different devices, the third node and first node are two nodes within the same device; when the first node and second node are two nodes within the same device, the third node and first node are two nodes in different devices. The multiple nodes included in the above data exchange system can be divided into multiple communication groups, and the data exchange process can be implemented between the multiple nodes included in each communication group.

[0074] In the aforementioned data exchange system, links between nodes in different devices constitute one communication plane, while links between nodes within the same device constitute another. When multiple nodes in the data exchange system need to exchange data, the aforementioned data exchange method allows for simultaneous data exchange between nodes on different devices and between nodes within the same device. Therefore, this method can utilize the bandwidth of both communication planes simultaneously, improving system bandwidth utilization, reducing system latency, and increasing system computational efficiency. Furthermore, by simultaneously utilizing the bandwidth of both communication planes, the traffic during the entire data exchange process can be evenly distributed across both planes, achieving efficient use of bandwidth and further improving system bandwidth utilization.

[0075] The following sections will introduce the concepts based on the above. Figure 4 The corresponding methods implement the data exchange process between nodes in a data exchange system in different ways. For example, through... Figure 4 The corresponding methods implement collection communication such as reduce_scatter, allgather, and allreduce. For ease of description, this application refers to the communication link between multiple nodes within the same device as the first link, and the communication link between nodes in different devices as the second link.

[0076] In one possible implementation, if the second node and the first node are nodes within the same device, and the third node and the first node are nodes in different devices, after completing the calculation and obtaining the result, multiple nodes in different devices and multiple nodes within the same device begin exchanging data. For the second node, the calculation result is sliced ​​into multiple first data segments, which are then sequentially sent to the first node via the first link. For the first node, while receiving multiple first data segments sent by the second node within the same device via the first link, multiple second data segments are sent to the third node via the second link. Specifically, after receiving a first data segment, the first node performs a reduce operation on this first data segment and a third data segment, using the result of the reduce operation as a second data segment, which is then sent to the third node in the different device. The third node, upon receiving the second data segment, also performs the same reduce operation. The reduce operation can be summation, finding the maximum or minimum value, etc., and this application does not specifically limit it. A third node in a different device refers to a node in a different device than the device where the second node is located.

[0077] The first node slices the calculation result to obtain multiple fourth data sets. While receiving multiple first data sets sent by the second node, the first node also sends these multiple fourth data sets to the second node. After receiving each fourth data set, the second node performs a reduction operation on one fourth data set and one fifth data set to obtain the reduction result. The reduction result is then sent to the fourth nodes in different devices via the second link. The fourth node and the third node are located on the same device.

[0078] The fourth node also slices the calculation result, obtaining multiple seventh data points. The fourth node sends these seventh data points to the third node. Each time the third node receives a seventh data point, it performs a reduction operation to obtain a sixth data point, and then sends this sixth data point to the first node. The first node performs a reduction operation on the received multiple sixth data points. The third node also slices the calculation result, obtaining multiple eighth data points, and sends these eighth data points to the fourth node. Each time the fourth node receives an eighth data point, it performs a reduction operation to obtain a ninth data point, and then sends this ninth data point to the second node.

[0079] The above method enables reduce_scatter communication based on the butterfly algorithm. See also... Figure 5 , Figure 5 This is a schematic diagram of an implementation of reduce_scatter provided in this application.Figure 5 The data exchange system in the system comprises four chips: chip1, chip2, chip3, and chip4. Each chip includes two dies: chip1 includes dies D0 and D1, chip2 includes dies D2 and D3, chip3 includes dies D4 and D5, and chip4 includes dies D6 and D7. The data exchange system contains a total of eight dies, therefore the data in each die is divided into eight parts. For example... Figure 5 As shown in Figure (a), the eight data points in D0 include a0 to a7, the eight data points in D1 include b0 to b7, the eight data points in D2 include c0 to c7, the eight data points in D3 include d0 to d7, the eight data points in D4 include e0 to e7, the eight data points in D5 include f0 to f7, the eight data points in D6 include g0 to g7, and the eight data points in D7 include h0 to h7. Based on the above... Figure 4 The corresponding method can achieve reduce_scatter in the following three steps.

[0080] It should be noted that in the above method, the communication mode between any three nodes in the first, second, third and fourth nodes is the same. In each case, one node receives data sent by another node in the same device, performs a reduction operation, and sends the result of the reduction operation to a node in another device. Therefore, the system dies will be divided into multiple communication groups of three in the following section to introduce the implementation process of reduce_scatter.

[0081] (1) Within the same communication group, two dies within the same chip exchange 2N of data with each other, and two dies in different chips exchange 2N of data with each other.

[0082] The eight dies are divided into eight communication groups, namely (D5, D4, D0), (D4, D5, D1), (D1, D0, D4), (D0, D1, D5), (D7, D6, D2), (D6, D7, D3), (D3, D2, D6) and (D2, D3, D7).

[0083] Taking the communication group (D5, D4, D0) as an example, the data exchange process of this communication group is described. In the communication group (D0, D4, D5), D4 is equivalent to the first node, D5 is equivalent to the second node, and D0 is equivalent to the third node. During the reduce_scatter process, D5 sends data f0 to D4 in multiple parts. After receiving a piece of data from D5, D4 performs a reduction operation on that data and a piece of data e0, and then sends the reduced data to D0. For example, if each die further slices the eight parts of data, dividing each part into three parts, then D5 divides data f0 into f00, f01, and f02, and D4 also divides data e0 into e00, e01, and e02.

[0084] D5 sends f00 to D4. After receiving f00, D4 reduces f00 and e00, and then sends the result to D0. After sending f00 to D4, D5 continues to send f01 to D4. After receiving f01, D4 reduces f01 and e01, and then sends the result to D0. After sending f01 to D4, D5 continues to send f02 to D4. After receiving f02, D4 reduces f02 and e02, and then sends the result to D0. After receiving data from D4 each time, or after receiving all the data from D4, D0 reduces the received data with a0 to obtain data a8. That is, a8 is the result of reducing f0, e0, and a0. After D5 sends data f0 to D4, D5 no longer retains data f0. D4 completes the reduction operation of data f0 and e0, and D4 no longer retains data e0 either.

[0085] D5 can also perform the same operation on another part of the data (any part of f1 to f7) as on data f0. For example, performing the above operation on f2 will give D0 the data a9, which is the result of reducing f2, e2 and a2.

[0086] In step (1), the three dies in each of the other communication groups can perform the operations performed by D5, D4 and D0 as described above. Specifically, for a communication group including (D4, D5, D1), D5 is equivalent to the first node, D4 ​​is equivalent to the second node, and D1 is equivalent to the third node. After the communication group performs step (1), D1 can obtain data b8 and b9.

[0087] For a communication group consisting of (D1, D0, D4), where D0 is equivalent to the first node, D1 is equivalent to the second node, and D4 is equivalent to the third node, after step (1) is executed, D4 can obtain data e8 and e9. For a communication group consisting of (D0, D1, D5), where D1 is equivalent to the first node, D0 is equivalent to the second node, and D5 is equivalent to the third node, after step (1) is executed, D5 can obtain data f8 and f9.

[0088] For a communication group consisting of (D7, D6, D2), where D6 is equivalent to the first node, D7 is equivalent to the second node, and D2 is equivalent to the third node, after step (1) is executed, D2 can obtain data c8 and c9. For a communication group consisting of (D6, D7, D3), where D7 is equivalent to the first node, D6 is equivalent to the second node, and D3 is equivalent to the third node, after step (1) is executed, D3 can obtain data d8 and d9.

[0089] For a communication group consisting of (D3, D2, D6), where D2 is equivalent to the first node, D3 is equivalent to the second node, and D6 is equivalent to the third node, after step (1) is executed, D6 can obtain data g8 and g9. For a communication group consisting of (D2, D3, D7), where D3 is equivalent to the first node, D2 is equivalent to the second node, and D7 is equivalent to the third node, after step (1) is executed, D7 can obtain data h8 and h9.

[0090] After each communication group performs step (1), the data included in each die is as follows: Figure 5 As shown in Figure (b), for example, D0 includes data a8, a1, a9, and a3; D4 includes data e8, e5, e9, and e7. If the data in each die is divided into eight parts, and the amount of data in each part is N, then during the execution of step (1), within a communication group, the one-way flow between two dies of the same chip is 2N. For example, the amount of data sent from D5 to D4 is 2N, and the amount of data sent from D4 to D5 is 2N. The one-way flow between two dies of different chips in the same communication group is also 2N. For example, the amount of data sent from D4 to D0 is 2N, and the amount of data sent from D0 to D4 is 2N. That is, within the same communication group, two dies within the same chip exchange 2N of data with each other, and two dies in different chips exchange 2N of data with each other.

[0091] (2) Within the same communication group, two dies within the same chip exchange N data with each other, and two dies in different chips exchange N data with each other.

[0092] The eight dies are further divided into eight communication groups, namely (D3, D2, D0), (D2, D3, D1), (D1, D0, D2), (D0, D1, D3), (D7, D6, D4), (D6, D7, D5), (D5, D4, D6) and (D4, D5, D7).

[0093] Taking the communication group (D3, D2, D0) as an example, the data exchange process of the communication group in step (2) is introduced. In the communication group (D3, D2, D0), D2 is equivalent to the first node mentioned above, D3 is equivalent to the second node mentioned above, and D0 is equivalent to the third node mentioned above. D3 sends data d0 to D2 in multiple parts. After receiving a piece of data sent by D3, D2 performs a reduction operation on the data and a piece of data c8, and then D2 sends the reduced data to D0. For example, D3 divides data d0 into d00, d01 and d02, and D2 also divides data c8 into c80, c81 and c82.

[0094] D3 sends d00 to D2. After receiving d00, D2 performs a reduction operation on d00 and c80, and then sends the result to D0. After sending d00 to D2, D3 continues to send d01 to D2. After receiving d01, D4 performs a reduction operation on d01 and c81, and then sends the result to D0. After sending d01 to D2, D3 continues to send d02 to D2. After receiving d02, D2 performs a reduction operation on d02 and c82, and then sends the result to D0. After receiving data from D2 each time, or after receiving all the data sent by D2, D0 performs a reduction operation on the received data and a8 to obtain data a10. That is, a10 is the result of the reduction operation on d0, c8, and a8.

[0095] In step (2), the three dies in each of the other communication groups can perform the operations performed by D3, D2 and D0 as described above. Specifically, for a communication group including (D2, D3, D1), D3 is equivalent to the first node, D2 is equivalent to the second node, and D1 is equivalent to the third node. After the communication group performs step (1), D1 can obtain data b10.

[0096] For a communication group consisting of (D1, D0, D2), where D0 is equivalent to the first node, D1 is equivalent to the second node, and D2 is equivalent to the third node, after step (2) is executed, D2 can obtain data c10. For a communication group consisting of (D0, D1, D3), where D1 is equivalent to the first node, D0 is equivalent to the second node, and D3 is equivalent to the third node, after step (1) is executed, D3 can obtain data d10.

[0097] For a communication group consisting of (D7, D6, D4), where D6 is equivalent to the first node, D7 is equivalent to the second node, and D4 is equivalent to the third node, after step (2) is executed, D4 can obtain data e10. For a communication group consisting of (D6, D7, D5), where D7 is equivalent to the first node, D6 is equivalent to the second node, and D5 is equivalent to the third node, after step (1) is executed, D5 can obtain data f10.

[0098] For a communication group consisting of (D5, D4, D6), where D4 is equivalent to the first node, D5 is equivalent to the second node, and D6 is equivalent to the third node, after step (2) is executed, D6 can obtain data g10. For a communication group consisting of (D4, D5, D7), where D5 is equivalent to the first node, D4 ​​is equivalent to the second node, and D7 is equivalent to the third node, after step (2) is executed, D7 can obtain data h10.

[0099] After each communication group performs step (2), the data included in each die is as follows: Figure 5 As shown in Figure (c), D0 includes data a10 and a1, D1 includes data b0 and b10, D2 includes data c10 and c3, D3 includes data d2 and d10, D4 includes data e10 and e5, D5 includes data f4 and f10, D6 includes data g10 and g7, and D7 includes data h9 and h10. During step (2), the unidirectional flow between two dies of the same chip within a communication group is N. For example, the amount of data sent from D3 to D2 is N, and the amount of data sent from D2 to D3 is N. Similarly, the unidirectional flow between two dies of different chips within the same communication group is also N. For example, the amount of data sent from D2 to D0 is N, and the amount of data sent from D0 to D2 is N. That is, within the same communication group, two dies within the same chip exchange N amounts of data, and two dies in different chips exchange N amounts of data.

[0100] (3) The amount of data N is exchanged between two dies within the same chip.

[0101] Data is exchanged between dies within each chip. For D0 and D1 in chip1, D0 sends a1 to D1, and D1 reduces a1 and b10 to obtain data b; D1 sends b0 to D0, and D0 reduces a10 and b10 to obtain data a.

[0102] For D2 and D3 in chip2, D2 sends c3 to D3, and D3 reduces d10 and c3 to obtain data d; D3 sends d2 to D2, and D2 reduces c10 and d2 to obtain data c.

[0103] For D4 and D5 in chip3, D4 sends e5 to D5, and D5 reduces f10 and e5 to obtain data f; D5 sends f4 to D4, and D4 reduces e10 and f4 to obtain data e.

[0104] For D6 and D7 in chip4, D6 sends g7 to D7, and D7 reduces h10 and g7 to obtain data h; D7 sends h9 to D6, and D6 reduces g10 and h9 to obtain data g.

[0105] After each die performs step (3), the data contained in each die is as follows: Figure 5 As shown in Figure (d). During the execution of step (3), the flow rate between the two dies of the same chip is N. For example, the amount of data sent from D3 to D2 is N, and the amount of data sent from D2 to D3 is N.

[0106] In the process of implementing reduce_scatter, each step (1) and (2) utilizes the bandwidth of two communication planes simultaneously, and the amount of data passing through the two communication planes is the same. Therefore, the above method can make reasonable use of the bandwidth resources of the two communication planes, improve communication efficiency, and thus improve the efficiency of distributed computing.

[0107] In one possible implementation, if the second node and the first node are nodes in different devices, and the third node and the first node are nodes in the same device, after the nodes in each device complete the calculation and obtain the results, multiple nodes in different devices and multiple nodes in the same device begin to exchange data. For the second node, the second node slices the calculation result to obtain multiple pieces of first data, and sends these multiple pieces of first data to the first node sequentially through the second link. For the first node, the first node receives the multiple pieces of first data sent by the second node from different devices through the second link, and in the process of receiving the multiple pieces of first data sent by the second node, the first node sends multiple pieces of second data to the third node in the same device through the first link. Here, the second node from a different device means that the second node is a node in a different device than the device where the first node is located.

[0108] Optionally, the aforementioned multiple sets of second data are the same as the multiple sets of first data. After receiving a set of first data sent by the second node through the second link, the first node saves the set of first data, copies the set of first data, and sends the set of first data to the third node through the first link.

[0109] Because the bandwidth of the communication link between nodes within the same device is greater than the bandwidth of the communication link between devices, that is, the bandwidth of the first link is greater than the bandwidth of the second link, the first node, while receiving multiple sets of first data sent by the second node, can send multiple sets of fifth data to the third node in addition to sending the received sets of first data. For example, after receiving a set of first data and sending it to the third node, the first node can send a set of fifth data, and then send the next set of received first data to the third node. These multiple sets of fifth data are data that already existed in the first node before it received the multiple sets of first data; for example, the multiple sets of second data are the calculation results of the first node. This allows for full utilization of the bandwidth between nodes within the same device, improving data exchange efficiency.

[0110] Optionally, the aforementioned multiple sets of second data may also be different from the multiple sets of first data. For example, the multiple sets of second data may be data that already existed in the first node before the first node received the multiple sets of data sent by the second node.

[0111] It should be understood that if the first device containing the first node includes more than two nodes, then the first node sends these multiple copies of the second data to all other nodes within the same device.

[0112] The first node also slices the calculation results to obtain multiple sets of third data. While receiving multiple sets of first data, the first node also sends multiple sets of third data to the second node through the second link. While receiving multiple sets of third data, the second node sends these multiple sets of fourth data to the fourth node in the same device through the first link.

[0113] The aforementioned multiple sets of fourth data are the same as the multiple sets of third data. After the second node receives a set of third data sent by the first node through the second link, it saves the set of third data, copies the set of third data, and sends the set of third data to the fourth node through the first link. The aforementioned multiple sets of fourth data may also be different from the multiple sets of third data. For example, the multiple sets of fourth data may be data that already exists in the second node before the second node receives the multiple sets of data sent by the first node.

[0114] The above method enables allgather communication based on the butterfly algorithm. See also Figure 6 , Figure 6 This is a schematic diagram of an implementation of allgather provided in this application. The data in each die is as follows: Figure 6 As shown in Figure (a), for example, the data in D0 is 'a', and the data in D1 is 'b'. Based on the above... Figure 4 The corresponding method can achieve allgather in the following three steps. It should be noted that in the above method, the first, second, and third nodes are grouped into one communication group, and the first, second, and fourth nodes are grouped into another. The data exchange method between these two communication groups is the same: one node receives data sent by a node in another device and sends a copy of the data to other nodes in the same device. Therefore, the following describes the implementation process of allgather based on the butterfly algorithm, dividing the system's dies into multiple communication groups of three.

[0115] (1) Within the same communication group, two dies within the same chip exchange N data with each other, and two dies in different chips exchange N data with each other.

[0116] The eight dies are further divided into eight communication groups, namely (D0, D2, D3), (D1, D3, D2), (D2, D0, D1), (D3, D1, D0), (D4, D6, D7), (D5, D7, D6), (D6, D4, D5) and (D7, D5, D4).

[0117] Taking the communication group (D0, D2, D3) as an example, the data exchange process in step (1) of the communication group is introduced. In the communication group (D0, D2, D3), D2 is equivalent to the first node mentioned above, D0 is equivalent to the second node mentioned above, and D3 is equivalent to the third node mentioned above. D0 slices data a to obtain multiple data, and then sends the multiple data to D2 in sequence. After receiving the data sent by D0, D2 sends the data to D3. After receiving the data sent by D0, D2 sends the received data to D3. After D0 sends the multiple data to D2 and D2 receives the multiple data sent by D0, D2 obtains data a; D2 sends all of these multiple data to D3, and after D3 receives these multiple data, D3 also obtains data a.

[0118] In step (1), the three dies in each of the other communication groups can perform the operations described above by D0, D2, and D3. Specifically, for a communication group including (D1, D3, D2), D3 is equivalent to the first node, D1 is equivalent to the second node, and D2 is equivalent to the third node. D1 divides the data b into multiple parts and sends them to D3. After receiving each part of the data, D3 also sends the received data to D2. Finally, both D2 and D3 receive the data b.

[0119] For a communication group consisting of (D2, D0, D1), D0 is equivalent to the first node mentioned above, D2 is equivalent to the second node mentioned above, and D1 is equivalent to the third node mentioned above. D2 divides the data c into multiple parts and sends them to D0. After receiving each part of the data, D0 also sends the received data to D1. Eventually, both D0 and D1 receive the data c.

[0120] For a communication group consisting of (D3, D1, D0), D1 is equivalent to the first node mentioned above, D3 is equivalent to the second node mentioned above, and D0 is equivalent to the third node mentioned above. D3 divides the data d into multiple parts and sends them to D1. After receiving each part of the data, D1 also sends the received data to D0. Eventually, both D0 and D1 receive the data d.

[0121] For a communication group consisting of (D4, D6, D7), D6 is equivalent to the first node mentioned above, D4 ​​is equivalent to the second node mentioned above, and D7 is equivalent to the third node mentioned above. D4 divides the data e into multiple parts and sends them to D6. After receiving each part of the data, D6 also sends the received data to D7. Eventually, both D6 and D7 receive the data e.

[0122] For a communication group consisting of (D5, D7, D6), D7 is equivalent to the first node mentioned above, D5 is equivalent to the second node mentioned above, and D6 is equivalent to the third node mentioned above. D5 divides the data f into multiple parts and sends them to D7. After receiving each part of the data, D7 also sends the received data to D6. Eventually, both D6 and D7 receive the data f.

[0123] For a communication group consisting of (D6, D4, D5), D4 is equivalent to the first node mentioned above, D6 is equivalent to the second node mentioned above, and D5 is equivalent to the third node mentioned above. D6 divides the data g into multiple parts and sends them to D4. After receiving each part of the data, D4 also sends the received data to D5. Eventually, both D4 and D5 receive the data g.

[0124] For a communication group consisting of (D7, D5, D4), D5 is equivalent to the first node mentioned above, D7 is equivalent to the second node mentioned above, and D4 is equivalent to the third node mentioned above. D7 divides the data h into multiple parts and sends them to D5. After receiving each part of the data, D5 also sends the received data to D4. Eventually, both D4 and D5 receive the data h.

[0125] After each communication group performs step (1), the data included in each die is as follows: Figure 6 As shown in Figure (b), for example, D0 includes data a, c, and d, and D1 includes data b, c, and d. During the execution of step (1), the unidirectional flow between two dies of different chips within the same communication group is also N. For example, the amount of data sent from D0 to D2 is N, and the amount of data sent from D2 to D0 is N. The unidirectional flow between two dies of the same chip within a communication group is N. For example, the amount of data sent from D2 to D3 is N, and the amount of data sent from D3 to D2 is N. That is, within the same communication group, two dies within the same chip exchange N amounts of data with each other, and two dies in different chips exchange N amounts of data with each other.

[0126] (2) Within the same communication group, two dies in the same chip exchange 2N of data with each other, and two dies in different chips exchange 2N of data with each other.

[0127] The eight dies are further divided into eight communication groups, namely (D0, D4, D5), (D1, D5, D4), (D2, D6, D7), (D3, D7, D6), (D4, D0, D1), (D5, D1, D0), (D6, D2, D3) and (D7, D3, D2).

[0128] For a communication group consisting of (D0, D4, D5), D4 is equivalent to the first node mentioned above, D0 is equivalent to the second node mentioned above, and D5 is equivalent to the third node mentioned above. D0 divides data a and data c into multiple parts and sends them to D4. After receiving each part of the data, D4 also sends the received data to D5. In the end, both D4 and D5 receive data a and data c.

[0129] For a communication group consisting of (D1, D5, D4), D5 is equivalent to the first node mentioned above, D1 is equivalent to the second node mentioned above, and D4 is equivalent to the third node mentioned above. D1 divides data b and data d into multiple parts and sends them to D5. After receiving each part of the data, D5 also sends the received data to D4. In the end, both D4 and D5 receive data b and data d.

[0130] For a communication group consisting of (D2, D6, D7), D6 is equivalent to the first node mentioned above, D2 is equivalent to the second node mentioned above, and D7 is equivalent to the third node mentioned above. D2 divides data a and data c into multiple parts and sends them to D6. After receiving each part of the data, D6 also sends the received data to D7. In the end, both D6 and D7 receive data a and data c.

[0131] For a communication group consisting of (D3, D7, D6), D7 is equivalent to the first node mentioned above, D3 is equivalent to the second node mentioned above, and D6 is equivalent to the third node mentioned above. D3 divides data b and data d into multiple parts and sends them to D7. After receiving each part of the data, D7 also sends the received data to D6. In the end, both D6 and D7 receive data b and data d.

[0132] For a communication group consisting of (D4, D0, D1), D0 is equivalent to the first node mentioned above, D4 ​​is equivalent to the second node mentioned above, and D1 is equivalent to the third node mentioned above. D4 divides data e and data g into multiple parts and sends them to D0. After receiving each part of the data, D0 also sends the received data to D1. In the end, both D0 and D1 receive data e and data g.

[0133] For a communication group consisting of (D5, D1, D0), D1 is equivalent to the first node mentioned above, D5 is equivalent to the second node mentioned above, and D0 is equivalent to the third node mentioned above. D5 divides data f and data h into multiple parts and sends them to D1. After receiving each part of the data, D1 also sends the received data to D0. In the end, both D0 and D1 receive data f and data h.

[0134] For a communication group consisting of (D6, D2, D3), D2 is equivalent to the first node mentioned above, D6 is equivalent to the second node mentioned above, and D3 is equivalent to the third node mentioned above. D6 divides data e and data g into multiple parts and sends them to D2. After receiving each part of the data, D2 also sends the received data to D3. In the end, both D2 and D3 receive data e and data g.

[0135] For a communication group consisting of (D7, D3, D2), D3 is equivalent to the first node mentioned above, D7 is equivalent to the second node mentioned above, and D2 is equivalent to the third node mentioned above. D7 divides data f and data h into multiple parts and sends them to D3. After receiving each part of the data, D3 also sends the received data to D2. In the end, both D2 and D3 receive data f and data h.

[0136] After each communication group performs step (2), the data included in each die is as follows: Figure 6 As shown in Figure (c), for example, D0 includes data a, c, d, e, f, g, and h. During the execution of step (2), the one-way flow between two dies of different chips within the same communication group is also 2N. For example, the amount of data sent from D0 to D4 is 2N, and the amount of data sent from D4 to D0 is 2N. The one-way flow between two dies of the same chip within a communication group is 2N. For example, the amount of data sent from D4 to D5 is 2N, and the amount of data sent from D5 to D4 is 2N. That is, within the same communication group, two dies within the same chip exchange 2N of data with each other, and two dies in different chips exchange 2N of data with each other.

[0137] (3) The amount of data N is exchanged between two dies within the same chip.

[0138] Data is exchanged between dies within each chip. For D0 and D1 in chip 1, D0 sends 'a' to D1, and D1 sends 'b' to D0; for D2 and D3 in chip 2, D2 sends 'c' to D3, and D3 sends 'd' to D2; for D4 and D5 in chip 3, D4 sends 'e' to D5, and D5 sends 'f' to D4; for D6 and D7 in chip 4, D6 sends 'g' to D7, and D7 sends 'h' to D6.

[0139] After each die performs step (3), the data contained in each die is as follows: Figure 6 As shown in Figure (d). During the execution of step (3), the flow rate between the two dies of the same chip is N. For example, the amount of data sent from D3 to D2 is N, and the amount of data sent from D2 to D3 is N.

[0140] In the process of implementing allgather, each step (1) and (2) utilizes the bandwidth of two communication planes simultaneously, and the amount of data transmitted through the two communication planes is the same. Therefore, the above method can make reasonable use of the bandwidth resources of the two communication planes, improve communication efficiency, and thus improve the efficiency of distributed computing.

[0141] It should be noted that in step (3) above, the data exchanged between the two nodes within the same chip is data that already existed before each node performed step (1). Therefore, the data exchanged between the two nodes in step (3) can be completed in the first two steps. For example, for the data c sent by D4 to D5 in step (3), D0 divides data a into multiple parts in step (1) and sends them to D4 sequentially. D0 sends one part of the data to D4, and D4 receives this part of the data and sends it to D5. Since the bandwidth of the link between nodes within the same chip is greater than the bandwidth of the link between different chips, D4 can send a part of data e to D5 before receiving the next part of the data sent by D0, and then send the next part of the data sent by D0. Through the above method, the data exchanged in step (3) can be completed in the first two steps, improving communication efficiency.

[0142] In one possible implementation, if the second node and the first node are nodes in different devices, and the third node and the first node are nodes in the same device, after the nodes in each device complete the calculation and obtain the results, multiple nodes in different devices and multiple nodes in the same device begin to exchange data. For the second node, the second node slices the calculation result to obtain multiple pieces of first data, and sends these multiple pieces of first data to the first node sequentially through the second link. For the first node, the first node receives the multiple pieces of first data sent by the second node from different devices through the second link, and in the process of receiving the multiple pieces of first data sent by the second node, the first node sends multiple pieces of second data to the third node in the same device through the first link. Here, the second node from a different device means that the second node is a node in a different device than the device where the first node is located.

[0143] Optionally, the aforementioned multiple sets of second data are the same as the multiple sets of first data. After receiving a set of first data sent by the second node through the second link, the first node saves the set of first data, copies the set of first data, and sends the set of first data to the third node through the first link.

[0144] Because the bandwidth of the communication link between nodes within the same device is greater than the bandwidth of the communication link between devices, that is, the bandwidth of the first link is greater than the bandwidth of the second link, the first node, while receiving multiple sets of first data sent by the second node, can send multiple sets of fifth data to the third node in addition to sending the received sets of first data. For example, after receiving a set of first data and sending it to the third node, the first node can send a set of fifth data, and then send the next set of received first data to the third node. These multiple sets of fifth data are data that already existed in the first node before it received the multiple sets of first data; for example, the multiple sets of second data are the calculation results of the first node. This allows for full utilization of the bandwidth between nodes within the same device, improving data exchange efficiency.

[0145] Optionally, the aforementioned multiple sets of second data may also be different from the multiple sets of first data. For example, the multiple sets of second data may be data that already existed in the first node before the first node received the multiple sets of data sent by the second node.

[0146] It should be understood that if the first device containing the first node includes more than two nodes, then the first node sends these multiple copies of the second data to all other nodes within the same device.

[0147] The first node also slices the calculation results to obtain multiple sets of third data. While receiving multiple sets of first data, the first node also sends multiple sets of third data to the fourth node in different devices through the second link. While receiving multiple sets of third data, the fourth node sends these multiple sets of fourth data to the fifth node in the same device through the first link.

[0148] The aforementioned multiple sets of fourth data are the same as the multiple sets of third data. After the fourth node receives a set of third data sent by the first node through the second link, it saves the set of third data, copies the set of third data, and sends the set of third data to the fifth node through the first link. The aforementioned multiple sets of fourth data are different from the multiple sets of third data. For example, the multiple sets of fourth data are data that already exists in the fourth node before the fourth node receives the multiple sets of data sent by the first node.

[0149] The above method enables allgather communication based on a ring algorithm. See also Figure 7 , Figure 7 This is a schematic diagram of another implementation of allgather provided in this application. The data in each die is as follows: Figure 7As shown in Figure (a), for example, data 'a' in D0 and data 'b' in D1. The eight dies are further divided into eight communication groups: (D0, D2, D3), (D1, D3, D2), (D2, D4, D5), (D3, D5, D4), (D4, D6, D7), (D5, D7, D6), (D6, D0, D1), and (D7, D1, D0). Based on the above... Figure 4 The corresponding method can achieve allgather in the following four steps. It should be noted that in the above method, the first, second, and third nodes are grouped into one communication group, and the first, fourth, and fifth nodes are grouped into another. The data exchange method between these two communication groups is the same: one node receives data sent by a node in another device and sends a copy of the data to other nodes in the same device. Therefore, the following describes the implementation process of allgather based on the ring algorithm, dividing the system's dies into multiple communication groups of three.

[0150] (1) Within the same communication group, one die in different chips sends N data to another die, and one die in the same chip sends N data to another die.

[0151] In a communication group consisting of (D0, D2, D3), D2 is equivalent to the first node mentioned above, D0 is equivalent to the second node mentioned above, and D3 is equivalent to the third node mentioned above. D0 divides the data 'a' into multiple parts and sends them to D2. After receiving each part of the data, D2 also sends the received data to D3. Eventually, both D2 and D3 receive the data 'a'.

[0152] For a communication group consisting of (D1, D3, D2), D3 is equivalent to the first node mentioned above, D1 is equivalent to the second node mentioned above, and D2 is equivalent to the third node mentioned above. D1 divides the data b into multiple parts and sends them to D3. After receiving each part of the data, D3 also sends the received data to D2. Eventually, both D2 and D3 receive the data b.

[0153] For a communication group consisting of (D2, D4, D5), D4 is equivalent to the first node mentioned above, D2 is equivalent to the second node mentioned above, and D5 is equivalent to the third node mentioned above. D2 divides the data c into multiple parts and sends them to D4. After receiving each part of the data, D4 also sends the received data to D5. Eventually, both D4 and D5 receive the data c.

[0154] For a communication group consisting of (D3, D5, D4), D5 is equivalent to the first node mentioned above, D3 is equivalent to the second node mentioned above, and D4 is equivalent to the third node mentioned above. D3 divides the data d into multiple parts and sends them to D5. After receiving each part of the data, D5 also sends the received data to D4. Eventually, both D4 and D5 receive the data d.

[0155] For a communication group consisting of (D4, D6, D7), D6 is equivalent to the first node mentioned above, D4 ​​is equivalent to the second node mentioned above, and D7 is equivalent to the third node mentioned above. D4 divides the data e into multiple parts and sends them to D6. After receiving each part of the data, D6 also sends the received data to D7. Eventually, both D6 and D7 receive the data e.

[0156] For a communication group consisting of (D5, D7, D6), D7 is equivalent to the first node mentioned above, D5 is equivalent to the second node mentioned above, and D6 is equivalent to the third node mentioned above. D5 divides the data f into multiple parts and sends them to D7. After receiving each part of the data, D7 also sends the received data to D6. Eventually, both D6 and D7 receive the data f.

[0157] For a communication group consisting of (D6, D0, D1), D0 is equivalent to the first node mentioned above, D6 is equivalent to the second node mentioned above, and D1 is equivalent to the third node mentioned above. D6 divides the data g into multiple parts and sends them to D0. After receiving each part of the data, D0 also sends the received data to D1. Eventually, both D0 and D1 receive the data g.

[0158] For a communication group consisting of (D7, D1, D0), D1 is equivalent to the first node mentioned above, D7 is equivalent to the second node mentioned above, and D0 is equivalent to the third node mentioned above. D7 divides the data h into multiple parts and sends them to D1. After receiving each part of the data, D1 also sends the received data to D0. Eventually, both D0 and D1 receive the data h.

[0159] After each communication group performs step (1), the data included in each die is as follows: Figure 7 As shown in Figure (b), for example, D0 includes data a, g, and h, and D1 includes data b, g, and h. During the execution of step (1), the unidirectional flow between two dies of different chips within the same communication group is also N. For example, the amount of data sent from D0 to D2 is N, and the amount of data sent from D2 to D0 is N. The unidirectional flow between two dies of the same chip within a communication group is N. For example, the amount of data sent from D2 to D3 is N, and the amount of data sent from D3 to D2 is N.

[0160] (2) Within the same communication group, one die in different chips sends N data to another die, and one die within the same chip sends N data to another die. For a communication group including (D2, D4, D5), D4 is equivalent to the first node mentioned above, D2 is equivalent to the second node mentioned above, and D5 is equivalent to the third node mentioned above. D2 divides the data 'a' into multiple parts and sends them to D4. After receiving one part of the data, D4 also sends the received data to D5. Finally, both D4 and D5 receive the data 'a'.

[0161] For a communication group consisting of (D3, D5, D4), D5 is equivalent to the first node mentioned above, D3 is equivalent to the second node mentioned above, and D4 is equivalent to the third node mentioned above. D3 divides the data b into multiple parts and sends them to D5. After receiving each part of the data, D5 also sends the received data to D4. Eventually, both D4 and D5 receive the data b.

[0162] For a communication group consisting of (D4, D6, D7), D6 is equivalent to the first node mentioned above, D4 ​​is equivalent to the second node mentioned above, and D7 is equivalent to the third node mentioned above. D4 divides the data c into multiple parts and sends them to D6. After receiving each part of the data, D6 also sends the received data to D7. Eventually, both D6 and D7 receive the data c.

[0163] For a communication group consisting of (D5, D7, D6), D7 is equivalent to the first node mentioned above, D5 is equivalent to the second node mentioned above, and D6 is equivalent to the third node mentioned above. D5 divides the data d into multiple parts and sends them to D7. After receiving each part of the data, D7 also sends the received data to D6. Eventually, both D6 and D7 receive the data d.

[0164] For a communication group consisting of (D6, D0, D1), D0 is equivalent to the first node mentioned above, D6 is equivalent to the second node mentioned above, and D1 is equivalent to the third node mentioned above. D6 divides the data e into multiple parts and sends them to D0. After receiving each part of the data, D0 also sends the received data to D1. Eventually, both D0 and D1 receive the data e.

[0165] For a communication group consisting of D7, D1, and D0, D1 is equivalent to the first node mentioned above, D7 is equivalent to the second node mentioned above, and D0 is equivalent to the third node mentioned above. D7 divides the data f into multiple parts and sends them to D1. After receiving each part of the data, D1 also sends the received data to D0. Eventually, both D0 and D1 receive the data f.

[0166] In a communication group consisting of (D0, D2, D3), D2 is equivalent to the first node mentioned above, D0 is equivalent to the second node mentioned above, and D3 is equivalent to the third node mentioned above. D0 divides the data g into multiple parts and sends them to D2. After receiving each part of the data, D2 also sends the received data to D3. Eventually, both D2 and D3 receive the data g.

[0167] For a communication group consisting of (D1, D3, D2), D3 is equivalent to the first node mentioned above, D1 is equivalent to the second node mentioned above, and D2 is equivalent to the third node mentioned above. D1 divides the data h into multiple parts and sends them to D3. After receiving each part of the data, D3 also sends the received data to D2. Eventually, both D2 and D3 receive the data h.

[0168] After each communication group performs step (2), the data included in each die is as follows: Figure 7 As shown in Figure (c), for example, D0 includes data a, e, f, g, and h, and D1 includes data b, e, f, g, and h. During the execution of step (2), the unidirectional flow between two dies of different chips within the same communication group is also N. For example, the amount of data sent from D0 to D2 is N, and the amount of data sent from D2 to D0 is N. The unidirectional flow between two dies of the same chip within a communication group is N. For example, the amount of data sent from D2 to D3 is N, and the amount of data sent from D3 to D2 is N.

[0169] (3) Within the same communication group, one die in different chips sends N data to another die, and one die in the same chip sends N data to another die.

[0170] For a communication group consisting of (D4, D6, D7), D6 is equivalent to the first node mentioned above, D4 ​​is equivalent to the second node mentioned above, and D7 is equivalent to the third node mentioned above. D4 divides the data 'a' into multiple parts and sends them to D6. After receiving each part of the data, D6 also sends the received data to D7. Eventually, both D6 and D7 receive the data 'a'.

[0171] For a communication group consisting of (D5, D7, D6), D7 is equivalent to the first node mentioned above, D5 is equivalent to the second node mentioned above, and D6 is equivalent to the third node mentioned above. D5 divides the data b into multiple parts and sends them to D7. After receiving each part of the data, D7 also sends the received data to D6. Eventually, both D6 and D7 receive the data b.

[0172] For a communication group consisting of (D6, D0, D1), D0 is equivalent to the first node mentioned above, D6 is equivalent to the second node mentioned above, and D1 is equivalent to the third node mentioned above. D6 divides the data c into multiple parts and sends them to D0. After receiving each part of the data, D0 also sends the received data to D1. Eventually, both D0 and D1 receive the data c.

[0173] For a communication group consisting of D7, D1, and D0, D1 is equivalent to the first node mentioned above, D7 is equivalent to the second node mentioned above, and D0 is equivalent to the third node mentioned above. D7 divides the data d into multiple parts and sends them to D1. After receiving each part of the data, D1 also sends the received data to D0. Eventually, both D0 and D1 receive the data d.

[0174] In a communication group consisting of (D0, D2, D3), D2 is equivalent to the first node mentioned above, D0 is equivalent to the second node mentioned above, and D3 is equivalent to the third node mentioned above. D0 divides the data e into multiple parts and sends them to D2. After receiving each part of the data, D2 also sends the received data to D3. Eventually, both D2 and D3 receive the data e.

[0175] For a communication group consisting of (D1, D3, D2), D3 is equivalent to the first node mentioned above, D1 is equivalent to the second node mentioned above, and D2 is equivalent to the third node mentioned above. D1 divides the data f into multiple parts and sends them to D3. After receiving each part of the data, D3 also sends the received data to D2. Eventually, both D2 and D3 receive the data f.

[0176] After each communication group performs step (3), the data included in each die is as follows: Figure 7 As shown in Figure (d), for example, D0 includes data a, c, d, e, f, g, and h, while D1 includes data b, c, d, e, f, g, and h. During the execution of step (3), the unidirectional flow between two dies of different chips within the same communication group is also N. For example, the amount of data sent from D0 to D2 is N, and the amount of data sent from D2 to D0 is N. The unidirectional flow between two dies of the same chip within a communication group is N. For example, the amount of data sent from D2 to D3 is N, and the amount of data sent from D3 to D2 is N.

[0177] (4) The amount of data N is exchanged between two dies within the same chip.

[0178] Data is exchanged between dies within each chip. For D0 and D1 in chip 1, D0 sends 'a' to D1, and D1 sends 'b' to D0; for D2 and D3 in chip 2, D2 sends 'c' to D3, and D3 sends 'd' to D2; for D4 and D5 in chip 3, D4 sends 'e' to D5, and D5 sends 'f' to D4; for D6 and D7 in chip 4, D6 sends 'g' to D7, and D7 sends 'h' to D6.

[0179] After each die performs step (4), the data contained in each die is as follows: Figure 7 As shown in Figure (e). During the execution of step (4), the flow rate between the two dies of the same chip is N. For example, the amount of data sent from D3 to D2 is N, and the amount of data sent from D2 to D3 is N.

[0180] In the process of implementing allgather, each step (1) to (3) utilizes the bandwidth of two communication planes simultaneously, and the amount of data passing through the two communication planes is the same. Therefore, the above method can make reasonable use of the bandwidth resources of the two communication planes, improve communication efficiency, and thus improve the efficiency of distributed computing.

[0181] Step (4) above and Figure 6 In the corresponding embodiment, step (3) is the same, where nodes within the same chip exchange data that existed before step (1) is executed. Therefore, the data exchanged in step (4) can also be completed in the first 3 steps, which will not be elaborated here.

[0182] The above Figure 6 and Figure 7 The examples all illustrate how to implement allgather using a chip containing two dies. If a chip contains more than two dies, when a die within a chip receives data from another chip, since the dies within a chip are fully interconnected, that die can broadcast the data to other dies within the same chip.

[0183] For example, see Figure 8 , Figure 8 This is a schematic diagram of another implementation of AllGather provided in this application. The data exchange system includes 3 chips, and each chip includes 4 dies. Chip 1 includes D0 to D3, chip 2 includes D4 to D7, and chip 3 includes D8 to D11. The raw data in each die is as follows: Figure 8As shown in Figure (a), for example, data a in D0 and data b in D1. The 12 dies are further divided into 12 communication groups: (D0, D4, D5, D6, D7), (D1, D5, D4, D6, D7), (D2, D6, D4, D5, D7), (D3, D7, D4, D5, D6), (D4, D8, D9, D10, D11), (D5, D9, D8, D10, D11), (D6, D10, D8, D9, D11), (D7, D11, D8, D9, D10), (D8, D0, D1, D2, D3), (D9, D1, D0, D2, D3), (D10, D2, D0, D1, D3), and (D11, D3, D0, D1, D2). Based on the above... Figure 4 The corresponding method can achieve allgather based on the ring algorithm through the following three steps.

[0184] (1) Within the same communication group, one die in different chips sends N data to another die, and one die in the same chip broadcasts N data to other dies.

[0185] In a communication group consisting of (D0, D4, D5, D6, D7), D4 is equivalent to the first node mentioned above, D0 is equivalent to the second node mentioned above, and D5, D6, and D7 are equivalent to the third node mentioned above. D0 divides the data 'a' into multiple parts and sends them to D4. After receiving each part of the data, D4 broadcasts the received data to D5, D6, and D7. Eventually, D4, D5, D6, and D7 all receive the data 'a'.

[0186] In the communication group (D1, D5, D4, D6, D7), D5 is equivalent to the first node mentioned above, D1 is equivalent to the second node mentioned above, and D4, D6, and D7 are equivalent to the third node mentioned above. D1 divides the data b into multiple parts and sends them to D5. After receiving each part of the data, D5 broadcasts the received data to D4, D6, and D7. Eventually, D4, D5, D6, and D7 all receive the data b.

[0187] In the communication group (D2, D6, D4, D5, D7), D6 is equivalent to the first node mentioned above, D2 is equivalent to the second node mentioned above, and D4, D5, and D7 are equivalent to the third node mentioned above. D2 divides the data c into multiple parts and sends them to D6. After receiving each part of the data, D6 broadcasts the received data to D4, D5, and D7. Eventually, D4, D5, D6, and D7 all receive the data b.

[0188] In a communication group consisting of (D3, D7, D4, D5, D6), D7 is equivalent to the first node mentioned above, D3 is equivalent to the second node mentioned above, and D4, D5, and D6 are equivalent to the third node mentioned above. D3 divides the data d into multiple parts and sends them to D7. After receiving each part of the data, D7 broadcasts the received data to D4, D5, and D6. Eventually, D4, D5, D6, and D7 all receive the data d.

[0189] In the communication group (D4, D8, D9, D10, D11), D8 is equivalent to the first node mentioned above, D4 ​​is equivalent to the second node mentioned above, and D9, D10, and D11 are equivalent to the third node mentioned above. D4 divides the data e into multiple parts and sends them to D8. After receiving each part of the data, D8 broadcasts the received data to D9, D10, and D11. Eventually, D8, D9, D10, and D11 all receive the data e.

[0190] In the communication group (D5, D9, D8, D10, D11), D9 is equivalent to the first node mentioned above, D5 is equivalent to the second node mentioned above, and D8, D10, and D11 are equivalent to the third node mentioned above. D5 divides the data f into multiple parts and sends them to D9. After receiving each part of the data, D9 broadcasts the received data to D8, D10, and D11. Eventually, D8, D9, D10, and D11 all receive the data f.

[0191] In the communication group (D6, D10, D8, D9, D11), D10 is equivalent to the first node mentioned above, D6 is equivalent to the second node mentioned above, and D8, D9, and D11 are equivalent to the third node mentioned above. D6 divides the data g into multiple parts and sends them to D10. After receiving each part of the data, D10 broadcasts the received data to D8, D9, and D11. Eventually, D8, D9, D10, and D11 all receive the data g.

[0192] In the communication group (D7, D11, D8, D9, D10), D11 is equivalent to the first node mentioned above, D7 is equivalent to the second node mentioned above, and D8, D9, and D10 are equivalent to the third node mentioned above. D7 divides the data h into multiple parts and sends them to D11. After receiving each part of the data, D11 broadcasts the received data to D8, D9, and D10. Eventually, D8, D9, D10, and D11 all receive the data h.

[0193] In a communication group consisting of (D8, D0, D1, D2, D3), D0 is equivalent to the first node mentioned above, D8 is equivalent to the second node mentioned above, and D1, D2, and D3 are equivalent to the third node mentioned above. D8 divides the data i into multiple parts and sends them to D0. After receiving each part of the data, D0 broadcasts the received data to D1, D2, and D3. Eventually, D0, D1, D2, and D3 all receive the data i.

[0194] In the communication group (D9, D1, D0, D2, D3), D1 is equivalent to the first node mentioned above, D9 is equivalent to the second node mentioned above, and D0, D2, and D3 are equivalent to the third node mentioned above. D8 divides the data j into multiple parts and sends them to D1. After receiving each part of the data, D1 broadcasts the received data to D0, D2, and D3. Eventually, D0, D1, D2, and D3 all receive the data j.

[0195] In a communication group consisting of (D10, D2, D0, D1, D3), D2 is equivalent to the first node mentioned above, D10 is equivalent to the second node mentioned above, and D0, D1, and D3 are equivalent to the third node mentioned above. D10 divides the data k into multiple parts and sends them to D2. After receiving each part of the data, D2 broadcasts the received data to D0, D1, and D3. Eventually, D0, D1, D2, and D3 all receive the data k.

[0196] In the communication group (D11, D3, D0, D1, D2), D3 is equivalent to the first node mentioned above, D11 is equivalent to the second node mentioned above, and D0, D1, and D2 are equivalent to the third node mentioned above. D11 divides the data m into multiple parts and sends them to D3. After receiving each part of the data, D3 broadcasts the received data to D0, D1, and D2. Eventually, D0, D1, D2, and D3 all receive the data k.

[0197] After each communication group performs step (1), the data included in each die is as follows: Figure 8 As shown in Figure (b), for example, D0 includes data a, i, j, k, and m, and D1 includes data a, i, j, k, and m. During the execution of step (1), the unidirectional flow between two dies of different chips within the same communication group is also N. For example, the amount of data sent from D0 to D4 is N, and the amount of data sent from D2 to D6 is N. The unidirectional flow between two dies of the same chip within a communication group is N. For example, the amount of data sent from D4 to D5, D6, and D7 is N.

[0198] (2) Within the same communication group, one die in different chips sends N data to another die, and one die in the same chip broadcasts N data to other dies.

[0199] In the communication group (D4, D8, D9, D10, D11), D8 is equivalent to the first node mentioned above, D4 ​​is equivalent to the second node mentioned above, and D9, D10, and D11 are equivalent to the third node mentioned above. D4 divides the data 'a' into multiple parts and sends them to D8. After receiving each part of the data, D8 broadcasts the received data to D9, D10, and D11. Eventually, D8, D9, D10, and D11 all receive the data 'a'.

[0200] In the communication group (D5, D9, D8, D10, D11), D9 is equivalent to the first node mentioned above, D5 is equivalent to the second node mentioned above, and D8, D10, and D11 are equivalent to the third node mentioned above. D5 divides the data b into multiple parts and sends them to D9. After receiving each part of the data, D9 broadcasts the received data to D8, D10, and D11. Eventually, D8, D9, D10, and D11 all receive the data b.

[0201] In the communication group (D6, D10, D8, D9, D11), D10 is equivalent to the first node mentioned above, D6 is equivalent to the second node mentioned above, and D8, D9, and D11 are equivalent to the third node mentioned above. D6 divides the data c into multiple parts and sends them to D10. After receiving each part of the data, D10 broadcasts the received data to D8, D9, and D11. Eventually, D8, D9, D10, and D11 all receive the data c.

[0202] In the communication group (D7, D11, D8, D9, D10), D11 is equivalent to the first node mentioned above, D7 is equivalent to the second node mentioned above, and D8, D9, and D10 are equivalent to the third node mentioned above. D7 divides the data d into multiple parts and sends them to D11. After receiving each part of the data, D11 broadcasts the received data to D8, D9, and D10. Eventually, D8, D9, D10, and D11 all receive the data d.

[0203] In a communication group consisting of (D8, D0, D1, D2, D3), D0 is equivalent to the first node mentioned above, D8 is equivalent to the second node mentioned above, and D1, D2, and D3 are equivalent to the third node mentioned above. D8 divides the data e into multiple parts and sends them to D0. After receiving each part of the data, D0 broadcasts the received data to D1, D2, and D3. Eventually, D0, D1, D2, and D3 all receive the data e.

[0204] In the communication group (D9, D1, D0, D2, D3), D1 is equivalent to the first node mentioned above, D9 is equivalent to the second node mentioned above, and D0, D2, and D3 are equivalent to the third node mentioned above. D8 divides the data f into multiple parts and sends them to D1. After receiving each part of the data, D1 broadcasts the received data to D0, D2, and D3. Eventually, D0, D1, D2, and D3 all receive the data f.

[0205] In a communication group consisting of (D10, D2, D0, D1, D3), D2 is equivalent to the first node mentioned above, D10 is equivalent to the second node mentioned above, and D0, D1, and D3 are equivalent to the third node mentioned above. D10 divides the data g into multiple parts and sends them to D2. After receiving each part of the data, D2 broadcasts the received data to D0, D1, and D3. Eventually, D0, D1, D2, and D3 all receive the data g.

[0206] In the communication group (D11, D3, D0, D1, D2), D3 is equivalent to the first node mentioned above, D11 is equivalent to the second node mentioned above, and D0, D1, and D2 are equivalent to the third node mentioned above. D11 divides the data h into multiple parts and sends them to D3. After receiving each part of the data, D3 broadcasts the received data to D0, D1, and D2. Eventually, D0, D1, D2, and D3 all receive the data h.

[0207] In a communication group consisting of (D0, D4, D5, D6, D7), D4 is equivalent to the first node mentioned above, D0 is equivalent to the second node mentioned above, and D5, D6, and D7 are equivalent to the third node mentioned above. D0 divides the data i into multiple parts and sends them to D4. After receiving each part of the data, D4 broadcasts the received data to D5, D6, and D7. Eventually, D4, D5, D6, and D7 all receive the data i.

[0208] In a communication group consisting of (D1, D5, D4, D6, D7), D5 is equivalent to the first node mentioned above, D1 is equivalent to the second node mentioned above, and D4, D6, and D7 are equivalent to the third node mentioned above. D1 divides the data j into multiple parts and sends them to D5. After receiving each part of the data, D5 broadcasts the received data to D4, D6, and D7. Eventually, D4, D5, D6, and D7 all receive the data j.

[0209] In the communication group (D2, D6, D4, D5, D7), D6 is equivalent to the first node mentioned above, D2 is equivalent to the second node mentioned above, and D4, D5, and D7 are equivalent to the third node mentioned above. D2 divides the data k into multiple parts and sends them to D6. After receiving each part of the data, D6 broadcasts the received data to D4, D5, and D7. Eventually, D4, D5, D6, and D7 all receive the data k.

[0210] In a communication group consisting of (D3, D7, D4, D5, D6), D7 is equivalent to the first node mentioned above, D3 is equivalent to the second node mentioned above, and D4, D5, and D6 are equivalent to the third node mentioned above. D3 divides the data m into multiple parts and sends them to D7. After receiving each part of the data, D7 broadcasts the received data to D4, D5, and D6. Eventually, D4, D5, D6, and D7 all receive the data m.

[0211] After each communication group performs step (2), the data included in each die is as follows: Figure 8 As shown in Figure (c), for example, D0 includes data a, e, f, g, h, i, j, k, and m; D1 includes data b, e, f, g, h, i, j, k, and m. During the execution of step (2), the unidirectional flow between two dies of different chips within the same communication group is also N. For example, the amount of data sent from D0 to D4 is N, and the amount of data sent from D2 to D6 is N. The unidirectional flow between two dies of the same chip within a communication group is N. For example, the amount of data sent from D4 to D5, D6, and D7 is N.

[0212] (3) Each die within the same chip broadcasts N data to other dies.

[0213] Data is exchanged between dies within each chip. Each die broadcasts its raw data to other dies within the same chip. For example, in chip 1, D0 broadcasts 'a' to D1, D2, and D3, and D1 broadcasts 'b' to D1, D2, and D3. In chip 2, D4 broadcasts 'e' to D5, D6, and D7.

[0214] After each die performs step (3), the data contained in each die is as follows: Figure 8 As shown in Figure (d). During the execution of step (3), the flow rate between the two dies of the same chip is N. For example, the amount of data sent from D3 to D2 is N, and the amount of data sent from D2 to D3 is N.

[0215] In the process of implementing allgather, each step (1) to (3) utilizes the bandwidth of two communication planes simultaneously, and the amount of data passing through the two communication planes is the same. Therefore, the above method can make reasonable use of the bandwidth resources of the two communication planes, improve communication efficiency, and thus improve the efficiency of distributed computing.

[0216] Step (3) above and Figure 6 In the corresponding embodiment, step (3) is the same, and the data that the nodes in the same chip interact with existed before step (1) was executed. Therefore, the data that the interaction in step (3) can also be completed in the first two steps, which will not be repeated here.

[0217] It should be understood that the all-gather communication can be decomposed into two processes: reduce_scatter and all-gather. Figures 5 to 8 This paper describes the process of implementing reduce_scatter and allgather using the method provided in this application. Therefore, the method provided in this application can achieve allreduce communication, for example, through... Figure 5 and Figure 6 The corresponding process implements AllRuduce communication, or through... Figure 5 and Figure 7 The corresponding process implements AllRuduce communication.

[0218] The above Figures 5 to 8 The process of implementing aggregate communication according to the data exchange method provided in this application is described. It should be understood that the data method provided in this application can also be applied to data exchange systems that include more devices and / or more nodes, which will not be elaborated here.

[0219] For the sake of simplicity, the above method embodiments are described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to this application. Other reasonable combinations of steps that those skilled in the art can conceive of based on the above description also fall within the scope of protection of this application. Again, those skilled in the art should be familiar with the fact that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to this application.

[0220] This application also provides a computing node, see [link to relevant documentation] Figure 9 , Figure 9 This is a schematic diagram of a computing node 900 provided in this application. The computing node 900 includes multiple processing circuits 910 and a communication interface 920. The computing node is as described above. Figure 3 The nodes in the data exchange system shown are, in essence, computing nodes, either dies in a chip or accelerator cards in a server. Multiple computing nodes can form a network such as... Figure 1 or Figure 3In the system shown, the various computing nodes 900 are connected via a communication interface 920. Multiple processing circuits 910 are used to perform calculations and obtain results. When the multiple computing nodes 900 complete their calculations and need to exchange data between them to synchronize the results, the processing circuits 910 perform calculations according to the above... Figures 4 to 8 Any of the data exchange methods described in the corresponding embodiments realizes data exchange between various computing nodes through the communication interface 920.

[0221] See Figure 10 , Figure 10 This is a schematic diagram of a computing device provided in this application, such as... Figure 10 As shown, the computing device 100 includes a bus 101, a processor 102, multiple accelerator cards 103, a communication interface 104, and a memory 105. The processor 102, the multiple accelerator cards 103, the communication interface 104, and the memory 105 communicate with each other via the bus 101. It should be understood that this application does not limit the number of processors 102, accelerator cards 103, and memory 105 in the computing device 100. The computing device can be a server, such as a server in a cloud data center, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer or a laptop computer.

[0222] Bus 101 can be a Peripheral Component Interconnect Express (PCIe) bus, an Extended Industry Standard Architecture (EISA) bus, a Unified Bus (Ubus or UB), a Compute Express Link (CXL) bus, a Cache Coherent Interconnect for Accelerators (CCIX) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 10 The bus 101 is represented by a single line, but this does not mean that there is only one bus or one type of bus. Bus 101 may include pathways for transmitting information between various components of computing device 100. A unified bus, for example, is a Lingqu bus.

[0223] The processor 102 may be a central processing unit (CPU), or may include a CPU and other hardware chips. The aforementioned hardware chips may be of various types, such as any one of the following: a programmable logic device (PLD), a complex programmable logic device (CPLD), a field programmable gate array (FPGA), or a digital signal processor (DSP). The computing device 100 may include one or more of the aforementioned types of hardware chips, or may include multiple types of the aforementioned hardware chips. This application embodiment does not make specific limitations.

[0224] The accelerator card 103 can be a graphics processing unit (GPU), a neural network processing unit (NPU), a tensor processing unit (TPU), or other dedicated AI processing chips, etc., and this application does not impose specific limitations. The accelerator card 103 includes the multiple processing circuits 910 and communication interface 920 included in the aforementioned computing node 900, and the accelerator card 103 can be used to implement the above... Figures 4 to 8 The corresponding data exchange method in the method embodiment will not be described in detail here.

[0225] Memory 105 can be non-volatile memory, such as read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Memory 105 can also be volatile memory, which can be random access memory (RAM) used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM). In addition, the memory 105 can also be implemented using storage class memory (SCM), phase change memory (PCM), or other types of storage media.

[0226] It should be noted that the same type of storage medium can be configured in the same computing device to realize the function of memory 105, or two or more types of storage media can be configured to realize the function of memory 105. This application does not limit this.

[0227] The memory 105 stores executable program code. The processor 102 executes this executable program code to issue a computing task to the accelerator card 103, so that the accelerator card completes the computing task and obtains the computing result. Figures 4 to 8 The data exchange operation is performed using any one of the methods shown in the embodiments.

[0228] The communication interface 104 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 100 and other devices or communication networks.

[0229] This application also provides a computer program product containing instructions, which may be a software or program product containing instructions capable of running on a computing device or stored on any usable medium. When the computer program product runs on a computing device, it causes the computing device to... Figures 4 to 8 The data exchange operation is performed using any one of the methods shown in the embodiments.

[0230] This application also provides a computer-readable storage medium, which can be any usable medium capable of being stored by a computing device or a data storage device such as a data center containing one or more usable media. The usable medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium, or a semiconductor medium (e.g., a solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to... Figures 4 to 8 The data exchange operation is performed using any one of the methods shown in the embodiments.

[0231] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of this application.

Claims

1. A data exchange method, characterized by, The application is applied to a data exchange system comprising a plurality of devices, each of the plurality of devices comprising a plurality of nodes, the plurality of nodes of the same device exchanging data through first links between nodes, and the nodes of different devices exchanging data through second links between devices, wherein a first node receives a plurality of first data sent by a second node of a different device through the second link in turn; the first node sends a plurality of second data to a third node in the same device through the first link in the process of receiving the plurality of first data; the first node sends a plurality of third data to the second node through the second link in the process of receiving the plurality of first data, so that the second node sends a plurality of fourth data to a fourth node in the same device through the first link in the process of receiving the plurality of third data.

2. The method of claim 1, wherein, The plurality of second data is the same as the plurality of first data, and the plurality of fourth data is the same as the plurality of third data. The first node sends a plurality of second data to a third node through the first link in the process of receiving the plurality of first data, comprising: the first node sends the received first data to the third node through the first link after receiving the first data sent by the second node each time.

3. The method of claim 2, wherein, Further comprising: the first node sends a plurality of fifth data to the third node through the first link in the process of receiving the plurality of first data, the plurality of fifth data being data generated by the first node.

4. The method of claim 1, wherein, The plurality of second data is data already existing in the first node before the first node receives the plurality of first data.

5. The method according to any one of claims 1 to 4, characterized in that, The device is a chip, and the node is a die; or the device is a server, and the node is an acceleration card.

6. The method of claim 5, wherein, The device is a chip, the node is a die, the first link between the nodes is a universal chip interconnection bus (UCIe), the second link between the devices is an NVLink bus, a peripheral component interconnect express (PCIe) bus or a uniform bus (UB); or the device is a server, the node is an acceleration card, the first link between the nodes is an NVLink bus, a PCIe bus or a UB bus, and the second link between the devices is an Ethernet link, an InfiniBand (IB) link or a fiber link.

7. The method according to any one of claims 1 to 6, characterized in that, The data exchange system is used for training of an artificial intelligence (AI) model, and data transmitted between the nodes of the data exchange system is data generated in the training process of the AI model.

8. A data exchange method characterized by, The application is applied to a data exchange system comprising a plurality of devices, each of the plurality of devices comprising a plurality of nodes, the plurality of nodes of the same device exchanging data through first links between nodes, and the nodes of different devices exchanging data through second links between devices, wherein a first node receives a plurality of first data sent by a second node of a different device through the second link in turn; the first node sends a plurality of second data to a third node in the same device through the first link in the process of receiving the plurality of first data; The first node sends, to a fourth node in a different device, multiple third data via the second link in the process of receiving the multiple first data, so that the fourth node sends, to a fifth node in the same device, multiple fourth data via the first link in the process of receiving the multiple third data.

9. The method of claim 8, wherein, The multiple second data are the same as the multiple first data, and the multiple fourth data are the same as the multiple third data. The first node sends, to a third node, multiple second data via the first link in the process of receiving the multiple first data, comprising: The first node sends, to the third node, the received one first data via the first link after receiving the one first data sent by the second node each time.

10. The method of claim 9, wherein, Further comprising: The first node sends, to the third node, multiple fifth data via the first link in the process of receiving the multiple first data, the multiple fifth data being data generated by the first node.

11. The method of claim 8, wherein, The multiple second data are data already existing in the first node before receiving the multiple first data.

12. The method according to any one of claims 8-11, characterized in that, The device is a chip, and the node is a die; or the device is a server, and the node is an acceleration card.

13. The method of claim 12, wherein, The device is a chip, the node is a die, the first link between the nodes is a universal chip interconnection bus (UCIe), the second link between the devices is an NVLink bus, a peripheral component interconnect express (PCIe) bus or a uniform bus (UB), or the device is a server, the node is an acceleration card, the first link between the nodes is an NVLink bus, a PCIe bus or a UB bus, and the second link between the devices is an Ethernet link, a wireless bandwidth IB link or a fiber link. The data exchange system is used for training of an artificial intelligence (AI) model, and data transmitted between the nodes included in the data exchange system is data generated in the training of the AI model.

14. The method according to any one of claims 8-13, characterized in that, The method is applied to a data exchange system including multiple devices, each of the multiple devices including multiple nodes, the multiple nodes of the same device exchanging data via first links between the nodes, and the nodes of different devices exchanging data via second links between the devices, wherein 15. A data exchange method characterized by, A first node receives multiple first data sent by a second node in the same device via the first links in sequence; After receiving the one first data sent by the second node each time, the first node performs a reduction operation on the one first data and one third data to obtain one second data; The first node sends the one second data to a third node in a different device via the second link. The method further comprises:

16. The method of claim 15, wherein, ​ The first node sends multiple fourth data to the second node through the first link, so that the second node performs a reduction operation on each received fourth data and a fifth data to obtain a reduction result, and sends the reduction result to a fourth node in a different device through the second link; the third node and the fourth node are located in the same device.

17. The method of claim 16, wherein, The method further comprises: The first node receives multiple sixth data sent by the third node through the second link; wherein each sixth data is obtained by performing a reduction operation on a seventh data sent by the fourth node and received by the third node; The first node performs a reduction operation on the received multiple sixth data to obtain a reduction result.

18. The method according to any one of claims 15-17, characterized by, The device is a chip, and the node is a die; or the device is a server, and the node is an acceleration card.

19. The method of claim 18, wherein, The device is a chip, the node is a die, and the first link between the nodes is a universal chip interconnection bus (UCIe); the second link between the devices is an NVLink bus, a peripheral component interconnect express (PCIe) bus, or a uniform bus (UB); Or, the device is a server, and the node is an acceleration card; the first link between the nodes is an NVLink bus, a PCIe bus, or a UB bus; and the second link between the devices is an Ethernet link, a wireless bandwidth IB link, or a fiber link.

20. The method according to any one of claims 15-19, characterized by, The data exchange system is used for training of an artificial intelligence (AI) model, and data transmitted between the nodes included in the data exchange system is data generated in the AI model training process.

21. A data exchange system, characterized by The system includes multiple devices, each device including multiple nodes; the multiple nodes of the same device exchange data through first links between the nodes, and the nodes of different devices exchange data through second links between the devices; each node is configured to perform the data exchange method of any one of claims 1-7; or each node is configured to perform the data exchange method of any one of claims 8-14; or each node is configured to perform the data exchange method of any one of claims 15-20.

22. A computing node, characterized in that, The computing node includes multiple processing circuits and a communication interface; the communication interface is configured to perform inter-node communication; and each processing circuit is configured to perform the data exchange method of any one of claims 1-7.

23. A computing node, characterized in that, The computing node includes multiple processing circuits and a communication interface; the communication interface is configured to perform inter-node communication; and each processing circuit is configured to perform the data exchange method of any one of claims 8-14.

24. A computing node, characterized in that, The computing node includes multiple processing circuits and a communication interface; the communication interface is configured to perform inter-node communication; and each processing circuit is configured to perform the data exchange method of any one of claims 15-20.

Images