Communication method and related device

Abstract

The embodiment of the invention provides a communication method and a related device, the communication method is applied to nodes of a data center network, all the nodes of the data center network comprise at least one node of a first group and at least one node of a second group, and the method comprises the following steps: the two groups of nodes execute a global protocol AllReduce in the groups according to different rules, the method comprises the following steps of: carrying out protocol on nodes in a first group and nodes in a second group to realize data synchronization of the nodes in the first group and the nodes in the second group, and then carrying out protocol Reduce through the nodes in the first group and the corresponding nodes in the second group to realize data synchronization between the two groups, so that data synchronization of all nodes of the data center network is realized through the steps, and the two groups are two set communication domains. And the intra-group data synchronization is mutually independent and does not interfere with each other.

Description

Technical Field

[0001] The present application relates to the field of communication technology, and in particular to a communication method and related devices. Background Art

[0002] Collective Communications is a global communication method in which all processes in a process group participate. Common collective communications include: one-to-many broadcast, many-to-one gathering, many-to-many full gathering, one-to-many scattering, many-to-one reduction, many-to-many global reduction, combined reduction scattering, etc. Among them, AllReduce is the most important collective communication method in data parallel training scenarios. It has multiple data senders and multiple data receivers. It performs the same Reduce operation on all nodes in the cluster, and then sends the results of the data reduction operations of all nodes in the cluster to all nodes. The AllReduce operation can be implemented by executing ReduceScatter and AllGather on the master node. Commonly used AllReduce communication algorithms include those based on the Ring algorithm and the Halving-Doubling (HD) algorithm.

[0003] However, when AllReduce is implemented based on the Ring algorithm and the HD algorithm, AllReduce is a many-to-many communication primitive with multiple data senders and multiple data receivers. It executes the same protocol operation on all nodes in the cluster, that is, during the execution of AllReduce, all nodes are located in the same collective communication domain to communicate in order to achieve data synchronization of all nodes. Summary of the invention

[0004] The present application provides a communication method and related devices, which divide all nodes of a data center network into two groups, and the two groups of nodes execute AllReduce according to different communication strategies to achieve data synchronization of nodes in each group, and then perform protocol Reduce between the nodes of the two groups to achieve data synchronization between the two groups, thereby achieving data synchronization of all nodes. When the data synchronization process of all nodes executes AllReduce, there is no need to place all nodes in the same collective communication domain. Data synchronization within the group is performed independently through two collective communication domains, which has higher collective communication flexibility.

[0005] In order to achieve the above purpose, this application adopts the following technical solutions:

[0006] In a first aspect, a communication method is provided, which is applied to nodes of a data center network, wherein all nodes of the data center network include at least one node of a first group and at least one node of a second group; the method comprises: the nodes of the first group execute a global AllReduce within the first group according to a first operation type; the nodes within the second group execute AllReduce within the second group according to a second operation type, the first operation type being used to indicate a communication strategy for the nodes within the first group to execute AllReduce, and the second operation type being used to indicate a communication strategy for the nodes within the second group to execute AllReduce; the nodes within the first group perform a protocol Reduce with the nodes within the second group.

[0007] In this way, by dividing all the nodes of the data center network into two groups, the nodes of the two groups independently execute AllReduce to achieve data synchronization of the nodes in the first group and the nodes in the second group, and then the nodes of the first group and the nodes of the second group perform Reduce to achieve data synchronization between the two groups. The above two steps are combined to achieve data synchronization of all nodes in the two data center networks, and the above two groups independently execute AllReduce. All the nodes of the data center network can be allocated to the two groups through preset rules, and then the nodes in each group that execute AllReduce can be controlled to improve the flexibility of collective communication.

[0008] Furthermore, the AllReduce executed by the first group of nodes and the second group of nodes can be the same (for example, both are AllReduce based on the HD algorithm and the AllReduce based on the HD algorithm includes ReduceScatter and AllGather), or they can be different types, as long as the nodes of the two groups achieve data synchronization within the group by executing AllReduce.

[0009] In some embodiments of the first aspect, the data center network also includes at least one first network device, and the node is connected to the first network device; the nodes of the first group execute the global protocol AllReduce within the first group according to the first operation type; the nodes in the second group execute AllReduce within the second group according to the second operation type, including: the first node in the first group sends a first message to the second node in the first group through the first network device, and receives a second message sent by the second node in the first group through the first network device, so that the nodes of the first group execute AllReduce within the first group according to the first operation type; the third node in the second group sends a third message to the fourth node in the second group through the first network device, and receives a fourth message sent by the third node in the second group through the first network device, so that the nodes of the second group execute AllReduce within the second group according to the second operation type.

[0010] In this way, the nodes in the first group send messages to each other to execute AllReduce in the first group, thereby achieving data synchronization of the nodes in the first group; the nodes in the second group send messages to each other to execute AllReduce in the second group, thereby achieving data synchronization of the nodes in the second group.

[0011] In some embodiments of the first aspect, the nodes in the first group perform protocol Reduce with the nodes in the second group, including: the nodes in the first group receive a fifth message sent by the corresponding nodes in the second group through the first network device, and perform Reduce based on the data of the fifth message, and the nodes in the second group receive a sixth message sent by the corresponding nodes in the first group through the first network device, and perform Reduce based on the data of the sixth message.

[0012] In this way, by sending and receiving messages between the nodes in the first group and the nodes in the second group, each node executes Reduce according to the data in the received message, so as to achieve data synchronization between the nodes in the first group and the nodes in the second group.

[0013] In some embodiments of the first aspect, before the nodes in the first group perform Reduce with the nodes in the second group, the method further includes: the nodes in the first group and the nodes in the second group perform a preset target number of AllReduce steps.

[0014] In this way, after the nodes in the first group and the nodes in the second group execute the preset target number of AllReduce steps, the nodes in the first group and the nodes in the second group perform Reduce, so as to ensure that the two groups synchronously perform inter-group data synchronization.

[0015] In some embodiments of the first aspect, the first message carries a first source address and a first destination address, the second message carries a second source address and a second destination address, the first destination address and the second source address are both addresses of a first node in the first group, and the second destination address and the first source address are both addresses of a second node in the first group;

[0016] The third message carries a third source address and a third destination address, the fourth message carries a fourth source address and a fourth destination address, the fourth destination address and the third source address are both the address of the third node in the second group, and the third destination address and the fourth source address are both the address of the fourth node in the second group.

[0017] In this way, by limiting the nodes corresponding to the source address and destination address of the first message and the second message to be nodes within the first group, and limiting the nodes corresponding to the source address and destination address of the third message and the fourth message to be nodes within the second group, the nodes in the first group can execute intra-group AllReduce through the first message and the second message, and the nodes in the second group can execute intra-group AllReduce through the third message and the fourth message, so as to achieve data synchronization of the nodes in the first group and the nodes in the second group.

[0018] In some embodiments of the first aspect, the fifth message carries a fifth source address and a fifth destination address, the sixth message carries a sixth source address and a sixth destination address, the fifth destination address is the address of the current node of the first group, the fifth source address is the address of a node in the second group, the sixth destination address is the address of the current node in the second group, and the sixth source address is the address of a node in the first group.

[0019] In this way, by limiting the source address and destination address of the fifth message and the sixth message to the node addresses of the two groups respectively, the nodes of the first group and the second group, that is, the nodes of the two groups send messages to each other, and then execute Reduce based on the data of the received messages to achieve inter-group data synchronization between the first group of nodes and the second group of nodes.

[0020] In some embodiments of the first aspect, the first message, the second message, the third message, and the fourth message all carry a collective communication type, and the collective communication type is AllReduce.

[0021] In this way, by carrying the collective communication type of AllReduce, the nodes in the first group are instructed to implement the execution of AllReduce of the nodes in the first group by sending and receiving the first message and the second message, and the nodes in the second group implement the AllReduce of the nodes in the second group by sending and receiving the third message and the fourth message.

[0022] In some embodiments of the first aspect, the first message and the second message both carry a first collective communication group identifier, and the first collective communication group identifier is used to indicate that the source node and the destination node corresponding to the first message or the second message both belong to the first group, and the third message and the fourth message both carry a second collective communication group identifier, and the second collective communication group identifier is used to indicate that the source node and the destination node corresponding to the third message or the fourth message both belong to the second group.

[0023] In this way, the first collective communication group identifier is carried in the first message and the second message to indicate that the first message and the second message are collective communications among nodes in the first group; and the second collective communication group identifier is carried in the third message and the fourth message to indicate that the third message and the fourth message are collective communications among nodes in the second group.

[0024] In some embodiments of the first aspect, the first message and the second message both carry the first operation type, and the third message and the fourth message both carry the second operation type.

[0025] In this way, by carrying the first operation type in the first message and the second message, the nodes in the first group are instructed to execute AllReduce according to the first operation type, and by carrying the second operation type in the third message and the fourth message, the nodes in the second group are instructed to execute AllReduce according to the second operation type.

[0026] In some embodiments of the first aspect, a node of the data center network receives configuration information, wherein the configuration information includes an address of a node in the first group or an address of a node in the second group.

[0027] In this way, by sending configuration information to the nodes in the data center network, the addresses of the nodes in the first group or the addresses of the nodes in the second group are configured in the nodes through the configuration information, so as to realize dividing the nodes in the data center network into the first group or the second group.

[0028] In some embodiments of the first aspect, if the node of the data center network is a node in the first group, the configuration information further includes a first correspondence between a first operation type, an address of a node in the first group, and a first set communication group number, the first set communication group number being a sequence number corresponding to the node in the first group, and the node in the first group executing a corresponding communication strategy according to the first correspondence and the first operation type;

[0029] If the node of the data center network is a node in the second group, the configuration information also includes a second correspondence between a second operation type, an address of the node in the second group, and a second collective communication group number, wherein the second collective communication group number is a serial number corresponding to the node in the second group, and the node in the second group executes a corresponding communication strategy based on the second correspondence and the second operation type.

[0030] In this way, the correspondence between the node address and the group number corresponding to the node is configured to determine the arrangement of the nodes in each group, and then the communication strategy corresponding to the operation type is executed according to the arrangement.

[0031] In some embodiments of the first aspect, if the node of the data center network is a node in the first group, the configuration information further includes at least one of a first collective communication group identifier and a collective communication type, and the collective communication type is AllReduce;

[0032] If the node of the data center network is a node in the second group, the configuration information further includes at least one of a second collective communication group identifier and a collective communication type.

[0033] In this way, by configuring the group identifier to the corresponding node, the node is divided into the first group or the second group, thereby realizing group allocation of the node. By configuring the collective communication type to the node, the node performs collective communication according to the configured collective communication type.

[0034] In some embodiments of the first aspect, the method further includes: a node in the first group receives a seventh message forwarded by the first network device, the seventh message carrying at least one of a first collective communication group identifier, a first operation type, a first correspondence between an address of a node in the first group and a first collective communication group number, and a collective communication type;

[0035] The nodes in the second group receive the eighth message forwarded by the first network device, and the eighth message carries at least one of the second correspondence between the second collective communication group identifier, the second operation type, the address of the node in the second group, and the second collective communication group number.

[0036] In this way, information is configured for all nodes in the data center network through the root node, and the above information is configured so that all nodes in the data center network execute AllReduce according to the corresponding operation type.

[0037] In some embodiments of the first aspect, the AllReduce includes a scatter reduction ReduceScatter and a full gathering AllGather;

[0038] The first operation type includes the nodes in the first group executing ReduceScatter according to the first rule and executing AllGather according to the second rule;

[0039] The second operation type includes the nodes in the second group executing ReduceScatter according to the second rule and executing AllGather according to the first rule;

[0040] The first rule and the second rule are communication strategies adopted by the node when executing ReduceScatter and AllGather respectively.

[0041] Furthermore, the first rule and the second rule are opposite. For example, the nodes of each group are numbered, and the difference between the corresponding numbers of the nodes is the corresponding distance between the nodes. Then the first rule can be from far to near, and the second rule can be from near to far, where far and near refers to the corresponding distance between the nodes.

[0042] In some embodiments of the first aspect, the data center network also includes at least one second network device, and the number of the first network devices is at least two; each of the first network devices has an upstream port and a downstream port, the node is connected to the first network device through the downstream port, and at least one of the second network devices is connected to the first network device through the upstream port, and the total effective upstream bandwidth of the first network device is less than the total effective downstream bandwidth of the first network device.

[0043] In this way, since the total effective uplink bandwidth of the first network device is less than the total effective downlink bandwidth of the first network device, the data center network is a converged network.

[0044] In some embodiments of the first aspect, the total number of nodes in the first group is the same as the total number of nodes in the second group.

[0045] In some embodiments of the first aspect, the node corresponding to the fifth source address and the node corresponding to the fifth destination address are both connected to the same first network device; the node corresponding to the sixth source address and the node corresponding to the sixth destination address are both connected to the same first network device.

[0046] In some embodiments of the first aspect, the nodes in the first group execute AllReduce within the first group according to a first operation type; the nodes in the second group execute AllReduce within the second group according to a second operation type, including: the nodes in the first group sequentially execute ReduceScatter and AllGather based on the halving and doubling HD algorithm within the first group according to the first operation type, and the nodes in the second group sequentially execute ReduceScatter and AllGather based on the HD algorithm of all nodes in the second group according to the second operation type, so that the nodes in the first group and the nodes in the second group implement all-reduce AllReduce, and the first operation type is different from the second operation type.

[0047] By adopting the above technical solution, AllReduce is executed by the nodes in the two groups according to different operation types to achieve data synchronization of the nodes in each group; then the data synchronization between the two groups is achieved by sending data between the nodes in the two groups. Compared with the AllReduce based on the Ring algorithm, this communication method only requires 2*log2(N)+1 steps (N is the number of nodes), which is easy to implement and has fewer implementation steps, thereby reducing the overall communication time.

[0048] In some embodiments of the first aspect, the data center network also includes at least one second network device, each of the first network devices has an upstream port and a downstream port, the total effective downstream bandwidth of the first network device is greater than the total effective upstream bandwidth, the node is connected to the first network device through the downstream port, and at least one of the second network devices is connected to the first network device through the upstream port.

[0049] With the above technical solution, since the total effective downlink bandwidth of the first network device is greater than the total effective uplink bandwidth, the data center network is a converged network. Since the collective communication of the converged network has a "barrel effect" at the bandwidth convergence between the first network device and the second network device, that is, the completion time of each AllReduce collective communication is determined by the slowest point-to-point communication time. If the convergence ratio of the data center network is 2:1, the AllReduce collective communication time is at least doubled. In this way, before the nodes of the data center network perform collective communication, all nodes are divided into two groups, and at least two nodes connected to each first network device are located in one of the groups. The two groups execute AllReduce according to different operation types, so that in each step of the two groups executing AllReduce, at least one group has two nodes in which the communication is performed inside the same first network device (that is, the communication between the nodes does not pass through the link between the first network device and the second network device), thereby avoiding all data traffic from crossing the first network device, and then avoiding the formation of link contention between the first network device and the second network device of the converged network, thereby reducing the overall communication time.

[0050] In some embodiments of the first aspect, the total number of nodes in the first group is the same as the total number of nodes in the second group, wherein all nodes connected to each of the first network devices include: at least two nodes in the first group and at least two nodes in the second group.

[0051] By adopting the above technical solution, by dividing all nodes of the data center network into a first group and a second group with the same number of nodes, at least two nodes connected to each first network device are divided into the first group, and at least two nodes connected to each first network device are divided into the second group, and then the nodes in the first group sequentially execute ReduceScatter and AllReduce based on the HD algorithm of all nodes in the first group according to the first rule, and the nodes in the second group sequentially execute ReduceScatter and AllReduce based on the HD algorithm of all nodes in the second group according to the second rule, and the nodes in the first group and the nodes in the second group independently implement AllReduce, thereby realizing data synchronization of all nodes in the first group and data synchronization of all nodes in the second group , and in the process of executing AllReduce by the nodes in the first group and the nodes in the second group, each node in the first group and the corresponding node in the second group realize the data synchronization of the node data of the first group and the node data of the second group by sending data to each other. The above steps cooperate to realize the data synchronization of all nodes in the data center network, and the nodes in the first group and the second group independently execute AllReduce according to different operation types, avoiding that in the communication process of AllReduce, all communication traffic is transmitted across the first network device, thereby causing the link contention between the first network device and the second network device in the converged network (the number of uplink ports of the first network device is less than the number of downlink ports), so that the communication time of the data center network increases due to the link bandwidth bottleneck. And the data synchronization process of all the nodes only requires 2*log2(N)+1 steps (N is the number of nodes), which is easy to implement and has fewer implementation steps.

[0052] In some embodiments of the first aspect, the topology of the data center network is a fat tree structure.

[0053] The above technical solution uses the HD algorithm to implement AllReduce on a data center network with a convergent fat tree structure, which will increase the communication time proportional to the convergence ratio. The communication method of the present application is implemented on a fat tree with a convergence ratio, which ensures fewer communication steps and less transmission data, while avoiding the increase in communication time due to bottleneck links.

[0054] In some embodiments of the first aspect, a total downlink bandwidth of the first network device is twice the total uplink bandwidth.

[0055] Using the above technical solution, since the total downstream bandwidth of the first network device is twice the total upstream bandwidth, the convergence ratio of the data center network is 2:1. In the data center network with a convergence ratio of 2:1, all of the data center network is divided into two groups, and the two groups independently implement AllReduce based on the HD algorithm to ensure that the collective communication between the two groups of nodes, at least part of the traffic is carried out in the first network device, resulting in link contention between the first network device and the second network device in the converged network (the number of upstream ports of the first network device is less than the number of downstream ports), which increases the communication time of the data center network due to the link bandwidth bottleneck.

[0056] In some embodiments of the first aspect, among all nodes connected to each of the first network devices: half of the nodes are divided into the first group, and the other half of the nodes are divided into the second group.

[0057] By adopting the above technical solution, all nodes connected to each network device are evenly divided into two groups. When data synchronization is performed between the two groups, half of the nodes connected to the network device communicate with the other half of the nodes connected to the network device to ensure that the collective communication performed by the two groups of nodes, at least part of the traffic is performed within the first network device, resulting in link contention between the first network device and the second network device of the converged network, which increases the communication time of the data center network due to the link bandwidth bottleneck.

[0058] In some embodiments of the first aspect, the first operation type includes the nodes in the first group executing ReduceScatter according to the first rule and executing AllGather according to the second rule; the second operation type includes the nodes in the second group executing ReduceScatter according to the second rule and executing AllGather according to the first rule; the first rule and the second rule are opposite.

[0059] By adopting the above technical scheme, since the nodes of the first group and the nodes of the second group implement different communication strategies of AllReduce based on the HD algorithm, and the communication strategies adopted by the two groups are opposite, that is, the first rule is the inverse rule of the second rule, for example, the first rule is from far to near, and the second rule is from near to far, the nodes in the first group execute ReduceScatter based on the HD algorithm from far to near and execute AllGather based on the HD algorithm from near to far, and the nodes in the second group execute ReduceScatter based on the HD algorithm from near to far and execute AllGather based on the HD algorithm from far to near, thereby avoiding that all traffic involved in the communication method is transmitted across the first network device.

[0060] In some embodiments of the first aspect, the nodes of the first group and the corresponding nodes of the second group are both connected to the same first network device.

[0061] By adopting the above technical solution, since each node in the first group and its corresponding node in the second group are connected to the same network device, data synchronization between the two groups is performed inside the first network device, and no traffic transmission across the first network device is generated, thereby avoiding the increase in communication time of the data center network due to link bandwidth bottlenecks.

[0062] In some embodiments of the first aspect, the node of the first group sends all data of the node to a corresponding node in the second group, and receives all data of the corresponding node sent by the corresponding node, and the node of the first group and the corresponding node respectively perform a reduction-reduce based on the received all data, including:

[0063] Before the nodes in the first group and the nodes in the second group implement AllReduce, the nodes in the first group send all the data of the nodes to the corresponding nodes in the second group, and receive all the data of the corresponding nodes sent by the corresponding nodes, and the nodes in the first group and the corresponding nodes respectively perform Reduce based on the received all the data.

[0064] By adopting the above technical solution, data synchronization is performed between the two groups of nodes before the two groups of nodes execute AllReduce based on the HD algorithm. The AllReduce based on the HD algorithm executed by each group is the same as the existing AllReduce based on the HD algorithm, with minor code changes, reducing adaptation complexity.

[0065] In some embodiments of the first aspect, the node of the first group sends all data of the node to a corresponding node in the second group, and receives all data of the corresponding node sent by the corresponding node, and the node of the first group and the corresponding node respectively perform a reduction-reduce based on the received all data, including:

[0066] After the nodes in the first group and the nodes in the second group implement AllReduce, the nodes in the first group send all the data of the nodes to the corresponding nodes in the second group, and receive all the data of the corresponding nodes sent by the corresponding nodes, and the nodes in the first group and the corresponding nodes respectively perform Reduce based on the received all the data.

[0067] By adopting the above technical solution, after the two groups of nodes execute AllReduce based on the HD algorithm, the data between the two groups of nodes are synchronized. The AllReduce based on the HD algorithm executed by each group is the same as the existing AllReduce based on the HD algorithm, with minor code changes, reducing the complexity of adaptation. And after the two groups of nodes execute AllReduce based on the HD algorithm, the data synchronization between the two groups is performed, reducing the amount of data transmitted by the two groups of nodes executing AllReduce based on the HD algorithm.

[0068] In some embodiments of the first aspect, the node of the first group sends all data of the node to a corresponding node in the second group, and receives all data of the corresponding node sent by the corresponding node, and the node of the first group and the corresponding node respectively perform a reduction-reduce based on the received all data, including:

[0069] After the nodes in the first group and the nodes in the second group implement ReduceScatter based on the HD algorithm, the nodes in the first group send all the data of the nodes to the corresponding nodes in the second group, and receive all the data of the corresponding nodes sent by the corresponding nodes, and the nodes in the first group and the corresponding nodes respectively execute Reduce according to the received all the data.

[0070] By adopting the above technical solution, after two groups of nodes execute ReduceScatter based on the HD algorithm, the data between the two groups of nodes are synchronized. The ReduceScatter based on the HD algorithm executed by each group is the same as the existing ReduceScatter based on the HD algorithm, with minor code changes, reducing the adaptation complexity.

[0071] In a second aspect, a communication method is provided, which is applied to a first network device of a data center network, wherein the data center network further includes a node connected to the first network device, and all nodes of the data center network include at least one node of a first group and at least one node of a second group, and the method includes:

[0072] receiving a first message sent by a first node in the first group, and sending the first message to a second node in the first group, receiving a second message sent by the second node in the first group, and sending the second message to the first node in the first group, so that the nodes in the first group execute the global protocol AllReduce in the first group according to a first operation type, wherein the first operation type is used to instruct the nodes in the first group to execute the communication strategy of AllReduce;

[0073] receiving a third message sent by a third node in the second group, and sending the third message to a fourth node in the second group, receiving a fourth message sent by the fourth node in the second group, and sending the fourth message to the third node in the second group, so that the nodes in the second group execute the global AllReduce in the first group according to a second operation type, wherein the second operation type is used to instruct the nodes in the second group to execute the communication strategy of AllReduce;

[0074] Receive the fifth message sent by the node in the first group, and send the fifth message to the node in the second group; receive the sixth message sent by the node in the second group, and send the sixth message to the node in the first group, so that the node in the first group and the node in the second group perform protocol Reduce.

[0075] In some embodiments of the second aspect, before the nodes in the first group and the nodes in the second group perform Reduce, the method further includes:

[0076] Determine whether the nodes in the first group and the nodes in the second group both execute a preset target number of steps of AllReduce;

[0077] If both the nodes in the first group and the nodes in the second group execute the preset target number of steps of AllReduce, the fifth message and the sixth message are forwarded to enable the nodes in the first group and the nodes in the second group to perform Reduce.

[0078] In some embodiments of the second aspect, the first message carries a first source address and a first destination address, the second message carries a second source address and a second destination address, the first destination address and the second source address are addresses of a first node in the first group, and the second destination address and the first source address are addresses of a second node in the first group;

[0079] The third message carries a third source address and a third destination address, the fourth message carries a fourth source address and a fourth destination address, the fourth destination address and the third source address are the addresses of the third node in the second group, and the third destination address and the fourth source address are the addresses of the fourth node in the second group.

[0080] In some embodiments of the second aspect, the fifth message carries a fifth source address and a fifth destination address, the sixth message carries a sixth source address and a sixth destination address, the fifth destination address is the address of the current node of the first group, the fifth source address is the address of a node in the second group, the sixth destination address is the address of the current node in the second group, and the sixth source address is the address of a node in the first group.

[0081] In some embodiments of the second aspect, the first message, the second message, the third message, and the fourth message all carry a collective communication type, and the collective communication type is AllReduce.

[0082] In some embodiments of the second aspect, the first message and the second message both carry a first collective communication group identifier, and the first collective communication group identifier is used to indicate that the source node and the destination node corresponding to the first message or the second message both belong to the first group, and the third message and the fourth message both carry a second collective communication group identifier, and the second collective communication group identifier is used to indicate that the source node and the destination node corresponding to the third message or the fourth message both belong to the second group.

[0083] In some embodiments of the second aspect, the first message and the second message both carry the first operation type, and the third message and the fourth message both carry the second operation type.

[0084] In a third aspect, a communication device is provided, comprising: a memory, the memory comprising computer-readable instructions;

[0085] A processor in communication with the memory, the processor being configured to execute the computer-readable instructions so that the node executes the communication method according to any one of the first aspect or the communication method according to any one of the second aspect.

[0086] In a fourth aspect, a computer-readable storage medium is provided, comprising a program or an instruction, which when executed by a processor implements the communication method as described in any one of the first aspect or the second aspect.

[0087] The beneficial effects brought about by each possible implementation method of the communication method provided in the second aspect, the communication device provided in the third aspect, and the computer-readable storage medium provided in the fourth aspect of the embodiments of the present application can be referred to the description of the various possible implementation methods in the first aspect, and will not be repeated here one by one. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] Figure 1 A system diagram of a data center network;

[0089] Figure 2A is a schematic diagram of an initial state of a node;

[0090] Figure 2B for Figure 2A Schematic diagram of the completion status after the middle node executes AllReduce;

[0091] Figure 3A schematic diagram of ReduceScatter that implements AllReduce based on the HD algorithm;

[0092] Figure 4 A schematic diagram of AllGather implementing AllReduce based on HD algorithm;

[0093] Figure 5 A flow chart of a communication method provided in an embodiment of the present application;

[0094] Fig. 6A A schematic diagram of implementing ReduceScatter based on the HD algorithm for a first group of internal nodes and a second group of internal nodes provided in an embodiment of the present application;

[0095] Figure 6B A schematic diagram of inter-group data synchronization between nodes in a first group and nodes in a second group provided in an embodiment of the present application.

[0096] Figure 6C A schematic diagram of a first group of internal nodes and a second group of internal nodes implementing AllGather based on the HD algorithm respectively provided in an embodiment of the present application;

[0097] Figure 7 A flow chart of a communication method provided in an embodiment of the present application;

[0098] Figure 8 A schematic diagram of the structure of a node provided in an embodiment of the present application. DETAILED DESCRIPTION

[0099] The technical solution in the present application will be described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of this specification, rather than all the embodiments.

[0100] To facilitate understanding, the terms involved in the embodiments of the present application are first briefly introduced.

[0101] 1. AllGather

[0102] AllGather is a many-to-many communication primitive with multiple data senders and multiple data receivers. It can collect data from multiple nodes in the cluster to a master node (Gather), and then distribute the collected data to other nodes (broadcast), that is, collect all the data in the cluster to all nodes.

[0103] 2. ReduceScatter

[0104] ReduceScatter is a many-to-many communication primitive with multiple data senders and multiple data receivers. It performs the same Reduce operation by dimension on all nodes in the cluster, and then scatters the results to all nodes in the cluster. Reduce-scatter is equivalent to performing Reduce operations for the number of nodes, and then performing scatter operations for the number of nodes.

[0105] 3. Global reduction (AllReduce)

[0106] AllReduce is a many-to-many communication primitive with multiple data senders and multiple data receivers. It performs the same Reduce operation on all nodes in the cluster and can send the results of data reduction operations on all nodes in the cluster to all nodes. The AllReduce operation can be implemented by executing ReduceScatter and AllGather on the nodes in sequence.

[0107] 4. Halving-Doubling (HD)

[0108] The HD algorithm is a collective communication algorithm used in AI training. All nodes in the collective communication communicate with each other two by two. At each step, the node can obtain the data of the communicating node, and this process is continuous, so that each node can obtain all the data of all nodes.

[0109] 5. Fat-Tree

[0110] Fat tree is a kind of CLOS network architecture. Fat tree is more like a real tree. The branches get thicker as you get closer to the root. The network bandwidth does not converge from leaves to roots. The basic concept of fat tree architecture is to use a large number of low-performance switches to build a large-scale non-blocking network. For any communication mode, there is always a path for their communication bandwidth to reach the network card bandwidth.

[0111] See also Figure 1 , Figure 1 A system diagram of a data center network. Figure 1 The data center network includes a first network device, a second network device and multiple nodes, wherein the first network device has an uplink port and a downlink port, the second network device is connected to the first network device through the uplink port, and the multiple nodes are connected to the first network device through the downlink port.

[0112] Figure 1The data center network includes two first network devices (first network device 1 to first network device 2), one second network device (second network device 1) and eight nodes (node ​​1 to node 8). The number of uplink ports of each first network device is 2 and the number of downlink ports is 4.

[0113] The uplink port and the downlink port here are valid ports of the first network device, that is, the uplink port corresponding to the connection between the first network device and the second network device, and the downlink port corresponding to the connection between the first network device and the node.

[0114] Optionally, Figure 1 The topology of the data center network in the figure is a fat tree structure and a two-layer fat tree network topology. It can be understood that in other embodiments, the topology of the data center network can also be a three-layer fat tree network topology. Then the data center network also includes a third network device (not shown), the third network device is connected to the uplink port of the second network device, and the second network device is connected to the first network device through its downlink port. Of course, the data center network can also be a four-layer fat tree network topology, or a P-layer fat tree network topology, where P is a positive integer greater than 4.

[0115] Optionally, Figure 1 The data center network in is a converged network and a Layer 2 network with a convergence ratio of 2:1. The ratio of the number of downlink ports to the number of uplink ports of each first network device is 2:1, and the total downlink effective bandwidth of the first network device is twice the total uplink bandwidth. It can be understood that in other embodiments, if the total downlink effective bandwidth of the first network device is the same as the total uplink bandwidth, the data center network is a non-convergent network.

[0116] in, Figure 1 The middle node is a server. It can be understood that the node can also be other electronic devices that can perform collective communication. The network device can be a device with data forwarding function such as a switch or a router.

[0117] It is easy to understand that AllReduce executes the same Reduce operation on all nodes in the cluster to synchronize data on all nodes. Figure 2A and Figure 2B Provide an example.

[0118] like Figure 2A As shown, Figure 2A is a schematic diagram of the initial state of a node, where Figure 2A There are four nodes in the cluster: node a to node d. The four nodes form a communication cluster. All four nodes are in the initial state. Among nodes a to node d, each node has a corresponding data block. Figure 2ANode a has four gradient data: a0, a1, a2 and a3; node b has four gradient data: b0, b1, b2 and b3; node c has four gradient data: c0, c1, c2 and c3; node d has four gradient data: d0, d1, d2 and d3.

[0119] See also Figure 2B , Figure 2B for Figure 2A Schematic diagram of the completion status after the middle node executes AllReduce. Figure 2A After all nodes complete AllReduce, we get Figure 2B In the completed state, each node has the specification of the data of all nodes, that is, the gradient data in all nodes are the same, and the four gradient data of nodes a to d in the completed state are the same, and are: a0+b0+c0+d0, a1+b1+c1+d1, a2+b2+c2+d2, and a3+b3+d3+d3.

[0120] Depend on Figure 2A and Figure 2B It can be seen that all nodes implement gradient synchronization among multiple nodes by executing AllReduce. The result of gradient synchronization within a node is stored in each node in the form of a specification.

[0121] Among them, the technical problem of implementing AllReduce based on the Ring algorithm is that the communication delay increases linearly with the increase of the number of nodes. It takes 2*(N-1) steps of communication to complete AllReduce, where N is the number of nodes. The increase in communication delay for smaller data packets is more obvious. However, AllReduce based on the HD algorithm only requires 2*log2(N) steps, which is much fewer steps than AllReduce based on the Ring algorithm.

[0122] Furthermore, the following is combined Figure 3 and Figure 4 The AllReduce for collective communication based on the HD algorithm is briefly introduced, and AllReduce includes ReduceScatter and AllGather. It can be understood that in other embodiments, AllReduce can also be implemented by executing Reduce and Broadcast in sequence.

[0123] It can be understood that the AllReduce involved in the embodiments of the present application can be applied to Figure 1 In the following embodiments, AllReduce is applied to the data center network shown in FIG. Figure 1 The data center network shown is used as an example for explanation and will not be described in detail later.

[0124] like Figure 3 As shown, Figure 3 A schematic diagram of ReduceScatter that implements AllReduce based on the HD algorithm.

[0125] If the data center network includes N nodes, the N nodes can form a node cluster (or collective communication domain), so that data synchronization of all nodes in the cluster can be achieved through AllReduce. ReduceScatter includes log2(N) steps:

[0126] Step 1, the distance is The two nodes send data blocks of half the message size of each node to each other, and each node performs reduction on the received data blocks.

[0127] Optionally, the N nodes are sorted according to a preset rule, and the distance between two nodes is the difference between the sorting numbers of all the sorted nodes. Figure 3 The 8 nodes are sorted in sequence, and the sorted nodes are: N0~N7, and the corresponding numbers of N0~N7 are 1~8. Then, the distance between node N1 and node N5 is the difference between the corresponding numbers of node N1 and node N5, that is, the distance is 4.

[0128] Among them, the preset rule can be sorting based on distance size, where the distance size can be the distance between the downlink ports of the first network device to which the two nodes are connected; of course, the preset rule can also be sorting based on the position of the nodes, and the position of the nodes can be the relative position between the nodes.

[0129] like Figure 3 As shown, Figure 3 The data center network in the example consists of 8 nodes. Assume that the message size to be executed for AllReduce is Size, and Figure 3 The size of the message sent by two nodes with a distance of 4 to each other (Size / 2 1 ) data block, N0 and N4, N1 and N5, N2 and N6, N3 and N7 node pairs send their own Size / 2 1 8 nodes perform Reduce on the received data blocks;

[0130] Step 2, the distance is The two nodes send 1 / 2 2 Each node performs Reduce on the received data blocks.

[0131] like Figure 3 As shown, Figure 3 The size of the message sent by two nodes with a distance of 2 to each other (Size / 22 ) data blocks: N0 and N2, N1 and N3, N4 and N6, N5 and N7 node pairs send each other data blocks of size Size / 2 2 Each node performs Reduce on the received data blocks.

[0132] Step 3, the distance is The two nodes send 1 / 2 3 Each node performs Reduce on the received data blocks.

[0133] like Figure 3 As shown, Figure 3 The size of the message sent by two nodes with a distance of 1 to each other (Size / 2 3 ), N0 and N1, N2 and N3, N4 and N5, N6 and N7 node pairs send data blocks of size Size / 2 to each other. 3 Each node performs Reduce on the received data blocks. In this way, 8 nodes complete the ReduceScatter of AllReduce based on the HD algorithm.

[0134] If the number of nodes N is greater than 8, execute step 4 and transfer 1 / 2 of the data block obtained in the previous step to 4 Between nodes whose distance is doubled (for example, the distance between two nodes is ) are sent to each other, and then the received data blocks are reduced, and then the cycle continues until each node has a reduction of the data blocks at the corresponding positions of all other nodes.

[0135] like Figure 4 As shown, Figure 4 The figure is a schematic diagram of the AllGather process of implementing AllReduce based on the HD algorithm.

[0136] N nodes in the data center network ( Figure 3 N is 8) Execution completed Figure 3 After the ReduceScatter of the HD algorithm-based AllReduce, N nodes then execute AllGather, where AllGather is the inverse process of AllReduce and AllGather includes log2(N) steps;

[0137] Step 1, the distance is The two nodes send each other their own message size Size data block, each node performs Reduce on the received data block;

[0138] like Figure 4 As shown, the node pairs with a distance of 1 are: N0 and N1, N2 and N3, N4 and N5, N6 and N7; the size of the message sent between the node pairs is Size / 2 3 Each node performs Reduce on the received data blocks.

[0139] Step 2, the distance is The two nodes send messages to each other Size data block, each node performs Reduce on the received data block.

[0140] like Figure 4 As shown, the node pairs with a distance of 2 are: N0 and N2, N1 and N3, N4 and N6, N5 and N7, and the size of the messages sent between the node pairs is Size / 2. 2 data blocks and perform Reduce on the received data blocks.

[0141] Step 3, the distance is The two nodes send Each node performs Reduce on the received data blocks of the respective message size.

[0142] like Figure 4 As shown, the node pairs with a distance of 4 are: N0 and N4, N1 and N5, N2 and N6, N3 and N7, and the size of the messages sent between the node pairs is Size / 2. 1 data blocks and perform Reduce on the received data blocks. In this way, 8 nodes complete the AllGather of AllReduce.

[0143] If the number of nodes N is greater than 8, execute step 4 to convert the data block obtained in the previous step into Between nodes with doubled distance (for example, the distance between two nodes is ) are sent to each other, and then the received data blocks are reduced, and then the cycle continues until each node has the Reduce of the data blocks of other nodes.

[0144] It can be understood that the nodes in the data center network execute AllReduce to achieve data synchronization among all nodes in the data center network. Figure 3 and Figure 4When the nodes in the data center execute AllReduce, all nodes must be in the same collective communication domain. The nodes in the same collective communication domain perform collective communication to achieve data synchronization of the nodes. However, in some scenarios, multiple data sending nodes and multiple data receiving nodes in the data center network may not all be able to perform direct data communication. For example, node A and node B in the data center network cannot communicate directly. If node A and node B are in the same collective communication domain, when the nodes in the data center network execute AllReduce, since node A and node B cannot directly exchange data to achieve data synchronization, AllReduce execution may fail.

[0145] Based on the above technical problems, an embodiment of the present application provides a communication method, which is applied to nodes of a data center network. All nodes of the data center network include at least one node of the first group and at least one node of the second group; the nodes of the first group and the nodes in the second group execute AllReduce in the group according to different operation types to achieve data synchronization of the nodes in the first group and the second group; then the nodes in the first group and the nodes in the second group perform protocol Reduce to achieve inter-group data synchronization of the nodes in the first group and the second group. By combining the above two steps, data synchronization of all nodes in the data center network is achieved. By dividing all nodes of the data center network into two groups, the two groups execute AllReduce independently. Since the two groups are independent collective communication domains, the data synchronization within the group is completed independently when executing AllReduce. After the data synchronization is achieved within the group, the nodes between the groups communicate with each other to achieve inter-group data synchronization of the nodes in the two groups, thereby achieving data synchronization of all nodes in the data center network. That is, when the nodes of the data center network execute AllReduce, it is not necessary to place all nodes in the same collective communication domain, which can improve the flexibility of collective communication of the data center network.

[0146] See also Figure 5 , Figure 5 A flow chart of a communication method provided in an embodiment of the present application, the communication method can be applied to Figure 1 The nodes of the data center network shown, all nodes of the data center network include at least one node of the first group and at least one node of the second group; it is easy to understand that before the nodes of the data center network communicate, all nodes of the data center network are divided into the first group and the second group, the first group includes at least one node and the second group includes at least one node.

[0147] See also Figure 5 , the method comprising:

[0148] S501. The nodes in the first group execute AllReduce in the first group according to the first operation type; the nodes in the second group execute AllReduce in the second group according to the second operation type. The first operation type is used to instruct the nodes in the first group to execute the communication strategy of AllReduce, and the second operation type is used to instruct the nodes in the second group to execute the communication strategy of AllReduce.

[0149] It is easy to understand that the first group of nodes and the second group of nodes execute AllReduce according to different operation types to achieve data synchronization of the nodes in the first group and data synchronization of the nodes in the second group.

[0150] It can be understood that AllReduce includes Ring-based AllReduce and HD algorithm-based AllReduce, and the HD algorithm-based AllReduce includes ReduceScatter and AllGather, and the HD algorithm-based AllReduce may also include Reduce and Broadcast. Then, the AllReduce executed by the nodes in the first group and the nodes in the second group may be the same, for example, the AllReduce executed by the nodes in the first group and the nodes in the second group are both Ring-based AllReduce or HD algorithm-based AllReduce. Of course, the AllReduce executed by the nodes in the first group and the nodes in the second group may be different, for example, the nodes in the first group execute AllReduce based on the HD algorithm, and the AllReduce includes ReduceScatter and AllGather, and the nodes in the second group also execute AllReduce based on the HD algorithm, and the AllReduce includes Reduce and Broadcast.

[0151] Optionally, since different AllReduces include different numbers of steps, in order to ensure that the nodes of the first group and the nodes of the second group execute the same number of steps of AllReduce, the nodes of the first group and the nodes of the second group may execute the same AllReduce, for example, the nodes in the first group and the nodes in the second group execute AllReduce based on the HD algorithm, and the AllReduce includes ReduceScatter and AllGather.

[0152] It is easy to understand that if the nodes in the first group or the second group of nodes execute the Ring-based AllReduce, the operation type can be the execution direction. For example, the nodes in the data center network execute the Ring-based AllReduce, and all nodes are arranged according to the preset rules (for example, all nodes are assigned numbers 1 to 4, and then arranged in a ring according to the numbers). The execution direction can be clockwise execution of AllReduce or counterclockwise execution of AllReduce. If the operation type is clockwise execution, each node receives the data block sent by the left node, and then reduces the received data block with the local corresponding data block; the intermediate value generated in the previous step is sent to the right node;

[0153] Optionally, if the nodes in the first group or the nodes in the second group execute AllReduce based on the HD algorithm, and AllReduce includes ReduceScatter and AllGather;

[0154] The first operation type includes the nodes in the first group executing ReduceScatter according to the first rule and executing AllGather according to the second rule;

[0155] The second operation type includes that the nodes in the second group execute ReduceScatter according to the second rule and execute AllGather according to the first rule; wherein the first rule and the second rule are communication strategies adopted when the nodes execute ReduceScatter and AllGather respectively.

[0156] Exemplarily, the nodes in the first group and the nodes in the second group are numbered respectively: [1, 2, 3, 4, 5, 6, 7, 8], and then the difference in node numbers is used as the distance between the nodes. Then, the nodes in the first group implement ReduceScatter based on the HD algorithm according to the first rule from far to near, and the nodes in the second group implement ReduceScatter based on the HD algorithm according to the second rule from near to far.

[0157] The distance in from near to far and from far to near here refers to the difference in numbers between the nodes in the first group and the nodes in the second group. For example, the larger the difference in numbers, the larger (farther) the corresponding distance between the two nodes.

[0158] S502: The nodes in the first group and the nodes in the second group perform Reduce.

[0159] It can be understood that the nodes in the first group and the nodes in the second group perform reduce to achieve data synchronization between the nodes in the first group and the nodes in the second group;

[0160] Optionally, if the total number of nodes in the first group is the same as the total number of nodes in the second group, data synchronization between the nodes in the first group and the nodes in the second group can be achieved by reducing each node in the first group with the corresponding node in the second group, wherein each node in the first group corresponds one-to-one to each node in the second group.

[0161] Optionally, if the total number of nodes in the first group is different from the total number of nodes in the second group, at least one node in the first group or the second group has at least two corresponding nodes in the other group, and each node is reduced with the corresponding node to achieve data synchronization between the nodes in the first group and the nodes in the second group. For example, there are 2 nodes in the first group and 3 nodes in the second group, then one of the nodes in the first group has two corresponding nodes in the first group, and another node in the first group has one corresponding node in the second group, and each node is reduced with the corresponding node to achieve data synchronization between the nodes in the first group and the nodes in the second group.

[0162] In this way, AllReduce is executed by the nodes in the first group and the nodes in the second group according to different operation types to achieve data synchronization of the nodes in the first group and the nodes in the second group. The nodes in the two groups independently execute data synchronization within the group, and then the two groups perform inter-group data synchronization to achieve data synchronization of all nodes in the data center network.

[0163] In some optional implementations, the data center network further includes at least one first network device, and the node is connected to the first network device. Then each node in the first group can send a message to the first network device, and the first network device forwards the message to the corresponding node in the first group. Each node in the second group can send a message to the first network device, and the first network device forwards the message to the corresponding node in the second group, that is, AllReduce of the nodes in the first group and the nodes in the second group is executed by sending and receiving messages, thereby realizing data synchronization of the nodes in the first group and the nodes in the second group. Then S501 includes:

[0164] The first node in the first group sends a first message to the second node in the first group through the first network device, and receives a second message sent by the second node in the first group through the first network device, so that the nodes in the first group execute AllReduce in the first group according to the first operation type;

[0165] The third node in the second group sends a third message to the fourth node in the second group through the first network device, and receives a fourth message sent by the third node in the second group through the first network device, so that the nodes in the second group execute AllReduce in the second group according to the second operation type.

[0166] It can be understood that the first node and the second node can be any nodes in the first group, and the third node and the fourth node can be any nodes in the second group. Then the first message and the second message are messages for communication between any two nodes in the first group.

[0167] In this way, by sending and receiving messages between nodes in the first group or the second group, the interaction of nodes in the group is achieved, thereby realizing the execution of AllReduce of nodes in the first group or the second group, and further realizing data synchronization of nodes in the first group and data synchronization of nodes in the second group.

[0168] It can be understood that the nodes within the group can perform AllReduce by sending and receiving messages to each other, and the nodes between the groups can also send and receive messages through the first network device, and perform Reduce according to the received messages to achieve inter-group data synchronization between the first group of nodes and the second group of nodes. Then S502 includes:

[0169] The nodes in the first group receive the fifth message sent by the corresponding nodes in the second group through the first network device, and perform Reduce based on the data of the fifth message. The nodes in the second group receive the sixth message sent by the corresponding nodes in the first group through the first network device, and perform Reduce based on the data of the sixth message.

[0170] It can be understood that inter-group data synchronization between the first group and the second group is achieved by sending and receiving inter-group messages between nodes in the first group and nodes in the second group, and then the nodes in each group execute Reduce based on the data carried in the received messages and the local data of the nodes; in order to achieve inter-group data synchronization between nodes in the first group and nodes in the second group, each node in the first group and each node in the second group must participate in the sending and receiving of messages (for example, the fifth message and the sixth message) to send the local data of the nodes of the current group to the corresponding nodes of the other group, and receive the data sent by the nodes of the other group, and achieve inter-group data synchronization between the first group of nodes and the second group of nodes by sending and receiving inter-group node messages in which all nodes of the two groups participate.

[0171] Understandably, Figure 5The communication method shown mainly includes two steps: implementing data synchronization within the first group and data synchronization within the second group through S501, and implementing data synchronization between the first group and the second group through S502. The execution order of the two steps can be: first implementing data synchronization between the first group and the second group through S502, and then implementing data synchronization within the first group and data synchronization within the second group through S501, or first implementing data synchronization within the first group and data synchronization within the second group through S501, and then implementing data synchronization between the first group and the second group through S502, or the nodes within the first group and the nodes within the second group first execute the preset target number of steps of AllReduce, and then implementing data synchronization between the first group and the second group through S502. Through the above pre-agreement or pre-configuration, the execution time of S501 and S502 of the first group of nodes and the second node can be synchronized.

[0172] Furthermore, the present application implements intra-group data synchronization by executing AllReduce by nodes within the group. For example, it takes 2*(N-1) steps for a node to execute AllReduce based on Ring; it takes 2*log2(N) steps for a node to execute AllReduce based on the HD algorithm, where N is the number of nodes in the first group or the second group. If the preset target number of steps is 0, data synchronization between the first group and the second group is first implemented through S502, and then data synchronization within the first group and data synchronization within the second group are implemented through S501; if the preset target number of steps is 2*(N-1) or 2*log2(N), S50 is first implemented through 1 implements data synchronization within the first group and data synchronization within the second group, and then implements data synchronization between the first group and the second group through S502; if the preset target number of steps is other integers between 0 and 2*(N-1) or 2*log2(N), then the first group and the second group are first implemented through S501. During the intra-group synchronization process (for example, after executing M steps, M can be an integer between 0 and 2*(N-1) or 2*log2(N)), the first group and the second group are implemented through S502. After the inter-group data synchronization is completed, the first group and the second group are continued to be implemented.

[0173] In order to achieve Figure 5 In order to synchronize the execution time of inter-group synchronization and intra-group synchronization, before S502, the method further includes: the first group of nodes and the second group of nodes execute AllReduce at a preset target number of steps. In this way, after limiting the preset target number of steps for the first group of nodes and the second group of nodes to execute AllReduce, S502 is executed again to achieve synchronous execution of S502 by the first group of nodes and the second group of nodes, that is, synchronization of the time of inter-group synchronization between the first group of nodes and the second group of nodes.

[0174] Exemplarily, the preset target number of steps is 3. After the nodes in the first group and the nodes in the second group execute 3 steps of AllReduce, a blocking instruction is inserted into the nodes in the first group or the second group so that the nodes in the first group and the nodes in the second group are stopped from executing collective communication. Then, after both groups complete 3 steps, the nodes in the first group and the nodes in the second group simultaneously trigger inter-group collective communication to achieve inter-group data synchronization.

[0175] Furthermore, if AllReduce is Ring-based AllReduce, the preset target number of steps is 0 or 2*(N-1); if AllReduce is HD algorithm-based AllReduce, the preset target number of steps is 0 or 2*log2(N) or log2(N), thereby ensuring the complete implementation of AllReduce for nodes in each group and reducing code modification and adaptation complexity.

[0176] It can be understood that in S501, by sending and receiving messages between nodes in the first group or the second group, AllReduce of the nodes in the first group or the second group is executed according to the data carried in the message to achieve data synchronization of the nodes in the first group and the nodes in the second group, by limiting the nodes corresponding to the source address and the destination address of the first message and the second message to be nodes in the first group, and limiting the nodes corresponding to the source address and the destination address of the third message and the fourth message to be nodes in the second group, the nodes in the first group execute AllReduce within the group through the first message and the second message, and the nodes in the second group execute AllReduce within the group through the third message and the fourth message, to achieve data synchronization of the nodes in the first group and the nodes in the second group.

[0177] Then the first message carries the first source address and the first destination address, the second message carries the second source address and the second destination address, the first destination address and the second source address are both the address of the first node in the first group, and the second destination address and the first source address are both the address of the second node in the first group; the third message carries the third source address and the third destination address, the fourth message carries the fourth source address and the fourth destination address, the fourth destination address and the third source address are both the address of the third node in the second group, and the third destination address and the fourth source address are both the address of the fourth node in the second group.

[0178] It can be understood that in S502, the first group of nodes and the second group of nodes send and receive messages to each other, and then the nodes of each group perform Reduce based on the data carried in the received message and the local data of the node; to achieve inter-group data synchronization between the nodes in the first group and the nodes in the second group, by limiting the source address and destination address of the fifth message and the sixth message to the node addresses of the two groups respectively, to achieve the nodes of the first group and the nodes of the second group, that is, the nodes of the two groups send messages to each other, and then perform Reduce based on the data of the received message, to achieve inter-group data synchronization between the nodes in the first group and the second group. Then the fifth message carries the fifth source address and the fifth destination address, the sixth message carries the sixth source address and the sixth destination address, the fifth destination address is the address of the current node in the first group, the fifth source address is the address of the node in the second group, the sixth destination address is the address of the current node in the second group, and the sixth source address is the address of the node in the first group.

[0179] In an optional implementation, the first message, the second message, the third message, and the fourth message all carry a collective communication type, and the collective communication type is AllReduce. By carrying the collective communication type of AllReduce, it is indicated that the nodes in the first group implement the execution of AllReduce of the nodes in the first group by sending and receiving the first message and the second message, and the nodes in the second group implement the AllReduce of the nodes in the second group by sending and receiving the third message and the fourth message. By carrying the collective communication type of AllReduce, it is indicated that the nodes in the second group implement the execution of AllReduce of the nodes in the second group by sending and receiving the third message and the fourth message.

[0180] Optionally, since AllReduce includes three implementation methods: Ring-based AllReduce, HD algorithm-based AllReduce, and HD algorithm-based AllReduce includes ReduceScatter and AllGather or AllReduce includes Reduce and Broadcast, the first message, the second message, the third message and the fourth message may also carry the implementation method of AllReduce to indicate the specific implementation method of AllReduce executed by the nodes in the first group or the nodes in the second group.

[0181] Furthermore, the first message, the second message, the third message, and the fourth message may carry the same AllReduce implementation method, or may carry different AllReduce implementation methods.

[0182] In an optional implementation, the first message and the second message both carry a first collective communication group identifier, which is used to indicate that the source node and the destination node corresponding to the first message or the second message both belong to the first group, and the third message and the fourth message both carry a second collective communication group identifier, which is used to indicate that the source node and the destination node corresponding to the third message or the fourth message both belong to the second group. In this way, by carrying the first collective communication group identifier in the first message and the second message, it is indicated that the first message and the second message are nodes in the first group for collective communication; by carrying the second collective communication group identifier in the third message and the fourth message, it is indicated that the third message and the fourth message are nodes in the second group for collective communication.

[0183] Optionally, the collective communication group identifier carried by the first message and the second message or the third message and the fourth message can carry the collective communication group identifier through a message header, that is, the collective communication group identifier is placed in the message header, or the collective communication group identifier can be placed in the data of the message (for example, in the payload of the message).

[0184] In an optional implementation, the first message and the second message both carry the first operation type, and the third message and the fourth message both carry the second operation type. In this way, by carrying the first operation type in the first message and the second message, the nodes in the first group are instructed to execute AllReduce according to the first operation type, and by carrying the second operation type in the third message and the fourth message, the nodes in the second group are instructed to execute AllReduce according to the second operation type.

[0185] It can be understood that in other implementations, one or more configuration information of the collective communication type, operation type, and collective communication group identifier can be configured in the node, and the node saves the configuration information locally. Then, during the message transmission process among the nodes in the group, the first message, the second message, the third message, and the fourth message may not need to carry the corresponding configuration information.

[0186] It is easy to understand that before the nodes in the data center network perform collective communication, configuration information needs to be sent to one or more nodes in the collective communication, for example, by sending configuration information to the nodes of the data center network through the network management or other management platform corresponding to the node, and configuring the node through the configuration information.

[0187] Optionally, the nodes of the data center network receive configuration information, and the configuration information includes the addresses of the nodes in the first group or the addresses of the nodes in the second group. By sending the configuration information to the nodes in the data center network, the addresses of the nodes in the first group or the addresses of the nodes in the second group are configured in the nodes through the configuration information, so as to realize dividing the nodes in the data center network into the first group or the second group.

[0188] Furthermore, the nodes in the data center network can be divided into the first group or the second group by sending configuration information to all nodes in the data center network. The configuration information can also be first sent to one or more nodes (such as a root node) in the data center network, and then the configuration information is sent to other nodes in the data center network through the root node to divide all nodes in the data center network into the first group or the second group.

[0189] It can be understood that before the nodes of the data center network perform collective communication, since all the nodes in the embodiment of the present application are divided into two groups, the nodes in each group can be numbered according to preset rules (for example, the collective communication group number), and then the nodes in the group can be arranged according to the numbers, and then the corresponding collective communication can be performed based on the arranged nodes in combination with the corresponding operation type.

[0190] Exemplarily, if the collective communication type of the node is AllReduce based on the HD algorithm, and AllReduce based on the HD algorithm includes ReduceScatter and AllGather, then the operation type of the collective communication is to execute ReduceScatter from far to near and to execute AllGather from near to far, where the far and near here refers to the difference in numbers corresponding to the nodes. For example, the greater the difference between the numbers corresponding to the two nodes, the farther the distance between the two nodes is. Correspondingly, the smaller the difference between the numbers corresponding to the two nodes is, the closer the distance between the two nodes is. That is, before executing the collective communication, the execution communication strategy of the operation type is determined according to the group number corresponding to each node.

[0191] That is, the group number and operation type corresponding to each node are configured in the node so that the node can execute the corresponding communication strategy according to the group number and operation type. If the node of the data center network is a node in the first group, the configuration information also includes the first operation type, the address of the node in the first group, and the first correspondence between the first collective communication group number, the first collective communication group number is the serial number corresponding to the node in the first group, and the node in the first group executes the corresponding communication strategy according to the first correspondence and the first operation type; if the node of the data center network is a node in the second group, the configuration information also includes the second operation type, the address of the node in the second group, and the second correspondence between the second collective communication group number, the second collective communication group number is the serial number corresponding to the node in the second group, and the second correspondence between the nodes in the second group and the second operation type executes the corresponding communication strategy. In this way, the correspondence between the node address and the group number corresponding to the node is configured to determine the arrangement of the nodes in each group, and then the communication strategy corresponding to the operation type is executed according to the arrangement.

[0192] In an optional implementation, the node of the data center network is a node in the first group, and the configuration information further includes at least one of a first collective communication group identifier and a collective communication type, and the collective communication type is AllReduce;

[0193] If the node of the data center network is a node in the second group, the configuration information also includes at least one of a second collective communication group identifier and a collective communication type. By configuring the group identifier to the corresponding node, the node is divided into the first group or the second group, thereby realizing group allocation of the node. By configuring the collective communication type to the node, the node performs collective communication according to the configured collective communication type.

[0194] Optionally, before all nodes in the data center network communicate with each other, the root node among all nodes may establish a communication connection with other nodes through the Transmission Control Protocol (TCP), and then transmit messages between nodes through the communication connection. It is understood that the root node and other nodes may also establish a communication connection through other protocols, such as the Remote Direct Memory Access (RDMA) protocol.

[0195] It can be understood that the configuration information can be configured to one of the nodes in the data center network (such as the root node), and then the configuration information can be distributed and configured to other nodes in the data center network through the root node. Then the communication method also includes: the nodes in the first group receive the seventh message forwarded by the first network device, and the seventh message carries at least one of the first collective communication group identifier, the first operation type, the first correspondence between the address of the node in the first group and the first collective communication group number, and the collective communication type; the nodes in the second group receive the eighth message forwarded by the first network device, and the eighth message carries at least one of the second correspondence between the second collective communication group identifier, the second operation type, the address of the node in the second group, and the second collective communication group number. In this way, all nodes in the data center network are configured with information through the root node, and the above information is configured so that all nodes in the data center network execute AllReduce according to the corresponding operation type.

[0196] In an optional implementation, the node corresponding to the fifth source address and the node corresponding to the fifth destination address are both connected to the same first network device; the node corresponding to the sixth source address and the node corresponding to the sixth destination address are both connected to the same first network device. In this way, since the data center network realizes data synchronization of the nodes in the first group and the nodes in the second group through the sending and receiving of the fifth message and the sixth message, and since the node corresponding to the fifth source address and the node corresponding to the fifth destination address are both connected to the same first network device; the node corresponding to the sixth source address and the node corresponding to the sixth destination address are both connected to the same first network device, when performing data synchronization between the first group and the second group, the source node and the destination node of the fifth message and the sixth message are both connected to the same first network device, that is, the messages for data synchronization between the groups are all transmitted in the same network device, that is, no message transmission across network devices is generated, thereby reducing traffic contention across network devices during collective communication. Especially when the data center is a convergent network, for example, the first network device is connected, and the total effective uplink bandwidth of the first network device is less than the total effective downlink bandwidth of the first network device. Since the convergent network is prone to traffic contention, the above solution can reduce the traffic contention generated during the communication process. Figure 3 and Figure 4 From the communication process, we can see that compared with AllReduce based on the ring algorithm, AllReduce based on the HD algorithm has fewer steps (if the number of nodes is N, only 2*log2(N) steps are required to implement AllReduce), which saves data transmission time.

[0197] However, since the implementation of AllReduce based on the HD algorithm requires cross-node transmission (that is, data transmission between nodes needs to be carried out between two first network devices), the interconnection bandwidth requirements between nodes are relatively high, and the overall communication time is easily increased due to the bottleneck of link bandwidth (for example, the link bandwidth between the first network device and the second network device).

[0198] For example, if Figure 1The total number of ports of the first network device in is 48, the number of nodes connected to the first network device through the downlink port is 32, and the number of second network devices connected to the first network device through the uplink port is 16. The second network device with a total number of ports of 32 and the first network device with a total number of ports of 48 can form a computing cluster of 1024 nodes (such as servers); if all nodes connected to the first network device perform data transmission across the first network device, for example, all nodes connected to the first network device 1 perform one-to-one communication with all nodes connected to the first network device 2, then the average real-time bandwidth can only reach 1 / 2 of the maximum bandwidth at most. The converged bandwidth between the first network device and the second network device has become a shortcoming that restricts the efficiency of AllReduce collective communication. This is because an AllReduce task contains multiple point-to-point communications, and the completion of AllReduce requires that all point-to-point communications are successfully completed. Therefore, there is a "barrel effect" in the collective communication at the bandwidth convergence point between the first network device and the second exchange, that is, the completion time of each AllReduce collective communication is determined by the slowest point-to-point communication time. If the convergence ratio of the data center network is 2:1, the AllReduce collective communication time will at least double.

[0199] The AllReduce executed by the first group of nodes and the second group of nodes in the above S501 is ReduceScatter and AllReduce based on the HD algorithm. The nodes in the first group execute ReduceScatter and AllReduce based on the HD algorithm of the first group in sequence according to the first operation type, and the nodes in the second group execute ReduceScatter and AllReduce based on the HD algorithm of the second group in sequence according to the second operation type. The nodes in the first group and the nodes in the second group independently implement AllReduce, thereby realizing data synchronization of all nodes in the first group and data synchronization of all nodes in the second group; then, in S502, the nodes in the first group and the corresponding nodes in the second group perform Reduce to realize data synchronization between the nodes in the first group and the node groups in the second group. The above steps are coordinated to realize data synchronization of all nodes in the data center network, and the data synchronization process of all the above nodes only requires 2*log2(N)+1 steps (N is the number of nodes), which is easy to implement and has fewer implementation steps.

[0200] The following is an explanation with reference to a schematic diagram of a scenario in which the communication method is applied.

[0201] Please also see Fig. 6A , Figure 6B and Figure 6C , Fig. 6A , Figure 6B and Figure 6CA schematic diagram of a communication method provided in an embodiment of the present application, which can be applied to Figure 1 The data center network shown. The data center network is a two-layer fat tree topology structure and includes 16 nodes. Among the 16 nodes, nodes 1 to 8 are all connected to the first network device 1, and nodes 9 to 16 are all connected to the first network device 2; all nodes are divided into two groups of equal number: [1, 2, 3, 4, 9, 10, 11, 12], [5, 6, 7, 8, 13, 14, 15, 16], so that among all nodes connected to each first network device: half of the number of nodes are in the first group, and the other half of the number of nodes are in the second group; then the nodes in the two groups execute AllReduce based on the HD algorithm according to different rules.

[0202] Optionally, obtain topology information of the data center network, the topology information including connection information between the node and the first network device, divide all nodes into a first group and a second group based on the topology information, and then send configuration information to the nodes of the data center to configure the nodes corresponding to the group to which each node belongs.

[0203] in, Fig. 6A , Figure 6B and Figure 6C The AllReduce executed by the middle node is based on the ReduceScatter and AllReduce of the HD algorithm.

[0204] Fig. 6A A schematic diagram of implementing ReduceScatter based on the HD algorithm for a first group of internal nodes and a second group of internal nodes provided in an embodiment of the present application;

[0205] First, the nodes in the first group and the nodes in the second group are numbered [1, 2, 3, 4, 5, 6, 7, 8] respectively, and then the difference in node numbers is used as the distance between nodes. Then, the nodes in the first group are ReduceScattered based on the HD algorithm from far to near, and the nodes in the second group are ReduceScattered based on the HD algorithm from near to far.

[0206] Among them, the distance in from near to far and from far to near here refers to the distance between the nodes in the first group and the second group. The distance can be the absolute distance between the two nodes, or the relative distance, or the number difference between the nodes in the group after sorting the nodes in the group based on preset rules as the distance between the nodes. For example, the larger the number difference, the larger (farther) the corresponding distance between the two nodes, and the smaller the number difference, the smaller (closer) the corresponding distance between the two nodes.

[0207] See also Fig. 6A, the number of nodes in the first group is 8, then the nodes in the first group implement ReduceScatter based on the HD algorithm including log2(8) steps, that is, 3 steps; the number of nodes in the second group is 8, then the nodes in the second group implement ReduceScatter based on the HD algorithm including log2(8) steps, that is, 3 steps.

[0208] Step 1: In the first group, nodes 1 and 9, 2 and 10, 3 and 11, and 4 and 12 send data blocks of 1 / 2 of their respective data sizes to each other. The 8 nodes in the first group perform Reduce on the received data blocks and their own data blocks. In the second group, nodes 5 and 6, 7 and 8, 13 and 14, and 15 and 16 send data blocks of 1 / 8 of their respective data sizes to each other. The 8 nodes in the second group perform Reduce on the received data blocks and their own data blocks.

[0209] Step 2: In the first group, nodes 1 and 3, 2 and 4, 9 and 11, and 10 and 12 send data blocks of 1 / 4 of their respective data sizes to each other. The 8 nodes in the first group perform Reduce on the received data blocks and their own data blocks. In the second group, nodes 5 and 7, 6 and 8, 13 and 15, and 14 and 16 send data blocks of 1 / 4 of their respective data sizes to each other. The 8 nodes in the second group perform Reduce on the received data blocks and their own data blocks.

[0210] Step 3: In the first group, nodes 1 and 2, 3 and 4, 9 and 10, 11 and 12 send data blocks of 1 / 8 of their respective data sizes to each other, and the 8 nodes in the first group perform Reduce on the received data blocks and their own data blocks. In the second group, nodes 5 and 13, 6 and 14, 7 and 15, 8 and 16 send data blocks of 1 / 2 of their respective data sizes to each other, and the 8 nodes in the second group perform Reduce on the received data blocks and their own data blocks.

[0211] In this way, through the above steps, the nodes in the first group and the nodes in the second group each implement ReduceScatter based on the HD algorithm of the nodes in the group.

[0212] See also Figure 6B , Figure 6B A schematic diagram of inter-group data synchronization between nodes in a first group and nodes in a second group provided in an embodiment of the present application.

[0213] After the nodes in the first group and the nodes in the second group independently implement ReduceScatter based on the HD algorithm, the nodes in the first group and the corresponding nodes in the second group send their own data to each other, and each node performs Reduce based on the received data and its own data to achieve synchronization of the data of the nodes in the first group and the data of the nodes in the second group.

[0214] Figure 6B The nodes in the first group and the corresponding nodes in the second group form node pairs: 1 and 5, 2 and 6, 3 and 7, 4 and 8, 9 and 13, 10 and 14, 11 and 15, 12 and 16. The node pairs send all the data of their own nodes to each other. For example, node 1 sends all its own data to node 5, and node 5 sends all its own data to node 1. Node 1 and node 5 perform Reduce on the received data and the data stored by themselves to achieve data synchronization between the nodes in the first group and the nodes in the second group.

[0215] See also Figure 6C , Figure 6C A schematic diagram of a first group of internal nodes and a second group of internal nodes implementing AllGather based on the HD algorithm respectively provided in an embodiment of the present application.

[0216] The nodes in the first group implement AllGather based on the HD algorithm from near to far, and the nodes in the second group implement AllGather based on the HD algorithm from far to near.

[0217] because Figure 6C If the number of nodes in the first group is 8, then the implementation of AllGather based on the HD algorithm by the nodes in the first group includes log2(8) steps, that is, 3 steps; if the number of nodes in the second group is 8, then the implementation of AllGather based on the HD algorithm by the nodes in the second group includes log2(8) steps, that is, 3 steps.

[0218] Step 1: In the first group, nodes 1 and 2, 3 and 4, 9 and 10, 11 and 12 send data blocks of 1 / 8 of their respective data sizes to each other. The 8 nodes in the first group perform Reduce on the received data blocks and their own data blocks. In the second group, nodes 5 and 13, 6 and 14, 7 and 15, 8 and 16 send data blocks of 1 / 2 of their respective data sizes to each other. The 8 nodes in the second group perform Reduce on the received data blocks and their own data blocks.

[0219] Step 2: In the first group, nodes 1 and 3, 2 and 4, 9 and 11, and 10 and 12 send data blocks of 1 / 4 of their respective data sizes to each other. The 8 nodes in the first group perform Reduce on the received data blocks and their own data blocks. In the second group, nodes 5 and 7, 6 and 8, 13 and 15, and 14 and 16 send data blocks of 1 / 4 of their respective data sizes to each other. The 8 nodes in the second group perform Reduce on the received data blocks and their own data blocks.

[0220] Step 3: In the first group, nodes 1 and 9, 2 and 10, 3 and 11, and 4 and 12 send data blocks of 1 / 2 of their respective data sizes to each other, and the 8 nodes in the first group perform Reduce on the received data blocks and their own data blocks. In the second group, nodes 5 and 6, 7 and 8, 13 and 14, and 15 and 16 send data blocks of 1 / 8 of their respective data sizes to each other, and the 8 nodes in the second group perform Reduce on the received data blocks and their own data blocks.

[0221] pass Fig. 6A , Figure 6B , Figure 6C This is to achieve AllReduce of the nodes in the data network center, that is, to achieve data synchronization of all nodes in the data center network.

[0222] Then S501 includes: the nodes in the first group sequentially execute ReduceScatter and AllGather based on the HD algorithm in the first group according to the first operation type, and the nodes in the second group sequentially execute ReduceScatter and AllGather based on the HD algorithm in the second group according to the second operation type, and the first operation type is different from the second operation type.

[0223] Among them, since AllReduce includes two processes: ReduceScatter and AllGather, the nodes in the two groups execute ReduceScatter and AllGather based on the HD algorithm in sequence, so that the nodes in the first group and the nodes in the second group execute AllReduce of the corresponding nodes in the group, thereby realizing data synchronization of all nodes in the first group and data synchronization of all nodes in the second group; since the two groups execute AllReduce according to different rules, data synchronization of all nodes in the first group and data synchronization of all nodes in the second group can be realized.

[0224] Among them, since AllReduce includes two processes: ReduceScatter and AllGather, the nodes in the two groups execute ReduceScatter and AllGather based on the HD algorithm in sequence, so that the nodes in the first group and the nodes in the second group execute AllReduce of the corresponding nodes in the group, thereby realizing data synchronization of all nodes in the first group and all nodes in the second group; since the two groups execute AllReduce according to different operation types, and each group has at least two nodes connected to the same first network device, in each step of the two groups executing AllReduce, there are at least two nodes whose communication is executed inside the same first network device, thereby avoiding all data traffic from crossing the first network device (that is, transmitting communication data through the link between the first network device and the second switch), thereby avoiding link contention between the first network device and the second network device of the converged network.

[0225] For example, see Fig. 6A , since nodes 1 to 8 are connected to the first network device 1, and nodes 9 to 16 are connected to the first network device 2, Fig. 6A In step 1, the collective communication of all nodes in the first group is performed across the first network device 1 and the first network device 2, and the collective communication of all nodes in the second group is performed within the first network device 1 and the second network device 2; Fig. 6A In step 2, collective communication of all nodes in the first group and the second group is performed in the first network device 1 and the second network device 2; Fig. 6A In step 3, the collective communication of all nodes in the first group is performed within the first network device 1 and the second network device 2, and the collective communication of all nodes in the second group is performed across the first network device 1 and the first network device 2. That is, in each step of AllReduce, Fig. 6A In the communication between all nodes, at least half of the data traffic is carried out inside the first network device, thereby avoiding traffic contention between the first network device and the second network device.

[0226] In this way, before the nodes of the data center network perform collective communication, all nodes are divided into two groups, and the two groups execute AllReduce according to different rules to achieve data synchronization of all nodes in the first group and data synchronization of all nodes in the second group. The above communication method only takes 2*log2(N)+1 steps to complete. Compared with the ring-based AllReduce, the data synchronization of all nodes in the data center network in this application requires fewer steps, saving communication time.

[0227] In an optional implementation, the data center network also includes at least one second network device, each first network device has an uplink port and a downlink port, the number of downlink ports is greater than the number of uplink ports, the node is connected to the first network device via the downlink port, and at least one second network device is connected to the first network device via the uplink port. Since the number of downlink ports of the first network device is greater than the number of uplink ports, and the uplink effective bandwidth of the first network device is less than the downlink effective bandwidth, the data center network is a convergence network. Since the collective communication of the convergence network has a "barrel effect" at the point where the bandwidth of the first network device and the second exchange converge, that is, the completion time of each AllReduce collective communication is determined by the slowest point-to-point communication time. If the convergence ratio of the data center network is 2:1, the AllReduce collective communication time is at least doubled. In this way, before the nodes of the data center network perform collective communication, all the nodes are divided into two groups, and at least two nodes connected to each first network device are located in one of the groups. The two groups execute AllReduce according to different rules, so that in each step of the two groups executing AllReduce, the communication between at least two nodes in one group is carried out within the same first network device (that is, the communication between the nodes does not pass through the link between the first network device and the second network device), thereby avoiding that all data traffic crosses the first network device, and further avoiding link contention between the first network device and the second network device of the converged network, thereby reducing the overall communication time.

[0228] In some embodiments of the first aspect, the total number of nodes in the first group is the same as the total number of nodes in the second group, wherein, among all nodes connected to each of the first network devices: the number of nodes divided into the first group or the second group is at least two. In this way, by dividing all nodes of the data center network into a first group and a second group with the same number of nodes, at least two nodes connected to each first network device are divided into the first group, and at least two nodes connected to each first network device are divided into the second group, and then the nodes in the first group sequentially execute ReduceScatter and AllReduce based on the HD algorithm for all nodes in the first group according to the first rule, and the nodes in the second group sequentially execute ReduceScatter and AllReduce based on the HD algorithm for all nodes in the second group according to the second rule, and the nodes in the first group and the nodes in the second group independently implement AllReduce, thereby achieving data synchronization of all nodes in the first group and all nodes in the second group, and During the AllReduce process of the nodes in the first group and the nodes in the second group, each node in the first group and the corresponding node in the second group achieve data synchronization of the node data of the first group and the node data of the second group by sending data to each other. The above steps cooperate to achieve data synchronization of all nodes in the data center network, and the nodes in the first group and the second group independently execute AllReduce according to different rules, which avoids that during the communication process of AllReduce, all communication traffic is transmitted across the first network device, thereby causing the link between the first network device and the second network device in a converged network (the number of uplink ports of the first network device is less than the number of downlink ports) to form link contention, which increases the communication time of the data center network due to the link bandwidth bottleneck. In addition, the data synchronization process of all the nodes is easy to implement and has fewer implementation steps.

[0229] For example, see Figure 1 , Figure 1 The total number of nodes is 8, and the 8 nodes are divided into two groups. The number of nodes in the first group and the second group are four, and the nodes in the first group are: node 1, node 3, node 5, node 7; the nodes in the second group are: node 2, node 4, node 6, node 8. Among the four nodes connected to the first network device 1, two nodes are divided into the first group, and the other two nodes are divided into the second group.

[0230] Optionally, since the number of nodes in the first group is the same as the number of nodes in the second group, each node in the first group has a unique corresponding node in the second group. The corresponding node of each node can be configured by the user or randomly assigned according to the scenario, as long as there is a one-to-one correspondence between each node in the first group and its corresponding node in the second group.

[0231] In an optional implementation, when executing Figure 5 Before the steps shown, all nodes of the data center network need to be grouped so that half of all nodes connected to each first network device are divided into the first group and the other half are divided into the second group. Then, AllReduce is executed in the nodes of the two groups according to different rules, so that at least half of all data traffic in the communication between the two groups is carried out in the first network device, which can further avoid the link contention between the first network device and the second network device of the converged network, thereby reducing the overall communication time.

[0232] For example, please refer again to Fig. 6A , Fig. 6A In the three steps, at least half of the data traffic in each step is performed in the first network device 1 or the first network device 2, thereby avoiding link contention caused by all communication data traffic passing through the link between the first network device and the second network device during the AllReduce process.

[0233] In an optional implementation, when executing Figure 5 Before the steps shown, all nodes in the data center network need to be grouped. All nodes connected to each first network device can be numbered in the order of the downstream ports to which the nodes are connected, and all nodes can be grouped by number size. For example, the nodes corresponding to the first half of the number are divided into the first group, and the nodes corresponding to the second half of the number are divided into the second group.

[0234] Exemplarily, the nodes connected to the first network device are numbered N0 to N7, and N0 to N3 are classified into the first group, and N4 to N7 are classified into the second group.

[0235] Optionally, nodes connected to the downlink ports at odd positions are divided into the first group, and nodes connected to the downlink ports at even positions are divided into the second group; it can be understood that all nodes connected to each first network device can also be grouped according to other rules, as long as half of all the nodes connected to the first network device can be divided into the first group and the other half into the second group.

[0236] Optionally, before grouping all nodes of the data center network, topology information of the data center network may be obtained, and then all nodes of the data center network may be grouped according to the topology information. The topology information includes connection information, and the connection information includes a connection relationship between the node and the first network device. For example, Figure 1 The first network device 1 is connected to the node 1 and the node 2 through the downlink port respectively;

[0237] Furthermore, the topology information also includes link bandwidth information, which is the bandwidth of the link between the node and the first network device, and the bandwidth between the first network device and the second network device, such as the bandwidth of the link between node 1 and the first network device 1, and the bandwidth of the link between the first network device 1 and the second network device 1. The link contention between the first network device and the second network device is predicted based on the bandwidth information. The bandwidth information of all links in the data center network can be detected by a bandwidth detector.

[0238] In an optional implementation, the first rule and the second rule are opposite, that is, the first rule is the inverse rule of the second rule, for example, the first rule is from far to near, and the second rule is from near to far, where far and near refer to the distance between nodes, and the nodes in the first group and the nodes in the second group can be numbered respectively, and the difference in the numbers corresponding to the nodes is the distance between the nodes; the nodes in the first group execute ReduceScatter and AllGather based on the HD algorithm for all nodes in the first group in sequence according to the first rule, including: the nodes in the first group execute ReduceScatter based on the HD algorithm from far to near and execute AllGather based on the HD algorithm from near to far, and the nodes in the second group execute ReduceScatter and AllGather based on the HD algorithm for all nodes in the second group in sequence according to the second rule, including: the nodes in the second group execute ReduceScatter based on the HD algorithm from near to far and the nodes execute AllGather based on the HD algorithm from far to near;

[0239] For another example, if the first rule is from large to small, then the second rule is from small to large, where large and small here may refer to the size of the node numbers in the group, or the size of the distance, etc.; then the nodes in the first group execute ReduceScatter and AllGather based on the HD algorithm for all the nodes in the first group in sequence according to the first rule, including: the nodes in the first group execute ReduceScatter based on the HD algorithm from large to small and execute AllGather based on the HD algorithm from small to large, and the nodes in the second group execute ReduceScatter and AllGather based on the HD algorithm for all the nodes in the second group in sequence according to the second rule, including: the nodes execute ReduceScatter based on the HD algorithm from small to large and the nodes execute AllGather based on the HD algorithm from large to small.

[0240] Since the first rule and the second rule are opposite, the first group of nodes and the second group of nodes execute ReduceScatter and AllGather based on the HD algorithm according to different rules, so as to avoid that all traffic involved in the communication method is transmitted across the first network device. Fig. 6A When the nodes in the first group execute ReduceScatter based on the HD algorithm from far to near, the nodes in the second group execute ReduceScatter based on the HD algorithm from near to far. The communication between the two farthest nodes is transmitted across the first network device, and the communication between the two closest nodes is transmitted across the first network device, thereby avoiding that all traffic is transmitted across the first network device.

[0241] It can be understood that in S501, the nodes in the first group and the nodes in the second group implement AllReduce based on the HD algorithm according to different rules, which can prevent all traffic in the collective communication process of the data center network from crossing the first network device, thereby avoiding traffic contention between the first network device and the second network device and causing an increase in communication time.

[0242] Since the first group of nodes and the second group of nodes implement AllReduce based on the HD algorithm based on different rules, for example Fig. 6A In the embodiment, the nodes in the first group implement ReduceScatter based on the HD algorithm from far to near, and the nodes in the second group implement ReduceScatter based on the HD algorithm from near to far, so that in each operation process, at least half of the data traffic is carried out inside the first network device, thereby avoiding traffic contention between the first network device and the second network device. Then S501, since the nodes in the first group and the nodes in the second group execute AllReduce based on the HD algorithm including 2log2(N) steps, the ReduceScatter executed by the nodes in the first group and the nodes in the second group both include log2(N) steps, and the AllGather executed by the nodes in the first group and the nodes in the second group both include log2(N) steps, then the number of steps of AllReduce based on the HD algorithm executed by the nodes in the first group is the same as the number of steps of AllReduce based on the HD algorithm executed by the nodes in the second group, and the same number of steps are carried out simultaneously, so as to achieve that at least half of the data traffic is carried out inside the first network device, thereby avoiding traffic contention between the first network device and the second network device.

[0243] In an optional implementation, in S502, the node data of the first group and the corresponding nodes of the second group are synchronized by sending data to each other, then each node in the first group and the corresponding node in the second group are connected to the same first network device, then when data is transmitted between the node pairs in the two groups, it is carried out in the first network device, and the data transmission does not cross the first network device, so as to avoid link contention caused by the link between the first network device and the second network device during the data transmission process.

[0244] For example, see Figure 6B , Figure 6B The node pairs are: 1 and 5, 2 and 6, 3 and 7, 4 and 8, 9 and 13, 10 and 14, 11 and 15, 12 and 16. The node pairs send all data to each other to synchronize the node data in the first group and the node data in the second group. Since nodes 1 to 8 are all connected to the first network device 1, and nodes 9 to 16 are all connected to the first network device 2, that is, Figure 6B The communication processes are all executed in the first network device 1 and the first network device 2, without passing through the link between the first network device and the second network device, thereby avoiding link contention between the first network device and the second network device during data transmission.

[0245] See also Figure 7 , Figure 7 A flow chart of a communication method provided in an embodiment of the present application, the communication method is applied to a first network device of a data center network, the data center network also includes a node connected to the first network device, all nodes of the data center network include at least one node of the first group and at least one node of the second group, the method includes:

[0246] S701, receiving a first message sent by a first node in the first group, and sending the first message to a second node in the first group, receiving a second message sent by the second node in the first group, and sending the second message to the first node in the first group, so that the nodes in the first group execute the global AllReduce in the first group according to a first operation type, where the first operation type is used to instruct the nodes in the first group to execute the communication strategy of AllReduce;

[0247] S702, receiving a third message sent by a third node in the second group, and sending the third message to a fourth node in the second group, receiving a fourth message sent by the fourth node in the second group, and sending the fourth message to the third node in the second group, so that the nodes in the second group execute the global AllReduce in the first group according to a second operation type, where the second operation type is used to instruct the nodes in the second group to execute the communication strategy of AllReduce;

[0248] S703, receiving the fifth message sent by the node in the first group, and sending the fifth message to the node in the second group; receiving the sixth message sent by the node in the second group, and sending the sixth message to the node in the first group, so that the node in the first group and the node in the second group perform Reduce.

[0249] Optionally, before the nodes in the first group and the nodes in the second group perform Reduce, the method further includes:

[0250] Determine whether the nodes in the first group and the nodes in the second group both execute a preset target number of steps of AllReduce;

[0251] If both the nodes in the first group and the nodes in the second group execute the preset target number of steps of AllReduce, the fifth message and the sixth message are forwarded to enable the nodes in the first group and the nodes in the second group to perform Reduce.

[0252] Optionally, the first message carries a first source address and a first destination address, the second message carries a second source address and a second destination address, the first destination address and the second source address are addresses of a first node in the first group, and the second destination address and the first source address are addresses of a second node in the first group;

[0253] The third message carries a third source address and a third destination address, the fourth message carries a fourth source address and a fourth destination address, the fourth destination address and the third source address are the addresses of the third node in the second group, and the third destination address and the fourth source address are the addresses of the fourth node in the second group.

[0254] Optionally, the fifth message carries a fifth source address and a fifth destination address, the sixth message carries a sixth source address and a sixth destination address, the fifth destination address is the address of the current node of the first group, the fifth source address is the address of a node in the second group, the sixth destination address is the address of the current node in the second group, and the sixth source address is the address of a node in the first group.

[0255] Optionally, the first message, the second message, the third message and the fourth message all carry a collective communication type, and the collective communication type is AllReduce.

[0256] Optionally, the first message and the second message both carry a first collective communication group identifier, and the first collective communication group identifier is used to indicate that the source node and the destination node corresponding to the first message or the second message both belong to the first group, and the third message and the fourth message both carry a second collective communication group identifier, and the second collective communication group identifier is used to indicate that the source node and the destination node corresponding to the third message or the fourth message both belong to the second group.

[0257] Optionally, the first message and the second message both carry the first operation type, and the third message and the fourth message both carry the second operation type.

[0258] The communication method provided by the present application divides all the nodes of the data center network into two groups, and the nodes in the two groups independently execute the ReduceScatter process and the AllGather process based on the HD algorithm. After executing the ReduceScatter process, the data of the nodes in the first group is exchanged with the data of the nodes in the second group. The AllReduce process based on the HD algorithm is implemented through the above process. Since the nodes in the two groups execute the ReduceScatter process and the AllGather process, data exchange across the first network device is avoided in the above process, thereby avoiding link contention between the first network device and the second network device, which causes the data center network to increase the communication time due to the link bandwidth bottleneck. The above AllReduce process requires 2*log2(N)+1 steps (the number of nodes is 2N), which is easy to implement and has fewer implementation steps.

[0259] It should be understood that the above is only to help those skilled in the art better understand the embodiments of the present application, rather than to limit the scope of the embodiments of the present application. According to the above examples given, those skilled in the art can obviously make various equivalent modifications or changes, for example, some steps in each method of the above method may not be necessary, or some new steps may be added. Or a combination of any two or any multiple embodiments of the above. Such modifications, changes or combined solutions also fall within the scope of the embodiments of the present application.

[0260] It should also be understood that the division of the methods, situations, categories and embodiments in the embodiments of the present application is only for the convenience of description and should not constitute a special limitation. The features of various methods, categories, situations and embodiments can be combined without contradiction.

[0261] It should also be understood that the various numerical numbers involved in the embodiments of the present application are only for the convenience of description and are not used to limit the scope of the embodiments of the present application. The size of the sequence number of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0262] It should also be understood that the above description of the embodiments of the present application focuses on emphasizing the differences between the various embodiments. The same or similar points that are not mentioned can be referenced to each other. For the sake of brevity, they will not be repeated here.

[0263] The above combination Figure 1-Figure 7 The method and data center network embodiments provided by the embodiments of the present application are described, and the communication device provided by the embodiments of the present application is described below.

[0264] In this embodiment, the above-mentioned communication device can be divided into functional modules according to the above-mentioned method. For example, it can be divided into various functional modules corresponding to various functions, or two or more functions can be integrated into one processing module. The above-mentioned integrated module can be implemented in the form of hardware. It should be noted that the division of modules in this embodiment is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0265] It should be noted that the relevant contents of each step involved in the above method embodiment can all be referred to the functional description of the corresponding functional module, which will not be repeated here.

[0266] The communication device provided in the embodiment of the present application is used to execute the communication method provided in the above method embodiment, so the same effect as the above implementation method can be achieved. In the case of an integrated unit, the node may include a processing module, a storage module and a communication module. Among them, the processing module can be used to control and manage the actions of the node. For example, it can be used to support the node to execute the steps performed by the processing unit. The storage module can be used to support the storage of program codes and data, etc. The communication module can be used to support the communication of the communication device with other communication devices.

[0267] Among them, the processing module can be a processor or a controller. It can be a device that implements or executes various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. The processor can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and a microprocessor, and so on. The storage module can be a memory. The communication module can specifically be a device that interacts with other communication devices, such as a radio frequency circuit, a Bluetooth chip, a Wi-Fi chip, etc.

[0268] See also Figure 8 , Figure 8 FIG. 1 shows a schematic diagram of the structure of an exemplary communication device of the present application. Figure 8 The communication device shown can execute the steps in the communication method executed by any communication device provided in the embodiments of the present application, wherein the communication device can be the node or network device described in the above embodiments.

[0269] The communication device 800 includes at least one processor 801 , a memory 803 , and at least one network interface 804 .

[0270] The processor 801 is, for example, a general-purpose CPU, a digital signal processor (DSP), a network processor (NP), a GPU, a neural network processor (NPU), a data processing unit (DPU), a microprocessor, or one or more integrated circuits or application-specific integrated circuits (ASIC) for implementing the solution of the present application, a programmable logic device (PLD) or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. The PLD is, for example, a complex programmable logic device (CPLD), a field programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof. It can implement or execute various logic blocks, modules, and circuits described in conjunction with the disclosure of the present application. The processor can also be a combination that implements a computing function, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like.

[0271] Optionally, the communication device 800 further includes a bus 802. The bus 802 is used to transmit information between the components of the communication device 800. The bus 802 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus 802 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0272] The memory 803 is, for example, a read-only memory (ROM) or other types of storage devices that can store static information and instructions, or a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 803 is, for example, independent and connected to the processor 801 via the bus 802. The memory 803 can also be integrated with the processor 801.

[0273] The network interface 804 uses any transceiver-like device to communicate with other devices or communication networks, and the communication network can be Ethernet, radio access network (RAN) or wireless local area network (WLAN), etc. The network interface 804 may include a wired network interface and may also include a wireless network interface. Specifically, the network interface 804 may be an Ethernet interface, such as a fast Ethernet (FE) interface, a gigabit Ethernet (GE) interface, an asynchronous transfer mode (ATM) interface, a WLAN interface, a cellular network interface or a combination thereof. The Ethernet interface may be an optical interface, an electrical interface or a combination thereof. In some embodiments of the present application, the network interface 804 may be used for the communication device 800 to communicate with other devices.

[0274] In a specific implementation, as some embodiments, the processor 801 may include one or more CPUs. Each of these processors may be a single-core processor or a multi-core processor. The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0275] In a specific implementation, as some embodiments, the communication device 800 may include multiple processors. Each of these processors may be a single-core processor or a multi-core processor. The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (such as computer program instructions).

[0276] In some implementations, the memory 803 is used to store program instructions for executing the solution of the present application, and the processor 801 can execute the program instructions stored in the memory 803. That is, the communication device 800 can implement the method provided by the method embodiment shown in the above embodiment through the processor 801 and the program instructions in the memory 803. The program instructions may include one or more software modules. Optionally, the processor 801 itself may also store program instructions for executing the solution of the present application.

[0277] In the specific implementation process, the processor 801 in the communication device 800 of the present application reads the instructions in the memory 803, so that Figure 8 The communication device 800 shown can execute all or part of the steps in the communication method executed by the communication device in the above embodiment.

[0278] Among them, each step of the method described in the above embodiment is completed by the hardware integrated logic circuit or software instruction in the processor of the communication device 800. The steps of the method embodiment disclosed in the present application can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor. The software module can be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method embodiment in combination with its hardware. To avoid repetition, it is not described in detail here.

[0279] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. It is worth noting that the processor may be a processor supporting the advanced RISC machines (ARM) architecture.

[0280] Further, in an optional embodiment, the memory may include a read-only memory and a random access memory, and provide instructions and data to the processor. The memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type.

[0281] The memory may be a volatile memory or a nonvolatile memory, or may include both volatile and nonvolatile memory. Among them, the nonvolatile memory may be a read only memory (ROM), a programmable read only memory (PROM), an erasable programmable read only memory (EPROM), an electrically erasable programmable read only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available. For example, static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM) and direct memory bus random access memory (DR RAM).

[0282] An embodiment of the present application further provides a data center network, including a communication device, wherein the communication device can execute the steps in the communication method executed by any communication device provided in the embodiment of the present application.

[0283] An embodiment of the present application provides a computer program (product), which includes: computer program code, when the computer program code is executed by a computer, the computer executes the steps in the communication method performed by any communication device provided in the embodiment of the present application.

[0284] An embodiment of the present application provides a computer-readable storage medium, which stores a program or instruction. When the program or instruction runs on a computer, the communication method performed by any communication device provided in the embodiment of the present application is executed.

[0285] An embodiment of the present application provides a chip, including a processor, for calling and executing instructions stored in a memory from the memory, so that a communication device equipped with the chip executes a communication method executed by any communication device provided in the embodiment of the present application.

[0286] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in this application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integration. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk), etc.

[0287] In this application, the terms "first", "second", etc. are used to distinguish between identical or similar items with substantially the same effects and functions. It should be understood that there is no logical or temporal dependency between "first", "second", and "nth", nor is there a limitation on the quantity and execution order. It should also be understood that although the following description uses the terms first, second, etc. to describe various elements, these elements should not be limited by the terms. These terms are only used to distinguish one element from another element.

[0288] It should also be understood that in the various embodiments of the present application, the size of the serial number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0289] The term "at least one" in this application means one or more, and the term "multiple" in this application means two or more, for example, multiple second devices means two or more second devices. The terms "system" and "network" are often used interchangeably herein.

[0290] It should be understood that the terms used in the description of the various examples herein are only for describing specific examples and are not intended to be limiting. As used in the description of the various examples and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0291] It should also be understood that the term "and / or" used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. The term "and / or" is a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this application generally indicates that the associated objects before and after are in an "or" relationship.

[0292] It should also be understood that the terms "if" and "if" may be interpreted to mean "when" or "upon" or "in response to determining" or "in response to detecting." Similarly, the phrases "if it is determined that ..." or "if [a stated condition or event] is detected" may be interpreted to mean "upon determining that ..." or "in response to determining that ..." or "upon detecting [a stated condition or event]" or "in response to detecting [a stated condition or event]," depending on the context.

[0293] The above description is only an embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present application shall be included in the protection scope of the present application.

Claims

1. A communication method, characterized in that: Applied to a node of a data center network, wherein all nodes of the data center network include at least one node of a first group and at least one node of a second group; the method comprises: The nodes of the first group execute the global All Reduce in the first group according to the first operation type; the nodes in the second group execute the All Reduce in the second group according to the second operation type, the first operation type is used to instruct the nodes in the first group to execute the communication strategy of AllReduce, and the second operation type is used to instruct the nodes in the second group to execute the communication strategy of AllReduce; The nodes in the first group perform Reduce with the nodes in the second group.

2. The method according to claim 1, characterized in that The data center network further includes at least one first network device, and the node is connected to the first network device; The nodes of the first group execute the global All Reduce in the first group according to the first operation type; The nodes in the second group perform All Reduce in the second group according to the second operation type, including: The first node in the first group sends a first message to the second node in the first group through the first network device, and receives a second message sent by the second node in the first group through the first network device, so that the nodes in the first group perform All Reduce in the first group according to the first operation type; The third node in the second group sends a third message to the fourth node in the second group through the first network device, and receives a fourth message sent by the third node in the second group through the first network device, so that the nodes in the second group perform All Reduce in the second group according to the second operation type.

3. The method according to claim 2, characterized in that The nodes in the first group and the nodes in the second group perform Reduce, including: The nodes in the first group receive the fifth message sent by the corresponding nodes in the second group through the first network device, and perform Reduce according to the data of the fifth message. The nodes in the second group receive the sixth message sent by the corresponding nodes in the first group through the first network device, and perform Reduce according to the data of the sixth message.

4. The method according to any one of claims 1 to 3, characterized in that Before the nodes in the first group and the nodes in the second group perform Reduce, the method further includes: The nodes within the first group and the nodes within the second group execute a preset target number of steps of All Reduce.

5. The method according to claim 2, characterized in that: The first message carries a first source address and a first destination address, the second message carries a second source address and a second destination address, the first destination address and the second source address are both addresses of a first node in the first group, and the second destination address and the first source address are both addresses of a second node in the first group; The third message carries a third source address and a third destination address, the fourth message carries a fourth source address and a fourth destination address, the fourth destination address and the third source address are both the address of the third node in the second group, and the third destination address and the fourth source address are both the address of the fourth node in the second group.

6. The method according to claim 3, characterized in that: The fifth message carries a fifth source address and a fifth destination address, the sixth message carries a sixth source address and a sixth destination address, the fifth destination address is the address of the current node of the first group, the fifth source address is the address of a node in the second group, the sixth destination address is the address of the current node in the second group, and the sixth source address is the address of a node in the first group.

7. The method according to claim 2, characterized in that The first message, the second message, the third message and the fourth message all carry a collective communication type, and the collective communication type is All Reduce.

8. The method according to claim 2, characterized in that: The first message and the second message both carry a first collective communication group identifier, and the first collective communication group identifier is used to indicate that the source node and the destination node corresponding to the first message or the second message both belong to the first group. The third message and the fourth message both carry a second collective communication group identifier, and the second collective communication group identifier is used to indicate that the source node and the destination node corresponding to the third message or the fourth message both belong to the second group.

9. The method according to claim 2, characterized in that: The first message and the second message both carry the first operation type, and the third message and the fourth message both carry the second operation type.

10. The method according to any one of claims 1 to 9, characterized in that The node of the data center network receives configuration information, where the configuration information includes an address of a node in the first group or an address of a node in the second group.

11. The method according to claim 10, characterized in that If the node of the data center network is a node in the first group, the configuration information also includes a first correspondence between a first operation type, an address of a node in the first group, and a first collective communication group number, the first collective communication group number being a sequence number corresponding to the node in the first group, and the node in the first group executing a corresponding communication strategy according to the first correspondence and the first operation type; If the node of the data center network is a node in the second group, the configuration information also includes a second correspondence between a second operation type, an address of the node in the second group, and a second collective communication group number, wherein the second collective communication group number is a serial number corresponding to the node in the second group, and the node in the second group executes a corresponding communication strategy based on the second correspondence and the second operation type.

12. The method according to claim 10 or 11, characterized in that: If the node of the data center network is a node in the first group, the configuration information further includes at least one of a first collective communication group identifier and a collective communication type, and the collective communication type is All Reduce; If the node of the data center network is a node in the second group, the configuration information further includes at least one of a second collective communication group identifier and a collective communication type.

13. The method according to claim 2 or 3, characterized in that: The method further comprises: The node in the first group receives the seventh message forwarded by the first network device, where the seventh message carries at least one of the first collective communication group identifier, the first operation type, the first correspondence between the address of the node in the first group and the first collective communication group number, and the collective communication type; The nodes in the second group receive the eighth message forwarded by the first network device, and the eighth message carries at least one of the second correspondence between the second collective communication group identifier, the second operation type, the address of the node in the second group, and the second collective communication group number.

14. The method according to any one of claims 1 to 13, characterized in that The All Reduce includes scatter reduction ReduceScatter and all gather All Gather; The first operation type includes the nodes in the first group executing ReduceScatter according to the first rule and executing All Gather according to the second rule; The second operation type includes the nodes in the second group executing ReduceScatter according to the second rule and executing All Gather according to the first rule; The first rule and the second rule are communication strategies adopted by the node when executing ReduceScatter and AllGather respectively.

15. The method according to claim 2 or 3, characterized in that: The data center network further includes at least one second network device, and the number of the first network devices is at least two; Each of the first network devices has an upstream port and a downstream port, the node is connected to the first network device via the downstream port, at least one of the second network devices is connected to the first network device via the upstream port, and the total effective upstream bandwidth of the first network device is less than the total effective downstream bandwidth of the first network device.

16. The method according to any one of claims 1 to 15, characterized in that The total number of nodes in the first group is the same as the total number of nodes in the second group.

17. The method according to claim 6, characterized in that The node corresponding to the fifth source address and the node corresponding to the fifth destination address are both connected to the same first network device; the node corresponding to the sixth source address and the node corresponding to the sixth destination address are both connected to the same first network device.

18. A communication method, characterized in that: A first network device applied to a data center network, the data center network further comprising a node connected to the first network device, all nodes of the data center network comprising at least one node of a first group and at least one node of a second group, the method comprising: receiving a first message sent by a first node in the first group, and sending the first message to a second node in the first group, receiving a second message sent by the second node in the first group, and sending the second message to the first node in the first group, so that the nodes in the first group execute the global All Reduce in the first group according to a first operation type, where the first operation type is used to instruct the nodes in the first group to execute the All Reduce communication strategy; receiving a third message sent by a third node in the second group, and sending the third message to a fourth node in the second group, receiving a fourth message sent by the fourth node in the second group, and sending the fourth message to the third node in the second group, so that the nodes in the second group execute the global All Reduce in the first group according to a second operation type, where the second operation type is used to instruct the nodes in the second group to execute the All Reduce communication strategy; Receive the fifth message sent by the node in the first group, and send the fifth message to the node in the second group; receive the sixth message sent by the node in the second group, and send the sixth message to the node in the first group, so that the node in the first group and the node in the second group perform protocol Reduce.

19. The method according to claim 18, characterized in that Before the nodes in the first group and the nodes in the second group perform Reduce, the method further includes: Determine whether the nodes in the first group and the nodes in the second group both execute a preset target number of All Reduce steps; If both the nodes in the first group and the nodes in the second group perform the preset target number of steps of All Reduce, the fifth message and the sixth message are forwarded to enable the nodes in the first group and the nodes in the second group to perform Reduce.

20. The method according to claim 18 or 19, characterized in that The first message carries a first source address and a first destination address, the second message carries a second source address and a second destination address, the first destination address and the second source address are addresses of a first node in the first group, and the second destination address and the first source address are addresses of a second node in the first group; The third message carries a third source address and a third destination address, the fourth message carries a fourth source address and a fourth destination address, the fourth destination address and the third source address are the addresses of the third node in the second group, and the third destination address and the fourth source address are the addresses of the fourth node in the second group.

21. The method according to any one of claims 18 to 20, characterized in that The fifth message carries a fifth source address and a fifth destination address, the sixth message carries a sixth source address and a sixth destination address, the fifth destination address is the address of the current node of the first group, the fifth source address is the address of a node in the second group, the sixth destination address is the address of the current node in the second group, and the sixth source address is the address of a node in the first group.

22. The method according to any one of claims 18 to 21, characterized in that The first message, the second message, the third message and the fourth message all carry a collective communication type, and the collective communication type is AllReduce.

23. The method according to any one of claims 18 to 22, characterized in that The first message and the second message both carry a first collective communication group identifier, and the first collective communication group identifier is used to indicate that the source node and the destination node corresponding to the first message or the second message both belong to the first group. The third message and the fourth message both carry a second collective communication group identifier, and the second collective communication group identifier is used to indicate that the source node and the destination node corresponding to the third message or the fourth message both belong to the second group.

24. The method according to any one of claims 18 to 23, characterized in that The first message and the second message both carry the first operation type, and the third message and the fourth message both carry the second operation type.

25. A communication device, characterized in that: include: a memory including computer-readable instructions; A processor in communication with the memory, the processor being configured to execute the computer-readable instructions so that the communication device executes the communication method of any one of claims 1-17 or the communication method of any one of claims 18-24.

26. A computer-readable storage medium, characterized in that: The method comprises a program or an instruction, which, when executed by a processor, implements the communication method according to any one of claims 1 to 17 or the communication method according to any one of claims 18 to 24.

Images