Collective communication method and apparatus, computing cluster

Abstract

The application discloses a kind of collection communication method and device, computing cluster, belong to communication technical field.The computing cluster includes first computing group and second computing group, first computing node in first computing group obtains first data, and after first part data in first data is sent to second computing node in second computing group, second part data in first data is sent to third computing node in second computing group.The first data is the data obtained by executing first collection communication operation in first computing group, the first part data corresponds to second computing node, and the second part data corresponds to third computing node, and the first part data and the second part data are used to realize second collection communication operation executed between different computing groups in the computing cluster.This application can realize hierarchical all-reduce algorithm in asymmetric computing cluster, and application scene is wide.

Description

Technical Field

[0001] This application relates to the field of communication technology, and in particular to a collective communication method and apparatus, and a computing cluster. Background Technology

[0002] Collective communication (CC) is an algorithm used to transfer data between multiple computing nodes in a computing cluster. The all-reduce algorithm is a type of collective communication algorithm that performs a reduction operation on data across multiple computing nodes in the cluster and sends the reduction results back to those nodes, ensuring that all nodes have the same reduction result.

[0003] The all-reduce algorithm includes the hierarchical all-reduce algorithm, also known as the multi-level all-reduce algorithm. The hierarchical all-reduce algorithm generally includes three operations: a reduce-scatter operation performed within each computation group in the computing cluster; an all-reduce operation performed between computation nodes with the same number in different computation groups in the computing cluster (e.g., computation nodes with the same number are called homologous computation nodes); and an all-gather operation performed within each computation group in the computing cluster.

[0004] However, current hierarchical all-reduce algorithms are only applicable to symmetric computing clusters and not to asymmetric computing clusters, thus limiting their application scenarios. A symmetric computing cluster refers to a computing cluster where all computing groups have an equal number of computing nodes, while an asymmetric computing cluster refers to a computing cluster where at least two computing groups have unequal numbers of computing nodes. Summary of the Invention

[0005] This application provides a collective communication method and apparatus, and a computing cluster. The technical solution of this application is as follows.

[0006] Firstly, a set communication method is provided, applied to a computing cluster comprising n computing groups, each computing group comprising at least one computing node, where n is an integer greater than 1. The n computing groups include a first computing group and a second computing group. The method comprises: a first computing node within the first computing group acquiring first data, which is data corresponding to the first computing node obtained by performing a first set communication operation within the first computing group; the first computing node sending a first portion of the first data to a second computing node within the second computing group, and sending a second portion of the first data to a third computing node within the second computing group, wherein the first portion of the data corresponds to the second computing node, and the second portion of the data corresponds to the third computing node. The first and second portions of the data are used to implement a second set communication operation performed between different computing groups within the n computing groups, where the second and third computing nodes are different. The first data is the result data obtained by performing the first set communication operation, also referred to as the result data of the first set communication operation.

[0007] The technical solution provided in this application involves a first portion of data (i.e., the result data of the first set communication operation) corresponding to a first computing node obtained by performing a first set communication operation within a first computing group, which corresponds to a second computing node within a second computing group. A second portion of this first data corresponds to a third computing node within the second computing group. This first and second portion of data are used to implement a second set communication operation between different computing groups within the n computing groups. In other words, different portions of the first data correspond to different computing nodes within the second computing group, and these different portions are used to implement a second set communication operation between different computing groups within the n computing groups. The first computing node sends these different portions of data to the corresponding computing nodes within the second computing group, facilitating the implementation of a second set communication operation between different computing groups within the n computing groups based on these different portions of data. This, in turn, facilitates the implementation of a hierarchical all-reduce algorithm within the computing cluster. The set communication method provided in this application can optimize the hierarchical all-reduce algorithm, enabling its application in asymmetric computing clusters and expanding its application scenarios.

[0008] Optionally, the method further includes: a first computing node sending a third portion of the first data to a fourth computing node within a second computing group. This third portion of data corresponds to the fourth computing node and is also used to implement a second set communication operation between different computing groups within the n computing groups. The second, third, and fourth computing nodes are all different. That is, the first, second, and third portions of the first data correspond one-to-one with the three computing nodes within the second computing group. For example, multiple portions of the first data correspond one-to-one with multiple computing nodes within the second computing group.

[0009] The technical solution provided in this application involves different portions of data in the first data corresponding to the first computing node obtained by performing a first set communication operation within the first computing group, which correspond to different computing nodes within the second computing group. These different portions of data are used to implement a second set communication operation between different computing groups in the n computing groups. The first computing node sends these different portions of data to the different computing nodes in the second computing group, which facilitates the implementation of the second set communication operation between different computing groups in the n computing groups based on these different portions of data. This, in turn, facilitates the implementation of a hierarchical all-reduce algorithm (i.e., an optimized hierarchical all-reduce algorithm) within the computing cluster.

[0010] Optionally, the method further includes: the first computing node sending a third portion of the first data to a third computing node within the second computing group, wherein the third portion of the data also corresponds to a third computing node, and this third portion of data is also used to implement a second set communication operation between different computing groups within the n computing groups. That is, the first portion of the first data corresponds to one computing node within the second computing group, and the second and third portions of the first data correspond to the same computing node within the second computing group. For example, multiple portions of the first data correspond to multiple computing nodes within the second computing group, and at least two of these multiple portions correspond to the same computing node within the second computing group.

[0011] The technical solution provided in this application involves different portions of the first data corresponding to the first computing node obtained by performing a first set communication operation within the first computing group, which correspond to different computing nodes within the second computing group. Furthermore, at least two portions of the different portions of the data can correspond to the same computing node within the second computing group. These different portions of the data are used to implement a second set communication operation between different computing groups within the n computing groups. The first computing node sends these different portions of the data to the corresponding different computing nodes within the second computing group. This facilitates the implementation of a second set communication operation between different computing groups within the n computing groups based on these different portions of the data, thereby facilitating the implementation of a hierarchical all-reduce algorithm (i.e., an optimized hierarchical all-reduce algorithm) within the computing cluster.

[0012] Optionally, the first computing node sends a first portion of the first data to a second computing node within the second computing group, and sends a second portion of the first data to a third computing node within the second computing group. This includes: the first computing node sequentially sending the first portion of the first data to the second computing node within the second computing group, and then sequentially sending the second portion of the first data to the third computing node within the second computing group. That is, the process of the first computing node sending the first portion of the first data to the second computing node and the process of the first computing node sending the second portion of the first data to the third computing node are sequential. This avoids congestion on the first computing node.

[0013] When the third part of the first data corresponds to the fourth computing node in the second computing group, the first computing node sequentially sends the first part of the first data to the second computing node in the second computing group, the second part of the first data to the third computing node in the second computing group, and the third part of the first data to the fourth computing node in the second computing group. That is, the processes of the first computing node sending the first part of the first data to the second computing node, the second part of the first data to the third computing node, and the third part of the first data to the fourth computing node are sequential. This avoids congestion on the first computing node.

[0014] When the third part of the first data corresponds to the third computing node within the second computing group, the first computing node sequentially sends the first part of the first data to the second computing node within the second computing group, the second part of the first data to the third computing node within the second computing group, and the third part of the first data to the third computing node within the second computing group. That is, the processes of the first computing node sending the first part of the first data to the second computing node, the second part of the first data to the third computing node, and the third part of the first data to the third computing node are sequential. This avoids congestion on the first computing node.

[0015] Optionally, the transmission order of the first and second data portions is determined based on the priorities of the first and second data portions. If the first data portion also includes a third data portion, the transmission order of the first, second, and third data portions is determined based on the priorities of the first, second, and third data portions.

[0016] The technical solution provided in this application determines the sending order of the first part of data and the second part of data based on the priority of the first part of data and the priority of the second part of data. This facilitates the first computing node to send the first part of data to the second computing node in the second computing group in sequence, and to send the second part of data to the third computing node in the second computing group, thereby avoiding congestion of the first computing node.

[0017] Optionally, the first computing node corresponds to a first logical node and a second logical node. The first part of the data corresponds to the first logical node, and the second part of the data corresponds to the second logical node. The priority of the first part of the data is determined based on its data arrangement order within the logical group to which the first logical node belongs, and the priority of the second part of the data is determined based on its data arrangement order within the logical group to which the second logical node belongs. For example, if the first logical node belongs to the first logical group and the second logical node belongs to the second logical group, the priority of the first part of the data is determined based on its data arrangement order within the first logical group, and the priority of the second part of the data is determined based on its data arrangement order within the second logical group. As another example, if both the first and second logical nodes belong to the first logical group, the priority of the first part of the data is determined based on its data arrangement order within the first logical group, and the priority of the second part of the data is determined based on its data arrangement order within the first logical group. Furthermore, the first computing node also corresponds to another logical node besides the first and second logical nodes, and the third part of the first data corresponds to the other logical node. The priority of the third part of the data is determined based on the data arrangement order of the third part of the data in the logical group to which the other logical node belongs.

[0018] The order of any one of the first, second, and third data portions within the logical group to which the corresponding logical node belongs is determined by the starting offset address of that portion of data. For example, the smaller the starting offset address, the earlier the data portion appears in the logical group to which the corresponding logical node belongs. Conversely, the larger the starting offset address, the later the data portion appears in the logical group to which the corresponding logical node belongs.

[0019] The technical solution provided in this application determines the priority of the first part of the data based on the data arrangement order of the first part of the data in the logical group to which the first logical node belongs, and determines the priority of the second part of the data based on the data arrangement order of the second part of the data in the logical group to which the second logical node belongs. This facilitates the first computing node in determining the sending order of the first part of the data and the second part of the data based on their respective priorities, and then sends the first part of the data to the second computing node in the second computing group and the second part of the data to the third computing node in the second computing group in sequence, thereby avoiding congestion at the first computing node.

[0020] Optionally, the number of computing nodes in the minimum computing group is G. minThe minimum computational group is the computational group with the fewest computational nodes among the n computational groups, and the computational nodes in the minimum computational group are related to G. min Each logical group corresponds one-to-one with the logical node corresponding to each computational node within that smallest computational group, and the logical node belongs to the logical group corresponding to that computational node. This G min Each logical group in the logical group includes at least one logical node.

[0021] Optionally, each computing node within each of the n computing groups corresponds to at least one logical node. The second set communication operation is a set communication operation performed between different computing groups within the n computing groups for each logical node. When the same computing node corresponds to multiple logical nodes, the process of performing the second set communication operation between different computing groups within the n computing groups for these multiple logical nodes is serialized. This avoids situations where the same computing node sends data to the same computing node in parallel, resulting in "one-to-many" traffic, and where multiple computing nodes send data to the same computing node in parallel, resulting in "many-to-one" traffic, thus preventing congestion on the same computing node.

[0022] Optionally, the first logical node belongs to the first logical group, the second logical node belongs to the second logical group, the fifth logical node in the first computing group corresponds to the third logical node, and the third logical node also belongs to the first logical group (that is, the first logical node and the third logical node belong to the same logical group), and the sixth logical node in the first computing group corresponds to the fourth logical node, and the fourth logical node also belongs to the second logical group (that is, the second logical node and the fourth logical node belong to the same logical group, and the fourth logical node and the first logical node belong to different logical groups). The process of performing the second set communication operation on the first logical node between different computing groups in the n computing groups is serial with the process of performing the second set communication operation on the third logical node between different computing groups in the n computing groups. The process of performing the second set communication operation on the first logical node between different computing groups in the n computing groups is parallel with the process of performing the second set communication operation on the fourth logical node between different computing groups in the n computing groups. In other words, the process of performing second-set communication operations between different computing groups within the same logical group for logical nodes is serial, while the process of performing second-set communication operations between different computing groups within the same n computing groups for logical nodes in different logical groups is parallel. This avoids the problems of "one-to-many" and "many-to-one" traffic during the second-set communication operations between different computing groups within the n computing groups, while simultaneously improving the efficiency of performing second-set communication operations between different computing groups within the n computing groups.

[0023] Optionally, before the first computing node sends the first portion of the first data to the second computing node in the second computing group and sends the second portion of the first data to the third computing node in the second computing group, the method further includes: the first computing node segments the first data based on the boundary of the first set communication operation result data corresponding to the multiple computing nodes to obtain the aforementioned first portion of data and the aforementioned second portion of data. The multiple computing nodes are located in computing groups other than the first computing group among the n computing groups. The first set communication operation result data corresponding to each of the multiple computing nodes is data obtained by performing a first set communication operation within the computing group where each computing node is located. There is no boundary aligned with the boundary of the first data for the first set communication operation result data corresponding to each of the multiple computing nodes. That is, the boundary of the first set communication operation result data corresponding to each of the multiple computing nodes is offset from the boundary of the first data. Specifically, the boundary of the first portion of data obtained by segmenting the first data is aligned with the boundary of the first set communication operation result data corresponding to at least one of the multiple computing nodes, and the boundary of the second portion of data obtained by segmenting the first data is aligned with the boundary of the first set communication operation result data corresponding to at least one of the multiple computing nodes. The first data is the result data of the first set communication operation corresponding to the first computing node, obtained by performing the first set communication operation within the first computing group. The boundary of any data is determined based on the starting offset address and the size of any data. Data with the same starting offset address and the same size are aligned at their boundaries.

[0024] Furthermore, the first computing node can divide the first data based on the boundary of the first set communication operation result data corresponding to the plurality of computing nodes to obtain a third part of data. The boundary of the third part of data is aligned with the boundary of the first set communication operation result data corresponding to at least one of the plurality of computing nodes.

[0025] The technical solution provided in this application involves a first computing node segmenting first data based on the boundaries of first set communication operation result data corresponding to multiple computing nodes. This results in the boundaries of the segmented partial data (including a first part of data, a second part of data, and possibly a third part of data) being aligned with the boundaries of the first set communication operation result data corresponding to at least one of the multiple computing nodes. This facilitates the implementation of a second set communication operation between different computing groups in the aforementioned n computing groups based on the segmented partial data.

[0026] Optionally, the first computing node includes w data blocks, which are used to implement the first set communication operation executed within the first computing group, where w = n × L, and L represents the least common multiple of the number of computing nodes in the n computing groups; or, w = n × G max G max This represents the number of computing nodes in the largest computing group, which is the computing group with the most computing nodes among the n computing groups.

[0027] The first set communication operation result data obtained by performing the first set communication operation within the first computing group, corresponding to each computing node in the first computing group, includes at least one data block. When w = n × L: the number of data blocks included in the first set communication operation result data corresponding to any two computing nodes in the first computing group is equal. Therefore, the execution time of the first set communication operation by multiple computing nodes in the first computing group can be synchronized (these multiple computing nodes can exchange data during the execution of the first set communication operation, and the time taken for them to exchange data is equal). When w = n × G max In the following cases: If the first computing group is the largest computing group, the number of data blocks included in the result data of the first set communication operation corresponding to any two computing nodes in the first computing group is equal. Therefore, the time for multiple computing nodes in the first computing group to perform the first set communication operation in the first computing group can be synchronized (these multiple computing nodes can send data to each other during the execution of the first set communication operation, and the time taken for the multiple computing nodes to send data to each other is equal); if the first computing group is not the largest computing group, the number of data blocks included in the result data of the first set communication operation corresponding to at least two computing nodes in the first computing group is not equal.

[0028] Optionally, the method further includes: a first computing node acquiring second data, the second data being data corresponding to the first computing node obtained by performing a second set communication operation between different computing groups in the n computing groups, the second data being used to implement a third set communication operation performed within the first computing group; and the first computing node sending the second data to computing nodes other than the first computing node within the first computing group.

[0029] The technical solution provided in this application allows the first computing node to send second data to computing nodes other than the first computing node within the first computing group, which facilitates the implementation of third set communication operations within the first computing group.

[0030] Optionally, the second data corresponding to the first computing node is data obtained by performing a second set communication operation between different computing groups in the n computing groups. This second data is used to implement a third set communication operation performed within the first computing group. The first set communication operation is a reduce-scatter operation; the second set communication operation is an all-reduce operation; and the third set communication operation is an all-gather operation. Thus, in this application, after performing a reduce-scatter operation within a computing group, based on the result data obtained from performing the reduce-scatter operation within the computing group, an all-reduce operation is performed between different computing groups in the n computing groups. Furthermore, based on the result data obtained from performing the all-reduce operation between different computing groups in the n computing groups, an all-gather operation is performed within the computing group, thereby realizing a hierarchical all-reduce algorithm.

[0031] Optionally, at least two of the n computing groups may have unequal numbers of computing nodes. That is, the computing cluster is an asymmetric computing cluster. Therefore, the set communication method provided in this application can be applied to asymmetric computing clusters.

[0032] Optionally, the computing nodes in the computing cluster include at least one of the following: servers, neural processing units (NPUs), graphics processing units (GPUs), tensor processing units (TPUs), or central processing units (CPUs). Therefore, the aggregated communication method provided in this application can be flexibly applied to aggregated communication between multiple servers, multiple NPUs, multiple GPUs, multiple TPUs, or multiple CPUs. When the computing node is an NPU, GPU, TPU, or CPU, the computing node is also referred to as a computing card, computing unit, or similar terminology, and this application does not limit this.

[0033] Secondly, a set communication method is provided for a computing cluster, comprising n computing groups, each of which includes at least one computing node. At least two of the n computing groups have unequal numbers of computing nodes (i.e., the computing cluster is an asymmetric computing cluster), where n is an integer greater than 1. The n computing groups include a first computing group, which is not the smallest computing group among the n groups. The smallest computing group is the computing group with the fewest computing nodes among the n groups. The method includes: k computing nodes within the first computing group, based on raw data corresponding to each of the k computing nodes, performing a first set communication operation within the first computing group, to obtain G... min The first data corresponding to each computing node, G min Each computing node is a subset of the k computing nodes, and is related to G. min The first data corresponding to each computing node is used to implement the second set communication operation between different computing groups in the n computing groups, G min G represents the number of computing nodes within this minimum computing group. min Both k and k are positive integers.

[0034] Among them, with the G min The first data corresponding to any two computing nodes in the k computing nodes is different. The data obtained by performing a first set communication operation within the first computing group based on the original data corresponding to each of the k computing nodes does not include data that is different from the original data of the k computing nodes except for G. min Data corresponding to computing nodes other than the k computing nodes. For example, based on the original data corresponding to each of the k computing nodes, a first set communication operation is performed within the first computing group to obtain only the data corresponding to the G nodes. min The first data corresponding to each computing node. This G min Each computing node corresponds one-to-one with a computing node within the smallest computing group, and this G... min The first data corresponding to each computing node in the computing nodes is aligned with the boundary of the data corresponding to the computing node corresponding to each computing node, obtained by performing a first set communication operation within the smallest computing group.

[0035] The data obtained from performing the first set communication operation is also called the result data of the first set communication operation. The first data is the data obtained from performing the first set communication operation; therefore, the first data is the result data of the first set communication operation.

[0036] The technical solution provided in this application executes a first set communication operation within a first computing group to obtain first data corresponding to a subset of computing nodes within the first computing group. The number of these subset of computing nodes is equal to the number of computing nodes in the smallest computing group. This first data, corresponding to these subset of computing nodes, is used to implement a second set communication operation between different computing groups within the n computing groups. Therefore, this application can combine the number of computing nodes in the smallest computing group to execute the first set communication operation in each computing group other than the smallest computing group. This results in first set communication operation result data corresponding to a subset of computing nodes within each computing group. The number of these subset of computing nodes is equal to the number of computing nodes in the smallest computing group, and these subset of computing nodes corresponds one-to-one with the computing nodes in the smallest computing group. Furthermore, based on the first set communication operation result data corresponding to these subset of computing nodes, a second set communication operation can be implemented between different computing groups within the n computing groups, facilitating the implementation of a hierarchical all-reduce algorithm within the computing cluster. The set communication method provided in this application can optimize the hierarchical all-reduce algorithm, enabling it to be applied to asymmetric computing clusters and expanding its application scenarios.

[0037] Optionally, the k computing nodes, based on the original data corresponding to each of the k computing nodes, are used to perform the first set communication operation within the first computing group, and obtain the data corresponding to the G. min The first data corresponding to each of the k computing nodes includes: each computing node divides the original data corresponding to that computing node into w data blocks, where w = n × G. min The k computing nodes, based on the w data blocks corresponding to each of the k computing nodes, perform a first set communication operation within the first computing group to obtain the data related to G. min The first data corresponding to each computing node, and the G min The first data corresponding to each of the computing nodes includes at least one data block. (This is related to the G...) min The number of data blocks included in the first data of any two computing nodes in a given computing node is equal. (This is in contrast to the G...) min The first data corresponding to any two computing nodes in a given computing node is of equal size.

[0038] The technical solution provided in this application is based on w = n × G min The original data is divided into segments, resulting in a small number of data blocks, and the size of each data block is not too small, which can avoid data fragmentation.

[0039] Optionally, the G min Each computing node corresponds one-to-one with G. min There are n logical nodes. The second set communication operation is a set communication operation performed between different computing groups within the n computing groups for each logical node. min The process of each logical node performing the second set of communication operations is parallelized. This improves the efficiency of communication between different computation groups within the n computation groups for the G. min The efficiency of each logical node in performing the second set of communication operations.

[0040] Optionally, the method further includes: the G min Each computing node in the computing nodes obtains the second data corresponding to that computing node, and the G... min The second data corresponding to each of the n computing nodes is the data obtained by performing a second set communication operation between different computing groups in the n computing groups, and is related to the G. min The second data corresponding to each computing node in the G is used to implement the third set communication operation executed within the first computing group; min Each computing node in the first computing group sends its corresponding second data to the other computing nodes in the first computing group.

[0041] The technical solution provided in this application, the G in the first computing group min Each computing node in the first computing group sends its corresponding second data to other computing nodes in the first computing group, which facilitates the implementation of third set communication operations within the first computing group.

[0042] Optional, with the G min The second data corresponding to each of the n computing nodes is the data obtained by performing a second set communication operation between different computing groups in the n computing groups, and is related to the G. minThe second data corresponding to each computing node in the n computing nodes is used to implement the third set communication operation executed within the first computing group; the first set communication operation is a reduce-scatter operation; the second set communication operation is an all-reduce operation; and the third set communication operation is an all-gather operation. Therefore, in this application, after executing the reduce-scatter operation within the computing group, based on the result data obtained from the reduce-scatter operation within the computing group, an all-reduce operation is executed between different computing groups in the n computing groups. Furthermore, based on the result data obtained from the all-reduce operation between different computing groups in the n computing groups, an all-gather operation is executed within the computing group, thus implementing a hierarchical all-reduce algorithm in an asymmetric computing cluster.

[0043] Optionally, the computing nodes in the computing cluster include at least one of the following: servers, NPUs, GPUs, TPUs, or CPUs. Therefore, this aggregated communication method can be flexibly applied to aggregated communication between multiple servers, multiple NPUs, multiple GPUs, multiple TPUs, or multiple CPUs.

[0044] Thirdly, a collective communication device is provided, applied to a computing node, comprising a transceiver module and a processing module. The transceiver module performs receiving and / or sending-related operations as described in the first aspect or any optional implementation thereof; the processing module performs operations other than receiving and / or sending-related operations as described in the first aspect or any optional implementation thereof. The transceiver module and / or processing module can be implemented based on software, hardware, or a combination of software and hardware. Optionally, the transceiver module includes a receiving module and / or a sending module. The receiving module performs receiving-related operations, and the sending module performs sending-related operations.

[0045] Optionally, the aggregated communication device is applied to a first computing node within a first computing group. The computing cluster includes n computing groups, each of which includes at least one computing node, where n is an integer greater than 1. The n computing groups include a first computing group and a second computing group. A processing module is used to acquire first data, which is data corresponding to the first computing node obtained by performing a first aggregated communication operation within the first computing group. A transceiver module is used to send a first portion of the first data to a second computing node within a second computing group and to send a second portion of the first data to a third computing node within a second computing group. The first portion of the data corresponds to the second computing node, and the second portion of the data corresponds to the third computing node. The first portion of the data and the second portion of the data are used to implement a second aggregated communication operation between different computing groups within the n computing groups. The second computing node and the third computing node are different.

[0046] Optionally, the transceiver module is also used to send the third part of the first data to the fourth computing node in the second computing group. The third part of the data corresponds to the fourth computing node. The third part of the data is also used to implement the second set communication operation between different computing groups in the n computing groups. The second computing node, the third computing node and the fourth computing node are different.

[0047] Optionally, the transceiver module is also used to send the third part of the first data to the third computing node in the second computing group. The third part of the data also corresponds to the third computing node. The third part of the data is also used to implement the second set communication operation between different computing groups in the n computing groups.

[0048] Optionally, a transceiver module is used to sequentially send the first part of the first data to the second computing node in the second computing group, and to send the second part of the first data to the third computing node in the second computing group.

[0049] When the third part of the first data corresponds to the fourth computing node in the second computing group, the transceiver module is used to send the first part of the first data to the second computing node in the second computing group, send the second part of the first data to the third computing node in the second computing group, and send the second part of the first data to the fourth computing node in the second computing group in sequence.

[0050] When the third part of the first data corresponds to the third computing node in the second computing group, the transceiver module is used to send the first part of the first data to the second computing node in the second computing group in sequence, send the second part of the first data to the third computing node in the second computing group, and send the third part of the first data to the third computing node in the second computing group.

[0051] Optionally, the transmission order of the first and second data portions is determined based on the priorities of the first and second data portions. If the first data portion also includes a third data portion, the transmission order of the first, second, and third data portions is determined based on the priorities of the first, second, and third data portions.

[0052] Optionally, the first computing node corresponds to a first logical node and a second logical node. The first part of the data corresponds to the first logical node, and the second part of the data corresponds to the second logical node. The priority of the first part of the data is determined based on its data arrangement order within the logical group to which the first logical node belongs, and the priority of the second part of the data is determined based on its data arrangement order within the logical group to which the second logical node belongs. For example, if the first logical node belongs to the first logical group and the second logical node belongs to the second logical group, the priority of the first part of the data is determined based on its data arrangement order within the first logical group, and the priority of the second part of the data is determined based on its data arrangement order within the second logical group. As another example, if both the first and second logical nodes belong to the first logical group, the priority of the first part of the data is determined based on its data arrangement order within the first logical group, and the priority of the second part of the data is determined based on its data arrangement order within the first logical group. Furthermore, the first computing node also corresponds to another logical node besides the first and second logical nodes, and the third part of the first data corresponds to the other logical node. The priority of the third part of the data is determined based on the data arrangement order of the third part of the data in the logical group to which the other logical node belongs.

[0053] Optionally, the number of computing nodes in the minimum computing group is G. min The minimum computational group is the computational group with the fewest computational nodes among the n computational groups, and the computational nodes in the minimum computational group are related to G. min Each logical group corresponds one-to-one with the logical node corresponding to each computational node within that smallest computational group, and the logical node belongs to the logical group corresponding to that computational node. This G min Each logical group in the logical group includes at least one logical node.

[0054] Optionally, each computing node within each of the n computing groups corresponds to at least one logical node. The second set communication operation is a set communication operation performed between different computing groups within the n computing groups for each logical node. In the case where the same computing node corresponds to multiple logical nodes, the process of performing the second set communication operation between different computing groups within the n computing groups for these multiple logical nodes is as follows:

[0055] Optionally, the first logical node belongs to the first logical group, the second logical node belongs to the second logical group, the fifth logical node in the first computing group corresponds to the third logical node, and the third logical node also belongs to the first logical group (that is, the first logical node and the third logical node belong to the same logical group), and the sixth logical node in the first computing group corresponds to the fourth logical node, and the fourth logical node also belongs to the second logical group (that is, the second logical node and the fourth logical node belong to the same logical group, and the fourth logical node and the first logical node belong to different logical groups). The process of performing the second set communication operation on the first logical node between different computing groups in the n computing groups is serial with the process of performing the second set communication operation on the third logical node between different computing groups in the n computing groups. The process of performing the second set communication operation on the first logical node between different computing groups in the n computing groups is parallel with the process of performing the second set communication operation on the fourth logical node between different computing groups in the n computing groups.

[0056] Optionally, the processing module is further configured to, before the transceiver module sends the first part of the first data to the second computing node in the second computing group and sends the second part of the first data to the third computing node in the second computing group, divide the first data based on the boundary of the first set communication operation result data corresponding to multiple computing nodes to obtain the first part of the data and the second part of the data. The multiple computing nodes are located in computing groups other than the first computing group among the n computing groups. The first set communication operation result data corresponding to each of the multiple computing nodes is the data obtained by performing the first set communication operation within the computing group where each computing node is located. There is no boundary aligned with the boundary of the first data for the first set communication operation result data corresponding to each of the multiple computing nodes.

[0057] In addition, the processing module can divide the first data based on the boundary of the first set communication operation result data corresponding to the multiple computing nodes to obtain a third part of the data. The boundary of the third part of the data is aligned with the boundary of the first set communication operation result data corresponding to at least one of the multiple computing nodes.

[0058] Optionally, the first computing node includes w data blocks, which are used to implement the first set communication operation executed within the first computing group, where w = n × L, and L represents the least common multiple of the number of computing nodes in the n computing groups; or, w = n × G max G max This represents the number of computing nodes in the largest computing group, which is the computing group with the most computing nodes among the n computing groups.

[0059] Optionally, the processing module is also used to acquire second data, which is data corresponding to the first computing node obtained by performing a second set communication operation between different computing groups in the n computing groups. The second data is used to implement a third set communication operation performed within the first computing group. The transceiver module is also used to send the second data to computing nodes in the first computing group other than the first computing node.

[0060] Optionally, the second data corresponding to the first computing node is the data obtained by performing a second set communication operation between different computing groups in the n computing groups. This second data is used to implement a third set communication operation performed within the first computing group. The first set communication operation is a reduce-scatter operation; the second set communication operation is an all-reduce operation; and the third set communication operation is an all-gather operation.

[0061] Optionally, at least two of the n computing groups may have unequal numbers of computing nodes. That is, the computing cluster is an asymmetric computing cluster.

[0062] Optionally, the computing nodes in the computing cluster may include at least one of the following: a server, an NPU, a GPU, a TPU, or a CPU.

[0063] Fourthly, a collective communication device is provided, comprising a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program stored in the memory to cause the collective communication device to perform the method provided in the first aspect or any optional implementation thereof. The processor may be one or more, and the memory may be one or more.

[0064] Optionally, the memory can be integrated with the processor, or the memory can be set up separately from the processor.

[0065] In the specific implementation process, the memory can be a non-transitory memory, such as read-only memory (ROM), which can be integrated with the processor on the same chip or set on different chips. This application does not limit the type of memory or the way the memory and processor are set.

[0066] Fifthly, a computing cluster is provided, comprising n computing groups, each of the n computing groups comprising at least one computing node, where n is an integer greater than 1; the at least one computing node in the computing cluster comprises the aggregate communication device provided in the third or fourth aspect. Optionally, the computing node in the computing cluster comprises at least one of the following: a server, an NPU, a GPU, a TPU, or a CPU.

[0067] Sixthly, a computing cluster is provided, comprising n computing groups, each of the n computing groups comprising at least one computing node, wherein at least two of the n computing groups have unequal numbers of computing nodes, where n is an integer greater than 1, and the n computing groups include a first computing group, which is not the smallest computing group among the n computing groups, but rather the computing group with the fewest computing nodes among the n computing groups; the k computing nodes in the first computing group are used to: perform a first set communication operation within the first computing group based on the original data corresponding to each of the k computing nodes, thereby obtaining G within the first computing group. min The first data corresponding to each computing node, G min Each computing node is a subset of the k computing nodes, and is related to G. min The first data corresponding to each computing node is used to implement the second set communication operation between different computing groups in the n computing groups. min G represents the number of computing nodes within this minimum computing group. min Both k and k are positive integers.

[0068] Optionally, each of the k computing nodes is used to: divide the original data corresponding to each computing node into w data blocks, where w = n × G. min ;

[0069] The k computing nodes are used to: perform a first set communication operation within a first computing group based on the w data blocks corresponding to each of the k computing nodes, to obtain the data from the G... min The first data corresponding to each computing node, and the G min The first data corresponding to each of the computing nodes includes at least one data block.

[0070] Optionally, the G min Each computing node corresponds one-to-one with G. min There are n logical nodes. The second set communication operation is a set communication operation performed between different computing groups within the n computing groups for each logical node.min The process of each logical node performing the second set of communication operations is parallel.

[0071] Optionally, the G min Each of the computing nodes is also used for:

[0072] Obtain the second data corresponding to each computing node, and the G min The second data corresponding to each of the n computing nodes is the data obtained by performing a second set communication operation between different computing groups in the n computing groups, and is related to the G. min The second data corresponding to each computing node in the computing node is used to realize the third set communication operation executed within the first computing group;

[0073] Send the second data corresponding to each computing node to the other computing nodes in the first computing group.

[0074] Optional, with the G min The second data corresponding to each of the n computing nodes is the data obtained by performing a second set communication operation between different computing groups in the n computing groups, and is related to the G. min The second data corresponding to each computing node in the computing node is used to implement the third set communication operation executed within the first computing group; the first set communication operation is a reduce-scatter operation; the second set communication operation is an all-reduce operation; and the third set communication operation is an all-gather operation.

[0075] Optionally, the computing nodes in the computing cluster may include at least one of the following: a server, an NPU, a GPU, a TPU, or a CPU.

[0076] In a seventh aspect, a computer-readable storage medium is provided, wherein a computer program is stored therein, which, when executed, implements the collective communication method provided by the first aspect or any alternative method thereof, or implements the collective communication method provided by the second aspect or any alternative method thereof.

[0077] Eighthly, a computer program product is provided, the computer program product comprising a program or code that, when executed, implements the collective communication method provided as in the first aspect or any alternative method of the first aspect, or implements the collective communication method provided as in the second aspect or any alternative method of the second aspect.

[0078] In a ninth aspect, a chip is provided, the chip including programmable logic circuitry and / or program instructions, the chip being used to implement the collective communication method provided by the first aspect or any alternative method of the first aspect, or to implement the collective communication method provided by the second aspect or any alternative method of the second aspect.

[0079] It should be understood that the beneficial effects achieved by the third to ninth aspects and their corresponding possible implementations described above can be referenced to the beneficial effects achieved by the first to second aspects and their corresponding possible implementations described above, and will not be repeated here. Furthermore, the aggregated communication device mentioned in the third and fourth aspects can be the chip mentioned in the ninth aspect, and this application does not limit it in this regard. Attached Figure Description

[0080] Figure 1 This is a schematic diagram of a reduce-scatter operation;

[0081] Figure 2 This is a schematic diagram of an all-reduce operation;

[0082] Figure 3 This is a schematic diagram of an all-gather operation;

[0083] Figure 4 This is a schematic diagram of a computing cluster provided in an embodiment of this application;

[0084] Figure 5 This is a schematic diagram of another computing cluster provided in an embodiment of this application;

[0085] Figure 6 This is a flowchart of a collection communication method provided in an embodiment of this application;

[0086] Figure 7 This application provides an embodiment of a method in which... Figure 4 The diagram illustrates the execution of reduce-scatter operations within a compute group in a compute cluster.

[0087] Figure 8 This application provides a method based on... Figure 7 A schematic diagram showing the logical nodes of the provided reduce-scatter result data;

[0088] Figure 9 This application provides an embodiment of a method for different computing groups to... Figure 8 A diagram illustrating the result of the all-reduce operation performed on the divided logical nodes;

[0089] Figure 10This application provides a method based on... Figure 9 A diagram illustrating the results of the all-reduce operation within the computation group after executing all-gather;

[0090] Figures 11 to 12 This application provides an embodiment of a method in which... Figure 5 The diagram illustrates the execution of reduce-scatter operations within a compute group in a compute cluster.

[0091] Figure 13 This application provides a method based on... Figure 11 A schematic diagram showing the logical nodes of the provided reduce-scatter result data;

[0092] Figure 14 This is a flowchart of another collection communication method provided in the embodiments of this application;

[0093] Figures 15 to 16 This is another embodiment provided in this application. Figure 5 The diagram illustrates the execution of reduce-scatter operations within a compute group in a compute cluster.

[0094] Figure 17 This application provides a method based on... Figure 16 A schematic diagram showing the logical nodes of the provided reduce-scatter result data;

[0095] Figure 18 This is a schematic diagram of a collection communication device provided in an embodiment of this application;

[0096] Figure 19 This is a schematic diagram of another collection communication device provided in an embodiment of this application;

[0097] Figure 20 This is a schematic diagram of another collection communication device provided in the embodiments of this application. Detailed Implementation

[0098] The embodiments of this application will now be described in further detail with reference to the accompanying drawings.

[0099] Collective communication (CC) is an algorithm used to transmit data between multiple computing nodes in a computing cluster. For example, a computing cluster may consist of multiple computing groups, each containing at least one computing node. Computing nodes within each group are interconnected, and computing nodes in different groups are also interconnected. The collective communication algorithm is used to implement data transmission between computing nodes within the same computing group and between computing nodes in different computing groups.

[0100] To meet the demands of high computing power, the computing cluster can be expanded both vertically (scale-up) and horizontally (scale-out). This gives the computing cluster a hierarchical structure (e.g., a network layer within a computing group, a network layer between different computing groups, multiple computing groups forming a larger computing group, and a network layer between different larger computing groups), with differences in network bandwidth and networking characteristics across different network layers. Generally, the higher the layer, the smaller the network bandwidth, and the lower the layer, the larger the network bandwidth.

[0101] An all-reduce algorithm is a collective communication algorithm that performs a reduction operation on data across multiple computing nodes in a computing cluster and sends the reduction results back to those nodes, ensuring that all nodes achieve the same reduction result. All-reduce algorithms include single-level all-reduce algorithms and hierarchical all-reduce algorithms. Single-level all-reduce algorithms treat all computing groups in the cluster as a whole (logically considering all computing groups as a single computing group). Examples of single-level all-reduce algorithms include ring algorithms, binary search, and halving and doubling (HD) algorithms. Hierarchical all-reduce algorithms are also called multi-level all-reduce algorithms. The term hierarchical all-reduce is used in contrast to single-level all-reduce algorithms. The hierarchical all-reduce algorithm treats each computation group in the computing cluster as a single computation group. After performing a reduce-scatter operation within each computation group, it performs an all-reduce operation between computation nodes with the same ID in different computation groups (e.g., nodes with the same ID are called "equal-numbered computation nodes") based on the results of the reduce-scatter operation within the computation group. Then, based on the results of the all-reduce operation between these nodes, it performs an all-gather operation within the computation group, ultimately achieving the same effect as the single-layer all-reduce algorithm. Therefore, the hierarchical all-reduce algorithm consists of three operations: reduce-scatter operations performed within each computation group in the computing cluster, all-reduce operations performed between computation nodes with the same ID in different computation groups, and all-gather operations performed within each computation group in the computing cluster. The hierarchical all-reduce algorithm is composed of reduce-scatter operations performed within each computation group, all-reduce operations performed between different computation groups, and all-gather operations performed within each computation group.

[0102] However, in single-layer all-reduce algorithms, the amount of data requiring cross-computation group communication is large, and the number of cross-computation group communications is high (for example, the communication data volume of the ring algorithm is 2*[(M-1) / M]*S, and the number of communication steps is 2*(M-1), where M is the number of compute nodes and S is the amount of data in a single communication). This results in poor performance of single-layer all-reduce algorithms in network convergence scenarios of compute clusters (e.g., long-distance communication latency reaches the millisecond level). Current hierarchical all-reduce algorithms are only applicable to symmetric compute clusters and cannot be applied to asymmetric compute clusters, thus limiting the application scenarios of current hierarchical all-reduce algorithms. A symmetric compute cluster refers to a compute cluster where the number of compute nodes in all compute groups is equal, while an asymmetric compute cluster refers to a compute cluster where the number of compute nodes in at least two compute groups is unequal.

[0103] Large-scale artificial intelligence (AI) training clusters are computing clusters that typically employ distributed parallelism for model training. This necessitates the use of ensemble communication algorithms to facilitate data transfer between computing nodes within the cluster. For example, large-scale AI training clusters may employ data parallelism (dp), expert parallelism (ep), or serialize parallelism (sp) for model training. dp, ep, or sp may span different network layers, leading to a significant performance degradation of traditional ensemble communication algorithms (such as the ring algorithm). Furthermore, considering user usability and dynamic allocation requirements, resources across different network layers may become fragmented, posing new demands on ensemble communication algorithms.

[0104] This application provides a set communication method that optimizes the hierarchical all-reduce algorithm, enabling its application in asymmetric computing clusters and expanding its application scenarios. This set communication method can be considered an optimized hierarchical all-reduce algorithm. It exhibits good communication performance and efficiency, is suitable for asymmetric computing clusters, and avoids resource fragmentation issues.

[0105] The collection communication method provided in this application involves reduce-scatter operation, all-reduce operation, and all-gather operation. To facilitate understanding of the technical solutions provided in this application, the reduce-scatter operation, all-reduce operation, and all-gather operation will be described below.

[0106] Reduce-scatter operations refer to performing a dimensional reduction operation on data across multiple computing nodes in a computing cluster, distributing the results of the reduction operations across different computing nodes, so that each computing node has a result for one dimension of the reduction operation. Reduction operations include, but are not limited to: summation, product, maximum value, minimum value, logical AND, bitwise AND, logical OR, bitwise OR, logical XOR, and bitwise XOR. The following example illustrates the reduction operation of summation. Figure 1 A schematic diagram of the reduce-scatter operation is shown. For example... Figure 1 As shown, computing node 1 includes data A1 to D1, computing node 2 includes data A2 to D2, computing node 3 includes data A3 to D3, and computing node 4 includes data A4 to D4. Data A1 to D1 are data with four different dimensions, data A2 to D2 are data with four different dimensions, data A3 to D3 are data with four different dimensions, and data A4 to D4 are data with four different dimensions. Furthermore, data A1 to A4 are data with the same dimension, data B1 to B4 are data with the same dimension, data C1 to C4 are data with the same dimension, and data D1 to D4 are data with the same dimension. Performing a reduce-scatter operation on the data in computing nodes 1 to 4 includes: calculating the sum of data A1 to A4... Figure 1 In this context, "A1+A2+A3+A4" represents the sum of data A1 to A4. The sum of data B1 to B4 is calculated. Figure 1 In this context, "B1+B2+B3+B4" represents the sum of data B1 to B4. The sum of data C1 to C4 is calculated. Figure 1 In this context, "C1+C2+C3+C4" represents the sum of data C1 to C4. The sum of data D1 to D4 is calculated. Figure 1 In this context, "D1+D2+D3+D4" represents the cumulative sum of data D1 to D4. The cumulative sums of data A1 to A4, data B1 to B4, data C1 to C4, and data D1 to D4 are distributed to different computing nodes 1 to 4, so that each of computing nodes 1 to 4 has a reduction operation result of one dimension. Figure 1The following example illustrates how to distribute the sums of data A1 to A4 across computing node 1, the sums of data B1 to B4 across computing node 2, the sums of data C1 to C4 across computing node 3, and the sums of data D1 to D4 across computing node 4. Figure 1 As shown, after the reduce-scatter operation is executed, compute node 1 includes the sum of data A1 to A4 "A1+A2+A3+A4", compute node 2 includes the sum of data B1 to B4 "B1+B2+B3+B4", compute node 3 includes the sum of data C1 to C4 "C1+C2+C3+C4", and compute node 4 includes the sum of data D1 to D4 "D1+D2+D3+D4".

[0107] All-reduce operation: This refers to performing a reduction operation on data across multiple compute nodes in a computing cluster and sending the reduction operation results back to those nodes, ensuring that all compute nodes have the same reduction result. Reduction operations include, but are not limited to: calculating the sum, calculating the product, finding the maximum value, finding the minimum value, logical AND, bitwise AND, logical OR, bitwise OR, logical XOR, and bitwise XOR. Let's take calculating the sum as an example. Figure 2 A schematic diagram of the all-reduce operation is shown. For example... Figure 2 As shown, compute node 1 includes data A, compute node 2 includes data B, compute node 3 includes data C, and compute node 4 includes data D. Performing an all-reduce operation on the data in compute nodes 1-4 includes: calculating the sum of data A-D. Figure 1 In this context, "A+B+C+D" represents the sum of data A to D, and the sum of data A to D is sent to computing nodes 1 to 4 so that computing nodes 1 to 4 have the same reduction operation result "A+B+C+D".

[0108] The all-gather operation refers to collecting data from multiple compute nodes in a computing cluster, so that each compute node has access to all the data from the other compute nodes. For example... Figure 3 A schematic diagram of the all-gather operation is shown. For example... Figure 3 As shown, computing node 1 includes data A, computing node 2 includes data B, computing node 3 includes data C, and computing node 4 includes data D. Performing an all-gather operation on the data of computing nodes 1 to 4 includes: collecting data A to D into computing nodes 1 to 4, so that computing nodes 1 to 4 all have data A to D.

[0109] The application scenarios of the embodiments of this application are described below.

[0110] This application provides a computing cluster, which can be a high-performance computing (HPC) cluster, an AI training cluster, or a data center computing cluster, and can be a distributed computing cluster. The computing cluster includes multiple computing groups, each including at least one computing node. The computing nodes within each computing group are communicatively connected, and computing nodes in different computing groups are also communicatively connected. This application embodiment is used to implement aggregated communication between computing nodes in the computing cluster.

[0111] A computing node can be a computing device or a computing component deployed within a computing device that has computing capabilities. The computing device can be a server. Computing components deployed within a computing device that have computing capabilities can include a neural processing unit (NPU), a graphics processing unit (GPU), a tensor processing unit (TPU), or a central processing unit (CPU). NPUs, GPUs, TPUs, and CPUs are also referred to as computing chips or computing cards. Furthermore, computing components deployed within a computing device that have computing capabilities can also include processor cores (e.g., CPU cores), microprocessors, integrated circuits, etc. For example, a computing node can be a server, an NPU, a GPU, a TPU, a CPU, a CPU core, a microprocessor, or one or more integrated circuits used to implement the computing functions of this application. The integrated circuit can be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or any combination thereof. The PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof. In the computing cluster provided in this application embodiment, the types of different computing nodes can be the same or different. For example, some computing nodes in the computing cluster are NPUs, and others are GPUs; or all computing nodes in the computing cluster are NPUs; or all computing nodes in the computing cluster are GPUs. This application embodiment does not limit the type of computing nodes. In other embodiments, computing nodes can also be network devices such as switches and routers that support aggregated communication; this application embodiment does not limit this.

[0112] In the case where a computing node is a computing component (e.g., a computing card) deployed within a computing device, all or some of the computing nodes in the same computing group are located within the same computing device. Computing nodes within the same computing device are connected via an internal network (e.g., a communication bus) of that computing device. Different computing devices are connected via local area networks (LANs), the internet, or private networks. In the case where a computing node is a computing device, different computing nodes are connected via LANs, the internet, or private networks.

[0113] In optional embodiments, the computing node further includes storage space for storing program code. The computing node can execute the program code stored in its storage space to implement the functions related to this application. The program code includes one or more software modules. In specific embodiments, the computing node further includes a collective communication library (CCL) module. The CCL module can execute the program code stored in the computing node's storage space to implement the functions related to this application. This embodiment of the application does not limit this aspect.

[0114] Please refer to Figure 4 The diagram illustrates a computing cluster 10 according to an embodiment of this application. The computing cluster 10 includes computing groups 1 to 3. Computing group 1 includes 3 computing nodes, specifically computing nodes 11 to 13. Computing group 2 includes 3 computing nodes, specifically computing nodes 21 to 23. Computing group 3 includes 2 computing nodes, specifically computing nodes 31 to 32. Computing nodes 11 to 13, 21 to 23, and 31 to 32 are connected via network communication. For example, computing nodes 11-13 are located in the same computing device (e.g., referred to as computing device 1), computing nodes 21-23 are located in the same computing device (e.g., referred to as computing device 2), and computing nodes 31-32 are located in the same computing device (e.g., referred to as computing device 3). Computing nodes 11-13 are connected through a network (e.g., a communication bus) within computing device 1, computing nodes 21-23 are connected through a network (e.g., a communication bus) within computing device 2, and computing nodes 31-32 are connected through a network (e.g., a communication bus) within computing device 3. Computing devices 1, 2, and 3 are connected via a local area network, the Internet, or a private network.

[0115] Please refer to Figure 5The diagram illustrates a computing cluster 20 according to an embodiment of this application. The computing cluster 20 includes computing groups 1 to 3. Computing group 1 includes 5 computing nodes, specifically computing nodes 11 to 15. Computing group 2 includes 4 computing nodes, specifically computing nodes 21 to 24. Computing group 3 includes 3 computing nodes, specifically computing nodes 31 to 33. Computing nodes 11 to 15, 21 to 24, and 31 to 33 are connected via network communication. For example, computing nodes 11-15 are located in the same computing device (e.g., referred to as computing device 1), computing nodes 21-24 are located in the same computing device (e.g., referred to as computing device 2), and computing nodes 31-33 are located in the same computing device (e.g., referred to as computing device 3). Computing nodes 11-15 are connected through a network (e.g., a communication bus) within computing device 1, computing nodes 21-24 are connected through a network (e.g., a communication bus) within computing device 2, and computing nodes 31-33 are connected through a network (e.g., a communication bus) within computing device 3. Computing devices 1, 2, and 3 are connected via a local area network, the Internet, or a private network.

[0116] It should be noted that, Figure 4 and Figure 5 This example illustrates a computing cluster comprising three computing groups, and that the cluster is an asymmetric computing cluster. Those skilled in the art will readily understand that this is merely an example and does not constitute a limitation on the technical solutions provided in this application. In practical applications, the number of computing groups within a computing cluster and the number of computing nodes within each group are determined based on actual circumstances; the number of computing nodes in different computing groups may be equal or unequal. Furthermore, the computing cluster may also include other devices. For example, the computing cluster may also include a management node (…). Figure 4 and Figure 5 (Not shown in the diagram), the computing nodes are managed by a management node. The management node and at least one computing node may reside in the same computing device, or the management node may be a device independent of the computing nodes; this application embodiment does not limit this. Those skilled in the art will understand that the computing cluster can be appropriately adjusted as business needs change; this application embodiment does not list such adjustments here. The above-mentioned terms for computing groups are merely exemplary; a computing group can also be called a sub-cluster, and a computing group can also be a supernode.

[0117] The above describes the application scenarios of the embodiments of this application. The following describes the embodiments of the collection communication method of this application.

[0118] Please refer to Figure 6The diagram illustrates a flowchart of a collective communication method provided in an embodiment of this application. This collective communication method is applied to a computing cluster comprising n computing groups, each of which includes at least one computing node, where n is an integer greater than 1. At least two of the n computing groups have unequal numbers of computing nodes, and the n computing groups include a first computing group and a second computing group. For example, the computing cluster is as follows: Figure 4 The computing cluster 10 shown is or is as follows Figure 5 The computing cluster 20 shown has a first computing group and a second computing group, which are any two computing groups from computing groups 1 to 3. The first computing node mentioned below can be any computing node in the first computing group, and the second computing node and the third computing node mentioned below can be any two computing nodes in the second computing group.

[0119] For example, the computing cluster is as follows Figure 4 The computing cluster 10 shown has the following configurations: First computing group 1, second computing group 3, first computing node 12, second computing node 31, and third computing node 32. Alternatively, the first computing group 2, second computing group 3, first computing node 22, second computing node 31, and third computing node 32. Or, the first computing group 3, second computing group 1, first computing node 31, second computing node 11, and third computing node 12. Or, the first computing group 3, second computing group 2, first computing node 31, second computing node 21, and third computing node 22. Or, the first computing group 3, second computing group 1, first computing node 32, second computing node 12, and third computing node 13. Alternatively, the first computation group is computation group 3, the second computation group is computation group 2, the first computation node is computation node 32, the second computation node is computation node 22, and the third computation node is computation node 23.

[0120] For example, the computing cluster is as follows: Figure 5The computing cluster 20 shown has the following configurations: First computing group is computing group 2, second computing group is computing group 1, first computing node is computing node 24, second computing node is computing node 14, and third computing node is computing node 15. Alternatively, the first computing group is computing group 3, the second computing group is computing group 1, the first computing node is computing node 33, the second computing node is computing node 14, and the third computing node is computing node 15. Or, the first computing group is computing group 3, the second computing group is computing group 1, the first computing node is computing node 32, the second computing node is computing node 12, and the third computing node is computing node 13. Or, the first computing group is computing group 3, the second computing group is computing group 2, the first computing node is computing node 32, the second computing node is computing node 22, and the third computing node is computing node 23.

[0121] like Figure 6 As shown, the collection communication method includes the following steps S601 to S602.

[0122] S601. The first computing node in the first computing group obtains the first data, which is the data corresponding to the first computing node obtained by performing the first set communication operation in the first computing group.

[0123] The first data is the result data obtained by performing the first set communication operation within the first computing group. For ease of description, the result data obtained by performing the first set communication operation is called the first set communication operation result data (or first set communication operation data), and the data used to implement the first set communication operation is called the original data. For example, the first set communication operation is a reduce-scatter operation, and the result data of the first set communication operation is reduce-scatter result data (or reduce-scatter data).

[0124] In a specific embodiment, multiple computing nodes within the first computing group each include raw data for executing a first set communication operation within the first computing group. Based on the raw data corresponding to each of the multiple computing nodes (i.e., included by each of the multiple computing nodes), the multiple computing nodes perform the first set communication operation within the first computing group, obtaining first set communication operation result data corresponding one-to-one with each of the multiple computing nodes. Each of the multiple computing nodes acquires the first set communication operation result data corresponding to itself. Here, the multiple computing nodes include a first computing node; therefore, the first computing node acquires the first set communication operation result data corresponding to itself, and the first set communication operation result data corresponding to the first computing node is also the first data. For example, the multiple computing nodes within the first computing group are all the computing nodes within the first computing group.

[0125] In an optional embodiment, the first computing node includes w data blocks. These w data blocks are used to implement a first set communication operation executed within the first computing group. The w data blocks can be obtained by the first computing node dividing the original data corresponding to it for implementing the first set communication operation within the first computing group. The sizes of the w data blocks can be equal. In another optional embodiment, each of the plurality of computing nodes within the first computing group includes w data blocks. The w data blocks corresponding to each of these plurality of computing nodes (i.e., the w data blocks included in each computing node) are used to implement the first set communication operation executed within the first computing group. The w data blocks corresponding to each of these plurality of computing nodes can be obtained by each computing node dividing the original data corresponding to it for implementing the first set communication operation within the first computing group. The sizes of the w data blocks corresponding to each computing node can be equal. Furthermore, the sizes of each data block corresponding to any two of the plurality of computing nodes can be equal. That is, the multiple computing nodes within the first computing group each correspond to w data blocks used to implement the first set communication operation within the first computing group, and the size of the data blocks corresponding to the multiple computing nodes can be equal. The boundaries of the w data blocks corresponding to the multiple computing nodes can be aligned one-to-one; for example, the boundary of the first data block corresponding to the multiple computing nodes is aligned, the boundary of the second data block is aligned, the boundary of the third data block is aligned, and so on, until the boundary of the w-th data block corresponding to the multiple computing nodes is aligned. The boundary of any data (including data blocks) can be determined based on the starting offset address and the size of that data; data with the same offset address and the same size are aligned.

[0126] In a specific embodiment, the original data corresponding to any two computing nodes in the first computing group for performing the first set communication operation within the first computing group is of equal size. Each computing node in the plurality of computing nodes segments the original data corresponding to itself for performing the first set communication operation within the first computing group to obtain w data blocks corresponding to itself. Based on the w data blocks corresponding to itself, the plurality of computing nodes perform the first set communication operation within the first computing group to obtain first set communication operation result data corresponding one-to-one with the plurality of computing nodes. The first set communication operation result data corresponding to each of the plurality of computing nodes includes at least one data block.

[0127] for Figure 7 In the illustrated embodiment, w = n × L or w = n × G max L represents the least common multiple of the number of computation nodes in the above n computation groups, and G max This indicates the number of computing nodes within the largest computing group, which is the computing group with the most computing nodes among the n computing groups. For example, the n computing groups are as follows: Figure 4 The calculation groups 1-3 shown have the following structure: Calculation group 1 has 3 calculation nodes, calculation group 2 has 3 calculation nodes, and calculation group 3 has 2 calculation nodes. The least common multiple of the number of calculation nodes in calculation groups 1-3 is 6 (i.e., L = 6), so w = n × L = 3 × 6 = 18. For example, the n calculation groups are as follows: Figure 5 The computation groups shown are 1 to 3. Computation group 1 has 5 computational nodes, computation group 2 has 4 computational nodes, and computation group 3 has 3 computational nodes. The largest computational group among these three is computation group 1, which has 5 computational nodes (i.e., G). max =5), w = n × G max =3×5=15.

[0128] When w = n × L, the aforementioned multiple computing nodes within the first computing group, based on w data blocks corresponding to each of their respective computing nodes, perform a first set communication operation within the first computing group. The resulting data from this first set communication operation, corresponding one-to-one with each of the multiple computing nodes, includes data blocks of equal size for any two nodes. This ensures time synchronization for the multiple computing nodes performing the first set communication operation within the first computing group (these nodes can exchange data during the operation, and the time taken for data exchange is equal).

[0129] In w = n × G maxIn the case where the first computing group is the largest computing group, the multiple computing nodes within the first computing group, based on w data blocks corresponding to each of the multiple computing nodes, perform a first set communication operation within the first computing group. The resulting data from this first set communication operation, corresponding one-to-one with each of the multiple computing nodes, includes data blocks of equal size for any two of the multiple computing nodes. This ensures time synchronization for the multiple computing nodes performing the first set communication operation within the first computing group (the multiple computing nodes can exchange data during the first set communication operation, and the time taken for each data exchange is equal).

[0130] In w = n × G max In the case where the first computing group is not the largest computing group, the number of data blocks included in the first set communication operation result data corresponding to at least two of the computing nodes within the first computing group are not equal, and the number of data blocks included in the first set communication operation result data corresponding to any two of the at least two computing nodes is an integer multiple of the number of data blocks (depending on the actual situation, it may not be an integer multiple). That is, the number of data blocks included in the first set communication operation result data corresponding to one of the computing nodes is an integer multiple of the number of data blocks included in the first set communication operation result data corresponding to the other computing node. It is easy to understand that in w = n × G max In the case where the first computing group is not the largest computing group, the size of the first set communication operation result data obtained by performing the first set communication operation within the first computing group is not equal to the size of the first set communication operation result data corresponding to at least two of the above-mentioned multiple computing nodes within the first computing group, and is an integer multiple of the size of the first set communication operation result data corresponding to any two of the at least two computing nodes (depending on the actual situation, it may not be an integer multiple).

[0131] The above description of step S601 uses the execution of a first set communication operation within a first computing group as an example. In practical applications, the first set communication operation can be executed within each computing group in a computing cluster. For example, the above-mentioned n computing groups in the computing cluster also include a second computing group, and the first set communication operation can also be executed within the second computing group. In a specific embodiment, multiple computing nodes within the second computing group each include raw data for implementing the first set communication operation within the second computing group; based on the raw data corresponding to each of the multiple computing nodes (i.e., included by each of the multiple computing nodes), the multiple computing nodes perform the first set communication operation within the second computing group to obtain first set communication operation result data corresponding one-to-one with the multiple computing nodes; each of the multiple computing nodes obtains the first set communication operation result data corresponding to each of the multiple computing nodes. In a specific embodiment, each of the plurality of computing nodes in the second computing group segments the original data corresponding to itself, used to perform the first set communication operation within the second computing group, to obtain w data blocks corresponding to each computing node. The w data blocks can be of equal size, and the size of each data block corresponding to any two of the plurality of computing nodes can be equal. The boundaries of the w data blocks corresponding to each of the plurality of computing nodes can be aligned one-to-one. Based on the w data blocks corresponding to each of the plurality of computing nodes, the plurality of computing nodes perform the first set communication operation within the second computing group to obtain first set communication operation result data corresponding one-to-one with each of the plurality of computing nodes. The first set communication operation result data corresponding to each of the plurality of computing nodes includes at least one data block. Where w = n × L or w = n × G max When w = n × L, the number of data blocks included in the first set communication operation result data corresponding to any two computing nodes within the second computing group is equal. When w = n × G max In the following cases, if the second computing group is the largest computing group, the number of data blocks included in the first set communication operation result data corresponding to any two computing nodes among the plurality of computing nodes in the second computing group is equal; if the second computing group is not the largest computing group, the number of data blocks included in the first set communication operation result data corresponding to at least two computing nodes among the plurality of computing nodes in the second computing group is not equal, and the number of data blocks included in the first set communication operation result data corresponding to any two computing nodes among the at least two computing nodes is an integer multiple of the number of data blocks included in the first set communication operation result data. Here, the plurality of computing nodes in the second computing group can be all computing nodes within the second computing group.

[0132] The above only briefly describes the process of executing the first set communication operation within the second computing group. For a detailed description of the specific process of executing the first set communication operation within the second computing group, please refer to the detailed process and related description of executing the first set communication operation within the first computing group; this embodiment will not repeat the details here. It should be noted that the boundaries of the first set communication operation result data obtained by executing the first set communication operation within the second computing group, corresponding one-to-one with the aforementioned multiple computing nodes within the second computing group, and the boundaries of the first set communication operation result data obtained by executing the first set communication operation within the first computing group, corresponding one-to-one with the aforementioned multiple computing nodes within the first computing group, may be aligned or not aligned (i.e., the boundaries are misaligned); this embodiment does not limit this. The process of executing the first set communication operation result within the second computing group and the process of executing the first set communication operation result within the first computing group can be parallelized, thereby improving the efficiency of executing the first set communication operation within both the first and second computing groups. In the aforementioned n computing groups, the specific process and related description of executing the first set communication operation within each computing group other than the first and second computing groups can be found by referring to the process of executing the first set communication operation within the first computing group. The boundaries of the first set communication operation result data obtained by executing the first set communication operation within any two computing groups in the n computing groups can be aligned or unaligned. The process of executing the first set communication operation within different computing groups in the n computing groups can be parallelized, thereby improving the efficiency of executing the first set communication operation within different computing groups. In the embodiments of this application, the process of executing the first set communication operation within any computing group in the n computing groups can be implemented based on a single-layer algorithm (e.g., ring algorithm, HD algorithm, etc.), thereby effectively utilizing the link bandwidth between computing nodes within any computing group.

[0133] As mentioned earlier, the first set communication operation can be a reduce-scatter operation. To facilitate understanding of step S601, the following will take the example of the first set communication operation being a reduce-scatter operation, and will combine it with... Figure 4 and Figure 5 The computing cluster shown below introduces the content of S601. In the following description, the reduction operation is used as an example to illustrate the summation, and the result data of the first set communication operation is referred to as the reduce-scatter result data.

[0134] Please refer to Figure 7 It illustrates an embodiment of the present application that provides a method for... Figure 4 This diagram illustrates the execution of reduce-scatter operations within a computing group in computing cluster 10. Figure 7 Let's take w = n × L as an example. Because in... Figure 4In the computing cluster 10 shown, the number of computing groups is 3. The number of computing nodes in computing group 1 is 3, the number of computing nodes in computing group 2 is 3, and the number of computing nodes in computing group 3 is 2. Therefore, n = 3, L = 6, and w = n × L = 3 × 6 = 18.

[0135] like Figure 7 As shown, computation group 1 includes computation nodes 11 to 13. Each of these nodes contains raw data for performing reduce-scatter operations within computation group 1. The size of the raw data corresponding to any two of these nodes (i.e., any two nodes within each of these nodes) is equal. Computation node 11 divides its corresponding raw data into 18 data blocks, with data A1 to X1. Computation node 12 divides its corresponding raw data into 18 data blocks, with data A2 to X2. Computation node 13 divides its corresponding raw data into 18 data blocks, with data A3 to X3. Computation nodes 11-13 perform reduce-scatter operations within computation group 1 based on the 18 data blocks corresponding to each of computation nodes 11-13, obtaining reduce-scatter result data that is one-to-one with the reduce-scatter result data corresponding to any two of the computation nodes 11-13. The size of the reduce-scatter result data is equal to that of any two of the computation nodes 11-13. The reduce-scatter result data corresponding to computation node 11 includes 6 data blocks, whose data are A1+A2+A3, B1+B2+B3, C1+C2+C3, D1+D2+D3, E1+E2+E3, and F1+F2+F3, respectively. The reduce-scatter result data corresponding to computation node 12 also includes 6 data blocks, whose data are G1+G2+G3, H1+H2+H3, J1+J2+J3, K1+K2+K3, P1+P2+P3, and Q1+Q2+Q3, respectively. The reduce-scatter result data corresponding to computing node 13 includes 6 data blocks, namely R1+R2+R3, S1+S2+S3, T1+T2+T3, U1+U2+U3, V1+V2+V3 and X1+X2+X3.

[0136] like Figure 7 As shown, computation group 2 includes computation nodes 21 to 23. Each of these nodes contains raw data for performing reduce-scatter operations within computation group 2. The size of the raw data corresponding to any two of these nodes (i.e., any two nodes included in each of these nodes) is equal. Computation node 21 divides its corresponding raw data into 18 data blocks, with data values ​​A4 to X4. Computation node 22 divides its corresponding raw data into 18 data blocks, with data values ​​A5 to X5. Computation node 23 divides its corresponding raw data into 18 data blocks, with data values ​​A6 to X6. Computation nodes 21-23 perform reduce-scatter operations within computation group 2 based on the 18 data blocks corresponding to each of the nodes 21-23, obtaining reduce-scatter result data that is one-to-one with the reduce-scatter result data corresponding to any two of the nodes 21-23. The reduce-scatter result data corresponding to computation node 21 includes 6 data blocks: A4+A5+A6, B4+B5+B6, C4+C5+C6, D4+D5+D6, E4+E5+E6, and F4+F5+F6. The reduce-scatter result data corresponding to computation node 22 also includes 6 data blocks: G4+G5+G6, H4+H5+H6, J4+J5+J6, K4+K5+K6, P4+P5+P6, and Q4+Q5+Q6. The reduce-scatter result data corresponding to computing node 23 includes 6 data blocks, namely R4+R5+R6, S4+S5+S6, T4+T5+T6, U4+U5+U6, V4+V5+V6 and X4+X5+X6.

[0137] like Figure 7As shown, computation group 3 includes computation nodes 31 and 32. Each of computation nodes 31 and 32 contains raw data for performing reduce-scatter operations within computation group 3. The size of the raw data for performing reduce-scatter operations within computation group 3 is equal to the size of any two computation nodes 31 and 32 (i.e., any two computation nodes 31 and 32 contain the same data). Computation node 31 divides the raw data for performing reduce-scatter operations within computation group 3 into 18 data blocks, with data values ​​A7 to X7 respectively. Computation node 32 divides the raw data for performing reduce-scatter operations within computation group 3 into 18 data blocks, with data values ​​A8 to X8 respectively. Computation nodes 31 and 32 perform reduce-scatter operations within computation group 3 based on the 18 data blocks corresponding to each of them. This yields reduce-scatter result data corresponding to each of the 18 data blocks, with the result data being equal in size to the reduce-scatter result data corresponding to any two of the 18 data blocks. The reduce-scatter result data corresponding to computation node 31 consists of 9 data blocks: A7+A8, B7+B8, C7+C8, D7+D8, E7+E8, F7+F8, G7+G8, H7+H8, and J7+J8. The reduce-scatter result data corresponding to computation node 32 also consists of 9 data blocks: K7+K8, P7+P8, Q7+Q8, R7+R8, S7+S8, T7+T8, U7+U8, V7+V8, and X7+X8.

[0138] like Figure 7As shown, in the reduce-scatter results obtained by performing the reduce-scatter operation within each of the computation groups 1-3, corresponding one-to-one with computation nodes 11-13, 21-23, and 31-31, the boundary of the reduce-scatter result data corresponding to computation node 11 is aligned with the boundary of the reduce-scatter result data corresponding to computation node 21; the boundary of the reduce-scatter result data corresponding to computation node 12 is aligned with the boundary of the reduce-scatter result data corresponding to computation node 22; and the boundary of the reduce-scatter result data corresponding to computation node 13 is aligned with the boundary of the reduce-scatter result data corresponding to computation node 23. The boundaries of the reduce-scatter result data corresponding to other computation nodes are not aligned. For example, the boundary of the reduce-scatter result data corresponding to computation node 31 is not aligned with the boundary of the reduce-scatter result data corresponding to each of the computation nodes 11-13 and 21-23, and the boundary of the reduce-scatter result data corresponding to computation node 32 is not aligned with the boundary of the reduce-scatter result data corresponding to each of the computation nodes 11-13 and 21-23.

[0139] It should be noted that, for example Figure 4 As shown in calculation 10 above, the computing nodes divide the original data according to w = n × L. All computing nodes in each computing group 1 to 3 participate in the reduce-scatter operation. The size of the reduce-scatter result data obtained by performing the reduce-scatter operation in each computing group 1 to 3 is equal to the size of the reduce-scatter result data corresponding to any two computing nodes in that computing group. This kind of reduce-scatter operation performed within the computing group is also called the intra-group uniform reduce-scatter operation.

[0140] Please refer to Figure 11 and Figure 12 It illustrates an embodiment of the present application that provides a method for... Figure 5 This diagram illustrates the execution of reduce-scatter operations within a computing group in computing cluster 20. Figure 11 and Figure 12 With w = n × G max For example, to illustrate. Because in such... Figure 5In the computing cluster 20 shown, there are 3 computing groups. Computing group 1 has 5 computing nodes, computing group 2 has 4 computing nodes, and computing group 3 has 3 computing nodes. Therefore, the largest computing group is computing group 1, which has 5 computing nodes. n = 3, G max =5, w=n×G max =3×5=15.

[0141] like Figure 11 As shown, computation group 1 includes computation nodes 11 to 15. Each of these nodes contains raw data for performing reduce-scatter operations within computation group 1. The size of the raw data corresponding to any two of these nodes (i.e., any two nodes within each of these nodes) is equal. Computation node 11 divides its corresponding raw data into 15 data blocks, with data A1 to T1. Computation node 12 divides its corresponding raw data into 15 data blocks, with data A2 to T2. Computation node 13 divides its corresponding raw data into 15 data blocks, with data A3 to T3. Compute node 14 divides the raw data corresponding to it, used for performing the reduce-scatter operation within compute group 1, into 15 data blocks, with data in these 15 blocks ranging from A4 to T4. Compute node 15 divides the raw data corresponding to it, used for performing the reduce-scatter operation within compute group 1, into 15 data blocks, with data in these 15 blocks ranging from A5 to T5. See also... Figure 11 and Figure 12Computation nodes 11-15 perform reduce-scatter operations within computation group 1 based on the 15 data blocks corresponding to each of the 11-15 nodes, obtaining reduce-scatter result data that corresponds one-to-one with the reduce-scatter result data corresponding to any two of the 11-15 nodes. The size of the reduce-scatter result data is equal to that corresponding to any two of the 11-15 nodes. The reduce-scatter result data corresponding to computation node 11 includes 3 data blocks, namely A1+A2+A3+A4+A5, B1+B2+B3+B4+B5, and C1+C2+C3+C4+C5. The reduce-scatter result data corresponding to computation node 12 also includes 3 data blocks, namely D1+D2+D3+D4+D5, E1+E2+E3+E4+E5, and F1+F2+F3+F4+F5. The reduce-scatter result data corresponding to computing node 13 consists of three data blocks: G1+G2+G3+G4+G5, H1+H2+H3+H4+H5, and J1+J2+J3+J4+J5. The reduce-scatter result data corresponding to computing node 14 consists of three data blocks: K1+K2+K3+K4+K5, P1+P2+P3+P4+P5, and Q1+Q2+Q3+Q4+Q5. The reduce-scatter result data corresponding to computing node 15 consists of three data blocks: R1+R2+R3+R4+R5, S1+S2+S3+S4+S5, and T1+T2+T3+T4+T5.

[0142] like Figure 11As shown, computation group 2 includes computation nodes 21 to 24. Each of these nodes contains raw data for performing reduce-scatter operations within computation group 2. The size of the raw data corresponding to any two of these nodes (i.e., any two nodes within each of these nodes) is equal. Computation node 21 divides its corresponding raw data into 15 data blocks, with data values ​​A6 to T6. Computation node 22 divides its corresponding raw data into 15 data blocks, with data values ​​A7 to T7. Computation node 23 divides its corresponding raw data into 15 data blocks, with data values ​​A8 to T8. Compute node 24 divides the raw data corresponding to it, used to perform the reduce-scatter operation within compute group 2, into 15 data blocks, with data in these 15 blocks ranging from A9 to T9. See also... Figure 11 and Figure 12Computation nodes 21-24 perform reduce-scatter operations within computation group 2 based on the 15 data blocks corresponding to each of computation nodes 21-24, obtaining reduce-scatter result data corresponding one-to-one with each of computation nodes 21-24. The size of the reduce-scatter result data corresponding to any two of computation nodes 21-23 is equal to the size of the reduce-scatter result data corresponding to computation node 24. The size of the reduce-scatter result data corresponding to computation node 24 is not equal to the size of the reduce-scatter result data corresponding to any one of computation nodes 21-23, and the size of the reduce-scatter result data corresponding to computation node 24 is twice the size of the reduce-scatter result data corresponding to any one of computation nodes 21-23. The reduce-scatter result data corresponding to computation node 21 includes 3 data blocks, namely A6+A7+A8+A9, B6+B7+B8+B9, and C6+C7+C8+C9. The reduce-scatter result data corresponding to compute node 22 includes three data blocks: D6+D7+D8+D9, E6+E7+E8+E9, and F6+F7+F8+F9. The reduce-scatter result data corresponding to compute node 23 includes three data blocks: G6+G7+G8+G9, H6+H7+H8+H9, and J6+J7+J8+J9. The reduce-scatter result data corresponding to compute node 24 includes six data blocks: K6+K7+K8+K9, P6+P7+P8+P9, Q6+Q7+Q8+Q9, R6+R7+R8+R9, S6+S7+S8+S9, and T6+T7+T8+T9.

[0143] like Figure 11As shown, computation group 3 includes computation nodes 31 to 33. Each of these nodes contains raw data for performing reduce-scatter operations within computation group 3. The size of the raw data corresponding to any two of these nodes (i.e., any two nodes included in any two of these nodes) is equal to the size of the raw data used for performing reduce-scatter operations within computation group 3. Computation node 31 divides the raw data corresponding to it into 15 data blocks, with data ranging from A10 to T10. Computation node 32 divides the raw data corresponding to it into 15 data blocks, with data ranging from A11 to T11. Computation node 33 divides the raw data corresponding to it, used to perform the reduce-scatter operation within computation group 3, into 15 data blocks, with data in these 15 blocks ranging from A12 to T12. See also... Figure 11 and Figure 12Computation nodes 31-33 perform reduce-scatter operations within computation group 3 based on the 15 data blocks corresponding to each of them, obtaining reduce-scatter result data corresponding to each of the 15 data blocks. The size of the reduce-scatter result data corresponding to computation node 32 is equal to the size of the reduce-scatter result data corresponding to computation node 33, while the size of the reduce-scatter result data corresponding to computation node 31 is not equal to the size of the reduce-scatter result data corresponding to computation node 32, and the size of the reduce-scatter result data corresponding to computation node 32 is twice the size of the reduce-scatter result data corresponding to computation node 31. The reduce-scatter result data corresponding to computation node 31 includes 3 data blocks, namely A10+A11+A12, B10+B11+B12, and C10+C11+C12. The reduce-scatter result data corresponding to computing node 32 includes 6 data blocks, namely D10+D11+D12, E10+E11+E12, F10+F11+F12, G10+G11+G12, H10+H11+H12, and J10+J11+J12. The reduce-scatter result data corresponding to computing node 33 includes 6 data blocks, namely K10+K11+K12, P10+P11+P12, Q10+Q11+Q12, R10+R11+R12, S10+S11+S12, and T10+T11+T12.

[0144] like Figure 11 and Figure 12As shown, in the reduce-scatter results obtained by performing the reduce-scatter operation in each of the computation groups 1 to 3, corresponding to computation nodes 11 to 15, 21 to 24, and 31 to 33, the boundaries of the reduce-scatter results corresponding to computation nodes 11, 21, and 31 are aligned. The boundaries of the reduce-scatter results corresponding to computation nodes 12 and 22 are aligned with those corresponding to computation nodes 22, 13 and 23, and 24 and 33, respectively. The boundaries of the reduce-scatter result data corresponding to other computing nodes are not aligned. For example, the boundary of the reduce-scatter result data corresponding to computing node 32 is not aligned with the boundary of the reduce-scatter result data corresponding to each of computing nodes 11-15 and 21-24, and the boundary of the reduce-scatter result data corresponding to computing node 32 is not aligned with the boundary of the reduce-scatter result data corresponding to each of computing nodes 11-15 and 21-23.

[0145] It should be noted that, for example Figure 5 In the calculation shown in calculation 20, as described above, all computing nodes in computing group 1 participate in the reduce-scatter operation, and the size of the reduce-scatter result data corresponding to any two computing nodes in computing group 1 is equal. This type of reduce-scatter operation performed in computing group 1 is also called an intra-group uniform reduce-scatter operation. In computing groups 2-3, the size of the reduce-scatter result data corresponding to at least two computing nodes in each computing group is not equal. This type of reduce-scatter operation performed in computing groups 2-3 is also called an intra-group non-uniform reduce-scatter operation.

[0146] Based on the above description of step S601, it can be seen that the computing nodes can divide the original data according to w = n × L, or according to w = n × G. max The original data is then partitioned. Typically, w = n × L is suitable for scenarios where the number of computing nodes within different computing groups in a computing cluster is relatively even (e.g., ...). Figure 4 The computing cluster shown is 10); w = n × G max Suitable for scenarios where the number of compute nodes in different compute groups within a compute cluster is not uniform (e.g.) Figure 5 The computing cluster 20 shown is also applicable to scenarios where the number of computing nodes in each computing group within the computing cluster is relatively even. For example, in scenarios where the number of computing nodes in each computing group within the computing cluster is not evenly distributed, the least common multiple of the number of computing nodes in multiple computing groups may be large (i.e., L is large). If the computing nodes divide the original data according to w = n × L, the number of data blocks obtained may be large, resulting in small individual data block sizes and easily leading to data fragmentation. For example, the computing cluster includes 3 computing groups with 6, 7, and 8 computing nodes respectively. The least common multiple of 6, 7, and 8 is L = 6 × 7 × 8 = 336. If the original data is divided according to w = n × L, the number of data blocks obtained is w = n × L = 3 × 336 = 1008, which easily leads to small individual data block sizes, thus causing data fragmentation. However, if the data is divided according to w = n × L... max The original data is divided into data blocks, and the number of data blocks obtained is w = n × G. max =3 × 8 = 24. The number of data blocks is relatively small, and the size of each data block is relatively large, which will not lead to data fragmentation. Therefore, for scenarios where the number of computing nodes within different computing groups in a computing cluster is not uniform, we can follow the formula w = n × G. max The original data is then segmented. Of course, in scenarios where the number of computing nodes in different computing groups within a computing cluster is not uniform, the original data can also be segmented according to w = n × L. This application embodiment does not limit the segmentation method of the original data.

[0147] S602. The first computing node sends the first part of the first data to the second computing node in the second computing group, and sends the second part of the first data to the third computing node in the second computing group. The first part of the data corresponds to the second computing node, and the second part of the data corresponds to the third computing node. The first part of the data and the second part of the data are used to implement the second set communication operation between different computing groups in n computing groups. The second computing node and the third computing node are different.

[0148] In an optional embodiment, the first data further includes a third portion of data, which is also used to implement the second set communication operation between different computing groups in the n computing groups. This third portion of data corresponds to a third computing node within the second computing group; that is, the first portion of data in the first data corresponds to a computing node within the second computing group, and the second and third portions of data in the first data correspond to the same computing node within the second computing group. Alternatively, the third portion of data corresponds to a fourth computing node within the second computing group, where the second, third, and fourth computing nodes are all different; that is, the first, second, and third portions of data in the first data correspond one-to-one to three different computing nodes within the second computing group. When the third portion of data corresponds to a third computing node, the first computing node also sends the third portion of data to the third computing node. When the third portion of data corresponds to a fourth computing node, the first computing node also sends the third portion of data to the fourth computing node. It is easy to understand that multiple data segments in the first data correspond to multiple computing nodes within the second computing group, and at least two of these data segments correspond to the same computing node within the second computing group. Alternatively, there is a one-to-one correspondence between the multiple data segments in the first data and the multiple computing nodes within the second computing group. The first computing node sends each data segment from the first data to its corresponding computing node within the second computing group.

[0149] In an optional embodiment, the first computing node sequentially sends a first portion of the first data to the second computing node and a second portion of the first data to the third computing node. For example, the first computing node first sends the first portion of the data to the second computing node, and then sends the second portion of the data to the third computing node. Alternatively, the first computing node first sends the second portion of the data to the third computing node, and then sends the first portion of the data to the second computing node. That is, the process of the first computing node sending the first portion of the data to the second computing node is sequential with the process of the first computing node sending the second portion of the data to the third computing node. This avoids the first computing node simultaneously sending data to both the second and third computing nodes, preventing a "one-to-many" traffic flow and thus avoiding congestion on the first computing node. Furthermore, the first set communication operation result data corresponding to the second computing node may also include data corresponding to the first computing node, and the second computing node may also send the data corresponding to the first computing node to the first computing node; the first set communication operation result data corresponding to the third computing node may also include data corresponding to the first computing node, and the third computing node may also send the data corresponding to the first computing node to the first computing node; and the process of the second computing node sending data to the first computing node and the process of the third computing node sending data to the first computing node can be serialized, thereby avoiding the "multiple-to-one" traffic caused by the second computing node and the third computing node simultaneously sending data to the first computing node, and thus avoiding congestion of the first computing node.

[0150] In an optional embodiment, the first data further includes a third portion of data. When this third portion of data corresponds to a third computing node within the second computing group, the first computing node sequentially sends the first portion of the first data to the second computing node within the second computing group, sends the second portion of the first data to the third computing node, and then sends the third portion of the data to the third computing node. This avoids the first computing node simultaneously sending data to both the second and third computing nodes, preventing a "one-to-many" traffic flow and thus avoiding congestion at the first computing node. When the third portion of data corresponds to a fourth computing node within the second computing group, the first computing node sequentially sends the first portion of the first data to the second computing node within the second computing group, sends the second portion of the first data to the third computing node, and then sends the third portion of the data to the fourth computing node. This also avoids the first computing node simultaneously sending data to the second, third, and fourth computing nodes, preventing a "one-to-many" traffic flow and thus avoiding congestion at the first computing node.

[0151] In this embodiment, the transmission order of the first part of data and the second part of data is determined based on the priorities of the first part of data and the second part of data, where the priorities of the first part of data and the second part of data are different. When the first data also includes a third part of data, the transmission order of the first part of data, the second part of data, and the third part of data is determined based on the priorities of the first part of data, the second part of data, and the third part of data. In an optional embodiment, the first computing node corresponds to a first logical node and a second logical node; the first part of data corresponds to the first logical node; the second part of data corresponds to the second logical node; the priority of the first part of data is determined based on its data arrangement order within the logical group to which the first logical node belongs; and the priority of the second part of data is determined based on its data arrangement order within the logical group to which the second logical node belongs. For example, the first logical node belongs to the first logical group, and the second logical node belongs to the second logical group; the priority of the first part of data is determined based on its data arrangement order within the first logical group, and the priority of the second part of data is determined based on its data arrangement order within the second logical group. For example, both the first logical node and the second logical node belong to the first logical group. The priority of the first part of the data is determined based on its data arrangement order within the first logical group, and the priority of the second part of the data is also determined based on its data arrangement order within the first logical group. In an optional embodiment, the first computing node also corresponds to another logical node besides the first and second logical nodes, and the third part of the first data corresponds to this other logical node. The priority of the third part of the data is determined based on its data arrangement order within the logical group to which this other logical node belongs. The data arrangement order of any part of the first, second, and third data within the logical group to which the corresponding logical node belongs is determined based on the starting offset address of that part of the data. For example, the smaller the starting offset address of any part of the data, the earlier it appears in the logical group to which the corresponding logical node belongs. Conversely, the larger the starting offset address of any part of the data, the later it appears in the logical group to which the corresponding logical node belongs.

[0152] In this embodiment, the number of computing nodes in the minimum computing group is G. min This minimum computational group is the computational group with the fewest computational nodes among the above n computational groups, and the computational nodes in this minimum computational group are related to G. min Each logical group corresponds one-to-one with the G. minEach logical group in the G group includes at least one logical node, and the logical node corresponding to each computational node within the smallest computational group belongs to the logical group corresponding to that computational node. min A logical group includes a first logical group and may also include a second logical group. In some embodiments, a logical node is also referred to as a logical card or a logical computing card.

[0153] In an optional embodiment, each computing node within each of the n computing groups corresponds to at least one logical node. The second set communication operation is a set communication operation performed between different computing groups within the n computing groups for each logical node (or, in other words, a set communication operation performed between different computing groups within the n computing groups and within each logical node). When the same computing node corresponds to multiple logical nodes, the process of performing the second set communication operation between different computing groups within the n computing groups for the multiple logical nodes is serial. Therefore, it avoids the situation where, during the second set communication operation between different computing groups within the n computing groups for the multiple logical nodes, the same computing node sends data to multiple computing nodes in parallel, resulting in "one-to-many" traffic, and multiple computing nodes send data to the same computing node in parallel, resulting in "many-to-one" traffic, thus preventing congestion on the same computing node. For example, the first and second parts of data are used to implement a second set of communication operations between different computing groups in the n computing groups. The first part of data corresponds to a first logical node, and the second part of data corresponds to a second logical node. The first computing node corresponds to both the first and second logical nodes. The process of performing the second set of communication operations between different computing groups in the n computing groups for the first logical node (including the first computing node sending the first part of data to the second computing node) and the process of performing the second set of communication operations between different computing groups in the n computing groups for the second logical node (including the first computing node sending the second part of data to the third computing node) are not sequential. Therefore, during the second set of communication operations between different computing groups in the n computing groups for the first and second logical nodes, it is possible to avoid the first computing node from simultaneously sending data to both the second and third computing nodes, resulting in a "one-to-many" traffic flow, and the second and third computing nodes from simultaneously sending data to the first computing node, resulting in a "many-to-one" traffic flow, thus preventing congestion on the first computing node.

[0154] In an optional embodiment, the process of performing the second set communication operation between different computing groups within the n computing groups for logical nodes within the same logical group is serial, while the process of performing the second set communication operation between different computing groups within the n computing groups for logical nodes within different logical groups is parallel. This avoids the problems of "one-to-many" and "many-to-one" traffic during the second set communication operation between different computing groups within the n computing groups, while improving the efficiency of performing the second set communication operation between different computing groups within the n computing groups. For example, the first logical node belongs to the first logical group, the second logical node belongs to the second logical group, the fifth computing node in the first computing group corresponds to the third logical node, and the third logical node also belongs to the first logical group (i.e., the first and third logical nodes belong to the same logical group), and the sixth computing node in the first computing group corresponds to the fourth logical node, and the fourth logical node also belongs to the second logical group (i.e., the second and fourth logical nodes belong to the same logical group, and the fourth logical node and the first logical node belong to different logical groups). The process of performing a second set communication operation between different computing groups in the n computing groups for the first logical node is serial with the process of performing a second set communication operation between different computing groups in the n computing groups for the third logical node. The process of performing a second set communication operation between different computing groups in the n computing groups for the first logical node is parallel with the process of performing a second set communication operation between different computing groups in the n computing groups for the fourth logical node.

[0155] In this embodiment, the logical node is determined based on the boundary of the first set communication operation result data (which is also the aforementioned first data) corresponding to the computing nodes within the n computing groups. The first set communication operation result data corresponding to any computing node is the result data obtained by performing the first set communication operation within the computing group where that computing node is located. The logical group is determined based on the logical node and the computing nodes within the smallest computing group among the n computing groups. Each logical node corresponds to n computing nodes, and these n computing nodes are located one-to-one in the n computing groups. The first set communication operation result data corresponding to each of the n computing nodes includes the data corresponding to each logical node. The data corresponding to each logical node in each of the n computing nodes can be the first set communication operation result data corresponding to each computing node, or it can be a part of the first set communication operation result data corresponding to each computing node. The boundary of the data corresponding to the same logical node is aligned (i.e., the offset address of the data corresponding to the same logical node is the same). For ease of description, the data corresponding to a logical node in the first set communication operation result data corresponding to any computing node is called a data slice. The data slice is used to implement the second set communication operation between different computing groups in the n computing groups. It is easy to understand that since the first part of the first data corresponds to the first logical node and the second part of the first data corresponds to the second logical node, the first part of the data and the second part of the data are each a data slice.

[0156] In a specific embodiment, the first computing node segments the first set communication operation result data corresponding to at least one computing node in the n computing groups based on the boundaries of the first set communication operation result data corresponding to the computing nodes within the n computing groups. The first computing node determines multiple data slices based on the segmentation results. The first computing node determines multiple logical nodes based on these multiple data slices. Each logical node corresponds to n computing nodes and n data slices. The n computing nodes are located one-to-one within the n computing groups, and the n data slices correspond one-to-one with the n computing nodes. Each data slice is either the first set communication operation result data corresponding to a corresponding computing node, or a portion of the first set communication operation result data corresponding to a corresponding computing node obtained by segmenting it. The boundaries of the n data slices are aligned. The n data slices corresponding to each logical node in the multiple logical nodes are used to implement the second set communication operation performed between different computing groups within the n computing groups for each logical node. The plurality of logical nodes includes a first logical node and a second logical node. The first part of the first data is a data slice obtained by segmenting the first data and corresponding to the first logical node, and this first part of the data corresponds to the first computing node. The second part of the first data is a data slice obtained by segmenting the first data and corresponding to the second logical node, and this second part of the data corresponds to the second computing node.

[0157] For example, a first computing node segments the first data based on the boundaries of the first set communication operation result data corresponding to multiple computing nodes, obtaining a first part of data and a second part of data (i.e., obtaining two data slices). These multiple computing nodes are located within computing groups other than the first computing group among the n computing groups. The first set communication operation result data corresponding to each of these multiple computing nodes does not have a boundary aligned with the boundary of the first data. Specifically, the boundary of the first part of the data obtained by segmenting the first data is aligned with the boundary of the first set communication operation result data corresponding to at least one of the multiple computing nodes, and the boundary of the second part of the data obtained by segmenting the first data is also aligned with the boundary of the first set communication operation result data corresponding to at least one of the multiple computing nodes. The first computing node determines a first logical node based on the first portion of data obtained from the segmentation. The first logical node corresponds to n computing nodes and n data slices. Each of the n computing nodes is located in one of the n computing groups. Each of the n computing nodes includes the first computing node. Each of the n data slices corresponds one-to-one with the n computing nodes and includes the first portion of data. The boundaries of the n data slices are aligned. These n data slices are used to implement a second set of communication operations between different computing groups within the n computing groups, targeting the first logical node. The first computing node determines a second logical node based on the second portion of data obtained from the segmentation. The second logical node corresponds to n computing nodes and n data slices. Each of the n computing nodes is located in one of the n computing groups. Each of the n computing nodes includes the first computing node. Each of the n data slices corresponds one-to-one with the n computing nodes and includes the second portion of data. The boundaries of the n data slices are aligned. These n data slices are used to implement a second set of communication operations between different computing groups within the n computing groups, targeting the second logical node. In an optional embodiment, the first computing node segments the first data based on the boundaries of the first set communication operation result data corresponding to the plurality of computing nodes, and can also obtain a third part of data (the third part of data is a data slice). The boundary of the third part of data is aligned with the boundary of the first set communication operation result data corresponding to at least one of the plurality of computing nodes. The first computing node can also determine a logical node based on the third part of data. The logical node corresponds to n computing nodes and n data slices. The n computing nodes are located one-to-one in the n computing groups. The n computing nodes include the first computing node. The n data slices are one-to-one with the n computing nodes. The n data slices include the third part of data. The boundaries of the n data slices are aligned. The n data slices are used to implement the second set communication operation performed between different computing groups in the n computing groups for the logical node.

[0158] After determining the aforementioned multiple logical nodes based on the multiple data slices determined by the partitioning results, the first computing node determines G based on these multiple logical nodes and the computing nodes within the smallest computing group among the n computing groups. min A logical group, the G min Each logical group in the G group includes at least one logical node. min Each logical group corresponds one-to-one with a computing node within the smallest computing group, and the logical node corresponding to each computing node within the smallest computing group belongs to the logical group corresponding to each computing node.

[0159] To facilitate understanding of the content related to logical nodes, the following will be combined with... Figure 4 and Figure 5 The computing cluster shown below introduces the content related to logical nodes. The reduce-scatter result data mentioned below is also the result data of the first set of communication operations.

[0160] In one embodiment, please refer to Figure 8 It illustrates an embodiment of this application based on Figure 7 The provided reduce-scatter result data (i.e., in) Figure 4The diagram illustrates the partitioning of logical nodes into logical nodes based on the result data of reduce-scatter operations performed within a computing group in the computing cluster. The first computing node, based on the boundaries of the reduce-scatter result data corresponding to computing nodes 11-13, 21-23, and 31-32, further partitions the reduce-scatter result data corresponding to computing nodes 12, 22, and 31-32. The first computing node then determines multiple data slices based on these partitions: data slice 11, data slices 121-122, data slice 13, data slice 21, data slices 221-222, data slice 23, data slices 311-312, and data slices 321-322. Data slice 11 represents the reduce-scatter result data corresponding to computing node 11. Data slices 121 and 122 are portions of the reduce-scatter result data corresponding to computing node 12, and they are distinct. Data slice 13 is the reduce-scatter result data corresponding to computing node 13. Data slice 21 is the reduce-scatter result data corresponding to computing node 21. Data slices 221 and 222 are portions of the reduce-scatter result data corresponding to computing node 22, and data slices 221 and 222 are different. Data slice 23 is the reduce-scatter result data corresponding to computing node 23. Data slices 311 and 312 are portions of the reduce-scatter result data corresponding to computing node 31, and data slices 311 and 312 are different. Data slices 321 and 322 are portions of the reduce-scatter result data corresponding to computing node 32, and data slices 321 and 322 are different.

[0161] See Figure 8After the first computing node determines the aforementioned multiple data slices based on the partitioning results, it then determines multiple logical nodes based on these data slices. These logical nodes are logical nodes 1 to 4. Logical node 1 corresponds to computing nodes 11, 21, and 31, and also corresponds to data slices 11, 21, and 311. Computing nodes 11, 21, and 31 are located in computing groups 1 to 3 in a one-to-one correspondence. Data slice 11 corresponds to computing node 11, data slice 21 corresponds to computing node 21, and data slice 311 corresponds to computing node 31. The boundaries of these three data slices are aligned. Data slices 11, 21, and 311 are used to implement the second set communication operation performed on logical node 1 between different computing groups in computing groups 1 to 3. Logical node 2 corresponds to compute nodes 12, 22, and 31, and also corresponds to data slices 121, 221, and 312. Computation nodes 12, 22, and 31 are located in compute groups 1 to 3 in a one-to-one correspondence. Data slice 121 corresponds to compute node 12, data slice 221 corresponds to compute node 22, and data slice 312 corresponds to compute node 31. The boundaries of these three data slices are aligned. Data slices 121, 221, and 312 are used to implement the second set communication operation performed on logical node 2 between different compute groups in compute groups 1 to 3. Logical node 3 corresponds to compute nodes 12, 22, and 32, and also corresponds to data slices 122, 222, and 321. Computation nodes 12, 22, and 32 are located in compute groups 1 to 3 in a one-to-one correspondence. Data slice 122 corresponds to compute node 12, data slice 222 corresponds to compute node 22, and data slice 321 corresponds to compute node 32. The boundaries of these three data slices are aligned. Data slices 122, 222, and 321 are used to implement the second set communication operation performed on logical node 3 between different compute groups in compute groups 1 to 3.Logical node 4 corresponds to compute nodes 13, 23, and 32, and also corresponds to data slices 13, 23, and 322. Computation nodes 13, 23, and 32 are located in compute groups 1 to 3 in a one-to-one correspondence. Data slice 13 corresponds to compute node 13, data slice 23 corresponds to compute node 23, and data slice 322 corresponds to compute node 32. The boundaries of these three data slices are aligned. Data slices 13, 23, and 322 are used to implement the second set communication operation performed on logical node 4 between different compute groups in compute groups 1 to 3.

[0162] See Figure 8 After the first computing node determines the multiple logical nodes based on the multiple data slices, the first computing node determines G based on these multiple logical nodes and the computing nodes (i.e., computing nodes 31-32) within the smallest computing group (i.e., computing group 3). min (here G) min =2) logic groups. For example, the G min The logic groups are logic group 1 and logic group 2. Logic group 1 corresponds to computing node 31 and includes logic nodes 1 to 2 corresponding to computing node 31. Logic group 2 corresponds to computing node 32 and includes logic nodes 3 to 4 corresponding to computing node 32.

[0163] See Figure 8 The first calculation group and the second calculation group can be any two calculation groups from calculation group 1 to 3. The first calculation node can be any calculation node in the first calculation group. The second calculation node and the third calculation node are two calculation nodes in the second calculation group.

[0164] See Figure 8In one example, the first computation group is computation group 1, the second computation group is computation group 3, the first computation node is computation node 12, the second computation node is computation node 31, and the third computation node is computation node 32. The first part of the data is data slice 121, and the second part of the data is data slice 122. Data slice 121 corresponds to computation node 31, and data slice 122 corresponds to computation node 32. The first logical node corresponding to the first part of the data is logical node 2, and the second logical node corresponding to the second part of the data is logical node 3. The first logical node belongs to logical group 1, and the second logical node belongs to logical group 2. Computation node 12 sends data slice 121 to computation node 31 and data slice 122 to computation node 32 in sequence. The sending order of data slice 121 and data slice 122 is determined based on the priority of data slice 121 and the priority of data slice 122. The priority of data slice 121 is determined based on the data arrangement order of data slice 121 within logical group 1, and the priority of data slice 122 is determined based on the data arrangement order of data slice 122 within logical group 2. This example uses computing node 31 as the second computing node and computing node 32 as the third computing node. The second computing node can also be computing node 32, and the third computing node can also be computing node 31. In this case, the first part of the data is data slice 122, the second part of the data is data slice 121, the first logical node corresponding to the first part of the data is logical node 3, the second logical node corresponding to the second part of the data is logical node 2, the first logical node belongs to logical group 2, and the second logical node belongs to logical group 1.

[0165] See Figure 8In another example, the first computation group is computation group 2, the second computation group is computation group 3, the first computation node is computation node 22, the second computation node is computation node 31, and the third computation node is computation node 32. The first part of the data is data slice 221, and the second part of the data is data slice 222. Data slice 221 corresponds to computation node 31, and data slice 222 corresponds to computation node 32. The first logical node corresponding to the first part of the data is logical node 2, and the second logical node corresponding to the second part of the data is logical node 3. The first logical node belongs to logical group 1, and the second logical node belongs to logical group 2. Computation node 22 sends data slice 221 to computation node 31 and data slice 222 to computation node 32 in sequence. The sending order of data slice 221 and data slice 222 is determined based on the priority of data slice 221 and the priority of data slice 222. The priority of data slice 221 is determined based on the data arrangement order of data slice 221 within logical group 1, and the priority of data slice 222 is determined based on the data arrangement order of data slice 222 within logical group 2. This example uses computing node 31 as the second computing node and computing node 32 as the third computing node. The second computing node can also be computing node 32, and the third computing node can also be computing node 31. In this case, the first part of the data is data slice 222, the second part of the data is data slice 221, the first logical node corresponding to the first part of the data is logical node 3, the second logical node corresponding to the second part of the data is logical node 2, the first logical node belongs to logical group 2, and the second logical node belongs to logical group 1.

[0166] See Figure 8In another example, the first computation group is computation group 3, the second computation group is computation group 1, the first computation node is computation node 31, the second computation node is computation node 11, and the third computation node is computation node 12. The first part of the data is data slice 311, and the second part of the data is data slice 312. Data slice 311 corresponds to computation node 11, and data slice 312 corresponds to computation node 12. The first logical node corresponding to the first part of the data is logical node 1, and the second logical node corresponding to the second part of the data is logical node 2. Both the first and second logical nodes belong to logical group 1. Computation node 31 sends data slice 311 to computation node 11 and data slice 312 to computation node 12 in sequence. The sending order of data slice 311 and data slice 312 is determined based on the priority of data slice 311 and the priority of data slice 312. The priority of data slice 311 is determined based on the data arrangement order of data slice 311 within logical group 1, and the priority of data slice 312 is determined based on the data arrangement order of data slice 312 within logical group 1. This example uses computing node 11 as the second computing node and computing node 12 as the third computing node. The second computing node can also be computing node 12, and the third computing node can also be computing node 11. In this case, the first part of the data is data slice 312, the second part of the data is data slice 311, the first logical node corresponding to the first part of the data is logical node 2, and the second logical node corresponding to the second part of the data is logical node 1. Both the first logical node and the second logical node belong to logical group 1.

[0167] See Figure 8In another example, the first computation group is computation group 3, the second computation group is computation group 2, the first computation node is computation node 31, the second computation node is computation node 21, and the third computation node is computation node 22. The first part of the data is data slice 311, and the second part of the data is data slice 312. Data slice 311 corresponds to computation node 21, and data slice 312 corresponds to computation node 22. The first logical node corresponding to the first part of the data is logical node 1, and the second logical node corresponding to the second part of the data is logical node 2. Both the first and second logical nodes belong to logical group 1. Computation node 31 sends data slice 311 to computation node 21 and data slice 312 to computation node 22 in sequence. The sending order of data slice 311 and data slice 312 is determined based on the priority of data slice 311 and the priority of data slice 312. The priority of data slice 311 is determined based on the data arrangement order of data slice 311 within logical group 1, and the priority of data slice 312 is determined based on the data arrangement order of data slice 312 within logical group 1. This example uses computing node 21 as the second computing node and computing node 22 as the third computing node. The second computing node can also be computing node 22, and the third computing node can also be computing node 21. In this case, the first part of the data is data slice 312, the second part of the data is data slice 311, the first logical node corresponding to the first part of the data is logical node 2, and the second logical node corresponding to the second part of the data is logical node 1. Both the first logical node and the second logical node belong to logical group 1.

[0168] See Figure 8In another example, the first computation group is computation group 3, the second computation group is computation group 1, the first computation node is computation node 32, the second computation node is computation node 12, and the third computation node is computation node 13. The first part of the data is data slice 321, and the second part of the data is data slice 322. Data slice 321 corresponds to computation node 12, and data slice 322 corresponds to computation node 13. The first logical node corresponding to the first part of the data is logical node 3, and the second logical node corresponding to the second part of the data is logical node 4. Both the first and second logical nodes belong to logical group 2. Computation node 32 sends data slice 321 to computation node 12 and data slice 322 to computation node 13 in sequence. The sending order of data slice 321 and data slice 322 is determined based on the priority of data slice 321 and the priority of data slice 322. The priority of data slice 321 is determined based on the data arrangement order of data slice 321 within logical group 2, and the priority of data slice 322 is determined based on the data arrangement order of data slice 322 within logical group 2. This example uses computing node 12 as the second computing node and computing node 13 as the third computing node. The second computing node can also be computing node 13, and the third computing node can also be computing node 12. In this case, the first part of the data is data slice 322, the second part of the data is data slice 321, the first logical node corresponding to the first part of the data is logical node 4, and the second logical node corresponding to the second part of the data is logical node 3. Both the first logical node and the second logical node belong to logical group 2.

[0169] See Figure 8In another example, the first computation group is computation group 3, the second computation group is computation group 2, the first computation node is computation node 32, the second computation node is computation node 22, and the third computation node is computation node 23. The first part of the data is data slice 321, and the second part of the data is data slice 322. Data slice 321 corresponds to computation node 22, and data slice 322 corresponds to computation node 23. The first logical node corresponding to the first part of the data is logical node 3, and the second logical node corresponding to the second part of the data is logical node 4. Both the first and second logical nodes belong to logical group 2. Computation node 32 sends data slice 321 to computation node 22 and data slice 322 to computation node 23 in sequence. The sending order of data slice 321 and data slice 322 is determined based on the priority of data slice 321 and the priority of data slice 322. The priority of data slice 321 is determined based on the data arrangement order of data slice 321 within logical group 2, and the priority of data slice 322 is determined based on the data arrangement order of data slice 322 within logical group 2. This example uses computing node 22 as the second computing node and computing node 23 as the third computing node. The second computing node can also be computing node 23, and the third computing node can also be computing node 22. In this case, the first part of the data is data slice 322, the second part of the data is data slice 321, the first logical node corresponding to the first part of the data is logical node 4, and the second logical node corresponding to the second part of the data is logical node 3. Both the first logical node and the second logical node belong to logical group 2.

[0170] See Figure 8The second set communication operation can be a set communication operation performed between different computing groups in computing groups 1-3 for each logical node in logical nodes 1-4. Since computing nodes 12 and 22 both correspond to logical nodes 2-3 (i.e., each computing node in computing nodes 12 and 22 corresponds to 2 logical nodes), the process of performing the second set communication operation between different computing groups in computing groups 1-3 for logical node 2 is serial to the process of performing the second set communication operation between different computing groups in computing groups 1-3 for logical node 3. Since computing node 31 corresponds to logical nodes 1-2, the process of performing the second set communication operation between different computing groups in computing groups 1-3 for logical node 1 is serial to the process of performing the second set communication operation between different computing groups in computing groups 1-3 for logical node 2. Since computing node 32 corresponds to logical nodes 3-4, the process of performing the second set communication operation between different computing groups in computing groups 1-3 for logical node 3 is serial to the process of performing the second set communication operation between different computing groups in computing groups 1-3 for logical node 4. Furthermore, since logical node 1 and logical node 2 belong to the same logical group (i.e., logical group 1), the process of performing the second set communication operation for logical node 1 between different computing groups in computing groups 1-3 is serial with the process of performing the second set communication operation for logical node 2 between different computing groups in computing groups 1-3. Since logical node 3 and logical node 4 belong to the same logical group (i.e., logical group 2), the process of performing the second set communication operation for logical node 3 between different computing groups in computing groups 1-3 is serial with the process of performing the second set communication operation for logical node 4 between different computing groups in computing groups 1-3. Moreover, the process of performing the second set communication operation for logical nodes within logical group 1 between different computing groups in computing groups 1-3 is parallel with the process of performing the second set communication operation for logical nodes within logical group 2 between different computing groups in computing groups 1-3. Based on this, it can be determined that the process of performing the second set communication operation for logical node 1 between different computing groups in computing groups 1 to 3 is parallel to the process of performing the second set communication operation for logical node 3 between different computing groups in computing groups 1 to 3; the process of performing the second set communication operation for logical node 1 between different computing groups in computing groups 1 to 3 is serial to the process of performing the second set communication operation for logical node 2 between different computing groups in computing groups 1 to 3; and the process of performing the second set communication operation for logical node 3 between different computing groups in computing groups 1 to 3 is serial to the process of performing the second set communication operation for logical node 4 between different computing groups in computing groups 1 to 3.For example, firstly, a second set communication operation is performed between different computing groups in computing groups 1-3 for logical node 1. Simultaneously, a second set communication operation is performed between different computing groups in computing groups 1-3 for logical node 3. After performing the second set communication operation between different computing groups in computing groups 1-3 for logical node 1, a second set communication operation is performed between different computing groups in computing groups 1-3 for logical node 2. After performing the second set communication operation between different computing groups in computing groups 1-3 for logical node 3, a second set communication operation is performed between different computing groups in computing groups 1-3 for logical node 4. In a specific example, for... Figure 8 The compute node 12 shown first performs a second set communication operation on logical node 3 based on data slice 122, and then performs a second set communication operation on logical node 2 based on data slice 121. For Figure 8 The computing node 22 shown first performs a second set communication operation on logical node 3 based on data slice 222, and then performs a second set communication operation on logical node 2 based on data slice 221. For Figure 8 The computing node 31 shown performs a second set communication operation on logical node 1 based on data slice 311, and then performs a second set communication operation on logical node 2 based on data slice 312. For Figure 8 The computing node 32 shown first performs a second set communication operation on logical node 3 based on data slice 321, and then performs a second set communication operation on logical node 4 based on data slice 322.

[0171] In another embodiment, please refer to Figure 13 It illustrates an embodiment of this application based on Figure 12 The provided reduce-scatter result data (i.e., in) Figure 5The diagram illustrates the partitioning of logical nodes into logical nodes based on the result data of reduce-scatter operations performed within a computing group in the computing cluster. The first computing node, based on the boundaries of the reduce-scatter result data corresponding to computing nodes 12-13, 22-23, and 14-15, further partitions the reduce-scatter result data corresponding to computing nodes 24 and 32-33. The first computing node then determines multiple data slices based on these partitions: data slices 11-15, 21-23, 241-242, 31, 321-322, and 331-332. Data slices 11-15 represent the reduce-scatter result data corresponding to computing nodes 11-15. Data slices 21-23 represent the reduce-scatter result data corresponding to computing nodes 21-23. Data slices 241 and 242 are portions of the reduce-scatter results corresponding to computation node 24, and they are different. Data slice 31 is the reduce-scatter result data corresponding to computation node 31. Data slices 321 and 322 are portions of the reduce-scatter results corresponding to computation node 32, and they are different. Data slices 331 and 332 are portions of the reduce-scatter results corresponding to computation node 33, and they are different.

[0172] See Figure 13After the first computing node determines the aforementioned multiple data slices based on the partitioning results, it then determines multiple logical nodes based on these data slices. These logical nodes are logical nodes 1 to 5. Logical node 1 corresponds to computing nodes 11, 21, and 31, and also corresponds to data slices 11, 21, and 31. Computing nodes 11, 21, and 31 are located in computing groups 1 to 3 in a one-to-one correspondence. Data slice 11 corresponds to computing node 11, data slice 21 corresponds to computing node 21, and data slice 31 corresponds to computing node 31. The boundaries of these three data slices are aligned. Data slices 11, 21, and 31 are used to implement the second set communication operation performed on logical node 1 between different computing groups in computing groups 1 to 3. Logical node 2 corresponds to compute nodes 12, 22, and 32, and also corresponds to data slices 12, 22, and 321. Computation nodes 12, 22, and 32 are located in compute groups 1 to 3 in a one-to-one correspondence. Data slice 12 corresponds to compute node 12, data slice 22 corresponds to compute node 22, and data slice 321 corresponds to compute node 32. The boundaries of these three data slices are aligned. Data slices 12, 22, and 321 are used to implement the second set communication operation performed on logical node 2 between different compute groups in compute groups 1 to 3. Logical node 3 corresponds to compute nodes 13, 23, and 32, and also corresponds to data slices 13, 23, and 322. Computation nodes 13, 23, and 32 are located in compute groups 1 to 3 in a one-to-one correspondence. Data slice 13 corresponds to compute node 13, data slice 23 corresponds to compute node 23, and data slice 322 corresponds to compute node 32. The boundaries of these three data slices are aligned. Data slices 13, 23, and 322 are used to implement the second set communication operation performed on logical node 3 between different compute groups in compute groups 1 to 3.Logical node 4 corresponds to compute nodes 14, 24, and 33, and also corresponds to data slices 14, 241, and 331. Computation nodes 14, 24, and 33 are located in compute groups 1 to 3 in a one-to-one correspondence. Data slice 14 corresponds to compute node 14, data slice 241 corresponds to compute node 24, and data slice 331 corresponds to compute node 33. The boundaries of these three data slices are aligned. Data slices 14, 241, and 331 are used to implement the second set communication operation performed on logical node 4 between different compute groups in compute groups 1 to 3. Logical node 5 corresponds to compute nodes 15, 24, and 33, and also corresponds to data slices 15, 242, and 332. Computation nodes 15, 24, and 33 are located in compute groups 1 to 3 in a one-to-one correspondence. Data slice 15 corresponds to compute node 15, data slice 242 corresponds to compute node 24, and data slice 332 corresponds to compute node 33. The boundaries of these three data slices are aligned. Data slices 15, 242, and 332 are used to implement the second set communication operation performed on logical node 5 between different compute groups in compute groups 1 to 3.

[0173] See Figure 13 After the first computing node determines the multiple logical nodes based on the multiple data slices, the first computing node determines G based on these multiple logical nodes and the computing nodes (i.e., computing nodes 31-33) within the smallest computing group (i.e., computing group 3). min (here G) min =3) logic groups. For example, the G min The logic groups are logic group 1, logic group 2, and logic group 3. Logic group 1 corresponds to computing node 31 and includes logic node 1 corresponding to computing node 31. Logic group 2 corresponds to computing node 32 and includes logic nodes 2 to 3 corresponding to computing node 32. Logic group 3 corresponds to computing node 33 and includes logic nodes 4 to 5 corresponding to computing node 33.

[0174] See Figure 13 The first calculation group and the second calculation group can be any two calculation groups from calculation group 1 to 3. The first calculation node can be any calculation node in the first calculation group. The second calculation node and the third calculation node are two calculation nodes in the second calculation group.

[0175] See Figure 13In one example, the first computation group is computation group 2, the second computation group is computation group 1, the first computation node is computation node 24, the second computation node is computation node 14, and the third computation node is computation node 15. The first part of the data is data slice 241, and the second part of the data is data slice 242. Data slice 241 corresponds to computation node 14, and data slice 242 corresponds to computation node 15. The first logical node corresponding to the first part of the data is logical node 4, and the second logical node corresponding to the second part of the data is logical node 5. Both the first and second logical nodes belong to logical group 3. Computation node 24 sends data slice 241 to computation node 14 and data slice 242 to computation node 15 in sequence. The sending order of data slice 241 and data slice 242 is determined based on the priorities of data slice 241 and data slice 242. The priority of data slice 241 is determined based on the data arrangement order of data slice 241 within logical group 3, and the priority of data slice 242 is determined based on the data arrangement order of data slice 242 within logical group 3. This example uses computing node 14 as the second computing node and computing node 15 as the third computing node. The second computing node can also be computing node 15, and the third computing node can also be computing node 14. In this case, the first part of the data is data slice 242, the second part of the data is data slice 241, the first logical node corresponding to the first part of the data is logical node 5, the second logical node corresponding to the second part of the data is logical node 5, and both the first and second logical nodes belong to logical group 3.

[0176] See Figure 13In another example, the first computation group is computation group 3, the second computation group is computation group 1, the first computation node is computation node 33, the second computation node is computation node 14, and the third computation node is computation node 15. The first part of the data is data slice 331, and the second part of the data is data slice 332. Data slice 331 corresponds to computation node 14, and data slice 332 corresponds to computation node 15. The first logical node corresponding to the first part of the data is logical node 4, and the second logical node corresponding to the second part of the data is logical node 5. Both the first and second logical nodes belong to logical group 3. Computation node 33 sends data slice 331 to computation node 14 and data slice 332 to computation node 15 in sequence. The sending order of data slice 331 and data slice 332 is determined based on the priority of data slice 331 and the priority of data slice 332. The priority of data slice 331 is determined based on the data arrangement order of data slice 331 within logical group 3, and the priority of data slice 332 is determined based on the data arrangement order of data slice 332 within logical group 3. This example uses computing node 14 as the second computing node and computing node 15 as the third computing node. The second computing node can also be computing node 15, and the third computing node can also be computing node 14. In this case, the first part of the data is data slice 332, the second part of the data is data slice 331, the first logical node corresponding to the first part of the data is logical node 5, the second logical node corresponding to the second part of the data is logical node 5, and both the first and second logical nodes belong to logical group 3.

[0177] See Figure 13In another example, the first computation group is computation group 3, the second computation group is computation group 1, the first computation node is computation node 32, the second computation node is computation node 12, and the third computation node is computation node 13. The first part of the data is data slice 321, and the second part of the data is data slice 322. Data slice 321 corresponds to computation node 12, and data slice 322 corresponds to computation node 13. The first logical node corresponding to the first part of the data is logical node 2, and the second logical node corresponding to the second part of the data is logical node 3. Both the first and second logical nodes belong to logical group 2. Computation node 32 sends data slice 321 to computation node 12 and data slice 322 to computation node 13 in sequence. The sending order of data slice 321 and data slice 322 is determined based on the priority of data slice 321 and the priority of data slice 322. The priority of data slice 321 is determined based on the data arrangement order of data slice 321 within logical group 2, and the priority of data slice 322 is determined based on the data arrangement order of data slice 322 within logical group 2. This example uses computing node 12 as the second computing node and computing node 13 as the third computing node. The second computing node can also be computing node 13, and the third computing node can also be computing node 12. In this case, the first part of the data is data slice 322, the second part of the data is data slice 321, the first logical node corresponding to the first part of the data is logical node 3, and the second logical node corresponding to the second part of the data is logical node 2. Both the first logical node and the second logical node belong to logical group 2.

[0178] See Figure 13In another example, the first computation group is computation group 3, the second computation group is computation group 2, the first computation node is computation node 32, the second computation node is computation node 22, and the third computation node is computation node 23. The first part of the data is data slice 321, and the second part of the data is data slice 322. Data slice 321 corresponds to computation node 22, and data slice 322 corresponds to computation node 23. The first logical node corresponding to the first part of the data is logical node 2, and the second logical node corresponding to the second part of the data is logical node 3. Both the first and second logical nodes belong to logical group 2. Computation node 32 sends data slice 321 to computation node 22 and data slice 322 to computation node 23 in sequence. The sending order of data slice 321 and data slice 322 is determined based on the priority of data slice 321 and the priority of data slice 322. The priority of data slice 321 is determined based on the data arrangement order of data slice 321 within logical group 2, and the priority of data slice 322 is determined based on the data arrangement order of data slice 322 within logical group 2. This example uses computing node 22 as the second computing node and computing node 23 as the third computing node. The second computing node can also be computing node 23, and the third computing node can also be computing node 22. In this case, the first part of the data is data slice 322, the second part of the data is data slice 321, the first logical node corresponding to the first part of the data is logical node 3, and the second logical node corresponding to the second part of the data is logical node 2. Both the first logical node and the second logical node belong to logical group 2.

[0179] See Figure 13The second set communication operation can be a set communication operation performed between different computing groups in computing groups 1-3 for each logical node in logical nodes 1-5. Since computing nodes 24 and 33 both correspond to logical nodes 4-5 (that is, each computing node in computing nodes 24 and 33 corresponds to 2 logical nodes), the process of performing the second set communication operation between different computing groups in computing groups 1-3 for logical node 4 is serial to the process of performing the second set communication operation between different computing groups in computing groups 1-3 for logical node 5. Since computing node 32 corresponds to logical nodes 2-3, the process of performing the second set communication operation between different computing groups in computing groups 1-3 for logical node 2 is serial to the process of performing the second set communication operation between different computing groups in computing groups 1-3 for logical node 3. Furthermore, since logical node 2 and logical node 3 belong to the same logical group (i.e., logical group 2), the process of performing the second set communication operation for logical node 2 between different computing groups in computing groups 1-3 is serial with the process of performing the second set communication operation for logical node 3 between different computing groups in computing groups 1-3. Since logical node 4 and logical node 5 belong to the same logical group (i.e., logical group 3), the process of performing the second set communication operation for logical node 4 between different computing groups in computing groups 1-3 is serial with the process of performing the second set communication operation for logical node 5 between different computing groups in computing groups 1-3. Moreover, the processes of performing the second set communication operation for logical nodes within logical group 1 between different computing groups in computing groups 1-3, for logical nodes within logical group 2 between different computing groups in computing groups 1-3, and for logical nodes within logical group 3 between different computing groups in computing groups 1-3 can be parallel. Based on this, it can be determined that the processes of performing the second set communication operation on logical node 1 between different computing groups in computing groups 1 to 3, the processes of performing the second set communication operation on logical nodes 2 to 3 between different computing groups in computing groups 1 to 3, and the processes of performing the second set communication operation on logical nodes 4 to 5 between different computing groups in computing groups 1 to 3 are parallel; the processes of performing the second set communication operation on logical node 2 between different computing groups in computing groups 1 to 3 are serial with the processes of performing the second set communication operation on logical node 3 between different computing groups in computing groups 1 to 3; and the processes of performing the second set communication operation on logical node 4 between different computing groups in computing groups 1 to 3 are serial with the processes of performing the second set communication operation on logical node 5 between different computing groups in computing groups 1 to 3.For example, firstly, a second set communication operation is performed between different computing groups in computing groups 1-3 for logical node 1. Simultaneously, a second set communication operation is performed between different computing groups in computing groups 1-3 for logical node 2, and also between different computing groups in computing groups 1-3 for logical node 4. After performing the second set communication operation between different computing groups in computing groups 1-3 for logical node 2, a second set communication operation is performed between different computing groups in computing groups 1-3 for logical node 3. After performing the second set communication operation between different computing groups in computing groups 1-3 for logical node 4, a second set communication operation is performed between different computing groups in computing groups 1-3 for logical node 5.

[0180] It should be noted that the above combination Figure 8 and Figure 13 The section introducing logical nodes only provides examples of the first and second logical nodes. For information on the third and fourth logical nodes, please refer to the sections on the first and second logical nodes. For example... Figure 8 As shown, the first logical node is one of logical nodes 2 and 3; the second logical node is the other logical node between logical nodes 2 and 3; the third logical node is one of logical nodes 1 and 4; and the fourth logical node is the other logical node between logical nodes 1 and 4. For example... Figure 13 As shown, the first logical node is one of logical nodes 4 and 5, the second logical node is the other logical node between logical nodes 4 and 5, and the third and fourth logical nodes are any two logical nodes from 1 to 3. For example... Figure 13 As shown, the first logical node is one of logical nodes 2 and 3, the second logical node is the other logical node of logical nodes 2 and 3, and the third and fourth logical nodes are any two logical nodes 1 and 4 to 5.

[0181] In this embodiment, the second set communication operation can be an all-reduce operation performed between different computing groups in n computing groups, and this all-reduce operation can be an all-reduce operation performed on a logical node. For ease of description, the result data obtained by performing the second set communication operation is referred to as the second set communication operation result data. When the second set communication operation is an all-reduce operation, the second set communication operation result data is also referred to as all-reduce result data. The second set communication operation result data corresponding to any computing node is used to implement the third set communication operation performed within the computing group where that computing node is located. For example, the second data corresponding to the first computing node is the data obtained by performing the second set communication operation between different computing groups in the n computing groups (that is, the second data corresponding to the first computing node is the second set communication operation result data corresponding to the first computing node), and this second data is used to implement the third set communication operation performed within the first computing group. The third set communication operation can be an all-gather operation. For ease of description, the result data obtained from executing the third set communication operation will be referred to as the third set communication operation result data. When the third set communication operation is an all-gather operation, the result data will also be called the all-gather result data. The following examples illustrate this with the second set communication operation being an all-reduce operation and the third set communication operation being an all-gather operation. Figure 4 and Figure 5 The illustrated computing cluster describes the execution of all-reduce operations between different computing groups within n computing groups, as well as the execution of all-gather operations within a computing group.

[0182] In one embodiment, see Figure 9 and combined Figure 8 , Figure 9 This application provides an embodiment of a method for computing groups 1 to 3 (i.e., Figure 4 The n computing groups in the computing cluster shown are interacting with each other. Figure 8This diagram illustrates the result of an all-reduce operation performed on the divided logical nodes. Data slice 11 corresponding to compute node 11, data slice 21 corresponding to compute node 21, and data slice 311 corresponding to compute node 31 are used to implement the all-reduce operation performed on logical node 1 within compute groups 1-3. Compute nodes 11, 21, and 31 perform the all-reduce operation on logical node 1 within compute groups 1-3 based on data slices 11, 21, and 311, obtaining all-reduce result data corresponding one-to-one with compute nodes 11, 21, and 31. The results obtained from the all-reduce operation on logical node 1 correspond to the data of compute nodes 11, 21, and 31. The all-reduce result data is the same. The all-reduce result data obtained by performing the all-reduce operation on logical node 1 and corresponding to each of the computing nodes 11, 21 and 31 includes 6 data blocks. The data of these 6 data blocks are A1+A2+A3+A4+A5+A6+A7+A8, B1+B2+B3+B4+B5+B6+B7+B8, C1+C2+C3+C4+C5+C6+C7+C8, D1+D2+D3+D4+D5+D6+D7+D8, E1+E2+E3+E4+E5+E6+E7+E8 and F1+F2+F3+F4+F5+F6+F7+F8. Data slice 121 corresponding to compute node 12, data slice 221 corresponding to compute node 22, and data slice 312 corresponding to compute node 31 are used to implement the all-reduce operation performed on logical node 2 in compute groups 1-3. Compute nodes 12, 22, and 31 perform the all-reduce operation on logical node 2 in compute groups 1-3 based on data slices 121, 221, and 312, obtaining all-reduce result data corresponding one-to-one with compute nodes 12, 22, and 31, respectively. Then, the operation is performed on logical node 2. The all-reduce operation yields the same all-reduce result data as compute nodes 12, 22, and 31. The all-reduce result data obtained by performing the all-reduce operation on logical node 2, corresponding to each of compute nodes 12, 22, and 31, includes three data blocks. The data in these three data blocks are G1+G2+G3+G4+G5+G6+G7+G8, H1+H2+H3+H4+H5+H6+H7+H8, and J1+J2+J3+J4+J5+J6+J7+J8, respectively.Data slice 122 corresponding to compute node 12, data slice 222 corresponding to compute node 22, and data slice 321 corresponding to compute node 32 are used to implement the all-reduce operation performed on logical node 3 between compute groups 1 and 3. Compute nodes 12, 22, and 32 perform the all-reduce operation on logical node 3 between compute groups 1 and 3 based on data slices 122, 222, and 321, obtaining all-reduce result data that corresponds one-to-one with compute nodes 12, 22, and 32, respectively. The operation is then performed on logical node 3. The all-reduce operation yields the same all-reduce result data as compute nodes 12, 22, and 32. The all-reduce result data obtained by performing the all-reduce operation on logical node 3, corresponding to each of compute nodes 12, 22, and 32, includes three data blocks. The data in these three data blocks are K1+K2+K3+K4+K5+K6+K7+K8, P1+P2+P3+P4+P5+P6+P7+P8, and Q1+Q2+Q3+Q4+Q5+Q6+Q7+Q8, respectively. Data slice 13 corresponding to compute node 13, data slice 23 corresponding to compute node 23, and data slice 322 corresponding to compute node 32 are used to implement the all-reduce operation performed on logical node 4 between compute groups 1 and 3. Compute nodes 13, 23, and 32 perform the all-reduce operation on logical node 4 between compute groups 1 and 3 based on data slices 13, 23, and 322, obtaining all-reduce result data that corresponds one-to-one with compute nodes 13, 23, and 32. The data slices 13, 23, and 32 obtained by performing the all-reduce operation on logical node 4 correspond to the data slices 13, 23, and 32. The all-reduce result data is the same. The all-reduce result data obtained by performing the all-reduce operation on logical node 4 and corresponding to each of the computing nodes in computing nodes 13, 23 and 32 includes 6 data blocks. The data of these 6 data blocks are R1+R2+R3+R4+R5+R6+R7+R8, S1+S2+S3+S4+S5+S6+S7+S8, T1+T2+T3+T4+T5+T6+T7+T8, U1+U2+U3+U4+U5+U6+U7+U8, V1+V2+V3+V4+V5+V6+V7+V8 and X1+X2+X3+X4+X5+X6+X7+X8.

[0183] See Figure 10 and combined Figure 9 , Figure 10 This application provides a method based on... Figure 9 The provided all-reduce result data (i.e., in) Figure 4 The n computing groups in the computing cluster shown are targeting... Figure 8 This diagram illustrates the result data (the data obtained from the all-reduce operation on the divided logical nodes) after performing an all-gather operation within the computation group. (See example...) Figure 9 As shown, the all-reduce result data corresponding to computing nodes 11-13 is used to implement the all-gather operation within computing group 1. Computing nodes 11-13 perform the all-gather operation within computing group 1 based on the all-reduce result data corresponding to their respective nodes, obtaining the all-gather result data corresponding to their respective nodes. Similarly, the all-reduce result data corresponding to computing nodes 21-23 is used to implement the all-gather operation within computing group 2. Computing nodes 21-23 perform the all-gather operation within computing group 2 based on the all-reduce result data corresponding to their respective nodes, obtaining the all-gather result data corresponding to their respective nodes. The all-reduce result data corresponding to compute nodes 31-32 is used to implement the all-gather operation within compute group 3. Compute nodes 31-32 perform the all-gather operation within compute group 3 based on the all-reduce result data corresponding to compute nodes 31-32, obtaining the all-gather result data corresponding to compute nodes 31-32. For example... Figure 10As shown, the all-gather result data corresponding to any two computing nodes among computing nodes 11-13, 21-23, and 31-32 is the same. The all-gather result data corresponding to each computing node among computing nodes 11-13, 21-23, and 31-32 includes 18 data blocks. The data of these 18 data blocks are A1+A2+A3+A4+A5+A6+A7+A8, B1+B2+B3+B4+B5+B6+B7+B8, C1+C2+C3+C4+C5+C6+C7+C8, D1+D2+D3+D4+D5+D6+D7+D8, E1+E2+E3+E4+E5+E6+E7+E8, F1+F2+F3+F4+F5+F6+F7+F8, G 1+G2+G3+G4+G5+G6+G7+G8, H1+H2+H3+H4+H5+H6+H7+H8, J1+J2+J3+J4+J5+J6+J7+J8, K1+K2+K3+K4+K5+K6+K7+K8, P1+P2+P3+P4+P5+P6+P7+P8, Q1+Q2+Q3+Q4+Q5+Q6+Q7+Q8, R1+R2+R3+R4+R5+R6+R7+R8, S1+S2+S3+S4+S5+S6+S7+S8, T1+T2+T3+T4+T5+T6+T7+T8, U1+U2+U3+U4+U5+U6+U7+U8, V1+V2+V3+V4+V5+V6+V7+V8 and X1+X2+X3+X4+X5+X6+X7+X8.

[0184] In another embodiment, see Figure 13Data slice 11 corresponding to compute node 11, data slice 21 corresponding to compute node 21, and data slice 31 corresponding to compute node 31 are used to implement the all-reduce operation performed on logical node 1 in compute groups 1 to 3. Compute nodes 11, 21, and 31 perform the all-reduce operation on logical node 1 in compute groups 1 to 3 based on data slices 11, 21, and 31, obtaining all-reduce result data that corresponds one-to-one with compute nodes 11, 21, and 31. The all-reduce operation performed on logical node 1 yields data corresponding to compute nodes 11, 21, and 31. The all-reduce result data corresponding to node 21 and compute node 31 are the same. The all-reduce result data obtained by performing the all-reduce operation on logical node 1 and corresponding to each compute node in compute node 11, compute node 21 and compute node 31 includes 3 data blocks. The data of these 3 data blocks are A1+A2+A3+A4+A5+A6+A7+A8+A9+A10+A11+A12, B1+B2+B3+B4+B5+B6+B7+B8+B9+B10+B11+B12 and C1+C2+C3+C4+C5+C6+C7+C8+C9+C10+C11+C12. Data slice 12 corresponding to compute node 12, data slice 22 corresponding to compute node 22, and data slice 321 corresponding to compute node 32 are used to implement the all-reduce operation performed on logical node 2 in compute groups 1-3. Compute nodes 12, 22, and 32 perform the all-reduce operation on logical node 2 in compute groups 1-3 based on data slices 12, 22, and 321, obtaining all-reduce result data that corresponds one-to-one with compute nodes 12, 22, and 32. The all-reduce operation performed on logical node 2 yields data corresponding to compute nodes 12, 22, and 32. The all-reduce result data corresponding to compute node 22 and compute node 32 are the same. The all-reduce result data obtained by performing the all-reduce operation on logical node 2 and corresponding to each compute node in compute node 12, compute node 22 and compute node 32 includes 3 data blocks. The data of these 3 data blocks are D1+D2+D3+D4+D5+D6+D7+D8+D9+D10+D11+D12, E1+E2+E3+E4+E5+E6+E7+E8+E9+E10+E11+E12 and F1+F2+F3+F4+F5+F6+F7+F8+F9+F10+F11+F12.Data slice 13 corresponding to compute node 13, data slice 23 corresponding to compute node 23, and data slice 322 corresponding to compute node 32 are used to implement the all-reduce operation performed on logical node 3 between compute groups 1 and 3. Compute nodes 13, 23, and 32 perform the all-reduce operation on logical node 3 between compute groups 1 and 3 based on data slices 13, 23, and 322, obtaining all-reduce result data that corresponds one-to-one with compute nodes 13, 23, and 32. The all-reduce operation performed on logical node 3 yields data corresponding to compute nodes 13, 23, and 32. The all-reduce result data corresponding to compute node 23 and compute node 32 are the same. The all-reduce result data obtained by performing the all-reduce operation on logical node 3 and corresponding to each compute node in compute node 13, compute node 23 and compute node 32 includes 3 data blocks. The data of these 3 data blocks are G1+G2+G3+G4+G5+G6+G7+G8+G9+G10+G11+G12, H1+H2+H3+H4+H5+H6+H7+H8+H9+H10+H11+H12 and J1+J2+J3+J4+J5+J6+J7+J8+J9+J10+J11+J12. Data slice 14 corresponding to compute node 14, data slice 241 corresponding to compute node 24, and data slice 331 corresponding to compute node 33 are used to implement the all-reduce operation performed on logical node 4 between compute groups 1 and 3. Compute nodes 14, 24, and 33 perform the all-reduce operation on logical node 4 between compute groups 1 and 3 based on data slices 14, 241, and 331, obtaining all-reduce result data that corresponds one-to-one with compute nodes 14, 24, and 33. The all-reduce operation performed on logical node 4 yields data corresponding to compute nodes 14, 24, and 33. The all-reduce result data corresponding to compute node 24 and compute node 33 are the same. The all-reduce result data obtained by performing the all-reduce operation on logical node 4 and corresponding to each compute node in compute node 14, compute node 24 and compute node 33 includes 3 data blocks. The data of these 3 data blocks are K1+K2+K3+K4+K5+K6+K7+K8+K9+K10+K11+K12, P1+P2+P3+P4+P5+P6+P7+P8+P9+P10+P11+P12 and Q1+Q2+Q3+Q4+Q5+Q6+Q7+Q8+Q9+Q10+Q11+Q12.Data slice 15 corresponding to compute node 15, data slice 242 corresponding to compute node 24, and data slice 332 corresponding to compute node 33 are used to implement the all-reduce operation performed on logical node 5 between compute groups 1 and 3. Compute nodes 15, 24, and 33 perform the all-reduce operation on logical node 5 between compute groups 1 and 3 based on data slices 15, 242, and 332, obtaining all-reduce result data that corresponds one-to-one with compute nodes 15, 24, and 33. The all-reduce operation performed on logical node 5 yields data corresponding to compute nodes 15, 24, and 33. The all-reduce result data corresponding to compute node 24 and compute node 33 are the same. The all-reduce result data obtained by performing an all-reduce operation on logical node 5, which is the same as the all-reduce result data corresponding to each of compute nodes 15, 24 and 33, includes 3 data blocks. The data of these 3 data blocks are R1+R2+R3+R4+R5+R6+R7+R8+R9+R10+R11+R12, S1+S2+S3+S4+S5+S6+S7+S8+S9+S10+S11+S12 and T1+T2+T3+T4+T5+T6+T7+T8+T9+T10+T11+T12. After performing an all-reduce operation on each logical node 1-5 across different computation groups 1-3, resulting in all-reduce result data corresponding to computation nodes 11-15, 21-24, and 31-33, computation nodes 11-15 perform an all-gather operation within computation group 1 based on their corresponding all-reduce result data, resulting in all-gather result data corresponding to their respective nodes. The all-reduce result data corresponding to computation nodes 21-24 is used to implement the all-gather operation performed within computation group 2. Computation nodes 21-24 then perform the all-gather operation within computation group 2 based on their corresponding all-reduce result data, resulting in all-gather result data corresponding to their respective nodes. The all-reduce result data corresponding to each of compute nodes 31 to 33 is used to implement the all-gather operation executed within compute group 3. Compute nodes 31 to 33 perform the all-gather operation within compute group 3 based on the all-reduce result data corresponding to each of compute nodes 31 to 33, and obtain the all-gather result data corresponding to each of compute nodes 31 to 33.The all-gather result data is identical to that of any two computing nodes among computing nodes 11-15, 21-24, and 31-33. The all-gather result data corresponding to each of these computing nodes comprises 15 data blocks, with the data in each block being A1+A2+A3+A4+A5+A6+A7+A8+A9+A10+A11+A12, B1+... B2+B3+B4+B5+B6+B7+B8+B9+B10+B11+B12, C1+C2+C3+C4+C5+C6+C7+C8+C9+C10+C11+C12, D1+D2+D3+D4+D5+ D6+D7+D8+D9+D10+D11+D12, E1+E2+E3+E4+E5+E6+E7+E8+E9+E10+E11+E12, F1+F2+F3+F4+F5+F6+F7+F8+F9+F 10+F11+F12, G1+G2+G3+G4+G5+G6+G7+G8+G9+G10+G11+G12, H1+H2+H3+H4+H5+H6+H7+H8+H9+H10+H11+H12, J 1+J2+J3+J4+J5+J6+J7+J8+J9+J10+J11+J12, K1+K2+K3+K4+K5+K6+K7+K8+K9+K10+K11+K12, P1+P2+P3+P4+P5 +P6+P7+P8+P9+P10+P11+P12, Q1+Q2+Q3+Q4+Q5+Q6+Q7+Q8+Q9+Q10+Q11+Q12, R1+R2+R3+R4+R5+R6+R7+R8+R9 +R10+R11+R12, S1+S2+S3+S4+S5+S6+S7+S8+S9+S10+S11+S12 and T1+T2+T3+T4+T5+T6+T7+T8+T9+T10+T11+T12.

[0185] It should be noted that, for the sake of brevity in the accompanying drawings, the embodiments of this application do not show the calculations between groups 1 and 3. Figure 13 The diagrams showing the logical nodes after the all-reduce operation and the all-gather operation are described in text only.

[0186] It should also be noted that during the execution of the third set communication operation (e.g., an all-gather operation) within any of the aforementioned n computing groups, the computing nodes within that group can send data to each other to implement the third set communication operation. For example, after executing the second set communication operation (e.g., an all-reduce operation) between different computing groups among the n computing groups, the first computing node obtains the second data, which is the data corresponding to the first computing node obtained from the second set communication operation between different computing groups among the n computing groups (i.e., the second data is the result data of the second set communication operation corresponding to the first computing node). This second data is used to implement the third set communication operation executed within the first computing group. During the execution of the third set communication operation within the first computing group, the first computing node sends the second data to computing nodes within the first computing group other than itself (or, the process of executing the third set communication operation within the first computing group includes the first computing node sending the second data to computing nodes within the first computing group other than itself). Furthermore, the process of executing the third set communication operation within different computing groups among the n computing groups can be parallelized, thereby improving the efficiency of executing the third set communication operation within different computing groups.

[0187] In this embodiment, the process of performing a second set communication operation (e.g., an all-reduce operation) on logical nodes between different computing groups in the n computing groups can be implemented based on algorithms with fewer communication steps, such as the HD algorithm, thereby reducing the number of communication steps required to perform the second set communication operation between different computing groups. The process of performing a third set communication operation within any of the n computing groups can be implemented based on a single-layer algorithm (e.g., the ring algorithm, the HD algorithm, etc.), thereby effectively utilizing the link bandwidth between different computing nodes.

[0188] based on Figure 6 As shown in the embodiments, the aggregate communication method provided in this application firstly performs a reduce-scatter operation (i.e., the first aggregate communication operation) within each computing group in the computing cluster. Then, based on the result data obtained from the reduce-scatter operation within each computing group in the computing cluster, an all-reduce operation (i.e., the second aggregate communication operation) is performed between different computing groups in the computing cluster. Finally, based on the result data obtained from the all-reduce operation between different computing groups in the computing cluster, an all-gather operation (i.e., the third aggregate communication operation) is performed within each computing group in the computing cluster. This achieves a hierarchical all-reduce algorithm.

[0189] In summary, the technical solution provided in this application provides that, in the first data corresponding to the first computing node obtained by performing a first set communication operation within the first computing group, a first part of the data corresponds to the second computing node within the second computing group, and a second part of the data corresponds to the third computing node within the second computing group. The first and second parts of the data are used to implement a second set communication operation between different computing groups in the n computing groups. That is, different parts of the first data correspond to different computing nodes within the second computing group, and these different parts of the data are used to implement a second set communication operation between different computing groups in the n computing groups. The first computing node sends these different parts of the data to the different computing nodes within the second computing group, which facilitates the implementation of a second set communication operation between different computing groups in the n computing groups based on these different parts of the data, thereby facilitating the implementation of the hierarchical all-reduce algorithm within the computing cluster. The set communication method provided in this application can optimize the hierarchical all-reduce algorithm, enabling its application in asymmetric computing clusters and expanding its application scenarios. Furthermore, the collection communication method provided in this application embodiment also has good communication performance and high communication efficiency.

[0190] Furthermore, in a scheme where computing nodes partition the original data according to w = n × L, the size of the reduce-scatter result data obtained by performing a reduce-scatter operation within each computing group is equal to the size of the reduce-scatter result data corresponding to any two computing nodes within that group. This type of reduce-scatter operation performed within a computing group is also called a uniform reduce-scatter operation within the group. Uniform reduce-scatter operation within the group ensures time synchronization of reduce-scatter operations performed by multiple computing nodes within the group. The process of performing reduce-scatter operations can maximize the utilization of the link bandwidth between these multiple computing nodes, thereby improving algorithm performance.

[0191] According to w = n × G max In the scheme of splitting the original data, the number of data blocks obtained from the splitting of the original data will not be too large, and the size of a single data block will not be too small. The data slices corresponding to multiple logical nodes obtained from the data partitioning based on the reduce-scatter result are of equal size (e.g., Figure 13 As shown in the figure, the data slices corresponding to each logical node are not too small, thus avoiding data fragmentation.

[0192] Please refer to Figure 14The diagram illustrates a flowchart of another collective communication method provided in an embodiment of this application. This collective communication method is applied to a computing cluster comprising n computing groups, each of which includes at least one computing node, where n is an integer greater than 1. At least two of the n computing groups have unequal numbers of computing nodes. The n computing groups include a first computing group, which is not the smallest computing group among the n groups; the smallest computing group is the computing group with the fewest computing nodes among the n computing groups. For example, the computing cluster is as follows: Figure 4 The computing cluster 10 shown is or is as follows Figure 5 The computing cluster 20 shown has a minimum computing group of computing group 3, and the first computing group is either computing group 1 or computing group 2. The k computing nodes within the first computing group, as described below, can be any k computing nodes within that first computing group. For example, this computing cluster is as follows: Figure 4 The computing cluster 10 shown has a first computing group of computing group 1, where the k computing nodes in the first computing group are any k computing nodes selected from computing nodes 11 to 13; or, the first computing group is computing group 2, where the k computing nodes in the first computing group are any k computing nodes selected from computing nodes 21 to 23. For example, the computing cluster is as follows: Figure 5 The computing cluster 20 shown has a first computing group of computing group 1, in which k computing nodes are any k computing nodes from computing nodes 11 to 15; or, the first computing group is computing group 2, in which k computing nodes are any k computing nodes from computing nodes 21 to 25.

[0193] like Figure 14 As shown, the collection communication method includes the following steps S1401 to S1402.

[0194] S1401. The k computing nodes in the first computing group obtain the raw data corresponding to each of the k computing nodes, which is used to implement the first set communication operation in the first computing group.

[0195] Here, the k computing nodes can be all computing nodes within the first computing group. The size of the original data corresponding to any two of the k computing nodes for implementing the first set communication operation within the first computing group can be equal.

[0196] In an optional embodiment, each of the k computing nodes receives raw data corresponding to that computing node, used to perform a first set communication operation within the first computing group.

[0197] S1402. Based on the original data corresponding to each of the k computing nodes, used to implement the first set communication operation within the first computing group, the first set communication operation is performed within the first computing group to obtain G within the first computing group. min The first data corresponding to each computing node, G min Each computing node is a subset of the k computing nodes, and is related to G. min The first data corresponding to each computing node is used to implement the second set communication operation between different computing groups in the n computing groups, G min G is the number of computing nodes in the smallest computing group. min Both k and k are positive integers.

[0198] Among them, with the G min The first data corresponding to any two computing nodes in the k computing nodes are different. The data obtained by the k computing nodes performing a first set communication operation within the first computing group based on the original data corresponding to each of the k computing nodes does not include data that is different from the original data of the k computing nodes except for G. min Data corresponding to computing nodes other than the k computing nodes. For example, the k computing nodes perform a first set communication operation within the first computing group based on the original data corresponding to each of the k computing nodes, and only obtain the data corresponding to G. min The first data corresponding to each computing node. This G min Each computing node corresponds one-to-one with a computing node within the smallest computing group, and this G... min The first data corresponding to each computing node in the minimum computing group is aligned with the boundary of the data corresponding to the computing node corresponding to that computing node, obtained by performing the first set communication operation within the minimum computing group. For ease of description, the result data obtained by performing the first set communication operation is referred to as the first set communication operation result data, and is aligned with the boundary of the G. min The first data corresponding to each computing node in the G is the communication operation result data with the first set corresponding to each computing node. min The first data corresponding to each computing node in the computing nodes is aligned with the boundary of the first set communication operation result data corresponding to the computing node corresponding to each computing node, obtained by performing the first set communication operation within the smallest computing group.

[0199] In an optional embodiment, the size of the original data corresponding to any two of the k computing nodes used to implement the first set communication operation within the first computing group is equal. min The first data corresponding to any two computing nodes in a given computing node is of equal size.

[0200] In an optional embodiment, each of the k computing nodes divides the original data corresponding to that computing node for performing the first set communication operation within the first computing group into w data blocks, where w = n × G. min The w data blocks can be of equal size, and the size of each data block corresponding to any two of the k computing nodes can be equal. The boundaries of the w data blocks corresponding to each of the k computing nodes can be aligned one-to-one. Based on the w data blocks corresponding to each of the k computing nodes, the k computing nodes perform a first set communication operation within the first computing group to obtain the data block corresponding to the G node. min The first data corresponding to each computing node, and the G min The first data corresponding to each of the computing nodes includes at least one data block, which is related to the G. min The number of data blocks included in the first data of any two compute nodes in a compute cluster can be equal. In one example, a compute cluster like this... Figure 4 As shown, the n calculation groups are Figure 4 The computational groups are 1 to 3. Computational group 1 has 3 computational nodes, computational group 2 has 3 computational nodes, computational group 3 has 2 computational nodes, and the smallest computational group is computational group 3, which has 2 computational nodes (i.e., G). min =2), w = n×G min =3 × 2 = 6. In another example, the computing cluster is as follows: Figure 5 As shown, the n calculation groups are Figure 5 The computational groups are 1 to 3. Computational group 1 has 5 computational nodes, computational group 2 has 4 computational nodes, and computational group 3 has 3 computational nodes. The smallest computational group is computational group 3, which has 3 computational nodes (i.e., G). min =3), w = n×G min =3×3=9.

[0201] The above description uses the execution of a first set communication operation within a first computing group as an example. In practical applications, the first set communication operation can be executed within each of the aforementioned n computing groups. The first computing group can be any computing group among the n computing groups except the smallest computing group. The process of executing the first set communication operation within each computing group other than the smallest computing group among the n computing groups can be referenced from the process of executing the first set communication operation within the first computing group, and will not be repeated here. The process of executing the first set communication operation within the smallest computing group includes: G within the smallest computing group min Each computing node (i.e., all computing nodes) is based on the G... minEach computing node corresponds to the original data used to perform the first set communication operation within the minimum computing group. The first set communication operation is performed within the minimum computing group to obtain the data corresponding to the G node. min The first set of communication operation results data corresponds one-to-one with each computing node. For example, the G within this minimum computing group. min Each of the computing nodes will divide the raw data corresponding to that computing node, used to perform the first set communication operation within the minimum computing group, into w data blocks, where w = n × G. min The sizes of the w data blocks can be equal, and, with the G... min The size of each data block corresponding to any two computing nodes in a given computing node can be equal to the size of the G. min The boundaries of the w data blocks corresponding to each computing node can be aligned one-to-one; this G min Each computing node is based on the G min Each computing node corresponds to w data blocks, and within this minimum computing group, the first set communication operation is performed to obtain the data from the G node. min The first set of communication operation results data corresponding one-to-one with each computing node, and the G min The first set of communication operation result data corresponding to each of the computing nodes includes at least one data block, which is related to the G. min The number of data blocks included in the first set communication operation result data corresponding to any two computing nodes in the computing nodes is equal.

[0202] It should be noted that the boundaries of the first set communication operation result data obtained by performing the first set communication operation in any two computing groups among the n computing groups can be aligned or unaligned. The process of performing the first set communication operation in different computing groups among the n computing groups can be parallelized, thereby improving the efficiency of performing the first set communication operation in different computing groups. In the embodiments of this application, the process of performing the first set communication operation in any computing group among the n computing groups can be implemented based on a single-layer algorithm (e.g., ring algorithm, HD algorithm, etc.), thereby effectively utilizing the link bandwidth between computing nodes within any computing group.

[0203] In this embodiment, the first set communication operation can be a reduce-scatter operation. For the sake of clarity, the following example uses the reduce-scatter operation as the first set communication operation. Figure 5 The computing cluster shown above will be described in detail. In the following description, the reduction operation, which is to calculate the cumulative sum, will be used as an example, and the result data of the first set of communication operations will be referred to as the reduce-scatter result data.

[0204] Please refer to Figure 15 and Figure 16 This illustrates another method provided by embodiments of this application, such as... Figure 5 This diagram illustrates the execution of reduce-scatter operations within a computing group in computing cluster 20. Figure 15 and Figure 16 With w = n × G min For example, to illustrate. Because in such... Figure 5 In the computing cluster 20 shown, there are 3 computing groups. Computing group 1 has 5 computing nodes, computing group 2 has 4 computing nodes, and computing group 3 has 3 computing nodes. Therefore, the smallest computing group is computing group 3, and the smallest computing group has 3 computing nodes. n = 3, G min =3, w=n×G min =3×3=9.

[0205] like Figure 15 As shown, computation group 1 includes computation nodes 11 to 15. Each of these nodes contains raw data for performing reduce-scatter operations within computation group 1. The size of the raw data corresponding to any two of these nodes (i.e., any two nodes included in any two of these nodes) is equal. Computation node 11 divides its corresponding raw data into nine data blocks, with data A1 to J1. Computation node 12 divides its corresponding raw data into nine data blocks, with data A2 to J2. Computation node 13 divides its corresponding raw data into nine data blocks, with data A3 to J3. Compute node 14 divides the raw data corresponding to it, used for performing the reduce-scatter operation within compute group 1, into 9 data blocks, with data in these 9 blocks designated A4 to J4. Compute node 15 divides the raw data corresponding to it, used for performing the reduce-scatter operation within compute group 1, into 9 data blocks, with data in these 9 blocks designated A5 to J5. See also... Figure 15 and Figure 16Computation nodes 11-15 perform reduce-scatter operations within computation group 1 based on the nine data blocks corresponding to each of computation nodes 11-15, obtaining reduce-scatter result data that corresponds one-to-one with computation nodes 11-13 (a subset of computation nodes 11-15). The size of the reduce-scatter result data is equal to that corresponding to any two computation nodes 11-13. The reduce-scatter result data corresponding to computation node 11 includes three data blocks: A1+A2+A3+A4+A5, B1+B2+B3+B4+B5, and C1+C2+C3+C4+C5. The reduce-scatter result data corresponding to computation node 12 also includes three data blocks: D1+D2+D3+D4+D5, E1+E2+E3+E4+E5, and F1+F2+F3+F4+F5. The reduce-scatter result data corresponding to compute node 13 includes three data blocks: G1+G2+G3+G4+G5, H1+H2+H3+H4+H5, and J1+J2+J3+J4+J5. The data obtained by performing the reduce-scatter operation within compute group 1 does not include the reduce-scatter result data corresponding to compute nodes 14-15.

[0206] like Figure 15As shown, computation group 2 includes computation nodes 21 to 24. Each of these nodes contains raw data for performing reduce-scatter operations within computation group 2. The size of the raw data corresponding to any two of these nodes (i.e., any two nodes within each of these nodes) is equal. Computation node 21 divides its corresponding raw data into nine data blocks, with data A6 to J6. Computation node 22 divides its corresponding raw data into nine data blocks, with data A7 to J7. Computation node 23 divides its corresponding raw data into nine data blocks, with data A8 to J8. Compute node 24 divides the raw data corresponding to it, used to perform the reduce-scatter operation within compute group 2, into nine data blocks, designated A9 to J9. (See also...) Figure 15 and Figure 16 Computation nodes 21-24 perform reduce-scatter operations within computation group 2 based on the nine data blocks corresponding to each of computation nodes 21-24. This yields reduce-scatter result data corresponding one-to-one with computation nodes 21-23 (a subset of computation nodes 21-24), with the result data being the same size as the reduce-scatter result data corresponding to any two of the computation nodes 21-23. The reduce-scatter result data corresponding to computation node 21 includes three data blocks: A6+A7+A8+A9, B6+B7+B8+B9, and C6+C7+C8+C9. The reduce-scatter result data corresponding to computation node 22 also includes three data blocks: D6+D7+D8+D9, E6+E7+E8+E9, and F6+F7+F8+F9. The reduce-scatter result data corresponding to compute node 23 includes three data blocks: G6+G7+G8+G9, H6+H7+H8+H9, and J6+J7+J8+J9. The data obtained by performing the reduce-scatter operation within compute group 2 does not include the reduce-scatter result data corresponding to compute node 24.

[0207] like Figure 15 As shown, computation group 3 includes computation nodes 31 to 33. Each of these nodes contains raw data for performing reduce-scatter operations within computation group 3. The size of the raw data corresponding to any two of these nodes (i.e., any two nodes included in each of these nodes) is equal. Computation node 31 divides its corresponding raw data into nine data blocks, with data A10 to J10. Computation node 32 divides its corresponding raw data into nine data blocks, with data A11 to J11. Computation node 33 divides its corresponding raw data into nine data blocks, with data A12 to J12. See also... Figure 15 and Figure 16 Computation nodes 31-33 perform reduce-scatter operations within computation group 3 based on the nine data blocks corresponding to each of the computation nodes 31-33, obtaining reduce-scatter result data that corresponds one-to-one with the reduce-scatter result data corresponding to any two of the computation nodes 31-33. The size of the reduce-scatter result data is equal to that corresponding to any two of the computation nodes 31-33. The reduce-scatter result data corresponding to computation node 31 includes three data blocks: A10+A11+A12, B10+B11+B12, and C10+C11+C12. The reduce-scatter result data corresponding to computation node 32 also includes three data blocks: D10+D11+D12, E10+E11+E12, and F10+F11+F12. The reduce-scatter result data corresponding to computing node 33 includes three data blocks: G10+G11+G12, H10+H11+H12, and J10+J11+J12, respectively.

[0208] like Figure 15 and Figure 16As shown, in each of the computation groups 1 to 3, the boundaries of the reduce-scatter results obtained by performing the reduce-scatter operation are aligned with the boundaries of the reduce-scatter results corresponding to computation nodes 11 to 13, 21 to 23, and 31 to 33. The boundaries of the reduce-scatter results corresponding to computation node 11, 21, and 31 are also aligned with the boundaries of the reduce-scatter results corresponding to computation node 12, 22, and 32. Finally, the boundaries of the reduce-scatter results corresponding to computation node 13, 23, and 33 are also aligned with the boundaries of the reduce-scatter results corresponding to computation node 23.

[0209] It should be noted that, in the above description, all computing nodes in computing group 1 participate in the reduce-scatter operation. However, performing the reduce-scatter operation within computing group 1 only yields reduce-scatter results corresponding to a subset of the computing nodes within group 1. This type of reduce-scatter operation performed within computing group 1 is also called an intra-group non-uniform reduce-scatter operation. Similarly, the aforementioned reduce-scatter operation performed within computing group 2 is also called an intra-group non-uniform reduce-scatter operation. For computing group 3, all computing nodes in computing group 3 participate in the reduce-scatter operation. Performing the reduce-scatter operation within computing group 3 yields reduce-scatter results corresponding to a subset of the computing nodes within group 3. min Each computing node corresponds to a one-to-one reduce-scatter result data, and is associated with the G... min The reduce-scatter result data corresponding to any two computing nodes in a computing group are of equal size. This type of reduce-scatter operation performed within computing group 3 is also called intra-group uniform reduce-scatter operation.

[0210] In this embodiment, according to w = n × G min The original data is segmented, resulting in a small number of data blocks, each with a relatively large size, thus avoiding data fragmentation. This is done according to w = n × G. minThe scheme of splitting the original data is suitable for scenarios where the number of computing nodes in different computing groups within a computing cluster is not uniform (e.g., Figure 5 The computing cluster 20 shown is also applicable to scenarios where the number of computing nodes in each computing group of the computing cluster is relatively uniform, but this application embodiment does not limit this.

[0211] In an optional embodiment, the G mentioned above in the first computing group min Each computing node corresponds one-to-one with G. min There are n logical nodes. The second set communication operation is a set communication operation performed between different computing groups within the aforementioned n computing groups for each logical node. min The process of each logical node performing the second set of communication operations is parallelized. This improves the efficiency of communication between different computation groups within the n computation groups for the G. min The efficiency of each logical node executing the second set communication operation. The logical node is determined based on the boundaries of the first set communication operation result data corresponding to the computing nodes within the n computing groups (the first set communication operation result data corresponding to the computing nodes within the first computing group is also the aforementioned first data). min Each logical node in the n logical nodes corresponds to n computation nodes. These n computation nodes are located one-to-one in the n computation groups. The first set communication operation result data corresponding to each of the n computation nodes is the data corresponding to each logical node, and is aligned with the boundary of the data corresponding to the same logical node (that is, the offset address of the data corresponding to the same logical node is the same). For ease of description, the first set communication operation result data corresponding to both a computation node and a logical node is called a data slice. The data slice is used to implement the second set communication operation between different computation groups in the n computation groups. It is not difficult to understand that the data corresponding to G in the first computation group... min The first data corresponding to each computing node in the computing nodes is a data slice.

[0212] The first computing node determines multiple data slices based on the boundaries of the communication operation result data of the first set corresponding to the computing nodes within the n computing groups. The first computing node determines G based on these multiple data slices. min A logical node, this G min Each logical node in the G group corresponds to n compute nodes and n data slices. The n compute nodes are located one-to-one within the n compute groups, and the n data slices are also one-to-one with the n compute nodes. Each data slice is the result data of a communication operation with the first set corresponding to its compute node, and the boundaries of the n data slices are aligned. minThe n data slices corresponding to each logical node in the n logical nodes are used to implement the second set communication operation performed on each logical node between different computing groups in the n computing groups. The first computing node can be the aforementioned G in the first computing group. min Any computing node in a computing node group, or any computing node in the smallest computing group. For example Figure 16 As shown, the first computing node can be any one of computing nodes 12-13, 22-23, and 31-33.

[0213] The following is combined Figure 5 The computing cluster shown below introduces the content related to logical nodes. The reduce-scatter result data mentioned below is also the result data of the first set of communication operations.

[0214] Please refer to Figure 17 It illustrates an embodiment of this application based on Figure 16 The provided reduce-scatter result data (i.e., in) Figure 5 The diagram illustrates the partitioning of logical nodes into logical nodes based on the result data of reduce-scatter operations performed within a compute group in the compute cluster. The first compute node defines multiple data slices based on the boundaries of the reduce-scatter result data corresponding to compute nodes 12-13, 22-23, and 31-33. These data slices are data slices 11-13, 21-23, and 31-33. Data slices 11-13 are the reduce-scatter result data corresponding to compute nodes 11-13. Data slices 21-23 are the reduce-scatter result data corresponding to compute nodes 21-23. Data slices 31-33 are the reduce-scatter result data corresponding to compute nodes 31-33.

[0215] See Figure 17After the first computing node determines the aforementioned multiple data slices, it determines multiple logical nodes based on these data slices. These logical nodes are logical nodes 1 to 3. Logical node 1 corresponds to computing nodes 11, 21, and 31, and also corresponds to data slices 11, 21, and 31. Computing nodes 11, 21, and 31 are located in computing groups 1 to 3 in a one-to-one correspondence. Data slice 11 corresponds to computing node 11, data slice 21 corresponds to computing node 21, and data slice 31 corresponds to computing node 31. The boundaries of these three data slices are aligned. Data slices 11, 21, and 31 are used to implement the second set communication operation performed on logical node 1 between different computing groups in computing groups 1 to 3. Logical node 2 corresponds to compute nodes 12, 22, and 32, and also corresponds to data slices 12, 22, and 32. Computation nodes 12, 22, and 32 are located in compute groups 1 to 3 in a one-to-one correspondence. Data slice 12 corresponds to compute node 12, data slice 22 corresponds to compute node 22, and data slice 32 corresponds to compute node 32. The boundaries of these three data slices are aligned. Data slices 12, 22, and 32 are used to implement the second set communication operation performed on logical node 2 between different compute groups in compute groups 1 to 3. Logical node 3 corresponds to compute nodes 13, 23, and 33, and also corresponds to data slices 13, 23, and 33. Computation nodes 13, 23, and 33 are located in compute groups 1 to 3 in a one-to-one correspondence. Data slice 13 corresponds to compute node 13, data slice 23 corresponds to compute node 23, and data slice 33 corresponds to compute node 33. The boundaries of these three data slices are aligned. Data slices 13, 23, and 33 are used to implement the second set communication operation performed on logical node 3 between different compute groups in compute groups 1 to 3. In this embodiment, the process of performing the second set communication operation on different logical nodes in logical nodes 1 to 3 between different compute groups in compute groups 1 to 3 is parallel.

[0216] In this embodiment, the second set communication operation can be an all-reduce operation performed between different computing groups in n computing groups, and this all-reduce operation can be an all-reduce operation performed on a logical node. For ease of description, the result data obtained from executing the second set communication operation is referred to as the second set communication operation result data. When the second set communication operation is an all-reduce operation, the result data is also referred to as all-reduce result data. The result data of the second set communication operation corresponding to any computing node is used to implement the third set communication operation performed within the computing group where that computing node is located. The third set communication operation can be an all-gather operation. For ease of description, the result data obtained from executing the third set communication operation is referred to as the third set communication operation result data. When the third set communication operation is an all-gather operation, the result data is also referred to as all-gather result data. The following example uses the second set communication operation as an all-reduce operation and the third set communication operation as an all-gather operation, combined with... Figure 5 The illustrated computing cluster describes the execution of all-reduce operations between different computing groups within n computing groups, as well as the execution of all-gather operations within a computing group.

[0217] See Figure 17Data slice 11 corresponding to compute node 11, data slice 21 corresponding to compute node 21, and data slice 31 corresponding to compute node 31 are used to implement the all-reduce operation performed on logical node 1 in compute groups 1 to 3. Compute nodes 11, 21, and 31 perform the all-reduce operation on logical node 1 in compute groups 1 to 3 based on data slices 11, 21, and 31, obtaining all-reduce result data that corresponds one-to-one with compute nodes 11, 21, and 31. The all-reduce operation performed on logical node 1 yields data corresponding to compute nodes 11, 21, and 31. The all-reduce result data corresponding to node 21 and compute node 31 are the same. The all-reduce result data obtained by performing the all-reduce operation on logical node 1 and corresponding to each compute node in compute node 11, compute node 21 and compute node 31 includes 3 data blocks. The data of these 3 data blocks are A1+A2+A3+A4+A5+A6+A7+A8+A9+A10+A11+A12, B1+B2+B3+B4+B5+B6+B7+B8+B9+B10+B11+B12 and C1+C2+C3+C4+C5+C6+C7+C8+C9+C10+C11+C12. Data slice 12 corresponding to compute node 12, data slice 22 corresponding to compute node 22, and data slice 32 corresponding to compute node 32 are used to implement the all-reduce operation performed on logical node 2 in compute groups 1-3. Compute nodes 12, 22, and 32 perform the all-reduce operation on logical node 2 in compute groups 1-3 based on data slices 12, 22, and 32, obtaining all-reduce result data that corresponds one-to-one with compute nodes 12, 22, and 32. The all-reduce operation performed on logical node 2 yields data corresponding to compute nodes 12, 22, and 32. The all-reduce result data corresponding to node 22 and compute node 32 are the same. The all-reduce result data obtained by performing the all-reduce operation on logical node 2 and corresponding to each compute node in compute node 12, compute node 22 and compute node 32 includes 3 data blocks. The data of these 3 data blocks are D1+D2+D3+D4+D5+D6+D7+D8+D9+D10+D11+D12, E1+E2+E3+E4+E5+E6+E7+E8+E9+E10+E11+E12 and F1+F2+F3+F4+F5+F6+F7+F8+F9+F10+F11+F12.Data slice 13 corresponding to compute node 13, data slice 23 corresponding to compute node 23, and data slice 33 corresponding to compute node 33 are used to implement the all-reduce operation performed on logical node 3 between compute groups 1 and 3. Compute nodes 13, 23, and 33 perform the all-reduce operation on logical node 3 between compute groups 1 and 3 based on data slices 13, 23, and 33, obtaining all-reduce result data that corresponds one-to-one with compute nodes 13, 23, and 33. The all-reduce operation performed on logical node 3 yields data corresponding to compute nodes 13, 23, and 33. The all-reduce result data corresponding to node 23 and compute node 33 are the same. The all-reduce result data obtained by performing the all-reduce operation on logical node 3 and corresponding to each compute node in compute node 13, compute node 23 and compute node 33 includes 3 data blocks. The data of these 3 data blocks are G1+G2+G3+G4+G5+G6+G7+G8+G9+G10+G11+G12, H1+H2+H3+H4+H5+H6+H7+H8+H9+H10+H11+H12 and J1+J2+J3+J4+J5+J6+J7+J8+J9+J10+J11+J12.

[0218] After performing an all-reduce operation on each logical node (1-3) across different computation groups (1-3), resulting in all-reduce result data corresponding to computation nodes 11-13, 21-23, and 31-33 respectively, the all-reduce result data corresponding to computation nodes 11-13 is used to implement the all-gather operation within computation group 1. Computation nodes 11-13 perform the all-gather operation within computation group 1 based on their corresponding all-reduce result data, resulting in all-gather result data corresponding to computation nodes 11-13. Similarly, the all-reduce result data corresponding to computation nodes 21-23 is used to implement the all-gather operation within computation group 2. Computation nodes 21-23 perform the all-gather operation within computation group 2 based on their corresponding all-reduce result data, resulting in all-gather result data corresponding to computation nodes 21-23. The all-reduce result data corresponding to each of compute nodes 31 to 33 is used to implement the all-gather operation executed within compute group 3. Compute nodes 31 to 33 perform the all-gather operation within compute group 3 based on the all-reduce result data corresponding to each of compute nodes 31 to 33, and obtain the all-gather result data corresponding to each of compute nodes 31 to 33.The all-gather result data corresponding to any two computing nodes among computing nodes 11-13, 21-23, and 31-33 is the same. The all-gather result data corresponding to each of these computing nodes comprises 9 data blocks, with the data in each block being A1+A2+A3+A4+A5+A6+A7+A8+A9+A10+A11+A12, B1+B2+B3+B4+B5+B6+B7+B8+B9+B10+B11+B12, and C1+C2+C3+C4+C5+C6+C7+C8+C9+C1 0+C11+C12, D1+D2+D3+D4+D5+D6+D7+D8+D9+D10+D11+D12, E1+E2+E3+E4+ E5+E6+E7+E8+E9+E10+E11+E12, F1+F2+F3+F4+F5+F6+F7+F8+F9+F10+F11 +F12、G1+G2+G3+G4+G5+G6+G7+G8+G9+G10+G11+G12、H1+H2+H3+H4+H5+H6 +H7+H8+H9+H10+H11+H12 and J1+J2+J3+J4+J5+J6+J7+J8+J9+J10+J11+J12.

[0219] It should be noted that, for the sake of brevity in the accompanying drawings, the embodiments of this application do not show the calculations between groups 1 and 3. Figure 16 The diagrams showing the logical nodes after the all-reduce operation and the all-gather operation are described in text only.

[0220] It should also be noted that the above embodiments are illustrated using the division of logical nodes as an example. For Figure 16As shown, the boundaries of the reduce-scatter result data corresponding to computing node 11, computing node 21, and computing node 31 are aligned; the boundaries of the reduce-scatter result data corresponding to computing node 12, computing node 22, and computing node 32 are aligned; and the boundaries of the reduce-scatter result data corresponding to computing node 13, computing node 23, and computing node 33 are aligned. Therefore, it is also possible to perform the second set communication operation directly between different computing groups without dividing logical nodes, and computing nodes 11-13, 21-23, and 31-31 can be used.

[0221] It should also be noted that during the execution of the third set communication operation (e.g., an all-gather operation) within any of the aforementioned n computing groups, the computing nodes within that group can send data to each other to implement the third set communication operation. For example, after the execution of the second set communication operation (e.g., an all-reduce operation) between different computing groups among the n computing groups, the aforementioned G in the first computing group... min Each computing node in the computing nodes obtains the second data corresponding to that computing node, and the G... min The second data corresponding to each computing node in the n computing groups is the data obtained by performing a second set communication operation between different computing groups in the n computing groups (that is, the second data is the result data of the second set communication operation), and is related to the G min The second data corresponding to each computing node in the first computing group is used to implement the third set communication operation executed within the first computing group. During the execution of the third set communication operation within the first computing group, the G within the first computing group... min Each computing node in the first computing group sends its corresponding second data to other computing nodes in the first computing group (or, in other words, the process of performing the third set communication operation within the first computing group includes the G in the first computing group). min Each of the n computing nodes sends its corresponding second data to the other computing nodes in the first computing group. Furthermore, the process of performing the third set communication operation within different computing groups can be parallelized, thereby improving the efficiency of performing the third set communication operation within different computing groups.

[0222] In this embodiment, the process of performing a second set communication operation (e.g., an all-reduce operation) between different computing groups in the n computing groups can be implemented based on algorithms with fewer communication steps, such as the HD algorithm, thereby reducing the number of communication steps required for performing the second set communication operation between different computing groups. The process of performing a third set communication operation within any of the n computing groups can be implemented based on a single-layer algorithm (e.g., the ring algorithm, the HD algorithm, etc.), thereby effectively utilizing the link bandwidth between different computing nodes.

[0223] As can be seen from the above embodiments, in the set communication method provided in this application, firstly, a reduce-scatter operation (i.e., the first set communication operation) is performed within each computing group in the computing cluster. Then, based on the result data obtained from the reduce-scatter operation within each computing group in the computing cluster, an all-reduce operation (i.e., the second set communication operation) is performed between different computing groups in the computing cluster. Finally, based on the result data obtained from the all-reduce operation between different computing groups in the computing cluster, an all-gather operation (i.e., the third set communication operation) is performed within each computing group in the computing cluster. This achieves a hierarchical all-reduce algorithm.

[0224] In summary, the technical solution provided by the embodiments of this application involves k computing nodes in a first computing group, which are based on raw data corresponding to each of the k computing nodes and used to perform a first set communication operation within the first computing group. The first set communication operation is performed within the first computing group to obtain first data that corresponds one-to-one with a portion of the k computing nodes. The number of the portion of computing nodes is equal to the number of computing nodes in the smallest computing group, and the first data corresponding to the portion of computing nodes is used to implement a second set communication operation between different computing groups in n computing groups. Therefore, this application embodiment can combine the number of computing nodes in the minimum computing group to perform a first set communication operation in each computing group other than the minimum computing group. This results in first set communication operation result data corresponding to a subset of computing nodes within each computing group. The number of these subset computing nodes is equal to the number of computing nodes in the minimum computing group, and each subset corresponds one-to-one with the computing nodes in the minimum computing group. Furthermore, a second set communication operation can be performed between different computing groups within the n computing groups based on the first set communication operation result data corresponding to these subset computing nodes, facilitating the implementation of the hierarchical all-reduce algorithm within the computing cluster. The set communication method provided by this application embodiment can optimize the hierarchical all-reduce algorithm, enabling its application in asymmetric computing clusters and expanding its application scenarios. Moreover, the set communication method provided by this application embodiment also exhibits good communication performance and high communication efficiency.

[0225] Furthermore, the embodiments of this application follow the formula w = n × G min The original data is split into data blocks. The number of data blocks obtained from this splitting process will not be too large, and the size of each data block will not be too small. The data slices corresponding to multiple logical nodes obtained from the reduce-scatter result data partitioning are of equal size (e.g., ...). Figure 13 As shown in the figure, the data slice corresponding to each logical node is not too small, thereby avoiding data fragmentation. Furthermore, in the embodiments of this application, there is no situation where one calculation corresponds to multiple logical nodes, which can avoid the occurrence of "multiple-to-one" traffic.

[0226] It should be noted that the execution of a collective communication operation (such as a first collective communication operation, a second collective communication operation, or a third collective communication operation) typically requires the participation of multiple computing nodes. In the above embodiments, the execution of a collective communication operation by a single computing node refers to the execution of the semantics corresponding to that single computing node within the collective communication operation (e.g., sending data). This application embodiment does not impose such limitations. Furthermore, the above embodiments use asymmetric computing clusters as examples; the collective communication method provided in this application embodiment is also applicable to symmetric computing clusters.

[0227] The above describes the method embodiments of this application. The apparatus embodiments of this application are described below. The apparatus of this application can be used to execute the method of this application. For details not disclosed in the apparatus embodiments of this application, please refer to the method embodiments.

[0228] Please refer to Figure 18 The diagram illustrates a collective communication device 800 provided in an embodiment of this application. The collective communication device 800 is applied to a computing node (e.g., a first computing node). Based on... Figure 18 The module shown, which integrates communication device 800, is capable of performing tasks such as... Figure 6 The illustrated method embodiments provide all or part of the operations. It should be understood that the aggregate communication device 800 may include more additional modules than those shown or omit some of the modules shown, and this application embodiment does not impose any limitations in this regard.

[0229] like Figure 18 As shown, the integrated communication device 800 includes a transceiver module 810 and a processing module 820. The transceiver module 810 is used to perform, for example... Figure 6 The illustrated embodiment provides a collection communication method for receiving and / or sending related operations. The processing module 820 is used to perform operations such as... Figure 6 Other operations besides the receiving and / or sending related operations in the aggregate communication method provided in the illustrated embodiment. The transceiver module 810 and / or processing module 820 can be implemented based on software, hardware, or a combination of software and hardware. The transceiver module 810 may include a receiving module and / or a sending module. The receiving module is used to perform receiving-related operations, and the sending module is used to perform sending-related operations.

[0230] In an optional embodiment, the aggregated communication device 800 is applied to a first computing node within a first computing group. The computing cluster includes n computing groups, each of which includes at least one computing node, where n is an integer greater than 1. The n computing groups include a first computing group and a second computing group. A processing module 820 is used to acquire first data, which is data corresponding to the first computing node obtained by performing a first aggregated communication operation within the first computing group. A transceiver module 810 is used to send a first portion of the first data to a second computing node within a second computing group, and to send a second portion of the first data to a third computing node within the second computing group. The first portion of the data corresponds to the second computing node, and the second portion of the data corresponds to the third computing node. The first and second portions of the data are used to implement a second aggregated communication operation between different computing groups within the n computing groups. The second and third computing nodes are different.

[0231] In an optional embodiment, the transceiver module 810 is further configured to send the third part of the first data to the fourth computing node in the second computing group. The third part of the data corresponds to the fourth computing node. The third part of the data is also used to implement the second set communication operation between different computing groups in the n computing groups. The second computing node, the third computing node and the fourth computing node are different.

[0232] In an optional embodiment, the transceiver module 810 is further configured to send the third part of the first data to the third computing node in the second computing group. The third part of the data also corresponds to the third computing node and is also used to implement the second set communication operation between different computing groups in the n computing groups.

[0233] In an optional embodiment, the transceiver module 810 is used to sequentially send the first part of the first data to the second computing node in the second computing group, and send the second part of the first data to the third computing node in the second computing group.

[0234] When the third part of the first data corresponds to the fourth computing node in the second computing group, the transceiver module 810 is used to send the first part of the first data to the second computing node in the second computing group, send the second part of the first data to the third computing node in the second computing group, and send the second part of the first data to the fourth computing node in the second computing group in sequence.

[0235] When the third part of the first data corresponds to the third computing node in the second computing group, the transceiver module 810 is used to send the first part of the first data to the second computing node in the second computing group in sequence, send the second part of the first data to the third computing node in the second computing group, and send the third part of the first data to the third computing node in the second computing group.

[0236] In an optional embodiment, the transmission order of the first portion of data and the second portion of data is determined based on the priority of the first portion of data and the priority of the second portion of data. If the first data also includes a third portion of data, the transmission order of the first portion of data, the second portion of data, and the third portion of data is determined based on the priority of the first portion of data, the priority of the second portion of data, and the priority of the third portion of data.

[0237] In optional embodiments, the first computing node corresponds to a first logical node and a second logical node. The first portion of data corresponds to the first logical node, and the second portion of data corresponds to the second logical node. The priority of the first portion of data is determined based on its data arrangement order within the logical group to which the first logical node belongs, and the priority of the second portion of data is determined based on its data arrangement order within the logical group to which the second logical node belongs. For example, if the first logical node belongs to the first logical group and the second logical node belongs to the second logical group, the priority of the first portion of data is determined based on its data arrangement order within the first logical group, and the priority of the second portion of data is determined based on its data arrangement order within the second logical group. As another example, if both the first and second logical nodes belong to the first logical group, the priority of the first portion of data is determined based on its data arrangement order within the first logical group, and the priority of the second portion of data is determined based on its data arrangement order within the first logical group. The first computing node may also correspond to another logical node besides the first and second logical nodes, and a third portion of the first data corresponds to this other logical node. The priority of this third portion of data is determined based on its data arrangement order within the logical group to which this other logical node belongs.

[0238] In an optional embodiment, the number of computing nodes within the smallest computing group is G. min The minimum computational group is the computational group with the fewest computational nodes among the n computational groups, and the computational nodes in the minimum computational group are related to G. min Each logical group corresponds one-to-one, and the logical node corresponding to each computing node within the smallest computing group belongs to the logical group corresponding to each computing node.

[0239] In an optional embodiment, each computing node in each of the n computing groups corresponds to at least one logical node. The second set communication operation is a set communication operation performed between different computing groups in the n computing groups for each logical node. When the same computing node corresponds to multiple logical nodes, the process of performing the second set communication operation between different computing groups in the n computing groups for the multiple logical nodes is serial.

[0240] In an optional embodiment, the first logical node belongs to the first logical group, the second logical node belongs to the second logical group, the fifth logical node in the first computing group corresponds to the third logical node, and the third logical node also belongs to the first logical group (that is, the first logical node and the third logical node belong to the same logical group), the sixth logical node in the first computing group corresponds to the fourth logical node, and the fourth logical node also belongs to the second logical group (that is, the second logical node and the fourth logical node belong to the same logical group, and the fourth logical node and the first logical node belong to different logical groups). The process of performing the second set communication operation on the first logical node between different computing groups in the n computing groups is serial with the process of performing the second set communication operation on the third logical node between different computing groups in the n computing groups. The process of performing the second set communication operation on the first logical node between different computing groups in the n computing groups is parallel with the process of performing the second set communication operation on the fourth logical node between different computing groups in the n computing groups.

[0241] In an optional embodiment, the processing module 820 is further configured to, before the transceiver module 810 sends the first part of the first data to the second computing node in the second computing group and sends the second part of the first data to the third computing node in the second computing group, divide the first data based on the boundary of the first set communication operation result data corresponding to multiple computing nodes to obtain the first part of the data and the second part of the data. The multiple computing nodes are located in computing groups other than the first computing group among the n computing groups. The first set communication operation result data corresponding to each of the multiple computing nodes is the data obtained by performing the first set communication operation in the computing group where each computing node is located. There is no boundary aligned with the boundary of the first data for the first set communication operation result data corresponding to each of the multiple computing nodes.

[0242] In addition, the processing module 820 can divide the first data based on the boundary of the first set communication operation result data corresponding to the plurality of computing nodes to obtain a third part of data. The boundary of the third part of data is aligned with the boundary of the first set communication operation result data corresponding to at least one of the plurality of computing nodes.

[0243] In an optional embodiment, the first computing node includes w data blocks, which are used to implement a first set communication operation executed within the first computing group, where w = n × L, and L represents the least common multiple of the number of computing nodes within the n computing groups; or, w = n × G max G max This represents the number of computing nodes in the largest computing group, which is the computing group with the most computing nodes among the n computing groups.

[0244] In an optional embodiment, the processing module 820 is further configured to acquire second data, which is data corresponding to the first computing node obtained by performing a second set communication operation between different computing groups in the n computing groups. The second data is used to implement a third set communication operation performed within the first computing group. The transceiver module 810 is further configured to send the second data to computing nodes in the first computing group other than the first computing node.

[0245] In an optional embodiment, the second data corresponding to the first computing node is data obtained by performing a second set communication operation between different computing groups in the n computing groups. This second data is used to implement a third set communication operation performed within the first computing group. The first set communication operation is a reduce-scatter operation; the second set communication operation is an all-reduce operation; and the third set communication operation is an all-gather operation.

[0246] In an optional embodiment, at least two of the n computing groups have unequal numbers of computing nodes. That is, the computing cluster is an asymmetric computing cluster.

[0247] In an optional embodiment, the computing nodes in the computing cluster include at least one of the following: server, NPU, GPU, TPU, or CPU.

[0248] It should be understood that the above Figure 18 The provided aggregate communication device 800 is illustrated only by the division of the above-described functional modules when implementing its functions. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the computing node can be divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process and beneficial effects are detailed in the method embodiments, which will not be repeated here.

[0249] It should be noted that the aggregate communication device mentioned in the embodiments of this application can be a computing node such as a server, NPU, GPU, TPU, or CPU, or it can be a component on a computing node. This application does not limit the scope of the aggregate communication device. The aggregate communication device provided in the embodiments of this application can also be implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD can be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof. The aggregate communication method provided in the above method embodiments can also be implemented in software. When the aggregate communication method provided in the above method embodiments is implemented in software, each module in the aggregate communication device can also be a software module.

[0250] This application provides a collective communication device. The collective communication device includes a memory and a processor. The memory stores a computer program. The processor executes the computer program stored in the memory to cause the collective communication device to perform actions such as... Figure 6 or Figure 14 The illustrated embodiment provides all or part of the steps of the collection communication method. The processor and memory are one or more. The memory is integrated with the processor, or the memory and processor are separately configured.

[0251] In one embodiment, please refer to Figure 19 This illustration shows a schematic diagram of another aggregated communication device 900 provided in an embodiment of this application. The aggregated communication device 900 can be a computing device (e.g., a server). The aggregated communication device 900 is used to perform actions such as... Figure 6 or Figure 14 The illustrated embodiment provides all or part of the steps of the collective communication method. The collective communication device 900 includes at least one processor 901. Figure 19 (Taking two processors 901 as an example), a communication bus 902, a memory 903, and at least one communication interface 904 are connected via the communication bus 902. The at least one processor 901, the memory 903, and the at least one communication interface 904 may also be connected using a connection method other than the communication bus 902.

[0252] The memory 903 is used to store the computer program that executes the technical solution of this application and is controlled by the processor 901 to execute it. The computer program stored in the memory 903 includes, but is not limited to, program code, program instructions, and data. The memory 903 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or it may be a non-volatile random access memory (NVRAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 903 can exist independently and be connected to the processor 901 via the communication bus 902. Alternatively, the memory 903 can be integrated with the processor 901; this embodiment does not limit the specific integration.

[0253] Processor 901 can be a general-purpose processor or a special-purpose processor. A general-purpose processor is a processor that performs specific steps and / or operations by reading and executing a computer program stored in memory (e.g., memory 903). In performing these steps and / or operations, the general-purpose processor may use the computer program stored in memory (e.g., memory 903), which can be executed to implement the relevant functions of the aforementioned processing module 820. General-purpose processors include, but are not limited to, central processing units (CPUs). Special-purpose processors are processors specifically designed to perform specific steps and / or operations. Special-purpose processors include, but are not limited to, NPUs, GPUs, TPUs, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. Processor 901 can implement or execute various logic blocks, modules, and circuits described in connection with the embodiments of this application. Processor 901 can also be a combination of computing functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0254] The communication bus 902 is used to transfer information between the processor 901, the communication interface 904, and the memory 903. The communication bus 902 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The communication bus 902 can be divided into an address bus, a data bus, and a control bus, etc. For ease of representation, Figure 19 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0255] Communication interface 904 includes input / output (I / O) interfaces, physical interfaces, and logical interfaces for interconnecting devices within the collective communication device 900, and interfaces for interconnecting the collective communication device 900 with other communication devices. Physical interfaces can be Ethernet interfaces, Fast Ethernet (FE) interfaces, Gigabit Ethernet (GE) interfaces, Terabit Ethernet (TbE) interfaces, 400GE interfaces, asynchronous transfer mode (ATM) interfaces, etc., used for interconnecting the collective communication device 900 with other devices. Logical interfaces are internal interfaces of the collective communication device 900, used for interconnecting devices within the collective communication device 900. It is easy to understand that communication interface 904 is used for communication between the collective communication device 900 and other devices or communication networks; for example, communication interface 904 is used for sending and receiving data between the collective communication device 900 and other devices or communication networks. Communication interface 904 can also be a device using any transceiver.

[0256] In a specific implementation, as one embodiment, the aggregated communication device 900 includes a plurality of processors 901, each of which can be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor. Here, a processor can refer to one or more devices, circuits, and / or processing cores used for processing data (e.g., computer program instructions).

[0257] In another embodiment, please refer to Figure 20 This illustration shows a schematic diagram of another collective communication device 2000 provided in an embodiment of this application. The collective communication device 2000 can be a network device supporting collective communication, such as a switch or router. The collective communication device 2000 is used to perform actions such as... Figure 6 or Figure 14 The illustrated embodiments provide all or part of the steps of the collection communication method. For example... Figure 20 As shown, the integrated communication device 2000 includes a main control board and one or more interface boards. Figure 1(Taking interface boards 1 and 2 as examples), the main control board and the interface boards are communicatively connected. The main control board, also called the main processing unit (MPU) or route processor card, is responsible for controlling and managing the various components in the aggregated communication device 2000, including route calculation, device management, and maintenance functions. The interface board, also called the line processing unit (LPU) or line card, is used for forwarding data. In some embodiments, the aggregated communication device 2000 may also include a switching network board, which is communicatively connected to the main control board and the interface boards. The switching network board is used for forwarding data between the interface boards; it can also be called a switchfabric unit (SFU). The interface board includes a central processing unit (CPU), memory, a forwarding chip, and a physical interface card (PIC). The CPU is communicatively connected to the memory, forwarding chip, and physical interface card, respectively. The memory is used to store the forwarding table. The forwarding chip forwards received data frames based on a forwarding table stored in memory. If the destination address of the data frame is the address of the aggregate communication device 2000, the data frame is sent to the CPU for processing. If the destination address of the data frame is not the address of the aggregate communication device 2000, the next hop and outgoing interface corresponding to the destination address are found in the forwarding table, and the data frame is forwarded to the outgoing interface corresponding to the destination address. The forwarding chip can be a network processor (NP) chip. The PIC, also known as a daughter card, can be installed on the interface board and is responsible for converting photoelectric signals into data frames, performing validity checks on the data frames, and forwarding them to the forwarding chip for processing. In some embodiments, the central processing unit can also perform the functions of the forwarding chip, such as implementing software forwarding based on a general-purpose CPU, thus eliminating the need for a forwarding chip on the interface board. Communication connections between the main control board, interface board, and switching network board can be achieved via a bus. The forwarding chip can be implemented using an ASIC or FPGA.

[0258] Logically, the aggregated communication device 2000 includes a control plane and a forwarding plane. The control plane includes a main control board and a central processing unit, while the forwarding plane includes various components that perform forwarding, such as memory, a PIC, and an NP chip. The control plane performs functions such as generating forwarding tables, processing signaling and protocol messages, and configuring and maintaining status. The control plane sends the generated forwarding tables to the forwarding plane. In the forwarding plane, the NP chip uses the forwarding tables sent by the control plane to look up and forward messages received by the PIC of the aggregated communication device 2000. The forwarding tables sent by the control plane can be stored in memory. In some embodiments, the control plane and the forwarding plane can be completely separated and not on the same device.

[0259] In one possible implementation, an inter-process communication (IPC) channel is established between the main control board and the interface board, and the main control board and the interface board communicate with each other through the IPC channel.

[0260] Based on the same inventive concept, embodiments of this application provide a computing cluster comprising n computing groups, each of the n computing groups comprising at least one computing node, where n is an integer greater than 1. The at least one computing node in the computing cluster includes, as follows: Figures 18 to 20 Any of the aggregated communication devices shown. The computing nodes in this computing cluster include at least one of the following: server, NPU, GPU, TPU, or CPU. For example, the computing cluster is as follows: Figure 4 The computing cluster 10 shown is or is as follows Figure 5 The computing cluster 20 is shown.

[0261] This application embodiment also provides another computing cluster, which includes n computing groups, each of the n computing groups including at least one computing node, and at least two of the n computing groups have unequal numbers of computing nodes, where n is an integer greater than 1. The n computing groups include a first computing group, which is not the smallest computing group among the n computing groups; the smallest computing group is the computing group with the fewest computing nodes among the n computing groups. The k computing nodes in the first computing group are used to perform operations such as... Figure 14 The illustrated embodiments provide all or part of the steps of the collection communication method.

[0262] In a specific embodiment, the k computing nodes within the first computing group are used to: perform the first set communication operation within the first computing group based on the original data corresponding to each of the k computing nodes, which is used to implement the first set communication operation within the first computing group, to obtain G within the first computing group. min The first data corresponding to each computing node, G min Each computing node is a subset of the k computing nodes, and is related to G.min The first data corresponding to each computing node is used to implement the second set communication operation between different computing groups in the n computing groups. min G represents the number of computing nodes within this minimum computing group. min Both k and k are positive integers.

[0263] In an optional embodiment, each of the k computing nodes is used to: divide the original data corresponding to each computing node into w data blocks, where w = n × G min The k computing nodes are used to: perform a first set communication operation within the first computing group based on the w data blocks corresponding to the k computing nodes, to obtain the data related to G. min The first data corresponding to each computing node, and the G min The first data corresponding to each of the computing nodes includes at least one data block.

[0264] In an optional embodiment, the G min Each computing node corresponds one-to-one with G. min There are n logical nodes. The second set communication operation is a set communication operation performed between different computing groups within the n computing groups for each logical node. min The process of each logical node performing the second set of communication operations is parallel.

[0265] In an optional embodiment, the G min Each of the computing nodes is further configured to: acquire second data corresponding to that computing node, and connect it with the G... min The second data corresponding to each of the n computing nodes is the data obtained by performing a second set communication operation between different computing groups in the n computing groups, and is related to the G. min The second data corresponding to each computing node in the first computing group is used to implement the third set communication operation executed within the first computing group; the second data corresponding to each computing node is sent to other computing nodes within the first computing group.

[0266] In an optional embodiment, with the G min The second data corresponding to each of the n computing nodes is the data obtained by performing a second set communication operation between different computing groups in the n computing groups, and is related to the G. min The second data corresponding to each computing node in the computing node is used to implement the third set communication operation executed within the first computing group; the first set communication operation is a reduce-scatter operation; the second set communication operation is an all-reduce operation; and the third set communication operation is an all-gather operation.

[0267] In an optional embodiment, the computing nodes in the computing cluster include at least one of the following: server, NPU, GPU, TPU, or CPU.

[0268] In an optional embodiment, the computing cluster is as follows: Figure 4 The computing cluster 10 shown is or is as follows Figure 5 The computing cluster 20 is shown.

[0269] Based on the same inventive concept, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed (e.g., by a computing node and / or a collective communication device), implements the following: Figure 6 At least some steps of the method embodiments shown, or implementations such as Figure 14 At least some steps of the method embodiment shown.

[0270] Based on the same inventive concept, embodiments of this application provide a computer program product, which includes a program or code that, when executed (e.g., executed by a computing node and / or a collective communication device), implements the following: Figure 6 At least some steps of the method embodiments shown, or implementations such as Figure 14 At least some steps of the method embodiment shown.

[0271] Based on the same inventive concept, embodiments of this application provide a chip, which includes programmable logic circuitry and / or program instructions, and which, when running, is used to implement, as... Figure 6 At least some steps of the method embodiments shown, or implementations such as Figure 14 At least some steps of the method embodiment shown.

[0272] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented entirely or partially as a computer program product, which includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium, or a semiconductor medium (e.g., solid-state drive), etc.

[0273] It should be understood that the term "at least one" in this application refers to one or more, and "multiple" refers to two or more. In this application, unless otherwise stated, the symbol " / " generally means "or," for example, A / B can mean A or B. The term "and / or" in this application is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, for clarity, this application uses terms such as "first," "second," and "third" to distinguish identical or similar items with substantially the same function and effect. Those skilled in the art will understand that the terms "first," "second," and "third" do not limit the quantity or order of execution.

[0274] The different types of embodiments, such as the method embodiments and device embodiments provided in this application, can be referenced to each other. The order of operations in the method embodiments can be adjusted appropriately, and the operations can be added or removed in response to the situation. Any variations that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application, and therefore will not be described in detail.

[0275] In the corresponding embodiments provided in this application, it should be understood that the disclosed devices, etc., can be implemented through other configurations. For example, the device embodiments described above are merely illustrative. For instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed between devices or modules may be through some interfaces, or indirect coupling or communication connection between devices or modules, which may be electrical or other forms. Modules described as separate components may or may not be physically separate, and components described as modules may or may not be physical modules; they may be located in one place or distributed across multiple network nodes. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0276] The above description is merely an exemplary embodiment of this application, but the scope of protection of this application is not limited thereto. Any equivalent modifications or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A collective communication method, characterized in that, Applied to a computing cluster, the computing cluster comprising n computing groups, each of the n computing groups comprising at least one computing node, where n is an integer greater than 1, and the n computing groups comprising a first computing group and a second computing group, the method includes: The first computing node in the first computing group acquires the first data, which is the data corresponding to the first computing node obtained by performing the first set communication operation in the first computing group. The first computing node sends a first portion of the first data to a second computing node in the second computing group, and sends a second portion of the first data to a third computing node in the second computing group. The first portion of the data corresponds to the second computing node, and the second portion of the data corresponds to the third computing node. The first portion of the data and the second portion of the data are used to implement a second set communication operation between different computing groups in the n computing groups. The second computing node and the third computing node are different.

2. The method according to claim 1, characterized in that, The method further includes: The first computing node sends the third part of the first data to the fourth computing node in the second computing group. The third part of the data corresponds to the fourth computing node. The third part of the data is also used to implement the second set communication operation between different computing groups in the n computing groups. The second computing node, the third computing node and the fourth computing node are different.

3. The method according to claim 1, characterized in that, The method further includes: The first computing node sends the third part of the first data to the third computing node. The third part of the data also corresponds to the third computing node. The third part of the data is also used to implement the second set communication operation between different computing groups in the n computing groups.

4. The method according to any one of claims 1 to 3, characterized in that, The first computing node sends a first portion of the first data to a second computing node within the second computing group, and sends a second portion of the first data to a third computing node within the second computing group, including: The first computing node sequentially sends the first part of the first data to the second computing node in the second computing group, and sends the second part of the first data to the third computing node in the second computing group.

5. The method according to claim 4, characterized in that, The transmission order of the first part of data and the second part of data is determined based on the priority of the first part of data and the priority of the second part of data.

6. The method according to claim 5, characterized in that, The first computing node corresponds to the first logical node and the second logical node. The first part of the data corresponds to the first logical node, and the second part of the data corresponds to the second logical node. The priority of the first part of the data is determined based on the data arrangement order of the first part of the data in the logical group to which the first logical node belongs. The priority of the second part of the data is determined based on the data arrangement order of the second part of the data in the logical group to which the second logical node belongs.

7. The method according to claim 6, characterized in that, The number of computing nodes in the smallest computing group is G. min The minimum computation group is the computation group with the fewest computation nodes among the n computation groups, and the computation nodes in the minimum computation group are related to G. min Each logical group corresponds one-to-one, and the logical node corresponding to each computing node in the smallest computing group belongs to the logical group corresponding to each computing node.

8. The method according to any one of claims 1 to 7, characterized in that, The first computing node includes w data blocks, which are used to implement the first set communication operation executed within the first computing group. w = n × L, where L represents the least common multiple of the number of computing nodes in the n computing groups; or, w = n × G max The G max This indicates the number of computing nodes in the largest computing group, which is the computing group with the most computing nodes among the n computing groups.

9. The method according to any one of claims 1 to 8, characterized in that, The second data corresponding to the first computing node is the data obtained by performing the second set communication operation between different computing groups in the n computing groups. The second data is used to implement the third set communication operation performed within the first computing group. The first set communication operation is a reduce-scatter operation; The second set communication operation is a full-reduce operation; The third set communication operation is an all-gather operation.

10. The method according to any one of claims 1 to 9, characterized in that, The number of computing nodes in at least two of the n computing groups is not equal.

11. A collective communication method, characterized in that, The method is applied to a computing cluster comprising n computing groups, each of the n computing groups comprising at least one computing node, wherein at least two of the n computing groups have unequal numbers of computing nodes, and n is an integer greater than 1. The n computing groups include a first computing group, which is not the smallest computing group among the n computing groups. The smallest computing group is the computing group with the fewest computing nodes among the n computing groups. The k computing nodes in the first computing group are based on the original data corresponding to each of the k computing nodes, used to perform the first set communication operation within the first computing group. The first set communication operation is performed within the first computing group to obtain G... min The first data corresponding to each computing node, the G min Each computing node is a subset of the k computing nodes, and is related to G. min The first data corresponding to each computing node is used to implement a second set communication operation between different computing groups in the n computing groups, wherein G min G is the number of computing nodes within the minimum computing group. min Both k and k are positive integers.

12. The method according to claim 11, characterized in that, The k computing nodes, based on the original data corresponding to each of the k computing nodes, are used to execute the first set communication operation within the first computing group. The first set communication operation is then performed within the first computing group to obtain data related to G. min The first data corresponding to each computing node includes: Each of the k computing nodes will divide the original data corresponding to each computing node into w data blocks, where w = n × G. min ; The k computing nodes, based on the w data blocks corresponding to each of the k computing nodes, perform the first set communication operation within the first computing group to obtain the data from the G... min The first data corresponding one-to-one with each computing node, and the G min The first data corresponding to each of the computing nodes includes at least one data block.

13. The method according to claim 11 or 12, characterized in that, With the G min The second data corresponding to each of the n computing nodes is the data obtained by performing the second set communication operation between different computing groups in the n computing groups, and is related to the G. min The second data corresponding to each computing node in the computing node is used to implement the third set communication operation executed within the first computing group; The first set communication operation is a reduce-scatter operation; The second set communication operation is a full-reduce operation; The third set communication operation is an all-gather operation.

14. The method according to any one of claims 1 to 13, characterized in that, The computing nodes in the computing cluster include at least one of the following: a server, a neural network processing unit (NPU), a graphics processing unit (GPU), a tensor processing unit (TPU), or a central processing unit (CPU).

15. A collective communication device, characterized in that, The aggregated communication device is applied to a computing node, and the aggregated communication device includes: The transceiver module is used to perform the receiving and / or sending related operations in the method according to any one of claims 1 to 10, 14; The processing module is configured to perform operations other than the receiving and / or sending related operations in the method described in any one of claims 1 to 10, 14.

16. A collective communication device, characterized in that, The aggregate communication device includes a memory and a processor; The memory is used to store computer programs; The processor is configured to execute a computer program stored in the memory to cause the collective communication device to perform the method as described in any one of claims 1 to 14.

17. A computing cluster, characterized in that, The computing cluster comprises n computing groups, and each of the n computing groups comprises at least one computing node, where n is an integer greater than 1. At least one computing node in the computing cluster includes the aggregate communication device as described in claim 15 or 16.

18. A computing cluster, characterized in that, The computing cluster includes n computing groups, each of the n computing groups includes at least one computing node, and the number of computing nodes in at least two of the n computing groups is not equal, where n is an integer greater than 1, and the n computing groups include a first computing group, which is not the smallest computing group among the n computing groups. The smallest computing group is the computing group with the fewest computing nodes among the n computing groups. The k computing nodes within the first computing group are used to: perform the first set communication operation within the first computing group based on the original data corresponding to each of the k computing nodes, thereby obtaining G within the first computing group. min The first data corresponding to each computing node, the G min Each computing node is a subset of the k computing nodes, and is related to G. min The first data corresponding to each computing node is used to implement a second set communication operation between different computing groups in the n computing groups, wherein G min G is the number of computing nodes within the minimum computing group. min Both k and k are positive integers.

19. The computing cluster according to claim 18, characterized in that, Each of the k computing nodes is used to: divide the original data corresponding to each computing node into w data blocks, where w = n × G min ; The k computing nodes are used to: perform the first set communication operation within the first computing group based on the w data blocks corresponding to the k computing nodes, to obtain the data from the G... min The first data corresponding one-to-one with each computing node, and the G min The first data corresponding to each of the computing nodes includes at least one data block.

20. The computing cluster according to claim 18 or 19, characterized in that, With the G min The second data corresponding to each of the n computing nodes is the data obtained by performing the second set communication operation between different computing groups in the n computing groups, and is related to the G. min The second data corresponding to each computing node in the computing node is used to implement the third set communication operation executed within the first computing group; The first set communication operation is a reduce-scatter operation; The second set communication operation is a full-reduce operation; The third set communication operation is an all-gather operation.

21. The computing cluster according to any one of claims 17 to 20, characterized in that, The computing nodes in the computing cluster include at least one of the following: a server, a neural network processing unit (NPU), a graphics processing unit (GPU), a tensor processing unit (TPU), or a central processing unit (CPU).

22. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed, implements the method as described in any one of claims 1 to 14.

23. A computer program product, characterized in that, The computer program product includes a program or code that, when executed, implements the method as described in any one of claims 1 to 14.

Images