A method of aggregated communication and related products

Abstract

The application provides a collection communication method and related products, wherein the collection communication method comprises the following steps: acquiring network architectures corresponding to a plurality of collection communication machines and bandwidth levels between the plurality of collection communication machines; dividing the plurality of collection communication machines into a plurality of communication groups with symmetrical structures according to the network architectures corresponding to the plurality of collection communication machines and the bandwidth levels between the plurality of collection communication machines; and finally completing collection communication between the plurality of collection communication machines based on the communication groups. The collection communication method is implemented, the complex network architecture (asymmetrical network architecture and network architecture with three or more bandwidth levels) is divided into a plurality of communication groups with symmetrical network architectures, so that efficient symmetrical network collection communication algorithms can be implemented in the communication groups, the communication logic is simplified, the bandwidth resources at different levels are fully utilized, and the overall collection communication efficiency is improved.

Description

Technical Field

[0001] This application relates to the field of collective communication technology, and in particular to a collective communication method and related products. Background Technology

[0002] Collective communication refers to a communication mode in a distributed system in which multiple collective communication machines (also known as computing units) work together to complete data interaction. Its core lies in the interaction and aggregation of data between multiple collective communication machines.

[0003] Current methods for implementing aggregated communication primarily utilize aggregated communication algorithms. For example, in a network architecture consisting of multiple aggregated communication machines, the RingAllReduce algorithm can be used to divide the data into N equal parts (N representing the number of aggregated communication machines). These N parts are then distributed to the N aggregated communication machines for processing. After N-1 rounds of interaction and N-1 rounds of aggregation, the aggregated communication of the entire system is finally completed.

[0004] Current aggregate communication methods primarily focus on networks with uniform bandwidth (symmetric architecture) or simple two-level convergence networks (with two or fewer bandwidth levels). When network architectures become more complex, such as those involving multi-level convergence (three or more bandwidth levels) or networks with uneven bandwidth (asymmetric architecture), current aggregate communication methods struggle to effectively utilize the bandwidth between the communicating machines, resulting in low overall system efficiency. For example, when using the RingAllReduce algorithm to handle multi-level convergence and asymmetric networks, the large number of bandwidth levels indicates significant bandwidth differences between the communicating machines. Since the total time required for each round of data transmission depends on the slowest link (the link with the lowest bandwidth), the overall performance of this method is limited by the lowest bandwidth in the aggregate communication system. Summary of the Invention

[0005] This application provides a collective communication method and related products. By dividing a complex network architecture (asymmetric network architecture and network architecture with three or more bandwidth levels) into multiple communication groups with symmetric network architecture, an efficient symmetric network collective communication algorithm can be implemented within the communication group. This not only simplifies the communication logic but also makes full use of the bandwidth resources at different levels, thereby improving the overall collective communication efficiency.

[0006] Firstly, this application provides a collective communication method applied to a collective communication machine group, wherein the collective communication machine group includes multiple collective communication machines, the network architecture corresponding to the collective communication machine group is an asymmetric network architecture, the asymmetric network architecture includes asymmetry in network topology or bandwidth, the collective communication machine group includes three or more bandwidth levels, the bandwidth levels are used to indicate the bandwidth size within a certain range, the method includes: receiving first information, the first information is used to indicate the structure of the collective communication machine group, the first information includes the bandwidth level corresponding to the bandwidth between every two collective communication machines in the multiple collective communication machines; based on the bandwidth level corresponding to the bandwidth between every two collective communication machines in the multiple collective communication machines, dividing the multiple collective communication machines into one or more first communication groups, wherein the first communication group includes all or some of the collective communication machines in the multiple collective communication machines; the network architecture corresponding to the first communication group is a symmetric network architecture; the symmetric network architecture includes symmetry in network topology or bandwidth; and based on one or more first communication groups, completing collective communication between the multiple collective communication machines.

[0007] Existing ensemble communication algorithms are mostly applicable to symmetric structures or network architectures with low bandwidth levels, and cannot be effectively applied to multi-level convergence and asymmetric network architectures.

[0008] Implementing the scheme shown in the first aspect, for complex network architectures (including asymmetric network architectures and network architectures with more than three bandwidth levels), the network is divided into multiple communication groups with symmetric network architectures based on the bandwidth levels. Since these communication groups have symmetric network architectures and contain fewer bandwidth levels, a set communication algorithm suitable for symmetric network architectures can be used to reduce internal communication logic and improve bandwidth utilization, thereby improving the overall set communication efficiency of the system.

[0009] In one possible implementation of the first aspect, the multiple collective communication machines are divided into one or more first communication groups based on the bandwidth level corresponding to the bandwidth between every two collective communication machines. This includes: generating a tree-shaped network topology for the multiple collective communication machines based on the bandwidth level corresponding to the bandwidth between every two collective communication machines, wherein leaf nodes in the tree-shaped network topology indicate the multiple collective communication machines, and non-leaf nodes indicate the convergence bandwidth level of the multiple collective communication machines, with non-leaf nodes closer to the root node indicating a smaller bandwidth level; dividing the tree-shaped network topology from the root node towards the leaf nodes to obtain multiple first network sets, wherein each first network set includes a portion of the structure in the tree-shaped network topology; and, if the network architecture corresponding to each of the multiple first network sets is a symmetrical network architecture, then the multiple first network sets are used as first communication groups.

[0010] In the above scheme, a tree-like network topology corresponding to a certain network architecture is constructed. In this tree-like topology, the bandwidth level decreases from the leaf to the root. The aggregated communication machines are grouped according to their bandwidth level. For example, aggregated communication machines with high bandwidth levels are grouped in the leaf subtrees of the tree-like topology, while aggregated communication machines with low bandwidth levels are grouped in levels closer to the root node. This ensures that when multiple aggregated communication machines are subsequently divided into multiple communication groups, the bandwidth differences between aggregated communication machines within the same communication group are small, thereby reducing bandwidth bottlenecks in data transmission and improving the overall aggregated communication efficiency of the system.

[0011] In one possible implementation of the first aspect, when the network architecture corresponding to the second network set is an asymmetric network architecture, the tree network topology corresponding to the second network set is divided into multiple third network sets, with the non-leaf node closest to the root node in the second network set as the dividing point. The multiple first network sets include the second network set, and the third network sets include a portion of the tree network topology. When the network architecture corresponding to each of the multiple third network sets is a symmetric network architecture, the multiple third network sets are used as the first communication group.

[0012] By implementing the above scheme, the asymmetry in the network architecture is eliminated through recursive partitioning.

[0013] In one possible implementation of the first aspect, when the network architecture corresponding to the fourth network set is an asymmetric network architecture, the multiple collective communication machines contained in the fourth network set are divided into multiple group blocks to make the network architecture corresponding to the fourth network set a symmetric network architecture. In this case, the multiple first network sets or multiple third network sets include the fourth network set, and the group blocks include all or part of the multiple collective communication machines contained in the fourth network set; the fourth network set is regarded as the first communication group.

[0014] Implementing the above scheme, for network architectures where asymmetry cannot be eliminated through recursive partitioning (the fourth network set mentioned above), the network architecture constructed from these multiple group communication machines can be divided into multiple packet blocks, thus achieving symmetry. It should be understood that the above scheme (hereinafter referred to as the packet block optimization scheme) and the recursive scheme can be used in parallel; that is, either the packet block optimization scheme or the recursive scheme can be used alone, or the recursive scheme can be used first, followed by the packet block optimization scheme.

[0015] For example, suppose there are 6 aggregate communication machines distributed across two regions (region A and region B) in an asymmetric network. Region A contains 4 aggregate communication machines, and region B contains 2 aggregate communication machines. As described above, the network architecture formed by regions A and B is asymmetric. By further dividing the aggregate communication machines in region A into two groups, group 1 and group 2 each contain the two aggregate communication machines in region A, the network architecture formed by group 1, group 2, and region B becomes symmetric.

[0016] In one possible implementation of the first aspect, the multiple collective communication machines contained in the fourth network set are divided into multiple group blocks, including: when the network architecture corresponding to the fourth network set is an asymmetric network architecture, determining multiple partitioning schemes corresponding to the fourth network set by enumeration, wherein the partitioning scheme indicates the specific partitioning method for the multiple collective communication machines contained in the fourth network set, and the partitioning scheme includes how many group blocks to divide into; calculating the communication cost corresponding to the multiple collective communication machines under the multiple partitioning schemes, wherein the communication cost includes the communication time required for the multiple collective communication machines to perform collective communication; and using a first partitioning scheme to divide the multiple collective communication machines contained in the fourth network set into multiple group blocks, wherein the communication cost corresponding to the first partitioning scheme is less than or equal to the communication cost corresponding to each of the multiple partitioning schemes, and the multiple partitioning schemes include the first partitioning scheme.

[0017] By implementing the above scheme, the partitioning schemes are enumerated and the communication cost (including the overall communication time) corresponding to each partitioning scheme is calculated. The appropriate partitioning scheme is selected based on the magnitude of the communication cost, avoiding the result of suboptimal overall performance due to local optima (when the bandwidth values ​​differ greatly, there may be local symmetry but overall asymmetry, resulting in low overall system aggregation communication efficiency). This maximizes the utilization of high-bandwidth links and improves the aggregation communication efficiency of the entire system.

[0018] For example, suppose there are 9 aggregated communication machines distributed across two regions (region A and region B) in an asymmetric network. Region A contains 6 aggregated communication machines with identical bandwidth; region B contains 3 aggregated communication machines with identical bandwidth. As described above, the network architecture formed by regions A and B is asymmetric. Through recursive partitioning, two communication groups are ultimately obtained: communication group 1 contains 6 aggregated communication machines, and communication group 2 contains 3 aggregated communication machines. Since the bandwidth within communication group 1 and communication group 2 is identical, their internal network architectures are symmetric. Therefore, there are 4 possible partitioning schemes for communication group 1. Scheme 1 involves dividing it into one block, which means treating the 6 aggregated communication machines as a single group. The communication group consists of 6 communication machines. Partition scheme 2 divides the group into 2 blocks, with each block containing 3 communication machines. Partition scheme 3 divides the group into 3 blocks, with each block containing 2 communication machines. Partition scheme 4 divides the group into 6 blocks, with each block containing 1 communication machine. Since communication group 1 is inherently a symmetrical network architecture, all partitioning schemes 1, 2, 3, and 4 result in symmetrical network architectures. However, for partitioning scheme 2, when the collective communication machines in region A are divided into 2 groups (each group includes 3 collective communication machines) for collective communication, from the perspective of the system as a whole, communication group 1 and communication group 2 also have network architecture symmetry. Therefore, under normal circumstances, after using partitioning scheme 2 to divide the collective communication machines in region A, the overall collective communication efficiency of the entire system will be higher.

[0019] In one possible implementation of the first aspect, the communication cost of multiple collective communication machines under multiple partitioning schemes is calculated, including: based on the amount of data participating in the collective computation, the structure of each of the multiple first communication groups, and the bandwidth level of each of the multiple first communication groups, the communication cost of the multiple first communication groups under the second partitioning scheme is calculated.

[0020] By implementing the above scheme, the communication cost of multiple communication groups under different partitioning schemes is calculated based on the amount of data participating in the ensemble computation, the structure of each communication group, and the bandwidth level of each communication group. This makes the evaluation of communication cost closer to the real communication scenario and avoids decision bias caused by a single indicator.

[0021] For example, the total data volume is x. After iterative partitioning, it includes two communication groups with a bandwidth of D1 (divided into a block). Communication group 1 includes M collective communication machines, and the bandwidth between these M collective communication machines is D2 (divided into a block). Communication group 2 includes N collective communication machines, and the bandwidth between these N collective communication machines is D3. Based on the above information, the communication costs for collective communication in communication group 1 and communication group 2 can be calculated as follows: the communication time between communication groups is x ÷ D1; the communication time within communication group 1 is x ÷ 2 ÷ M ÷ D2 × (M-1); and the communication time within communication group 2 is x ÷ 2 ÷ N ÷ D3 × (N-1).

[0022] This explains that since communication group 1 and communication group 2 can be executed in parallel, the larger of the two communication times is used when calculating the total communication time of the system.

[0023] In one possible implementation of the first aspect, the multiple partitioning schemes include a prime number partitioning scheme, which indicates that all or some of the multiple collective communication machines included in the fourth network set are used as transmission machines, and the transmission machines are used to transmit the data that they need to participate in collective communication to other collective communication machines in the same communication group.

[0024] In the above scheme, the core of the prime number partitioning scheme is to solve the indivisibility of prime numbers, thereby enabling the method provided in this application to be more flexibly applied in multi-level convergent asymmetric networks. For example, if a network set includes 7 collective communication machines, no matter how the multiple collective communication machines are partitioned, a symmetric network architecture cannot be obtained (including partitioning into one group of 4 and one group of 3, where the load between these two groups will be unbalanced). By designating one of the collective communication machines as the transmission machine, the remaining 6 collective communication machines can perform collective communication, and these 6 can be partitioned into a symmetric network architecture (including partitioning into two groups of 3 or three groups of 2). In this process, the transmission machine and the collective communication machines not only utilize high bandwidth resources but also avoid the load imbalance caused by prime number grouping.

[0025] In one possible implementation of the first aspect, the first information includes the bandwidth size between every two collective communication machines in the plurality of collective communication machines. After receiving the first information, the method further includes: calculating the bandwidth level corresponding to the bandwidth between every two collective communication machines in the plurality of collective communication machines based on the bandwidth size between every two collective communication machines in the plurality of collective communication machines.

[0026] By implementing the above scheme, the bandwidth level corresponding to the bandwidth between each pair of multiple aggregated communication machines is calculated based on the bandwidth size between each pair of aggregated communication machines. This makes the bandwidth level division more in line with the actual network conditions, reduces the deviation caused by manual intervention, and thus improves the scenario adaptability of the method in this application.

[0027] Secondly, this application provides a collective communication device, characterized in that the collective communication device is applied to a collective communication machine group, wherein the collective communication machine group includes multiple collective communication machines, the network architecture corresponding to the collective communication machine group is an asymmetric network architecture, the asymmetric network architecture includes asymmetry in network topology or bandwidth, the collective communication machine group includes three or more bandwidth levels, the bandwidth levels are used to indicate the bandwidth size within a certain range, the device includes: a receiving module, used to receive first information, the first information is used to indicate the structure of the collective communication machine group, the first information includes the bandwidth level corresponding to the bandwidth between every two collective communication machines in the multiple collective communication machines; a processing module, used to divide the multiple collective communication machines into one or more first communication groups based on the bandwidth level corresponding to the bandwidth between every two collective communication machines in the multiple collective communication machines, wherein the first communication group includes all or some of the collective communication machines in the multiple collective communication machines; the network architecture corresponding to the first communication group is a symmetric network architecture; the symmetric network architecture includes symmetry in network topology or bandwidth; and a collective communication module, used to complete collective communication between the multiple collective communication machines based on one or more first communication groups.

[0028] In one possible implementation of the second aspect, based on the bandwidth level corresponding to the bandwidth between every two collective communication machines in the multiple collective communication machines, the processing module is specifically used to: generate a tree-shaped network topology structure corresponding to the multiple collective communication machines based on the bandwidth level corresponding to the bandwidth between every two collective communication machines in the multiple collective communication machines. In the tree-shaped network topology structure, leaf nodes are used to indicate the multiple collective communication machines, and non-leaf nodes are used to indicate the convergence bandwidth level of the multiple collective communication machines. The bandwidth level indicated by a non-leaf node closer to the root node is smaller. Starting from the root node, the tree-shaped network topology structure is divided in the direction from the root node to the leaf nodes to obtain multiple first network sets. The first network sets include a part of the structure in the tree-shaped network topology structure. When the network architecture corresponding to each of the multiple first network sets is a symmetrical network architecture, the multiple first network sets are regarded as the first communication group.

[0029] In one possible implementation of the second aspect, the processing module is further configured to: when the network architecture corresponding to the second network set is an asymmetric network architecture, divide the tree network topology of the part corresponding to the second network set by taking the non-leaf node closest to the root node in the second network set as the dividing point to obtain multiple third network sets, wherein the multiple first network sets include the second network set, and the third network sets include a part of the tree network topology; when the network architecture corresponding to each of the multiple third network sets is a symmetric network architecture, the multiple third network sets are used as a first communication group.

[0030] In one possible implementation of the second aspect, the processing module is further configured to: when the network architecture corresponding to the fourth network set is an asymmetric network architecture, divide the multiple collective communication machines contained in the fourth network set into multiple group blocks, so that the network architecture corresponding to the fourth network set is a symmetric network architecture, wherein multiple first network sets or multiple third network sets include the fourth network set, and the group blocks include all or part of the collective communication machines in the multiple collective communication machines contained in the fourth network set; and regard the fourth network set as the first communication group.

[0031] In one possible implementation of the second aspect, the processing module is specifically used to: when the network architecture corresponding to the fourth network set is an asymmetric network architecture, determine multiple partitioning schemes corresponding to the fourth network set by enumeration. The partitioning scheme is used to indicate the specific partitioning method of the multiple collective communication machines contained in the fourth network set, and the partitioning scheme includes how many group blocks to divide it into; calculate the communication cost corresponding to the multiple collective communication machines under the multiple partitioning schemes, the communication cost includes the communication time required for the multiple collective communication machines to perform collective communication; and use the first partitioning scheme to divide the multiple collective communication machines contained in the fourth network set into multiple group blocks, wherein the communication cost corresponding to the first partitioning scheme is less than or equal to the communication cost corresponding to each of the multiple partitioning schemes, and the multiple partitioning schemes include the first partitioning scheme.

[0032] In one possible implementation of the second aspect, the processing module is specifically used to: calculate the communication cost of the multiple first communication groups under the second partitioning scheme based on the amount of data participating in the set calculation, the structure of each of the multiple first communication groups, and the bandwidth level corresponding to each of the multiple first communication groups.

[0033] In one possible implementation of the second aspect, the multiple partitioning schemes include a prime number partitioning scheme, which is used to indicate that all or some of the multiple collective communication machines included in the fourth network set are used as transmission machines, and the transmission machines are used to transmit the data that they need to participate in collective communication to other collective communication machines in the same communication group.

[0034] In one possible implementation of the second aspect, the receiving module is further configured to: calculate the bandwidth level corresponding to the bandwidth between each pair of multiple aggregated communication machines based on the bandwidth size between each pair of aggregated communication machines.

[0035] Thirdly, this application provides a computing device including a processor and a memory, the processor being configured to execute instructions stored in the memory to cause the computing device to perform a collection communication method as described in the first aspect and any possible implementation thereof.

[0036] Fourthly, this application provides a computing device cluster, which includes at least one computing device. Each computing device includes a processor and a memory. The processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster implements the collective communication method as described in the first aspect and any possible implementation of the first aspect.

[0037] Fifthly, this application provides a computer program product containing instructions that, when executed by a computing device, cause the computing device to perform a collection communication method as described in the first aspect and any possible implementation thereof.

[0038] In a sixth aspect, this application provides a computer-readable storage medium including computer program instructions, which, when executed by a computing device, enable the computing device to perform a collection communication method as described in the first aspect and any possible implementation thereof.

[0039] The second, third, fourth, fifth and sixth aspects mentioned above all have various possible designs similar to the first aspect and any possible implementation of the first aspect, and can produce corresponding technical effects, which will not be elaborated here.

[0040] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0041] Figure 1 This is a schematic diagram of a collective communication system architecture provided in this application;

[0042] Figure 2 This is an example diagram of a collective communication system deployed on a cloud platform, as provided in this application;

[0043] Figure 3 This is a schematic diagram of a collection communication method provided in this application;

[0044] Figure 4 This is a network architecture diagram corresponding to multiple collective communication machines provided in this application;

[0045] Figure 5 This is a schematic diagram of a tree network topology provided in this application;

[0046] Figure 6 This is a schematic diagram of the structure of a first network set provided in this application;

[0047] Figure 7 This is a schematic diagram of the structure of a fourth network set provided in this application;

[0048] Figure 8 This is a schematic diagram of the architecture of a collective communication device provided in this application;

[0049] Figure 9 This is a schematic diagram of the structure of a computing device provided in this application;

[0050] Figure 10 This is a schematic diagram of the structure of a computing device cluster provided in this application;

[0051] Figure 11 This is a schematic diagram of another structure of a computing device cluster provided in this application. Detailed Implementation

[0052] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0053] To better introduce the solutions involved in this application, the terms and technologies involved in this application will be introduced first.

[0054] Collective communication (CCM) refers to a communication mode in a distributed system where multiple CCM machines (also known as computing units) collaborate to complete data processing. Its core lies in the interaction and aggregation of data among these machines. CCM is widely used in scenarios such as distributed training. For example, CCM can be used for gradient synchronization of model parameters. When multiple computing units process tasks according to the model, each slave computing unit calculates its own gradient data and then sends it to a master computing unit. Finally, the master computing unit updates the model parameters within these slave computing units based on this gradient data.

[0055] Network architecture: Network architecture refers to the structure composed of multiple aggregated communication machines at the network layer. Network architectures include symmetric and asymmetric network architectures. A symmetric network architecture refers to a network architecture with a symmetric network topology or symmetric bandwidth, while an asymmetric network architecture refers to a network architecture with an asymmetric network topology or asymmetric bandwidth. For example, suppose a network architecture contains region A and region B, and there is a communication connection between them. If the bandwidth levels within regions A and B are different, then the network architecture composed of regions A and B can be described as an asymmetric network architecture. Alternatively, if region A includes 4 aggregated communication machines and region B includes 2 aggregated communication machines, then the network architecture composed of regions A and B can also be described as an asymmetric network architecture. A network architecture composed of regions A and B is symmetric only if the bandwidth levels within regions A and B are the same, and the number of aggregated communication machines in regions A and B is also the same.

[0056] Bandwidth tiers: Bandwidth tiers indicate the total bandwidth within a certain range. For example, bandwidth tiers can include high bandwidth, medium bandwidth, and low bandwidth. High bandwidth tiers indicate all bandwidth between 200Gbps and 400Gbps; medium bandwidth tiers indicate all bandwidth between 50Gbps and 150Gbps; and low bandwidth tiers indicate all bandwidth between 10Gbps and 30Gbps. This tiered classification helps to categorize the links in the system.

[0057] Current aggregated communication methods primarily target network architectures of aggregated communication machine groups (comprising multiple aggregated communication machines), selecting appropriate aggregated communication algorithms to implement aggregated communication. For example, for multiple aggregated communication machines converging at the same bandwidth level and located close to each other at the network layer, the RingAllReduce algorithm can be used to facilitate aggregated communication between these machines. Specifically, this involves: dividing the data into N equal parts (N representing the number of aggregated communication machines), distributing these N parts to the N machines for processing, and then performing N-1 rounds of interaction and N-1 rounds of aggregation to ultimately complete the aggregated communication for the entire system.

[0058] In current ensemble communication methods, when applied to asymmetric network architectures, the asymmetry of the network architecture may lead to some ensemble communication machines being overloaded while others are underloaded.

[0059] For example, suppose there are two regions with aggregated communication machines: region A has 4 aggregated communication machines, and region B has 2 aggregated communication machines. The bandwidth between any two aggregated communication machines within the same region is the same. When the entire system needs to process 100 Gbits of data, region A and region B will each receive 50 Gbits of data. Since region A has 4 aggregated communication machines, each machine in region A processes 50 ÷ 4 = 12.5 Gbits of data. Since region B has 2 aggregated communication machines, each machine in region B processes 50 ÷ 2 = 25 Gbits of data. It can be seen that the amount of data processed by the aggregated communication machines in region A is different from that in region B.

[0060] In current collective communication methods, when applied to network architectures with multiple bandwidth levels, the asymmetry of bandwidth may limit the collective communication efficiency of the entire system to the lowest bandwidth, thus preventing the reuse of high bandwidth resources.

[0061] For example, suppose there are two regions with interconnected communication machines. Region A includes four interconnected communication machines with a bandwidth of 40Gbps, and Region B includes two interconnected communication machines with a bandwidth of 20Gbps. When the entire system needs to process 100Gbit of data, Region A and Region B will each receive 50Gbit of data. Since Region A includes four interconnected communication machines with a bandwidth of 40Gbps between each pair of interconnected communication machines, the time required for Region A to process this 50Gbit of data can be expressed as 50÷4×(4-1)÷40=0.9375 seconds. Since Region B includes two interconnected communication machines with a bandwidth of 20Gbps between each pair of interconnected communication machines, the time required for Region B to process this 50Gbit of data can be expressed as 50÷2×(2-1)÷20=1.25 seconds. It can be seen that although Region A and Region B can process simultaneously, Region A needs to wait for Region B to finish processing before proceeding with subsequent steps.

[0062] To address the problems of current ensemble communication methods when dealing with asymmetric networks and network architectures with multiple bandwidth levels, this application provides an ensemble communication method and related products. The ensemble communication method includes the following steps:

[0063] ① Receive the network architecture corresponding to multiple aggregated communication machines and the bandwidth hierarchy between multiple aggregated communication machines.

[0064] ② Based on the bandwidth hierarchy of multiple aggregated communication machines, the network architecture of these multiple aggregated communication machines is divided into multiple communication groups with symmetrical network architecture.

[0065] ③ Based on these communication groups, complete the collective communication between multiple collective communication machines.

[0066] By implementing the above method, when handling aggregated communication in complex network architectures (including asymmetric network architectures and architectures with more than three bandwidth levels), the network architecture is divided into one or more communication groups. The goal of this transformation is to make these communication groups possess a symmetric network architecture. Since the communication groups have a symmetric network architecture, aggregated communication algorithms suitable for symmetric network architectures can be used to implement aggregated communication. On the one hand, because the characteristics of aggregated communication machines and links are less different under a symmetric network architecture, the communication algorithm can use unified and simple rules for data transmission and interaction, reducing the complex transmission and path planning issues caused by differences in nodes and links under asymmetric network architectures. On the other hand, under a symmetric architecture, data can be rationally allocated to each link, preventing some links from becoming congested while others remain idle.

[0067] The system architecture involved in the collective communication method provided in this application will be introduced below. (See also...) Figure 1 , Figure 1 This is a schematic diagram of a collective communication system architecture provided in this application, such as... Figure 1 As shown, the aggregated communication system architecture includes a client 100, an aggregated communication device 110, and an aggregated communication machine group 120, wherein the aggregated communication machine group 120 includes multiple aggregated communication machines (e.g., ...). Figure 1 The communication devices are: 1, 2, ..., n; 100 has a communication connection with the communication device 110, and the communication device 110 has a communication connection with the communication device group 120.

[0068] The communication connection between the client 100 and the aggregated communication device 110, and the communication connection between the aggregated communication device 110 and the aggregated communication machine group 120, can be either a wired or wireless connection. Wired connections include Ethernet, wired lines, cables, etc.; wireless connections include wireless local area networks (Wi-Fi), cellular networks, etc. Furthermore, it may also include the Internet, local area networks (LANs), etc., which simultaneously support wired and wireless connections. Of course, other connection types may also be included, but this application does not specifically limit them.

[0069] Client 100 can be deployed on computing devices. These devices can use virtualization technology to create virtual environments such as virtual machines and containers to manage and utilize computing resources. A virtual machine is a virtualization technology implemented at the computer software level. Its working principle is to create an abstraction layer on top of the physical computer hardware, dividing and managing physical hardware resources, including memory and hard disks, so that each virtual machine can independently use its corresponding hardware resources, meaning each virtual machine can independently run its operating system and applications. A container is a lightweight, portable software encapsulation technology. Its working principle is to encapsulate an application and all its dependencies (including databases, frameworks, and files) together to form an independent unit. This allows applications within a container to run in the same way in different environments, unaffected by external environments. Compared to virtual machines, containers do not need to simulate complete hardware and operating systems; instead, they directly utilize the operating system kernel of the device they reside on. Therefore, containers also have advantages over virtual machines, such as faster startup speeds and more flexible deployment.

[0070] The communication unit 110 can also be deployed on computing devices, including bare metal servers (BMS). BMS refers to general-purpose physical servers, including servers based on Advanced RISC Machines (ARM) architecture and x86 servers, which are used to deliver complete server resources directly to users, allowing users to operate and manage them completely autonomously. Compared with virtual machines, BMS does not need to simulate multiple virtual computing environments through software, so there is no performance loss after virtualization. Compared with containers, BMS does not need to encapsulate and isolate individual applications, so there are no encapsulation limitations of containers.

[0071] Client 100 can also be deployed on terminal devices, including computer terminal devices, mobile terminal devices, network terminal devices, and virtual terminal devices. Among them, computer terminal devices include personal computers, laptops, and tablets; mobile terminal devices include smartphones, smartwatches, and smart bracelets; network terminal devices include routers, switches, and modems; and virtual terminal devices include virtual reality (VR) devices.

[0072] The aggregated communication device 110 and aggregated communication machine group 120 can also be deployed in a computing device cluster, which includes multiple computing devices as described above.

[0073] This explains that the same computing devices used in the aforementioned aggregated communication device 110 and aggregated communication machine group 120 can also be deployed on different computing devices, for example, in... Figure 1 In the system architecture shown, the aggregated communication device 110 and the aggregated communication machine group 120 are deployed in different computing devices. Whether the aggregated communication device 110 and the aggregated communication machine group 120 are deployed in the same computing device or terminal device depends on the specific application environment, and this application does not make specific limitations here.

[0074] Optionally, the code or source code used to implement the functions of each module in the client 100 and the aggregated communication device 110 can be deployed in a storage device or storage array. The storage device is a hardware device used to store and retrieve digital data, specifically a hard disk drive (HDD), a solid state drive (SSD), a universal serial bus flash drive, and a memory card, etc. The storage array refers to a storage system formed by combining multiple storage devices in a specific way, specifically a redundant array of independent disks (RAID), a storage area network (SAN), and network attached storage (NAS), etc. This application does not specifically limit the specific form of the storage device and storage array, or the device on which the storage device and storage array are located.

[0075] The client 100 and the aggregated communication device 110 include a chip system composed of one or more chips to achieve the corresponding functions. Specifically, the client 100 and the aggregated communication device 110 can be devices implemented using a CPU, an application-specific integrated circuit (ASIC), or a programmable logic device (PLD). The PLD can be a complex programmable logical device (CPLD), an FPGA, or a generic array logic (GAL). While data processing units (DPUs), NPUs, GPUs, system-on-chips (SoCs), offloading cards, and accelerator cards are not traditionally considered PLDs, they can also be used to construct these devices, and this application does not specifically limit their use.

[0076] The preceding text has described in detail the possible deployment methods of client 100, aggregated communication device 110, and aggregated communication machine group 120. In actual deployment, any of the above methods can be flexibly deployed according to specific application scenarios and business requirements. The following section provides an exemplary illustration of the actual deployment methods of client 100, aggregated communication device 110, and aggregated communication machine group 120 in conjunction with specific application scenarios.

[0077] As an example of an application scenario where a collective communication system is deployed on a cloud platform, in this scenario, the collective communication device 110 can be deployed on the cloud platform, see [link to relevant documentation]. Figure 2 , Figure 2 This is an example diagram of a collective communication system deployed on a cloud platform, as provided in this application. Figure 2 As shown, a user can initiate a cloud service purchase request to the cloud platform through client 100. The cloud service includes using some of the aggregated communication machines (some of the aggregated communication machines in aggregated communication machine group 120) in the cloud platform to calculate data. After client 100 sends the purchase request to the cloud platform, the cloud platform can grant the user access rights to the cloud service to client 100. The user sends data to the cloud platform through client 100, and the data is processed and calculated through aggregated communication device 110 and aggregated communication machines in aggregated communication machine group 120.

[0078] The cloud platform also maintains various basic resources, including computing resources, storage resources, network resources, and security resources, to meet the computing needs of the aggregated communication device 110 and the aggregated communication machine group 120 under different scales and loads. Furthermore, these computing resources can be dynamically scaled according to the usage needs of the aggregated communication device 110 and the aggregated communication machine group 120, thereby ensuring the overall stable operation and providing reliable services to users.

[0079] Optionally, the aggregated communication device 110 and the aggregated communication machine group 120 can be services provided by the same cloud platform. Figure 2 The cloud platform shown includes Figure 1 The aggregated communication device 110 and aggregated communication machine group 120 are shown. Alternatively, the aggregated communication device 110 and aggregated communication machine group 120 can be services provided by different cloud platforms. In this case, a hybrid cloud architecture can be used to achieve data communication between different cloud services. That is, the aggregated communication device 110 is deployed on a first cloud platform, and the aggregated communication machine group 120 is deployed on a second cloud platform. The first cloud platform and the second cloud platform achieve data communication between the aggregated communication device 110 and the aggregated communication machine group 120 through their respective cloud platforms.

[0080] It should be understood that the above application scenarios are for illustrative purposes only. The client 100, the aggregated communication device 110, and the aggregated communication machine group 120 can be flexibly deployed according to actual business needs. They will not be listed one by one here.

[0081] The above describes the system architecture that may be involved in this application. The following is a brief explanation of how this system architecture implements the collection communication method provided in this application.

[0082] In the proposed aggregate communication method, when a user needs to process some data (e.g., classify the data using a specific model), the user can send the data to be calculated and the network architecture information corresponding to the aggregate communication machine group 120 to the aggregate communication device 110 through the client 100. The aggregate communication device 110 formulates corresponding grouping instructions and selects a suitable aggregate communication algorithm based on the network architecture information of the aggregate communication machine group 120. The aggregate communication device 110 divides the multiple aggregate communication machines in the aggregate communication machine group 120 into multiple communication groups according to the grouping instructions, and the aggregate communication algorithm and these multiple communication groups process the computational data input by the user.

[0083] It should be understood here that the aforementioned users cannot fail to specify which collective communication machines are involved in data processing, i.e., the aforementioned... Figure 1 In this process, the aggregated communication device 110 may also obtain the network architecture information of the aggregated communication machine group 120 directly from the aggregated communication machine group 120 instead of obtaining the network architecture information of the multiple aggregated communication machines from the client 100.

[0084] The network architecture information of the aforementioned group of communication machines 120 includes the number of communication machines, the type of communication machines, and the bandwidth hierarchy between each pair of communication machines. For a description of the bandwidth hierarchy, please refer to the beginning of the embodiments of this application, which will not be elaborated here.

[0085] The aforementioned grouping instruction refers to the "logical-level partitioning plan" generated by the aggregated communication device 110 based on the network architecture information of the aggregated communication machine group 120. For example, if the network architecture information of the aggregated communication machine group 120 includes three bandwidth levels (10-20Gbps, 50-100Gbps, and 200-400Gbps), then the grouping instruction can divide multiple aggregated communication machines in the aggregated communication machine group 120 into these three bandwidth levels, and ultimately divide these multiple aggregated communication machines into three communication groups.

[0086] The aforementioned suitable collective communication algorithm is selected by the collective communication device 110 after logically grouping multiple collective communication machines in the collective communication machine group 120, based on the characteristics of each communication group (including the number of collective communication machines in the communication group and the bandwidth level within the communication group). For example, for a communication group containing a high bandwidth level, the hierarchical AllReduce algorithm can be used to further optimize the collaborative efficiency within the communication group, thereby optimizing the efficiency of the entire collective communication machine group 120 in processing data.

[0087] The following is based on Figure 2 The system architecture shown, combined with Figure 3To introduce the collection communication method provided in this application, see [link to relevant documentation]. Figure 3 , Figure 3 This is a schematic diagram of a collection communication method provided in this application, such as... Figure 3 As shown, the steps of this collection communication method include:

[0088] S301: The communication device 110 receives the first information sent by the client 100.

[0089] The aforementioned first information includes the processing data sent by the user and all or part of the network architecture in the aggregated communication machine group 120. The processing data sent by the user refers to the raw data that the client 100 needs to complete or calculate through aggregated communication. For example, when training a model for animal recognition, the raw data may be multiple pictures of animals such as cats and dogs. Multiple computing units (aggregated communication machines) process a portion of the raw data through distributed computing and then update the model parameters synchronously through aggregated communication. The network architecture refers to the topology structure composed of multiple nodes (aggregated communication machines), including which nodes can communicate directly and the bandwidth levels between nodes. For an introduction to the bandwidth levels, please refer to the beginning of the embodiments, which will not be elaborated further in this application.

[0090] For the sake of clarity in the subsequent description of the embodiments of this application, this application will use the following... Figure 4 The network architecture shown is used as an example to illustrate the collection communication method provided in this application.

[0091] Please see Figure 4 , Figure 4 This application provides a network architecture diagram corresponding to multiple collective communication machines. Figure 4 The system comprises 12 nodes, each representing a set of communication machines, where:

[0092] All nodes marked "1" can communicate with each other, and the communication bandwidth is "Highbandwidth".

[0093] All nodes marked "2" can communicate with each other, and the communication bandwidth is "High bandwidth".

[0094] All nodes marked "3" can communicate with each other, and the communication bandwidth is "High bandwidth".

[0095] All nodes marked "4" can communicate with each other, and the communication bandwidth is "High bandwidth".

[0096] Each node marked "1" can communicate with each node marked "2", and the communication bandwidth is "Midbandwidth".

[0097] Each node labeled "3" can communicate with each node labeled "4", and the communication bandwidth is "Mid bandwidth".

[0098] Each node identified as "1", each node identified as "2", each node identified as "3", and each node identified as "4" can communicate, and the communication bandwidth is "Low bandwidth".

[0099] The terms "High bandwidth", "Mid bandwidth", and "Low bandwidth" refer to bandwidth levels, with "High bandwidth" indicating a high bandwidth level, "Mid bandwidth" indicating a medium bandwidth level, and "Low bandwidth" indicating a low bandwidth level.

[0100] according to Figure 4 It can be observed that the network architecture formed by these 12 communication machines is symmetrical in bandwidth, but asymmetrical in network topology. For example, the bandwidth hierarchy on the left is a 4:2 structure, while the bandwidth hierarchy on the right is a 3:3 structure.

[0101] This explains that, Figure 4 This is merely an example drawn in this application to illustrate the network architecture corresponding to multiple collective communication machines. The actual form of the network architecture can be adjusted according to the actual application scenario. For example, each node identified as "1", "2", "3" and "4" only represents a type of collective communication machine, and the identifier can also be replaced with the ID number corresponding to each collective communication machine.

[0102] The specific form of all or part of the network architecture in the aforementioned group of communication machines 120 can be a structural image constructed using methods such as ranktable or structure. Here, ranktable is a node information table used to record the code, IP address, port, and connection relationships between each communication machine and other machines; structure is a data structure that packages the network architecture into structured data by defining fields (such as "communication machine ID" and "bandwidth level"). The network architecture in the group of communication machines 120 is as follows: Figure 4In the case of the network architecture shown, the first information mentioned above may include "the bandwidth level between the collective communication machine 1 and the collective communication machine 2 is high bandwidth", "the bandwidth level between the collective communication machine 1 and the collective communication machine 3 is mid bandwidth", and "the bandwidth level between the collective communication machine 1 and the collective communication machine 2 is low bandwidth", etc.

[0103] Optionally, in all or part of the network architecture of the aforementioned group of communication machines 120, the bandwidth between the communication machines may not be a bandwidth level. For example, the bandwidth between the communication machines may be a specific bandwidth size, including "50Gbps", "12.5Gbps", and "135Gbps", etc. After receiving the above information, the communication device 110 converts the bandwidth size into the corresponding bandwidth level according to its internal bandwidth level classification rules. For example, the bandwidth level classification rules include bandwidth sizes of 10-20Gbps belonging to the low bandwidth level, bandwidth sizes of 40-60Gbps belonging to the medium bandwidth level, and bandwidth sizes of 100-200Gbps belonging to the high bandwidth level. When the communication device 110 receives a bandwidth size of 135Gbps between communication machine 1 and communication machine 2, the communication device 110 converts it into a high bandwidth level between communication machine 1 and communication machine 2.

[0104] It should be understood that the aggregated communication device 110 can calculate the bandwidth level corresponding to the bandwidth between each pair of aggregated communication machines based on the bandwidth size between each pair of aggregated communication machines. This makes the bandwidth level division more in line with the actual network conditions, reduces the deviation caused by manual intervention, and thus improves the scenario adaptability of the method of this application.

[0105] Optionally, all or part of the network architecture in the aforementioned collective communication machine group 120 can be obtained directly from the collective communication device 110. That is, the aforementioned client 100 can send only the processing data to the collective communication device 110. The collective communication device 110 determines the number of collective communication machines that need to participate in the calculation based on the data type and data volume of the processing data. The collective communication device 110 obtains the network architecture information corresponding to the multiple collective communication machines that match the number from the collective communication machine group 120.

[0106] S302: The aggregated communication device 110 divides multiple aggregated communication machines into one or more first communication groups based on the first information.

[0107] The first communication group mentioned above is a part of the network architecture composed of multiple collective communication machines. That is, the first communication group includes all or part of the collective communication machines in the collective communication machine group 120. The network architecture corresponding to the first communication group is a symmetrical network architecture. For a description of the symmetrical network architecture, please refer to the introductory section at the beginning of the embodiment. This application will not elaborate further here.

[0108] The aforementioned aggregated communication device 110 divides multiple aggregated communication machines into one or more first communication groups based on first information. Specifically, the aggregated communication device 110 generates a tree-shaped network topology corresponding to multiple aggregated communication machines based on the bandwidth level corresponding to the bandwidth between every two aggregated communication machines. Starting from the root node of the tree-shaped network topology, the aggregated communication device 110 divides the tree-shaped network topology from the root node to the leaf node to obtain multiple first network sets. When the network architecture corresponding to each of the multiple first network sets is a symmetrical network architecture, the aggregated communication device 110 uses the multiple first network sets as first communication groups.

[0109] In the above tree network topology, leaf nodes are used to indicate multiple sets of communication machines, and non-leaf nodes are used to indicate the bandwidth level at which multiple sets of communication machines converge. The closer a non-leaf node is to the root node, the smaller the bandwidth level it indicates. The above first network set includes a portion of the structure in the tree network topology.

[0110] The following is combined Figure 4 This section describes the specific implementation process of the aggregated communication device 110 dividing multiple aggregated communication machines into one or more first communication groups based on first information.

[0111] Combination Figure 4 It can be seen that, among the 12 collective communication machines, the bandwidth between all nodes marked "1" is "High bandwidth"; the bandwidth between all nodes marked "2" is "High bandwidth"; the bandwidth between all nodes marked "3" is "High bandwidth"; the bandwidth between all nodes marked "4" is "High bandwidth"; the communication bandwidth between each node marked "1" and each node marked "2" is "Mid bandwidth"; the communication bandwidth between each node marked "3" and each node marked "4" is "Mid bandwidth"; and the bandwidth between each node marked "1" and each node marked "2" and each node marked "3" and each node marked "4" is "Low bandwidth".

[0112] Therefore, the aggregated communication device 110 can construct a bandwidth hierarchy based on the bandwidth between each pair of aggregated communication machines among these 12 aggregated communication machines, such as... Figure 5 The tree network topology shown is shown.

[0113] See Figure 5 , Figure 5 This is a schematic diagram of a tree network topology provided in this application, such as... Figure 5 As shown, the tree-like network architecture includes 19 nodes, of which 12 are leaf nodes. Each leaf node represents a collection of communication machines, and each non-leaf node represents a bandwidth level. The bandwidth level represented by the root node is "Lowbandwidth". The bandwidth level represented by all non-leaf nodes in the layer closest to the root node is "Midbandwidth". The bandwidth level represented by all non-leaf nodes in the last layer is "Highbandwidth".

[0114] Since the bandwidth between all nodes labeled "1", "2", "3", and "4" is "High bandwidth", the leaf nodes corresponding to these 12 aggregate communication machines are distributed as child nodes of the last-level non-leaf nodes. Furthermore, it can be observed that when child nodes in different regions communicate, they traverse to higher-level non-leaf nodes. For example, when a node labeled "1" needs to communicate with a node labeled "2", all nodes labeled "1" along with their connected leaf nodes are treated as a new node, and all nodes labeled "2" along with their connected leaf nodes are treated as a new node. Figure 5 It can be observed that the bandwidth between these two new nodes is the "Mid bandwidth," which is... Figure 4 The network architecture shown corresponds to this.

[0115] according to Figure 5 It can also be observed that the network architecture formed by these 12 nodes is symmetrical in terms of bandwidth level, but asymmetrical in terms of network topology.

[0116] The communication device 110 is in the process of... Figure 4 The network architecture shown is transformed into Figure 5 Following the tree network topology shown, due to Figure 5 The tree network topology shown is asymmetric, so the aggregation communication device 110 needs to start from the root node of the tree network topology and try to divide the tree network topology into multiple first network sets from top to bottom.

[0117] See Figure 6 , Figure 6This is a schematic diagram of the structure of a first network set provided in this application, such as... Figure 6 As shown, the aggregated communication device 110 attempts to connect from top to bottom, starting from the root node. Figure 5 After the tree-shaped network topology shown is divided into multiple first network sets, there are a total of two first network sets. The first network set on the right has a symmetrical network architecture, which can be seen from the bandwidth level (two high bandwidth levels connected to one medium bandwidth level) and the network topology (the number of communication machines under the two high bandwidth levels is 3). The first network set on the left is symmetrical in terms of bandwidth level (two high bandwidth levels connected to one medium bandwidth level), but it is asymmetrical in terms of network topology (the number of communication machines under the two high bandwidth levels is 4 and 2 respectively). Therefore, the aggregation communication device 110 will only use the first network set on the right as the first communication group.

[0118] Optional, via Figure 6 It can be observed that after the aggregator communication device 110 attempts to divide the tree network topology into multiple first network sets starting from the root node and proceeding downwards, the multiple first network sets obtained are not necessarily symmetrical.

[0119] When the network architecture corresponding to the fourth network set is an asymmetric network architecture, the aggregate communication device 110 can divide the multiple aggregate communication machines contained in the fourth network set into multiple group blocks so that the network architecture corresponding to the fourth network set is a symmetric network architecture. At this time, the aggregate communication device 110 uses the processed fourth network set as the first communication group.

[0120] It should be understood that for network architectures where asymmetry cannot be eliminated through recursive partitioning (the fourth network set mentioned above), the network architecture constructed by the multiple group blocks can be made symmetric by dividing the internal set of communication machines into multiple group blocks.

[0121] The aforementioned fourth network set is a subset of the plurality of first network sets whose network architecture is asymmetric. The packet block includes all or some of the multiple set communication machines contained in the fourth network set.

[0122] See Figure 7 , Figure 7 This is a schematic diagram of the structure of a fourth network set provided in this application, such as... Figure 7 As shown, this fourth network set is... Figure 6In the first network set on the left, when the aggregation communication device 110 determines that a certain first network set is an asymmetric network architecture, the aggregation communication device 110 groups the aggregation communication machines (4 aggregation communication machines identified as "1" and 2 aggregation communication machines identified as "2") contained in the network set and attempts to build a symmetric network architecture.

[0123] from Figure 7 It can be observed that by dividing the four collective communication machines marked "1" into two groups, each group containing two collective communication machines marked "1", the internal bandwidth hierarchy of the fourth network set is symmetrical, and its network topology is also symmetrical (the group can be considered as a whole; since the bandwidth level corresponding to the bandwidth size between these two group blocks is higher than the previous level, these two group blocks can be understood as child nodes under two high-bandwidth level nodes, and these two high-bandwidth level nodes communicate through a medium-bandwidth level node. At this time, the corresponding network topology of the fourth network set is that the number of collective communication machines under the three high-bandwidth levels are 2, 2, and 2 respectively, which is symmetrical).

[0124] Optionally, since there may be multiple schemes for the above-mentioned aggregated communication device 110 to divide the multiple aggregated communication machines contained in the fourth network set into multiple group blocks, the aggregated communication device 110 can select the partitioning scheme with the lowest communication cost. The specific steps include: when the network architecture corresponding to the fourth network set is an asymmetric network architecture, the aggregated communication device 110 determines the multiple partitioning schemes corresponding to the fourth network set by enumeration; the aggregated communication device 110 calculates the communication cost corresponding to the multiple aggregated communication machines under the multiple partitioning schemes, the communication cost including the communication time required for the multiple aggregated communication machines to perform aggregated communication; the aggregated communication device 110 uses the first partitioning scheme to divide the multiple aggregated communication machines contained in the fourth network set into multiple group blocks, wherein the communication cost corresponding to the first partitioning scheme is less than or equal to the communication cost corresponding to each of the multiple partitioning schemes.

[0125] The aforementioned aggregated communication device 110 determines multiple partitioning schemes corresponding to the fourth network set through enumeration. Specifically, it determines the partitioning scheme based on the number of aggregated communication machines included in the fourth network set. By enumerating the partitioning schemes and calculating the communication cost (including the overall communication time) corresponding to each scheme, a suitable partitioning scheme is selected based on the magnitude of the communication cost. This avoids the situation where local optima lead to overall suboptimal results (when bandwidth values ​​differ significantly, local symmetry may occur, but overall asymmetry may result, leading to low overall aggregated communication efficiency of the system). This maximizes the utilization of high-bandwidth links and improves the overall aggregated communication efficiency of the system.

[0126] For example, in Figure 6 In the middle, the first network set on the left includes 6 group communication machines. Among them, 4 group communication machines marked "1" are the same non-leaf child node. Then, the group communication device 110 has three partitioning schemes for these 4 group communication machines marked "1": (1,1,1,1), (2,2) and (4).

[0127] The aforementioned aggregated communication device 110 calculates the communication costs corresponding to multiple aggregated communication machines under multiple partitioning schemes. Specifically, the aggregated communication device 110 inputs the fourth network set under different partitioning schemes into the mathematical model of communication costs to obtain the cost corresponding to that partitioning scheme. The following uses... Figure 7 The network architecture shown illustrates the process of calculating communication costs using this communication cost data model.

[0128] Assume that the amount of data required for communication by each set of communication machines is x. Since in Figure 7In the network architecture shown, the first layer consists of a high-bandwidth node, and the second layer consists of two medium-bandwidth nodes. Therefore, in the first layer, data is divided into two parts, and the communication time at this layer can be expressed as x÷2÷bw_min_1, where "x÷2" indicates that the amount of data being communicated is divided into two parts, and "bw_min_1" represents the minimum bandwidth at this bandwidth level (since the bandwidth level only represents a certain range of bandwidth, the bandwidth between various communication machines can be used as the specific value for calculation; it should also be understood that the minimum bandwidth can be the average value of the range represented by the bandwidth level). The second layer consists of three medium-bandwidth nodes (due to the packet block structure, the high-bandwidth node on the communication machine marked "1" can be divided into two). Therefore, in this layer, the amount of data being communicated is divided into three parts, and the communication time at this layer can be expressed as x÷2÷3÷bw_min_2, where "x÷2÷3" indicates that the data is further divided into three parts, and "bw_min_2" represents the minimum bandwidth at this bandwidth level. Therefore, the communication cost (communication time) corresponding to this partitioning scheme can be expressed as x÷2÷bw_min_1+x÷2÷3÷bw_min_2.

[0129] In one possible embodiment, the above-mentioned multiple partitioning schemes include a prime number partitioning scheme, which is used to indicate that all or some of the multiple collective communication machines included in the fourth network set are used as transmission machines, and the transmission machines are used to transmit the data that they need to participate in collective communication to other collective communication machines in the same communication group.

[0130] In the above scheme, the core of the prime number partitioning scheme is to solve the indivisibility of prime numbers, thereby enabling the method provided in this application to be more flexibly applied in multi-level convergent asymmetric networks. For example, if a network set includes 7 collective communication machines, no matter how the multiple collective communication machines are partitioned, a symmetric network architecture cannot be obtained (including partitioning into one group of 4 and one group of 3, where the load between these two groups will be unbalanced). By designating one of the collective communication machines as the transmission machine, the remaining 6 collective communication machines can perform collective communication, and these 6 can be partitioned into a symmetric network architecture (including partitioning into two groups of 3 or three groups of 2). In this process, the transmission machine and the collective communication machines not only utilize high bandwidth resources but also avoid the load imbalance caused by prime number grouping.

[0131] In one possible embodiment, when the network architecture corresponding to the second network set is an asymmetric network architecture, the aggregation communication device 110 divides the tree-like network topology of the second network set into multiple third network sets, using the non-leaf node closest to the root node in the second network set as the dividing point. If the network architecture corresponding to each of the multiple third network sets is a symmetric network architecture, the aggregation communication device 110 uses the multiple third network sets as a first communication group. It should be understood that asymmetry in the network architecture can also be eliminated through recursive partitioning.

[0132] The second network set mentioned above is a subset of the multiple first network sets whose network architecture is asymmetric, and the third network set is a subset of the second network set.

[0133] For example, for such Figure 6 The first network set shown on the left has an asymmetric network architecture. At this time, the aggregation communication device 110 takes the non-leaf node closest to the root node of the tree network topology in the first network set as the starting point and continues to divide it from top to bottom to obtain two third network sets. One third network set consists of 4 aggregation communication machines marked "1" and the other third network set consists of 2 aggregation communication machines marked "2". As can be seen from the definition of a symmetric network architecture, the network architectures corresponding to the above two third network sets are both symmetric network architectures.

[0134] This indicates that the above block-based scheme and recursive scheme can be used in parallel. That is, you can use either the block-based optimization scheme or the recursive scheme, or you can use the recursive scheme first and then use the block-based optimization scheme.

[0135] S303: The collective communication device 110, based on the collective communication algorithm and one or more first communication groups, completes collective communication between multiple collective communication machines.

[0136] The aforementioned set communication algorithm is a core technology in distributed computing used for collaborative data processing by multiple nodes (set communication machines). Its core concepts include data exchange and set-up, thereby achieving information synchronization between multiple nodes. Common set communication algorithms include the RingAllReduce algorithm and the hierarchical AllReduce algorithm.

[0137] The following is combined Figure 5 , Figure 6 as well as Figure 7 This will illustrate the change in overall system communication efficiency after using the aggregated communication method provided in this application. Figure 5In this network architecture, assuming that the amount of data required for communication by each set of communication machines in a single communication session is X, the bandwidth corresponding to the low bandwidth level can be represented as V1, the bandwidth corresponding to the medium bandwidth level can be represented as V2, and the bandwidth corresponding to the high bandwidth level can be represented as V3, and V3 > V2 > V1.

[0138] When using the RingAllReduce algorithm to process, such as Figure 5 When considering the network architecture shown, its time cost can be expressed as the following formula:

[0139]

[0140] In Formula 1 above, T1 represents the communication time required by the current ensemble communication method, and "X÷12" represents the amount of data allocated to each ensemble communication machine in a single transmission. Dividing this amount of data by the bandwidth at this bandwidth level yields the time required for multiple ensemble communication machines in a single transmission. 2×(12-1) is because the ensemble communication algorithm involves two data exchange phases (Scatter-Reduce and AllGather), each requiring 12-1 communication steps. Therefore, the total number of communication rounds required is 2×(12-1).

[0141] Using the collection communication method provided in this application Figure 5 After processing the network architecture shown, two communication groups are obtained, one of which has a network architecture similar to... Figure 6 The network architecture of the other communication group is similar to the first network set on the right. Figure 7 It is similar to the fourth network set in the text.

[0142] When using the RingAllReduce algorithm to process the network architecture shown for these two communication groups, the time cost can be expressed as the following formula:

[0143]

[0144] In Formula 2 above, T2 represents the communication time required by the aggregated communication method provided in this application, and "X÷2" represents the amount of data allocated to each aggregated communication machine in a single transmission. Since multiple group blocks are designed here, including group blocks with 3 aggregated communication machines and group blocks with 2 aggregated communication machines, these two communication groups can process data simultaneously. Therefore, when calculating the processing time, it is only necessary to select the one in this communication group that takes the most time. "X÷6=X÷2÷3" indicates that the data of this layer needs to be processed by three group blocks after it is left to the next layer. Therefore, the amount of data falling on each group block can be expressed as "X÷6". Finally, the data in the third layer needs to be processed by 2 communication groups. Therefore, the data falling between each communication group can be expressed as "X÷12=X÷2÷3÷2". The communication time of each layer is determined by the lowest bandwidth under that bandwidth level. In Formula 1 and Formula 2 above, V1, V2 and V3 can also be expressed as the lowest bandwidth size under that bandwidth level.

[0145] Since V3 > V2 > V1, we can set V3 = 3 × V1 and V2 = 2 × V1. The final result expressed by the above formula is T2 = 5 × X ÷ 6 ÷ V1. Since T1 = 11 × X ÷ 6 ÷ V1, we can find that T2 is less than T1, which proves that the ensemble communication method provided in this application can be better applied to ensemble communication systems with multi-level convergence and asymmetric network architecture compared with the prior art.

[0146] In summary, this application provides a collective communication method, which includes: a collective communication device 110 acquiring the network architecture corresponding to multiple collective communication machines in a collective communication machine group 120 and the bandwidth levels between the multiple collective communication machines; the collective communication device 110 dividing the multiple collective communication machines into multiple communication groups with symmetrical structures according to the network architecture corresponding to the multiple collective communication machines and the bandwidth levels between the multiple collective communication machines; and the collective communication device 110 completing collective communication between the multiple collective communication machines based on these communication groups. By implementing the above-described collective communication method, complex network architectures (asymmetric network architectures and network architectures with three or more bandwidth levels) are divided into multiple communication groups with symmetrical network architectures, efficient symmetrical network collective communication algorithms can be implemented within the communication groups. This not only simplifies the communication logic but also fully utilizes the bandwidth resources at different levels, thereby improving the overall collective communication efficiency.

[0147] For the sake of simplicity, the above method embodiments are described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to this application. Other reasonable combinations of steps that those skilled in the art can conceive of based on the above description also fall within the scope of protection of this application. Again, those skilled in the art should be familiar with the fact that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to this application.

[0148] See Figure 8 , Figure 8 This is a schematic diagram of the architecture of a collective communication device provided in this application, such as... Figure 8 As shown, the aggregated communication device 800 is used to receive first information sent by the aggregated communication machine group, and to divide multiple aggregated communication machines (aggregated communication machine 1, aggregated communication machine 2, ..., aggregated communication machine n) in the aggregated communication machine group according to the first information.

[0149] The integrated communication device 800 can be further divided into multiple unit modules, such as... Figure 8 As shown, the aggregated communication device 800 also includes a receiving module 801, a processing module 802, and an aggregated communication module 803. It should be understood that... Figure 8 The number and names of the unit modules included in the aggregate communication device 800 are merely examples provided in this application. The aggregate communication device 800 may include more or fewer unit modules, and the names of the unit modules are not limited to [specific examples]. Figure 8 The name of the unit module in, for example, Figure 8 The receiving module 801 and the collection communication module 803 can be combined into a single communication module. It should be understood that the above examples are for illustrative purposes only and should not be considered as specific limitations.

[0150] The receiving module 801, processing module 802, and collective communication module 803 can all be implemented in software or in hardware. For example, the implementation of the receiving module 801 will be described below. Similarly, the implementation of the processing module 802 and the collective communication module 803 can refer to the implementation of the receiving module 801.

[0151] As an example of a software functional unit, the receiving module 801 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the aforementioned computing instance may be one or more. For example, the receiving module 801 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code can be distributed in the same region or in different regions. Further, the multiple hosts, virtual machines, or containers used to run the code can be distributed in the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Similarly, the multiple hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Typically, a VPC is set up within one region. Communication between two VPCs within the same region, and between VPCs in different regions, requires a communication gateway within each VPC to achieve interconnection between VPCs.

[0152] As an example of a hardware functional unit, the receiving module 801 may include at least one computing device, such as a server. Alternatively, the receiving module 801 may be implemented using a CPU, an ASIC, or a PLD, where the PLD can be a CPLD, FPGA, GAL, DPU, NPU, SoC, offload card, accelerator card, or any combination thereof. It should be noted that the multiple computing devices included in the receiving module 801 can be distributed in the same Availability Zone (AZ) or different AZs. Similarly, the multiple computing devices included in the receiving module 801 can be distributed in the same Virtual Private Cloud (VPC) or multiple VPCs, where the multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, GALs, DPUs, NPUs, SoCs, offload cards, and inference cards.

[0153] The functions of the receiving module 801, processing module 802, and collective communication module 803 in the above-mentioned collective communication device 800 are described below.

[0154] The receiving module 801 is used to receive first information, which indicates the structure of the group of collective communication machines. The first information includes the bandwidth level corresponding to the bandwidth between every two collective communication machines in the group of collective communication machines.

[0155] The processing module 802 is used to divide the multiple collective communication machines into one or more first communication groups based on the bandwidth level corresponding to the bandwidth between every two collective communication machines in the multiple collective communication machines. The first communication group includes all or some of the collective communication machines in the multiple collective communication machines. The network architecture corresponding to the first communication group is a symmetrical network architecture. The symmetrical network architecture includes symmetry in network topology or bandwidth.

[0156] The collective communication module 803 is used to complete collective communication between multiple collective communication machines based on one or more first communication groups.

[0157] In one possible implementation, based on the bandwidth level corresponding to the bandwidth between every two collective communication machines in the multiple collective communication machines, the processing module 802 is specifically used to: generate a tree-shaped network topology structure corresponding to the multiple collective communication machines based on the bandwidth level corresponding to the bandwidth between every two collective communication machines in the multiple collective communication machines. In the tree-shaped network topology structure, leaf nodes are used to indicate the multiple collective communication machines, and non-leaf nodes are used to indicate the convergence bandwidth level of the multiple collective communication machines. The bandwidth level indicated by a non-leaf node closer to the root node is smaller. Starting from the root node, the tree-shaped network topology structure is divided in the direction from the root node to the leaf nodes to obtain multiple first network sets. The first network sets include a part of the structure in the tree-shaped network topology structure. When the network architecture corresponding to each of the multiple first network sets is a symmetrical network architecture, the multiple first network sets are regarded as the first communication group.

[0158] In one possible implementation, the processing module 802 is further configured to: when the network architecture corresponding to the second network set is an asymmetric network architecture, divide the tree network topology of the second network set into multiple third network sets by using the non-leaf node closest to the root node in the second network set as the dividing point, wherein the multiple first network sets include the second network set, and the third network sets include a portion of the tree network topology; and when the network architecture corresponding to each of the multiple third network sets is a symmetric network architecture, use the multiple third network sets as a first communication group.

[0159] In one possible implementation, the processing module 802 is further configured to: when the network architecture corresponding to the fourth network set is an asymmetric network architecture, divide the multiple collective communication machines contained in the fourth network set into multiple group blocks to make the network architecture corresponding to the fourth network set a symmetric network architecture, wherein multiple first network sets or multiple third network sets include the fourth network set, and the group blocks include all or part of the multiple collective communication machines contained in the fourth network set; and use the fourth network set as the first communication group.

[0160] In one possible implementation, the processing module 802 is specifically used to: when the network architecture corresponding to the fourth network set is an asymmetric network architecture, determine multiple partitioning schemes corresponding to the fourth network set by enumeration. The partitioning scheme is used to indicate the specific partitioning method of the multiple collective communication machines contained in the fourth network set, and the partitioning scheme includes how many group blocks to divide it into; calculate the communication cost corresponding to the multiple collective communication machines under the multiple partitioning schemes, the communication cost includes the communication time required for the multiple collective communication machines to perform collective communication; and use the first partitioning scheme to divide the multiple collective communication machines contained in the fourth network set into multiple group blocks, wherein the communication cost corresponding to the first partitioning scheme is less than or equal to the communication cost corresponding to each of the multiple partitioning schemes, and the multiple partitioning schemes include the first partitioning scheme.

[0161] In one possible implementation, the processing module 802 is specifically used to: calculate the communication cost of the multiple first communication groups under the second partitioning scheme based on the amount of data participating in the set calculation, the structure of each of the multiple first communication groups, and the bandwidth level corresponding to each of the multiple first communication groups.

[0162] In one possible implementation, the partitioning schemes include a prime number partitioning scheme, which indicates that all or some of the collective communication machines in the fourth network set are used as transmission machines, and the transmission machines are used to transmit the data that they need to participate in collective communication to other collective communication machines in the same communication group.

[0163] In one possible implementation, the receiving module 801 is further configured to: calculate the bandwidth level corresponding to the bandwidth between each pair of multiple collective communication machines based on the bandwidth size between each pair of multiple collective communication machines.

[0164] In summary, this application provides a collective communication device. The collective communication device 800 obtains the network architecture corresponding to multiple collective communication machines in the collective communication machine group 120 and the bandwidth levels between these machines through a receiving module 801. A processing module 802 divides these multiple collective communication machines into multiple communication groups with symmetrical structures based on the network architecture and bandwidth levels. Finally, a collective communication module 803 completes collective communication between the multiple collective communication machines based on these communication groups. By using the aforementioned collective communication device 800, complex network architectures (asymmetric network architectures and network architectures with three or more bandwidth levels) are divided into multiple communication groups with symmetrical network architectures. This allows for the implementation of efficient symmetrical network collective communication algorithms within the communication groups, simplifying communication logic and fully utilizing bandwidth resources at different levels, thereby improving overall collective communication efficiency.

[0165] The above text combines Figures 3 to 11 The collection communication method provided in this application is described in detail below, and will be combined with... Figures 9 to 11 This application describes the computing device and computing device cluster provided in accordance with this application.

[0166] Figure 9 This is a schematic diagram of the structure of a computing device provided in this application. The computing device 900 may be the collective communication device 800 mentioned above.

[0167] Furthermore, the computing device 900 includes a bus 901, a processor 902, a memory 903, and a communication interface 904. The processor 902, the memory 903, and the communication interface 904 communicate with each other via the bus 901. The computing device 900 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 900.

[0168] The 901 bus can be a Peripheral Component Interconnect Express (PCIe) bus, an Extended Industry Standard Architecture (EISA) bus, a Unified Bus (Ubus or UB), a Compute Express Link (CXL) bus, a Cache Coherent Interconnect for Accelerators (CCIX) bus, etc. The Unified Bus is also known as the Lingqu bus. Buses can be divided into address buses, data buses, control buses, etc. For ease of representation, Figure 9The symbol is represented by only one line, but this does not mean that there is only one bus or one type of bus.

[0169] Processor 902 may include any one or more computing devices such as central processing unit (CPU), graphics processing unit (GPU), microprocessor (MP) or digital signal processor (DSP), ASIC, FPGA, CPLD, NPU, SoC, offload card, accelerator card, etc.

[0170] The memory 903 stores program code, which is executed by the processor 902 to perform the above-mentioned tasks. Figure 3 The embodiment describes the processing steps of the collection communication method. The program code includes one or more software units, which are... Figure 10 Specifically, the receiving module 801, processing module 802, and collection communication module 803 in the embodiment are described. The receiving module 801 can be used to execute... Figure 3 Step S301 in the embodiment; the processing module 802 can be used to execute Figure 3 Step S302 in the embodiment; and the collection communication module 803 can be used to execute Figure 3 Step S303 in the embodiment will not be repeated here.

[0171] Memory 903 may include volatile memory, such as random access memory (RAM). Processor 902 may also include non-volatile memory, such as read-only memory (ROM), flash memory, HDD, or SSD. Furthermore, memory 903 may also be implemented using storage class memory (SCM), phase change memory (PCM), or other types of storage media.

[0172] It is worth noting that the same type of storage medium can be configured in the same computing device to realize the function of memory 903, or two or more types of storage media can be configured to realize the function of memory 903. This application does not limit this.

[0173] The communication interface 904 is a wired interface (e.g., an Ethernet interface), an internal interface (e.g., a Peripheral Component Interconnect express (PCIe) bus interface), a wired interface (e.g., an Ethernet interface), or a wireless interface (e.g., a cellular network interface or a wireless LAN interface) for communicating with other servers or units.

[0174] As one possible implementation, the computing device 900 may include multiple types of processors 902, meaning the computing device 900 is a heterogeneous device. For example, the computing device 900 may include a CPU and a GPU, and at least one of the processors 902 can execute the operation steps corresponding to the set communication method. For the sake of brevity, further details will not be elaborated here.

[0175] It needs to be explained that, Figure 9 This is merely one possible implementation of an embodiment of this application. In practical applications, the computing device 900 may include more or fewer components, and this is not a limitation. For content not shown or described in the embodiments of this application, please refer to the foregoing. Figure 3 The relevant descriptions in the embodiments will not be repeated here.

[0176] Figure 10 This is a schematic diagram of a computing device cluster provided in this application, which includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0177] like Figure 10 As shown, the computing device cluster includes at least one computing device 1000. The memory 1003 of one or more computing devices 1000 in the computing device cluster may store the same instructions for executing the aggregate communication method.

[0178] In some possible implementations, the memory 1003 of one or more computing devices 1000 in the computing device cluster may also store partial instructions for executing the collective communication method. In other words, a combination of one or more computing devices 1000 can jointly execute instructions for executing the collective communication method.

[0179] It should be noted that the memory 1003 in different computing devices 1000 within the computing device cluster can store different instructions, each used to execute a portion of the resource migration system's functions. That is, the instructions stored in the memory 1003 of different computing devices 1000 can implement the receiving module 801, the processing module 802, and the aggregated communication module 803. Specifically, the receiving module 801 can be used to execute... Figure 3 Step S301 in the embodiment; the processing module 802 can be used to execute Figure 3 Step S302 in the embodiment; and the collection communication module 803 can be used to execute Figure 3 Step S303 in the embodiment will not be repeated here.

[0180] The computing device 1000 includes a bus 1001, a processor 1002, a memory 1003, and a communication interface 1004. Further descriptions of the bus 1001, processor 1002, memory 1003, and communication interface 1004 can be found in [reference needed]. Figure 9 The descriptions of bus 901, processor 902, memory 903 and communication interface 904 in the embodiments will not be repeated here.

[0181] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc., as described below. Figure 11 One possible implementation method is shown.

[0182] Figure 11 This is another schematic diagram of a computing device cluster provided in this application, such as... Figure 11 As shown, computing devices 1000A and 1000B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this possible implementation, the memory 1003 in computing device 1000A stores instructions for implementing the receiving module 801 and the aggregation communication module 803. Meanwhile, the memory 1003 in computing device 1000B stores instructions for implementing the processing module 802.

[0183] It should be understood that Figure 11 The functions of computing device 1000A shown can also be performed by multiple computing devices 1000. Similarly, the functions of computing device 1000B can also be performed by multiple computing devices 1000.

[0184] It needs to be explained that, Figure 11 The implementation shown may be implemented when the processing power of the computing device 1000A is insufficient, or when the storage space of the computing device 1000A is insufficient, or in other business scenarios. This application does not make any specific limitations.

[0185] This application also provides another type of computing device cluster. The interconnection relationships between the computing devices in this computing device cluster can be similarly referenced. Figure 10 and Figure 11 The connection method of the computing device cluster. The difference is that the memory 1003 of one or more computing devices 1000 in the computing device cluster can store the same instructions for executing the collective communication method.

[0186] In some possible implementations, the memory 1003 of one or more computing devices 1000 in the computing device cluster may also store partial instructions for executing the collective communication method. In other words, a combination of one or more computing devices 1000 can jointly execute instructions for executing the collective communication method.

[0187] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product runs on at least one computing device, it causes the at least one computing device to perform a collection communication method.

[0188] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium capable of being stored by a computing device, or a data storage device such as a data center containing one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., high-density digital video discs (DVDs)), or semiconductor media (e.g., solid-state drives). The computer-readable storage medium includes instructions that instruct the computing device to perform a collective communication method.

[0189] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. A computer program product includes a plurality of computer instructions. When the computer program instructions are loaded or executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0190] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of this application.

Claims

1. A collective communication method, characterized in that, The method is applied to a group of aggregated communication machines, wherein the group of aggregated communication machines comprises multiple aggregated communication machines, the network architecture corresponding to the group of aggregated communication machines is an asymmetric network architecture, the asymmetric network architecture includes asymmetry in network topology or bandwidth, the group of aggregated communication machines includes three or more bandwidth levels, the bandwidth levels are used to indicate all bandwidth sizes within a certain range, and the method includes: Receive first information, the first information being used to indicate the structure of the aggregated communication machine group, the first information including the bandwidth level and / or bandwidth size corresponding to the bandwidth between every two aggregated communication machines in the plurality of aggregated communication machines, the bandwidth size being used to calculate the bandwidth level; Based on the bandwidth level corresponding to the bandwidth between every two collective communication machines, the collective communication machines are divided into one or more first communication groups, wherein the first communication group includes all or some of the collective communication machines; the network architecture corresponding to the first communication group is a symmetrical network architecture; the symmetrical network architecture includes symmetry in network topology or bandwidth. Based on the one or more first communication groups and the collective communication algorithm, collective communication between the multiple collective communication machines is completed.

2. The method according to claim 1, characterized in that, The step of dividing the plurality of aggregated communication machines into one or more first communication groups based on the bandwidth hierarchy corresponding to the bandwidth between every two aggregated communication machines includes: Based on the bandwidth level corresponding to the bandwidth between every two collective communication machines in the plurality of collective communication machines, a tree network topology corresponding to the plurality of collective communication machines is generated. In the tree network topology, leaf nodes are used to indicate the plurality of collective communication machines, and non-leaf nodes are used to indicate the bandwidth level at which the plurality of collective communication machines converge. The closer a non-leaf node is to the root node, the smaller the bandwidth level it indicates. Starting from the root node, the tree network topology is divided in the direction from the root node to the leaf node to obtain multiple first network sets, each of which includes a portion of the tree network topology. When the network architecture corresponding to each of the plurality of first network sets is a symmetric network architecture, the plurality of first network sets are regarded as the first communication group.

3. The method according to claim 2, characterized in that, When the network architecture corresponding to the second network set is an asymmetric network architecture, the tree network topology corresponding to the second network set is divided into multiple third network sets by taking the non-leaf node closest to the root node in the second network set as the dividing point. The multiple first network sets include the second network set, and the third network sets include a part of the tree network topology. When the network architecture corresponding to each of the plurality of third network sets is a symmetric network architecture, the plurality of third network sets are regarded as the first communication group.

4. The method according to claim 3, characterized in that, When the network architecture corresponding to the fourth network set is an asymmetric network architecture, the multiple collective communication machines contained in the fourth network set are divided into multiple group blocks to make the network architecture corresponding to the fourth network set a symmetric network architecture. The multiple first network sets or the multiple third network sets include the fourth network set, and the group block includes all or part of the multiple collective communication machines contained in the fourth network set. The fourth network set is designated as the first communication group.

5. The method according to claim 4, characterized in that, The multiple communication machines included in the fourth network set are divided into multiple packet blocks, including: When the network architecture corresponding to the fourth network set is an asymmetric network architecture, multiple partitioning schemes corresponding to the fourth network set are determined by enumeration. The partitioning scheme is used to indicate the specific partitioning method of the multiple set communication machines contained in the fourth network set, and the partitioning scheme includes how many packet blocks to divide it into. Calculate the communication cost of the plurality of collective communication machines under the plurality of partitioning schemes, wherein the communication cost includes the communication time required for the plurality of collective communication machines to perform collective communication; The first partitioning scheme is used to divide the multiple set communication machines contained in the fourth network set into multiple packet blocks, wherein the communication cost corresponding to the first partitioning scheme is less than or equal to the communication cost corresponding to each of the multiple partitioning schemes, and the multiple partitioning schemes include the first partitioning scheme.

6. The method according to claim 5, characterized in that, Calculate the communication costs of the plurality of collective communication machines under the plurality of partitioning schemes, including: Based on the amount of data involved in the set computation, the structure of each of the multiple first communication groups, and the bandwidth level corresponding to each of the multiple first communication groups, the communication cost of the multiple first communication groups under the multiple partitioning schemes is calculated.

7. The method according to claim 5 or 6, characterized in that, The plurality of partitioning schemes includes a prime number partitioning scheme, which is used to indicate that all or some of the collective communication machines in the plurality of collective communication machines included in the fourth network set are used as transmission machines. The transmission machines are used to transmit the data that they need to participate in collective communication to other collective communication machines in the same communication group.

8. A computing device cluster, characterized in that, The computing device cluster includes at least one computing device, each of the at least one computing device including a processor and a memory, wherein the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the computing device cluster to implement the operational steps of the method as described in any one of claims 1 to 7.

9. A computer program product containing instructions, characterized in that, When the instruction is executed by the computing device cluster, the computing device cluster causes the computing device cluster to perform the operation steps of the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It includes computer program instructions, which, when executed by a cluster of computing devices, perform the operational steps of the method as described in any one of claims 1 to 7.

Images