The method implemented by the first computing node, the first computing node, and the readable medium

Abstract

In distributed training, to avoid network congestion, a first computing node can determine, according to a node-aware halving doubling algorithm, a rendezvous identifier for sending a data packet from a first process to a second process, the first process and the second process belonging to different nodes connected to different tile switches under a particular network topology. The first computing node can then send the data packet from the first process to the second process through a rendezvous switch corresponding to the rendezvous identifier.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more particularly to a method implemented by a first computing node, the first computing node, and a readable medium. Background Technology

[0002] With the rapid development of neural networks such as Deep Neural Networks (DNNs), various application areas (e.g., computer vision, natural language processing, speech recognition) have flourished and benefited from the inherent versatility and flexibility of neural networks. However, due to the increasing complexity and stringent accuracy requirements of neural network applications, the size of neural network models and the amount of training data required to train them have also increased significantly. This inevitably leads to longer training times, negatively impacting the effectiveness and timeliness of training models to meet the ever-changing application environments.

[0003] To reduce the time required to train neural network models, a distributed training system employing parallel training can be used. Generally, a distributed training system can include a large number of computing nodes or servers distributed across a network, allocating a subset of computational tasks to these nodes or servers for parallel training. However, data communication between computing nodes or servers in a distributed training system creates a lower bound or bottleneck for the potential reduction in training time. This is especially true when the distributed training system includes various types of heterogeneous connections or interconnections within and between computing nodes or servers, exhibiting different characteristics in terms of latency, bandwidth, and topology. This heterogeneity of connections or interconnections increases the difficulty and complexity of designing the data communication network for the computing nodes or servers in a distributed training system.

[0004] Furthermore, excessive data flow through specific network switches or connections between compute nodes or servers in a distributed training system can cause network congestion, potentially extending training time due to delays in processing training results. This excessive data flow through specific network switches or connections may be due to a loss of control over the path selection of routing data sent between compute nodes or servers. Summary of the Invention

[0005] This application provides a method implemented by a first computing node, the first computing node, and a readable medium to solve the problem of network congestion.

[0006] In a first aspect, a method implemented by a first computing node is provided, comprising: determining an aggregation identifier for sending data packets from a first process to a second process according to a node-aware halving and doubling algorithm, wherein the first process and the second process belong to different nodes connected to different slice switches under a specific network topology; and sending the data packets from the first process to the second process through an aggregation switch corresponding to the aggregation identifier.

[0007] Secondly, one or more machine-readable media are provided, which store machine-readable instructions that, when executed by a first computing node, cause the first computing node to perform an action, the action including: determining a convergence identifier for sending data packets from a first process to a second process according to a node-aware halving and doubling algorithm, the first process and the second process belonging to different nodes connected to different slice switches under a specific network topology; and sending the data packets from the first process to the second process through a convergence switch corresponding to the convergence identifier.

[0008] Thirdly, a first computing node is provided, comprising: one or more processing units; and a memory storing machine-executable instructions that, when executed by the one or more processing units, cause the one or more processing units to perform an action, the action comprising: determining a convergence identifier for sending data packets from a first process to a second process according to a node-aware halving and doubling algorithm, the first process and the second process belonging to different nodes connected to different slice switches under a specific network topology; and sending data packets from the first process to the second process through a convergence switch corresponding to the convergence identifier.

[0009] Fourthly, a determining module is provided for determining a convergence identifier for sending data packets from a first process to a second process based on a node-aware halving and doubling algorithm, wherein the first process and the second process belong to different nodes connected to different slice switches under a specific network topology; and a sending module is provided for sending the data packets from the first process to the second process through the convergence switch corresponding to the convergence identifier.

[0010] Fifthly, a computer program product is provided, comprising a computer program that, when run in a computer, is used to implement the method described in the first aspect or any possible implementation thereof. Attached Figure Description

[0011] Please refer to the accompanying drawings for a detailed description. In the drawings, the leftmost number of the reference numerals indicates the first drawing in which that reference numeral appears. The same reference numerals are used in different drawings to indicate similar or identical designations.

[0012] Figure 1 An example environment in which the distributed training system can be applied is shown.

[0013] Figure 2 A more detailed example computing node is shown.

[0014] Figure 3A A ring configuration is shown, in which a predetermined number of nodes are interconnected.

[0015] Figure 3B This illustrates a configuration that halve or doubles the preset number of nodes connected to each other.

[0016] Figure 4 A schematic diagram of an exemplary cluster communication library is shown.

[0017] Figure 5 An exemplary topology-aware multi-stage algorithm is shown.

[0018] Figure 6 An exemplary ring-based algorithm is shown for the intra-node reduction scattering phase of a computation node.

[0019] Figure 7 An exemplary halving and doubling algorithm is shown for the intra-node reduction scattering phase of a computation node.

[0020] Figure 8 An exemplary halving and doubling algorithm for the global reduction phase between nodes is shown.

[0021] Figure 9 An exemplary halving and doubling algorithm for the global reduction phase between nodes is shown in more detail.

[0022] Figure 10 An exemplary ring-based cluster communication algorithm is shown.

[0023] Figure 11 Example scenarios are shown where the intra-node reduction scattering phase, the inter-node global reduction phase, and the intra-node global aggregation phase are performed in parallel or overlapping manner.

[0024] Figure 12 An exemplary fat-tree network topology is shown.

[0025] Figure 13 An example scenario using the first congestion avoidance method is shown.

[0026] Figure 14 An example scenario using the second congestion avoidance method is shown.

[0027] Figure 15 An exemplary topology-aware multi-stage method is illustrated.

[0028] Figure 16A first exemplary network congestion avoidance method is shown.

[0029] Figure 17 A second exemplary network congestion avoidance method is shown.

[0030] Figure 18 An exemplary parallel approach based on a hybrid architecture is shown in distributed training. Detailed Implementation

[0031] Overview

[0032] As mentioned above, existing distributed training systems suffer from performance bottlenecks in scalability due to data communication between computing nodes. Furthermore, the diversity of network architectures (including Ethernet, InfiniBand, PCIe, NVLink, NVSwitch, QPI / UPI, etc.) and the high degree of variation in network characteristics (e.g., latency, bandwidth, and topology) often prevent distributed training systems from effectively utilizing these heterogeneous connections or interconnects to perform cluster data operations within and between computing nodes, as well as data transmission between computing nodes. Moreover, network congestion can occur due to uncontrolled routing of data between computing nodes, leading to excessive data flow through specific network switches or connections between computing nodes in the distributed training system, and resulting in extended training time caused by latency in processing training results. Additionally, existing distributed training systems fail to differentiate between different underlying architectures of algorithms used for cluster operations, thus resulting in poor performance.

[0033] This disclosure describes an exemplary distributed training system. In implementation, the exemplary distributed training system may employ a structure-aware cluster communication library that enables the distributed training system to scale linearly. In implementation, the cluster communication library may be tailored to the communication algorithm, at least in part, based on analysis of the underlying structure and supporting network architecture, to achieve the desired or maximum efficiency. In implementation, the distributed training system may divide basic operations into multiple sub-operations, each using a type of structure.

[0034] In implementation, the exemplary distributed training system can implement hybrid algorithms, which allow multiple algorithms to coexist in a single cluster operation and selectively employ algorithms for specific architectures to improve or maximize the efficiency of the entire communication path. In implementation, the distributed training system can employ a dual-process parallel algorithm that starts two concurrent processes and pipelines the use of intra-node and inter-node connections, thereby improving communication efficiency by overlapping intra-node and inter-node communication.

[0035] In implementation, an exemplary distributed training system can employ a probe-based routing control mechanism that generates a mapping from connection to path. This mechanism distributes or spreads connections to different aggregation or intermediate switches in the communication network by reordering participants or processes in the cluster operation and mapping data streams on the distributed training system to specific physical links, thereby avoiding network congestion.

[0036] This application describes several different embodiments and implementations. The following sections describe exemplary frameworks suitable for practicing various implementations. Next, this application describes exemplary systems, devices, and processes for implementing a distributed training system.

[0037] Exemplary Environment

[0038] Figure 1 An exemplary environment 100 for implementing a distributed training system is shown. This environment 100 may include a distributed training system 102. In this example, the distributed training system 102 may include multiple computing nodes or servers 104-1, 104-2, ..., 104-K (hereinafter collectively referred to as computing nodes 104), where K is a positive integer greater than 1. In implementation, the multiple computing nodes 104 may communicate data with each other via a communication network 106.

[0039] The computing node 104 can be implemented as any variety of computing devices with computing / processing and communication capabilities, including but not limited to servers, desktop computers, laptops or portable computers, handheld devices, netbooks, internet devices, tablet computers, mobile devices (such as mobile phones, personal digital assistants, smartphones, etc.), or combinations thereof.

[0040] Communication network 106 can be a wireless or wired network, or a combination thereof. Network 106 can be a collection of independent networks interconnected and used as a single large network (e.g., the Internet or an intranet). Examples of such independent networks include, but are not limited to, telephone networks, cable networks, local area networks (LANs), wide area networks (WANs), and metropolitan area networks (MANs). Furthermore, independent networks can be wireless or wired networks, or a combination thereof. Wired networks can include electrical carrier connections (e.g., communication cables) and / or optical carriers or connections (e.g., fiber optic connections). Wireless networks can include, for example, WiFi networks, other radio frequency networks (e.g., Bluetooth). Zigbee, etc. In implementation, the communication network 106 may include multiple inter-node interconnects or switches 108-1, 108-2, ..., 108-L (hereinafter collectively referred to as inter-node switches 108) for providing connections between computing nodes 104, where L is a positive integer greater than 1.

[0041] In implementation, environment 100 may further include client device 110. The user can instruct distributed training system 102 to perform training on a specific learning model (e.g., a deep neural network model) based on data sent from client device 110 to distributed training system 102.

[0042] Exemplary computing node

[0043] Figure 2 A more detailed view of compute node 104 is shown. In implementations, compute node 104 may include, but is not limited to, one or more processing units 202, input / output (I / O) interfaces 204 and / or one or more network interfaces 206, and memory 208. In implementations, compute node 104 may further include one or more intra-node interconnects or switches 210.

[0044] In implementation, processing unit 202 may be configured to execute instructions stored in memory 208 and / or received from input / output interface 204 and / or network interface 206. In implementation, processing unit 202 may be implemented as one or more hardware processors, including, for example, microprocessors, dedicated instruction set processors, physical processing units (PPUs), central processing units (CPUs), graphics processing units, digital signal processors, tensor processing units, etc. Additionally or alternatively, the functions described herein may be performed at least in part by one or more hardware logic components. Examples of hardware logic components that may be used, such as, but not limited to, include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip systems (SOCs), complex programmable logic devices (CPLDs), etc.

[0045] Memory 208 may include machine-readable media in the form of volatile memory, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory 208 is an example of a machine-readable medium.

[0046] Machine-readable media can be volatile or non-volatile, removable or non-removable, and can be used to store information using any method or technology. This information can include machine-readable instructions, data structures, program modules, or other data. Examples of machine-readable media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random-access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), fast flash memory or other internal storage technologies, compact disk read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible to computing nodes. As defined herein, machine-readable media do not include any transient media, such as modulated data signals and carrier waves.

[0047] In implementation, network interface 206 can be configured to connect computing node 104 to other computing nodes via communication network 106. In implementation, network interface 206 can be established via a Network Interface Controller (NIC), which can use hardware and software to connect computing node 104 to communication network 106. In implementation, each type of NIC can use different types of fabrication or connectors to connect to the physical medium associated with communication network 106. Examples of various types of fabrication or connectors can be found in the IEEE 802 specification, and may include, for example, Ethernet (defined in 802.3), Token Ring (defined in 802.5), Wireless Network (defined in 802.11), InfiniBand, etc.

[0048] In implementation, the node switch 210 may include various types of interconnects or switches, including but not limited to high-speed serial computer expansion buses (such as PCIe), serial multi-channel short-range communication links (such as Nolan, which is a wired communication protocol-based serial multi-channel short-range communication link), switch chips with multiple ports (such as NVSwitch), point-to-point processor interconnects (such as Intel QPI / UPI), etc.

[0049] Although only hardware components are described in this example of compute node 104, in other instances, compute node 104 may further include other hardware components and / or other software components, such as program module 212 that executes instructions stored in memory 208 to perform various operations, and program data 214 for storing received data for training, intermediate and final results computed during training, etc.

[0050] Exemplary Cluster Communication Algorithm

[0051] Figure 3A and 3B An exemplary cluster communication algorithm that can be used in the distributed training system 102 is shown. In implementation, the cluster communication algorithm may include, but is not limited to, ring-based communication algorithms, halving and doubling communication algorithms, etc.

[0052] Figure 3AA ring configuration is illustrated, which interconnects a predetermined number of nodes (e.g., N nodes, where N is a positive integer greater than 1) with multiple connections (i.e., N connections), and divides data (e.g., packets or messages) into multiple data blocks (i.e., N data blocks) for transmission. This requires multiple communication steps (N-1 steps in this example) to complete the cluster operation. In each step, a node can receive data from one of its neighboring nodes, perform a specific operation on the received data to obtain a local result, and forward the received data to another neighboring node. After N-1 steps, each node in the ring has data from the other nodes in the ring, and the final result is distributed to all nodes, requiring another N-1 steps to broadcast their respective local results. For each node, the total forwarded data size is 2S, where S represents the data size or message size.

[0053] Figure 3B This illustrates a halving-doubled configuration for interconnecting a predetermined number of nodes (e.g., N nodes, where N is a positive integer greater than 1). In this halving configuration, nodes communicate with each other in pairs, requiring only N / 2 connections for each step of communication. In the first step, adjacent nodes are paired together, sending half of the message or data to their respective pairs and receiving the other half for processing. Thus, intermediate results can be distributed to paired nodes. In subsequent steps, new pairs are formed with increasing or doubling distances, and the amount of data used for processing is halved. After log2N steps of communication, the results are distributed among all nodes in the halving-doubled configuration. The local results within a node are then broadcast to other nodes via additional log2N steps of communication.

[0054] Exemplary cluster communication library

[0055] Figure 4 A schematic diagram depicts an example cluster communication library 400 that can be used by a distributed training system 102. In implementation, the cluster communication library is a communication library designed to provide high performance, high scalability, and strong availability, and can be configured to support not only standard cluster operations such as global allreduce and global allgather operations, but also other custom operations for custom applications. In implementation, the cluster communication library 400 can employ different types of interconnects or switches with different characteristics (e.g., latency, bandwidth, topology) and provide a mechanism to collect information about the underlying hardware in the network and computing nodes, thereby enabling the development of topology-aware algorithm designs based on one or more pieces of collected information.

[0056] In implementation, the cluster communication library 400 can provide flexibility to allow multiple algorithms to be executed in a single operation and improve performance (e.g., performance of communication and training) by leveraging the parallelism between intra-node and inter-node communication. Furthermore, the cluster communication library 400 can utilize multiple NICs in computing nodes with traditional or novel mapping algorithms and eliminate network congestion through a connected topology-aware arrangement.

[0057] In implementation, the cluster communication library 400 may include a software stack 402. In implementation, the software stack 402 may include multiple components, including but not limited to a transport component 404, a computation component 406, a communicator component 408, and a library context component 410. In implementation, the software stack 402 may be designed in a modular manner to allow for versatility and scalability.

[0058] In implementation, the transport component 404 can be responsible for the transfer or transmission of peer-to-peer (P2P) data in intra-node and inter-node communication. As an example, and not a limitation, the cluster communication library 400 can support TCP (Transmission Control Protocol) and RDMA (Remote Direct Memory Access) for inter-node communication, and P2P structures for intra-node communication, such as PCIe (Peripheral Component Interconnect Express), NVLink / NVSwitch, and QPI / UPI (Quick Path Interconnect / Ultra Path Interconnect). For RDMA communication, the transport component 404 can also be configured to manage memory regions (MRs) and corresponding memory buffers in processing units (such as graphics processing unit (GPU) devices) and host memory.

[0059] In implementation, the computing component 406 can provide a set of basic operations and various network algorithms. For example, the basic operations can be configured with algorithms supported by the cluster communication library 400. Furthermore, the computing component 406 allows users to define new operations based on these basic operations to achieve heterogeneity-aware operations, which can employ optimal or better algorithms for each type of architecture.

[0060] In implementation, communicator component 408 can be associated with a software process and configured to perform manipulation and processing on a processing unit (such as a GPU device). Communicator component 408 can store or record information about other peer components (e.g., sort ID, IP address, etc.) and maintain connections with peer components. In implementation, communicator component 408 can further collect intra-node and inter-node topology information and use this information to guide algorithm design. In implementation, intra-node information may include, but is not limited to, the type of interconnection, the distance between processing units, and the distance between processing units and network interface controllers. In implementation, inter-node information may include, but is not limited to, for example, the number of available network interface controllers, the topology of the cluster or compute nodes, and the location of compute nodes in the cluster.

[0061] In implementation, the library context component 410 can be configured to expose one or more application interfaces for setting system configurations (such as environment variables), managing the communicator component 408, and providing other functions such as logging.

[0062] Furthermore, in some cases, the cluster communication library 400 may further include or provide multiple tools and utilities 412 for topology-aware design, testing and evaluation, and availability improvement. As an example, and not a limitation, tools and utilities 412 may include performance testing tools for the transport component 404 to assist in algorithm design and evaluation, probe-based routing mechanisms to ensure system availability, and other functionalities such as device management capabilities that can be extended to support devices other than GPUs.

[0063] An exemplary topology-aware multi-stage algorithm for cluster communication

[0064] In implementation, cluster communication can be defined as communication involving a group of processing units or processes, and the operations of cluster communication can be executed jointly by all processing units or processes included in the group. Examples of cluster communication operations can include, but are not limited to, global reduction operations, global aggregation operations, and reduce-scatter operations. In implementation, global reduction operations are one of the many important foundations of cluster communication in distributed training and involve performing reduction on data across processes within a group. Examples of reduction can include, but are not limited to, summation operations, averaging operations, maximum value operations, and minimum value operations.

[0065] As an example, and not a limitation, this section uses global reduction as an example to illustrate how cluster operations can be divided into multiple micro-operations or sub-operations. In implementation, the distributed training system 102 can employ a topology-aware multi-stage algorithm that divides the global reduction operation into multiple micro-operations or sub-operations and selectively chooses one or more micro-operations or sub-operations as needed, thereby reducing the amount of data transmitted by eliminating potentially unnecessary micro-operations or sub-operations. In implementation, the distributed training system 102 can separate cluster communication algorithms from micro-operations or sub-operations and allow independent or separate matching between algorithms and micro-operations or sub-operations based on underlying structural information, thereby maximizing or optimizing bandwidth utilization by reducing the amount of data transmitted.

[0066] Figure 5 An exemplary topology-aware multi-stage algorithm 500 that can be used in a distributed training system 102 is shown. In implementation, the topology-aware multi-stage algorithm 500 may include multiple stages, such as an intra-node reduction scattering stage 502, an inter-node global reduction stage 504, and an intra-node global aggregation stage 506.

[0067] In implementation, the distributed training system 102 can first distribute the portions of the data to be processed for training to multiple computing nodes 104, so that each computing node 104 can receive the corresponding portion of the data. In implementation, each computing node 104 can divide the corresponding portion of the data into multiple data slices (e.g., N data slices, where N is a positive integer) and distribute these multiple data slices to multiple local processing units or processes included in the corresponding computing node 104 (e.g., N local processing units or processes).

[0068] In implementation, during the intra-node reduce scatter phase 502, each local processing unit or process included in each compute node 104 can divide the data slice allocated to it into multiple data blocks (e.g., M blocks). The local processing units or processes included in each compute node 104 can then collaboratively perform intra-node reduce scatter suboperations to obtain all reduce results for multiple data blocks in the corresponding compute node 104 in multiple steps or iterations according to a specific cluster communication algorithm. At the end of the intra-node reduce scatter phase 502, the local processing units or processes included in the compute node 104 may have different data blocks containing the reduce results (or reduce scatter results) of all processing units or processes included in that compute node 104.

[0069] As an example and not a limitation, two exemplary cluster communication algorithms, a ring-based algorithm and a halving-doubling algorithm, are used to describe a particular cluster communication algorithm to illustrate a specific mechanism or operation in the intra-node reduction scattering phase 502. However, other cluster communication algorithms can be used in this intra-node reduction scattering phase 502. For example, the distributed training system 102 can select a specific cluster communication algorithm to be used in the intra-node reduction scattering phase 502 based on information from multiple factors collected by the cluster communication library 400. In implementation, these multiple factors may include, but are not limited to, the type of interconnection between processing units (or processes) in the computing node, the number of interconnections in the computing node, etc.

[0070] For example, in the intra-node reduction and distribution phase 502, the distributed training system 102 may employ a first cluster communication algorithm for the first computing node and a second cluster communication algorithm for the second computing node, wherein the second computing node has the same or different processing and connectivity capabilities as the first computing node, and the first cluster communication algorithm may be the same or different from the second cluster communication algorithm. As an example, and not a limitation, for computing nodes interconnected using NVSwitch or PCIe and comprising multiple processing units or processes for training in a power of 2, the distributed training system 102 may employ a halving and doubling algorithm; while for another computing node interconnected using NVLink or others and comprising multiple processing units or processes for training in a number not exceeding a power of 2, the distributed training system 102 may employ a ring-based algorithm, and so on.

[0071] Figure 6 An exemplary ring-based algorithm 600 for computing nodes in the intra-node reduction scattering phase 502 is illustrated. For the purposes of brevity and description, the exemplary ring-based algorithm includes only a configuration of one ring. However, any ring-based algorithm including a configuration of more than one ring may be employed, for example, each ring processing a portion of a data block.

[0072] In this example, the described computing node comprises M processing units or processes (with sorting identifiers or numbers 1, 2, ..., M), and the data allocated to each processing unit or process is divided into M data blocks. In the first step, a processing unit or process (e.g., P1) can send one of its M data blocks to the next processing unit or process in the ring (e.g., P2), receive another data block from the previous processing unit or process in the ring (e.g., PM), and reduce the received data block with the corresponding local data block to obtain a partially reduced result. In each subsequent step (e.g., the k-th step), a processing unit or process (e.g., P1) can send the partially reduced result (in this example, the partially reduced result obtained by P1 in the (k-1)-th step) to the next processing unit or process in the ring (e.g., P2), receive the partially reduced result from the previous processing unit or process (e.g., PM) (in this example, the partially reduced result obtained by PM in the (k-1)-th step), and reduce the received partially reduced result with another local data block that was not previously sent or reduced with other data.

[0073] like Figure 6 As shown, in each step, different data blocks can be received, reduced, or sent by different processing units or processes within the computing node. Furthermore, each processing unit or process can send or receive and reduce different data blocks (or partial results) at different steps. At the end of the intra-node reduction distribution phase 502 (i.e., after M-1 steps), each processing unit or process can include a result data block that stores the reduction results of the M corresponding data blocks of the M processing units or processes in that computing node. For example, after M-1 steps, the data block at the "top position" P1 stores the reduction results of all data blocks corresponding to the M processing units or processes at that "top position," such as... Figure 6 As shown.

[0074] Figure 7An exemplary halving and doubling algorithm 700 for computing nodes is illustrated in the intra-node reduction scattering phase 502. In this example, the described computing node includes M processing units or processes (M is set to 8 in this example). In the first step, a processing unit or process (e.g., P1) can send half of the data assigned to it to another nearby processing unit or process (e.g., P2), receive half of the data assigned to that other processing unit or process (e.g., P2), and reduce the received data with the other half of the data assigned to its own processing unit or process (e.g., P1) to obtain a partial reduction result. In each subsequent step, a processing unit or process (e.g., P1) can send half of the partial reduction result obtained locally in the previous step to a different processing unit or process located further and further away from the processing unit or process (i.e., P1), and reduce the received partial reduction result with the other half of the partial reduction result obtained locally in the previous step to obtain a new partial reduction result for the processing unit or process (i.e., P1). At the end of the intra-node reduction scattering phase 502 (i.e., after log2M steps, i.e., at the end of the previous step), the partial reduction scattering phase 502 continues. Figure 7 Following the three steps in the example shown, each processing unit or process may include a result data block that stores M (in this example, such as) data blocks in the computing node. Figure 7 The reduction results of M corresponding data blocks for the 8 processing units or processes shown. For example, after log2M steps, the data block storage at the "bottom position" P1 corresponds to... Figure 7 The M "bottom positions" shown (in this example, such as...) Figure 7 The diagram shows the reduction results for all data blocks of 8 processing units or processes.

[0075] In implementation, during the inter-node global reduction phase 504, the inter-node global reduction sub-operations are node-based (i.e., between different computing nodes) and can be executed between processing units (or processes) included in different computing nodes. In implementation, processing units (or processes) of different computing nodes holding data blocks of the same reduction result (or reduction distribution result) form the same group, and within this group, they exchange their respective results to perform the inter-node global reduction sub-operations. At the end of the inter-node global reduction phase 504, each processing unit or process of each computing node in a specific group can possess a specific data block of the reduction results of all processing units or processes in that group, and processing units or processes in different groups possess different data blocks of the reduction results of the corresponding processing units or processes in different groups.

[0076] In implementation, the distributed training system 102 can select a specific cluster communication algorithm based on one or more selection criteria, and can implement global reducer operations between nodes based on the selected cluster communication algorithm. Examples of specific cluster communication algorithms may include, but are not limited to, ring-based algorithms (e.g., hierarchical ring algorithms, multi-ring algorithms, etc.), halving and doubling algorithms, etc. In implementation, one or more selection criteria may include, but are not limited to, the topology of the communication network connecting computing nodes (e.g., communication network 206), the number of switches used in the communication network, the type of switches used in the communication network, the network type of the communication network, etc.

[0077] As examples and not limitations, the ring-based algorithm and the halving-doubling algorithm are used as examples to describe a particular cluster communication algorithm to illustrate a specific mechanism or operation in the inter-node global reduction phase 504. However, other cluster communication algorithms may be used in this inter-node global reduction phase 504 based on one or more of the selection criteria described above.

[0078] Figure 8 and 9 An exemplary halving and doubling algorithm is provided for the global reduction phase 504 between nodes. In this example, such as Figure 8 As shown, for the purposes of brevity and description, the distributed training system 102 described includes multiple computing nodes (i.e., node 0, node 1, node 2, ..., node N-1, where N is the number of nodes in the system). Figure 8 In the diagram, denoted as 4), each computing node comprises eight processing units or processes, each with a corresponding sequence number (i.e., sequence 0, sequence 1, sequence 2, ..., sequence M-1, where M is used for illustration). Figure 8 (The middle is shown as 8). Figure 8 As shown, processing units or processes with the same sort number in the corresponding computing nodes include data blocks with the same reduction result (or reduction scatter result), and they form the same group. For example, a processing unit or process with sort number 0 in the corresponding computing node includes data blocks of the reduction result at the first position in each local data block, and they form the same group (e.g., group 0). In implementation, processing units or processes in different groups may not communicate with each other.

[0079] In implementation, the global reduction suboperation between nodes can be executed independently between processing units (or processes) in each group, allowing each processing unit (or process) in the group to obtain all reduction results for the same data block from all processing units (or processes) in the same group. Similar to the halving and doubling algorithm mechanism described above for the intra-node reduction scattering phase, processing units or processes in each group can iteratively send local reduction results of the corresponding data block to other processing units or processes in the same group, receive the local reduction results of the corresponding data block from other processing units or processes in the doubled or increased distance, and perform reduction operations on the received reduction results using the local reduction results.

[0080] Figure 9 An example scenario is shown where the halving-doubling algorithm is applied to eight computing nodes. In this example, as... Figure 9 As shown, the number of steps executed in the global reduction phase 504 between nodes using the halving and doubling algorithm is log2N = log28 = 3, where N is the number of computing nodes. In the first step, a first processing unit or process (e.g., the processing unit or process with sort number 0) in a certain group of the first computing node (e.g., node 0) can send its local reduction result to a second processing unit or process in the same group of the second computing node (e.g., node 1), receive the local reduction result from the second processing unit or process in the same group of the second computing node, and perform a reduction operation on its local reduction result and the received local reduction result to obtain a new local reduction result.

[0081] In the second step, the first processing unit or process in the first computing node (e.g., node 0) (e.g., the processing unit or process with sort number 0) can send its new local reduction result to the third processing unit or process in the same group in the third computing node (i.e., node 2 in this example) (e.g., sort number 0), receive the local reduction result from the third processing unit or process in the same group in the first computing node, and perform a reduction operation on its new local reduction result and the received local reduction result to obtain another new local reduction result.

[0082] In the third (or final) step, the same operation is performed on the first processing unit or process, but this time the same operation is performed on the fourth processing unit or process in the same group in the fourth computing node (i.e., node 4 in this example).

[0083] At the end of the global reduction phase 504 between nodes, each processing unit or process of each computing node in a specific group can have a specific data block of the reduction results of all processing units or processes in the same group, and processing units or processes in different groups have different data blocks of the reduction results of the corresponding processing units or processes in different groups.

[0084] Similar to the halving and doubling algorithm, the inter-node global reduction suboperation can be executed independently among processing units (or processes) in each group of multiple computing nodes (e.g., N computing nodes) using a ring-based algorithm. This allows each processing unit (or process) in a group to obtain all reduction results for the same data block from all processing units (or processes) within the same group. Similar to the ring-based algorithm mechanism described above for the intra-node reduction distribution phase, each processing unit or process in a computing node can iteratively send its local reduction result for the corresponding data block to the corresponding processing unit or process in the next computing node. It then receives the local reduction result for the corresponding data block from the processing unit or process in the previous computing node and performs a reduction operation on the received reduction result using its local reduction result. At the end of the inter-node global reduction phase 504 (i.e., after step N-1), each processing unit or process in each computing node within a specific group can possess a specific data block of the reduction results for all processing units or processes in that same group. Processing units or processes in different groups possess different data blocks of the reduction results for the corresponding processing units or processes in different groups.

[0085] In implementation, similar to the intra-node reduction distribution phase 502, in the intra-node global aggregation phase 506, global aggregation suboperations can be executed locally by processing units or processes on each of the multiple computing nodes of the distributed training system 102 to locally broadcast the corresponding reduction results obtained in the inter-node global reduction phase 504 to each other within the same computing node. At the end of the intra-node global aggregation phase 506, each processing unit or process on each computing node of the distributed training system 102 can have the reduction results of the entire data distributed across multiple computing nodes.

[0086] As an example and not a limitation, this paper uses a ring-based algorithm to illustrate how reduction results obtained by local processing units or processes within the compute nodes of the distributed training system 102 (in the inter-node global reduction phase 504) can be broadcast. However, the distributed training system 102 can employ different or the same cluster communication algorithms (e.g., halving and doubling algorithms) for different compute nodes. For example, the distributed training system 102 can employ different or the same cluster communication algorithms for different compute nodes based on multiple factors associated with each individual compute node. In implementation, these multiple factors may include, but are not limited to, the type of interconnection between processing units (or processes) within the compute node, the number of interconnections within the compute node, etc.

[0087] Figure 10 An exemplary ring-based cluster communication algorithm 1000 is illustrated for broadcasting individual reduction results of processing units or processes to each other within the computing nodes of a distributed training system 102. Figure 10As shown, in the first step, each of the M processing units or processes in the computing node (e.g., P1) can send its reduction result obtained in the inter-node global reduction phase 504 to one of its two adjacent processing units or processes (e.g., P2 in this example) according to a ring configuration, and receive a reduction result from the other of its two adjacent processing units or processes (e.g., PM in this example). In each subsequent step, each processing unit or process (e.g., P1) can send a newly received reduction result to one of its two adjacent processing units or processes (e.g., P2 in this example) according to a ring configuration, and receive another reduction result from the other of its two adjacent processing units or processes (e.g., PM in this example). At the end of the intra-node global aggregation phase 506 (i.e., after M-1 steps), each processing unit or process in the computing node may have a reduction result of the reduction results of all processing units or processes in the computing node.

[0088] Exemplary Parallel Algorithm

[0089] In implementation, the distributed training system 102 can execute multiple stages included in the topology-aware multi-stage algorithm, namely, an intra-node reduction and scattering stage 502, an inter-node global reduction stage 504, and an intra-node global aggregation stage 506, etc. Alternatively, in implementation, the distributed training system 102 can partially or substantially overlap some of the intra-node reduction and scattering stage 502, the inter-node global reduction stage 504, and the intra-node global aggregation stage 506, and execute some parts of these stages in parallel.

[0090] For example, since the intra-node reduction and scattering phase 502 and the intra-node global aggregation phase 506 involve intra-node data communication or transmission (i.e., data communication or transmission within the computing node), and the inter-node global reduction phase 504 involves inter-node data communication or transmission (i.e., data communication or transmission between computing nodes), in implementation, the distributed training system 102 may allow at least a portion of the intra-node reduction and scattering phase 502 and the inter-node global reduction phase 504, as well as a portion of the inter-node global reduction phase 504 and the intra-node global aggregation phase 506, to be executed in parallel, thereby improving the utilization of intra-node and inter-node links (or connections) and preventing intra-node links from being idle when inter-node links are used, and vice versa.

[0091] Figure 11 Example scenarios are shown where the intra-node reduction scattering phase, the inter-node global reduction phase, and the intra-node global aggregation phase are executed in parallel or overlapping manner. For example... Figure 11 As shown, the processing unit or process of a computing node can divide a data block into multiple blocks (as in this example). Figure 11The four blocks shown are distributed to at least two concurrent threads (such as the first thread 1102 and the second thread 1104). In this way, the processing unit or process can pipeline intra-node and inter-node suboperations so that they can be executed by at least two concurrent threads (in this example, the first thread 1102 and the second thread 1104).

[0092] As an example and not a limitation, the first thread 1102 may perform an inter-node global reduce suboperation (i.e., the operation in the inter-node global reduce phase 504) on a first data block (e.g., data block 1106), while the second thread 1104 may perform an intra-node reduce scatter suboperation (i.e., the operation in the intra-node reduce scatter phase 502) on a second data block (e.g., data block 1108). For example, the first thread 1102 may perform an intra-node global aggregation suboperation (i.e., the operation in the intra-node global aggregation phase 506) on a third data block (e.g., data block 1110), while the second thread 1104 may perform an inter-node global reduce suboperation on a fourth data block (e.g., data block 1112).

[0093] As an example, and not a limitation, another operation involved in distributed neural network training can be further used as an example. In implementation, the distributed training system 102 can divide the global aggregation operation involved in distributed neural network training into multiple sub-operations, namely, inter-node global aggregation sub-operations, intra-node global aggregation sub-operations, and data replication sub-operations. In implementation, the inter-node global aggregation sub-operation can be similar to the inter-node global reduction sub-operation described above, but instead of performing a reduction operation (e.g., reducing the received result with the local result), it broadcasts data (e.g., the reduction result) instead of performing a reduction operation. The inter-node global aggregation sub-operation can be similar to or equivalent to the inter-node global aggregation sub-operation described above. In implementation, the data replication sub-operation can include the operation of replicating the result data (e.g., the final reduction result) as an output parameter.

[0094] In implementation, the processing unit or process of a computing node can divide a data block into multiple blocks (e.g., four blocks) and distribute these blocks to at least two concurrent threads (e.g., the first thread and the second thread), and pipeline intra-node and inter-node sub-operations for execution by at least two concurrent threads.

[0095] For example, the first thread can perform inter-node global aggregate suboperation on the first data block, while the second thread performs intra-node global aggregate suboperation on the second data block. Furthermore, the first thread can perform data duplication suboperation on the third data block, while the second thread performs inter-node global aggregate suboperation on the fourth data block.

[0096] Exemplary congestion avoidance methods

[0097] In implementation, due to data transmission between multiple computing nodes in the distributed training system 102, data or traffic congestion may occur at some switches or links in the communication network 206. To avoid congestion, the distributed training system 102 can employ a predetermined congestion avoidance strategy to distribute or transfer data traffic among various switches or links in the communication network 206, thereby preventing excessive data from passing through a particular switch or link in the communication network 206 during training (e.g., inter-node global reduction suboperation or stage, or inter-node global aggregation suboperation or stage).

[0098] In implementation, the distributed training system 102 may employ a first congestion avoidance method, which includes a ring generation strategy and subsequent network flow routing management. Additionally or alternatively, the distributed training system 102 may employ a second congestion avoidance method, which includes a strategy for reordering node identifiers and subsequent network flow routing management. Depending on the type of network topology of the communication network 206 and the processing and communication capabilities of the multiple computing nodes 104, the distributed training system 102 may select one or more of the first or second congestion avoidance methods to route data flows between all or part of the multiple computing nodes in the distributed training system 102. Furthermore, the distributed training system 102 may selectively combine parts of the first and second congestion avoidance methods to implement new congestion avoidance methods. In implementation, both the first and second congestion avoidance methods can be designed to specify dedicated network paths for each direction of the inter-node data flow in a manner where there is little or no conflict between the inter-node data flows.

[0099] In implementation, the distributed training system 102 can pre-obtain or establish a mapping relationship between communication connections and routing paths (e.g., physical links). In implementation, connection path data structures in the form of tables, linked lists, etc., can be created and used to store the mapping relationship information. In implementation, the distributed training system 102 can selectively or strategically use specific paths to establish connections between any two computing nodes based on the connection path data structure.

[0100] In implementation, the distributed training system 102 can determine the mapping relationship between communication connections and routing paths by enabling each computing node of the distributed training system 102 to send probe packets to other computing nodes by changing the source / destination ports of probe packets, thereby exhausting all possible communication connections between the computing nodes of the distributed training system 102. Obviously, the distributed training system 102 can use other methods to explore the mapping relationship between communication connections and routing paths, which are not limited here.

[0101] As an example, and not a limitation, the first compute node can send multiple probe packets to the second compute node. Each probe packet has a different combination of source and destination ports, with the source and destination addresses being the addresses of the first and second compute nodes, respectively. Each probe packet can record the switches it traversed, so when the corresponding probe packet returns to the first compute node, the first compute node can know the entire routing path of the corresponding probe packet used for mapping. Accordingly, a connection path data structure (e.g., a connection path data structure) can be established between the first and second compute nodes. Similarly, a mapping relationship (and connection path data structure) between communication connections and routing paths for other compute node pairs in the distributed training system 102 can be established accordingly.

[0102] For the purposes of brevity and illustration, the exemplary network topology, namely the fat tree network (or particularly the two-layer Clos network architecture in a full mesh topology), is used herein as an example network topology for the communication network 206 associated with the distributed training system 102. However, the exemplary congestion avoidance strategy described herein can also be applied to other network topologies.

[0103] Figure 12 An exemplary fat-tree network topology 1200 is illustrated. In this example, the exemplary fat-tree network topology is a two-layer Clos network architecture within a full mesh topology. One layer corresponds to a layer of slice switches 1202 directly connected to compute nodes 1204, each slice switch 1202 connected to one or more compute nodes 1204. In implementation, compute nodes 1204 may include one or more network interface controllers (e.g., four network interface controllers) connected to one or more ports (e.g., four ports) of slice switches 1202. In implementation, the number of network interface controllers for each compute node 1204 may be the same or different. The other layer corresponds to a layer of aggregation switches 1206 (or backbone switches 1206) connected to one or more slice switches 1202.

[0104] In implementation, if two processing units or processes located in different computing nodes are connected under the same slice switch, data packets transmitted between these two processing units or processes will pass through that same slice switch without passing through any aggregation switch. Alternatively, if two processing units or processes located in different computing nodes are connected under different slice switches, data packets transmitted between these two processing units or processes will pass through one of the aggregation switches. Using the connection path data structure described above, by setting an appropriate combination of source and destination ports in the data packets, data packets transmitted between two processing units or processes can be routed through a designated aggregation switch. In implementation, the routing management of the first congestion avoidance method and / or the second congestion avoidance method can be designed to enable data flows from the same slice switch to different destination slice switches to pass through different aggregation switches, and / or enable data flows from different source slice switches to the same destination slice switch to pass through different aggregation switches, thereby avoiding conflicts between data flows and ensuring no network congestion at the aggregation switches.

[0105] In implementation, as described above, the first congestion avoidance method may include a ring generation strategy and subsequent network flow routing management. The first congestion avoidance method can support various ring-based algorithms, including but not limited to ring algorithms, ring partitioning algorithms, multi-ring algorithms, hierarchical ring algorithms, algorithms involving multiple hierarchical rings, and node-aware ring algorithms.

[0106] In implementation, the ring generation strategy may include a topology-aware strategy for ring generation. As an example and limitation, the topology-aware strategy for ring generation may include multiple rules to establish a ring or ring configuration for processing units or processes. In implementation, processing units or processes in a compute node can send / receive data to / from processing units or processes in another compute node via a network interface controller. In implementation, processing units or processes in a compute node may be associated with a single network interface controller or multiple network interface controllers to transfer data to processing units or processes in other compute nodes. Additionally or alternatively, multiple processing units or processes may be associated with a single network interface controller and use that network interface controller to transfer data to processing units or processes in other compute nodes.

[0107] In implementation, multiple rules may include, but are not limited to, the priority of the processing unit or process in the first computing node to select the adjacent processing unit or process, the conditions for the network interface controller in the first computing node to send or receive data, and the conditions for the network interface controller in the first computing node to route data to / from the network interface controller in the second computing node.

[0108] In implementation, the priority of selecting adjacent processing units or processes in the first computing node may include selecting processing units or processes in the first computing node in descending order of priority and using inter-process communication (if applicable), selecting processing units or processes in the second computing node connected to the same chip switch as the first computing node, and selecting processing units or processes in the third computing node connected to a different chip switch than the first computing node, wherein the first computing node is different from the second computing node and the third computing node.

[0109] In implementation, the conditions under which the network interface controller in the first computing node sends or receives data may include, for example, the network interface controller being able to send data only to the network interface controller in the second computing node, and / or the network interface controller being able to receive data only from the network interface controller in the third computing node, wherein the first computing node is different from the second computing node and the third computing node, and the second computing node may be the same as or different from the third computing node.

[0110] In implementation, the conditions for the network interface controller in the first computing node to route data to / from the network interface controller in the second computing node may include, for example, if the data is sent through the network interface controller in the first computing node, then data sent by processing units or processes belonging to multiple rings will be routed to the network interface controller in the second computing node. In implementation, the conditions for the network interface controller in the first computing node to route data to / from the network interface controller in the second computing node may further include, if the data is sent by processing units or processes belonging to multiple rings through the network interface controller in the second computing node, then the data will be received through the network interface controller in the first computing node.

[0111] In implementation, the routing management of the first congestion avoidance method can assign Network Interface Controller (NIC) identifiers to each NIC connected to or linked to the same network interface controller. The routing management of the first congestion avoidance method can also assign aggregation identifiers to each aggregation switch in the communication network 206. For a processing unit or process in a ring, the routing management can determine a routing identifier used to route packets originating from that processing unit or process.

[0112] For example, if the network interface controller of a processing unit or process and the network interface controller of the next processing unit or process in the ring are located in the same compute node, or directly connected or linked to the same slice switch, the routing identifier can be determined as a default or default identifier. This default routing identifier indicates that data is routed either within the compute node or via the slice switch, without passing through any aggregation switch in the communication network. Otherwise, the routing identifier can be determined as equal to the NIC identifier of the processing unit or process or other predefined value. The aggregation identifier can be determined based on the determined routing identifier, based on the mapping between the routing identifier and the aggregation identifier. In implementation, for example, the mapping between the routing identifier and the aggregation identifier can be predetermined using a probe-based routing mechanism (e.g., sending probe packets between compute nodes as described above).

[0113] In other words, data flows between processing units (or processes) within the same compute node or between network interface controllers of the same switch will not pass through any aggregation switches in the communication network. On the other hand, data flows between processing units (or processes) within different compute nodes and between network interface controllers of different switches will pass through designated aggregation switches based on predetermined mapping relationships, thereby achieving data flow routing control and management, and distributing data flows to different aggregation switches to avoid network congestion.

[0114] Figure 13 An example scenario using the first congestion avoidance method is illustrated. In this example, four inter-node rings (or ring configurations, R0, R1, R2, and R3) containing eight compute nodes (node ​​0, node 1, ..., node 7) are generated, and each ring uses a different aggregation switch to send and receive data (e.g., during the inter-node global reduction phase 504). Therefore, there are no collisions among these four rings. Furthermore, each shard switch in any ring has only one data flow entering and one data flow leaving, thus avoiding network congestion.

[0115] In implementation, as described above, the second congestion avoidance method may include a strategy for reordering node identifiers, as well as subsequent network flow routing management. In implementation, to minimize communication costs, the second congestion avoidance method may reorder the identifiers of computing nodes and processing units (or processes) based on the network topology connecting computing nodes and processing units (or processes) according to multiple rules.

[0116] In implementation, multiple rules may include, for example, grouping compute nodes by their respective switches. For instance, compute nodes connected to the same switch (e.g., compute nodes with network interface controllers linked to the same switch) are grouped together, and each compute node is assigned a node identifier. Because the compute nodes are connected to the same switch, these compute nodes are physically adjacent to each other.

[0117] In implementation, multiple rules may also include assigning sort identifiers (or sort numbers) to each processing unit or process in a compute node using the same sequential sequence. For example, k processing units (or processes) in a first compute node may be assigned sort identifiers 0, 1, ..., k-1, while k processing units (or processes) in a second compute node may be assigned sort identifiers k, k+1, ..., 2k-1, and so on, for other compute nodes. Processing units (or processes) in a compute node can be sorted according to the corresponding network interface controller used by the processing unit (or process), and processing units (or processes) using the same network interface controller are (physically) adjacent to each other.

[0118] In this scenario, during the first log2L steps, the data flow between processing units (or processes) within a compute node can be restricted to the corresponding shard switch, which has a better latency than the aggregation switch, thus preventing network congestion. In this implementation, L is the number of compute nodes per shard switch in the node-aware doubling-halving algorithm described above. In the implementation, for the conventional doubling-halving algorithm, L is the product of the number of compute nodes per shard switch and the number of processing units (or processes) per compute node.

[0119] In implementation, the routing management of the second congestion avoidance method may include determining a convergence identifier for data streams or packets sent from a first processing unit (or process) with a first ordering identifier in a first computing node with a first node identifier to a second processing unit (or process) with a second ordering identifier in a second computing node with a second node identifier, wherein the first computing node may be the same as or different from the second computing node.

[0120] In implementation, the aggregation identifier can be determined at least in part based on at least some of the following: the ordering identifier, the node identifier, the number of network interface controllers per compute node, and the maximum number of compute nodes per slice switch. As an example, and not a limitation, the aggregation identifier can be determined as: the first ordering identifier of the first processing unit (or process) sending the data stream or packet + (the first node identifier of the first compute node having the first processing unit (or process) % the maximum number of compute nodes per slice switch) × the number of network interface controllers per compute node, where % represents the modulo operator. Clearly, other methods of calculating the aggregation identifier are also applicable, provided consistent results are obtained. For example, the aggregation identifier can be determined based on a pre-defined mapping between the aggregation identifier and the combination of the ordering identifier and the node identifier.

[0121] In implementation, the routing management of the second congestion avoidance method may include pre-assigning convergence identifiers to each convergence switch in the communication network 206 associated with the distributed training system 102. If the first processing unit (or process) and the second processing unit (or process) are linked to the same switch or under the same switch (e.g., through their respective network controllers), the data flow or packets will pass through that switch without needing to go through any convergence switch in the communication network 206. If the first processing unit (or process) and the second processing unit (or process) are not linked to the same switch or are not under the same switch, the data flow or packets sent by the first processing unit (or process) to the second processing unit (or process) will pass through the convergence switch with the determined convergence identifier. In this example, the number of network interface controllers described includes in each compute node is four.

[0122] Figure 14 An example scenario using the second congestion avoidance method is illustrated. In this example, all compute nodes include the same number of processing units (or processes) and the same number of network interface controllers, each with the same number of processing units (or processes) to be associated. Furthermore, the number of network interface controllers linked to the slice switch is less than the number of aggregation switches in the network. In this example, there are 4 network interface controllers per compute node, and a maximum of 2 compute nodes per slice switch. In implementations, for node-aware doubling-halving algorithms, the number of compute nodes under the same slice switch can be a power of 2, and the number of network interface controllers included in the compute nodes under the same slice switch can be a power of 2. For conventional doubling-halving algorithms, the number of processing units (or processes) using the same network interface controller can be a power of 2. In this example, the number of network interface controllers included in each compute node is described as 4.

[0123] In this example, during the global reduction phase between nodes in the node-aware halving and doubling algorithm, the processing units (or processes) of compute nodes (nodes 0, 2, 4, and 6) will use convergence switches with convergence identifiers (e.g., A1, A2, A3, and A4), and the processing units (or processes) of compute nodes (nodes 1, 3, 5, and 7) will use convergence switches with convergence identifiers (e.g., A5, A6, A7, and A8). Accordingly, there are no collisions in the data flow between compute nodes, thus avoiding network congestion at any convergence switch in the network.

[0124] In implementation, at each step of the global reduction phase between nodes, a processing unit (or process) can communicate data with a new processing unit (or process). Synchronization can be performed to ensure that the data flow performed by a processing unit (or process) using the network interface controller in the current step does not overlap with the data flow performed by an adjacent processing unit (or process) using the same network interface controller in a previous step, thus avoiding micro-burst congestion.

[0125] Exemplary methods

[0126] Figure 15 A schematic diagram of an exemplary topology-aware multi-stage method is shown. Figure 16 A schematic diagram of a first exemplary network congestion avoidance method is shown. Figure 17 A schematic diagram of a second exemplary network congestion avoidance method is shown. Figure 18 A schematic diagram of an exemplary parallel method based on a hybrid architecture in distributed training is shown. Figure 15-18 The method can (but not necessarily) be used in Figure 1 Utilizing in the environment shown Figure 2 The computation nodes shown are implemented with the aid of the method and scenario illustrated in Figure 3-14. For ease of description, refer to... Figure 1-14 Methods 1500-1800 are described here. However, methods 1500-1800 can be alternatively implemented in other environments and / or using other systems.

[0127] Methods 1500-1800 are described in the general context of machine-executable instructions. Typically, machine-executable instructions can include routines, programs, objects, components, data structures, procedures, modules, functions, etc., that perform a specific function or implement a specific abstract data type. Furthermore, each example method is represented as a set of boxes in a logic flowchart representing a series of operations that can be implemented in hardware, software, firmware, or a combination thereof. The order in which the methods are described is not intended to be construed as limiting, and any number of described method boxes can be combined in any order to implement the method or alternative methods. Additionally, individual boxes may be omitted from the method without departing from the spirit and scope of the subject matter described herein. In the context of software, a box represents a computer instruction that performs the operation when executed by one or more processors. In the context of hardware, some or all of the boxes may represent an application-specific integrated circuit (ASIC) or other physical component that performs the operation.

[0128] refer to Figure 15 In block 1502, a first computing node (e.g., computing node 104) can perform a reduction scatter operation among a first set of processing units in the first computing node according to a first cluster communication algorithm.

[0129] In implementation, before performing the reduction scatter suboperation, the first computing node may select a first cluster communication algorithm based at least in part on the type or bandwidth of intra-node connections between the first set of processing units in the first computing node. In implementation, the first cluster communication algorithm may include, but is not limited to, ring-based algorithms or halving / doubling algorithms.

[0130] In implementation, performing a reduction scatter suboperation among the first set of processing units in the first computing node according to the first cluster communication algorithm may include dividing data into multiple data blocks; allocating the multiple data blocks to the first set of processing units; receiving data blocks from a second processing unit of the first set of processing units at a first processing unit of the first set of processing units according to the first cluster communication algorithm; and reducing the received data blocks with local data blocks at the first processing unit.

[0131] In box 1504, the first computing node can perform a global reducer operation between the first set of processing units in the first computing node and the second set of processing units in the second computing node according to the second cluster communication algorithm.

[0132] In implementation, before executing the global reducer operation, the first computing node may select the second cluster communication algorithm, at least in part, based on the type or bandwidth of the inter-node connections between the first computing node and other computing nodes, and / or the connection topology between the first computing node and other computing nodes. In implementation, the first cluster communication algorithm may include, but is not limited to, ring-based algorithms, or halving / doubling algorithms (such as node-aware halving / doubling algorithms).

[0133] In implementation, performing a global reduction sub-operation between the first set of processing units in the first computing node and the second set of processing units in the second computing node according to the second cluster algorithm may include: the first set of processing units receiving each part of the reduction scattering result obtained by the second set of processing units in the second computing node according to the second cluster algorithm; each processing unit in the first set of processing units forming a group with the corresponding processing unit in the second set of processing units, and receiving the corresponding part of the reduction scattering result from the corresponding processing unit; and the first set of processing units performing reduction on each part of the reduction scattering result by using the corresponding local part of the reduction scattering result obtained after performing the reduction scattering sub-operation between the first set of processing units.

[0134] In box 1506, the first computing node can perform a global aggregate suboperation among the first set of processing units in the first computing node according to the first cluster communication algorithm.

[0135] In implementation, performing a global aggregate suboperation between the first set of processing units in the first computing node according to the first cluster communication algorithm may include: receiving a data block from a second processing unit of the first set of processing units at the first processing unit of the first set of processing units according to the first cluster communication algorithm; and reducing the received data block with a local data block at the first processing unit.

[0136] refer to Figure 16 In box 1602, a first compute node (e.g., compute node 104) or a first process may determine a routing identifier for routing data from the first process to the second process, at least in part, based on whether the network interface controller associated with the first process and the network interface controller associated with the second process are located in the same compute node or linked to the same switch.

[0137] In implementation, the first and second processes can belong to a specific inter-node ring connecting multiple different nodes within a particular network topology. As an example and not a limitation, a particular network topology can include a fat tree topology.

[0138] In implementation, the network interface controller associated with the first process is configured to send data to or receive data from a second computing node in the ring topology, which is different from the first computing node.

[0139] In implementation, the network interface controller associated with the first process is also associated with one or more processes, wherein all data sent from the first process and one or more processes is sent through the network interface controller.

[0140] In implementation, in response to determining that the network interface controller associated with the first process and the network interface controller associated with the second process are located in the same compute node or linked to the same switch, the routing identifier can be set or determined as the default identifier.

[0141] In implementation, in response to determining that the network interface controller associated with the first process and the network interface controller associated with the second process are located in different compute nodes or linked to different slice switches, the routing identifier can be set or determined to be equal to the identifier of the network interface controller associated with the first process.

[0142] In box 1604, the first compute node or the first process can route data from the first process to the second process based on the routing identifier.

[0143] In implementation, routing data from a first process to a second process based on a routing identifier may include routing data from the first process to the second process at least through a slice switch connected to a network interface controller and an aggregation switch having an identifier that corresponds to the identifier of the network interface controller, which is associated with the first process.

[0144] refer to Figure 17 In box 1702, a first compute node (e.g., compute node 104) or a first process may determine an aggregation identifier for sending data packets from the first process to the second process, which belong to different nodes connected to different slice switches under a specific network topology, based on a node-aware halving and doubling algorithm.

[0145] In implementation, the first compute node can assign different aggregation identifiers to data packets directed to compute nodes connected to different slice switches, so that the data packets can be routed to nodes connected to different slice switches through different aggregation switches.

[0146] In implementation, the first computing node can, at least partially, assign source and destination ports to the aggregation switches associated with the aggregation identifier based on a predetermined mapping relationship. In implementation, the mapping relationship can record the relationships between aggregation identifiers of multiple aggregation switches and their corresponding source and destination port pairs. In implementation, a specific network topology may include a fat-tree topology.

[0147] In box 1704, the first compute node can send data packets from the first process to the second process through the aggregation switch corresponding to the aggregation identifier.

[0148] In implementation, the first computing node can also send data packets from the first set of processes included in the first computing node to the second set of processes included in the second computing node through multiple different aggregation switches corresponding to multiple different aggregation identifiers assigned to each data packet.

[0149] In implementation, the first computing node can also receive data packets from the second processing set included in the second computing node through multiple different aggregation switches corresponding to multiple different aggregation identifiers assigned to each data packet.

[0150] refer to Figure 18 In block 1802, a first computing node (e.g., computing node 104) or processing unit may divide a data block allocated to the processing unit into multiple data segments, which include at least a first data segment and a second data segment.

[0151] In box 1804, the first computing node or processing unit can allocate multiple data segments to multiple threads, which include at least the first thread and the second thread.

[0152] In box 1806, the first computing node or processing unit may use a first thread to perform intra-node sub-operations on a portion of the first data segment, and in parallel use a second thread to perform inter-node sub-operations on a portion of the second data segment.

[0153] In practice, performing intra-node suboperations on a portion of a first data segment using a first thread may include transferring the portion of the first data segment between a processing unit included in the first computing node and another processing unit via an intra-node connection.

[0154] In practice, using a second thread to perform inter-node suboperations on a portion of the second data segment may include transferring a portion of the second data segment between a processing unit and another processing unit included in a second computing node that is different from the first computing node via an inter-node connection.

[0155] In implementation, intra-node suboperations may include reduction scatter suboperations or global aggregation suboperations performed within the first computing node, and inter-node suboperations may include global reduction suboperations performed between the first computing node and a second computing node different from the first computing node.

[0156] In implementation, intra-node suboperations may include global aggregate suboperations or copy suboperations executed within the first computing node, and inter-node suboperations may include global aggregate suboperations executed between the first computing node and a second computing node different from the first computing node.

[0157] In implementation, the first computing node or processing unit may use a first thread to perform another inter-node sub-operation on a portion of the first data segment, and in parallel use a second thread to perform another intra-node sub-operation on a portion of the second data segment.

[0158] In implementation, a first thread performs intra-node sub-operations on a portion of the first data segment, and a second thread performs inter-node sub-operations on a portion of the second data segment in parallel, such that an intra-node connection is used to transfer a portion of the first data segment to another processing unit included in the first computing node, and an inter-node connection is used concurrently to transfer a portion of the second data segment to another processing unit included in a second computing node that is different from the first computing node.

[0159] Although the method blocks described above are executed in a certain order, in some implementations, some or all of the method blocks may be executed in a different order or in parallel.

[0160] Summarize

[0161] Although the embodiments are described using language specific to structural features and / or methodological actions, it should be understood that the claims are not necessarily limited to the specific features or actions described. Rather, these specific features and actions are disclosed as examples of implementing the claimed subject matter. Additionally or alternatively, some or all of the operations may be implemented by one or more ASICs, FPGAs, or other hardware.

[0162] This disclosure can be further understood using the following provisions:

[0163] Clause 1: A method implemented by a first computing node, the method comprising: performing a reduction scatter suboperation among a first set of processing units in the first computing node according to a first cluster communication algorithm; performing a global reduction suboperation among the first set of processing units in the first computing node and a second set of processing units in the second computing node according to a second cluster communication algorithm; and performing a global aggregation suboperation among the first set of processing units in the first computing node according to the first cluster communication algorithm.

[0164] Clause 2: The method described in Clause 1 further includes: selecting a first cluster communication algorithm based at least in part on the type or bandwidth of intra-node connections between the first set of processing units in the first computing node.

[0165] Clause 3: The method described in Clause 1 further includes: selecting a second cluster communication algorithm based at least in part on the type or bandwidth of the inter-node connection between the first computing node and other computing nodes, and / or the connection topology between the first computing node and other computing nodes.

[0166] Clause 4: The method described in Clause 1, wherein the first cluster communication algorithm includes a ring-based algorithm or a halving-doubling algorithm.

[0167] Clause 5: The method according to Clause 1, wherein performing a reduction scatter operation among a first set of processing units in a first computing node according to a first cluster communication algorithm includes: dividing data into multiple data blocks; allocating the multiple data blocks to the first set of processing units; receiving data blocks at a first processing unit of the first set of processing units from a second processing unit of the first set of processing units according to the first cluster communication algorithm; and reducing the received data blocks at the first processing unit using local data blocks.

[0168] Clause 6: The method according to Clause 1, wherein performing a global reduction sub-operation between a first set of processing units in a first computing node and a second set of processing units in a second computing node according to a second clustering algorithm comprises: the first set of processing units receiving portions of the reduction scattering result obtained by the second set of processing units in the second computing node according to the second clustering algorithm, wherein each processing unit of the first set of processing units forms a group with a corresponding processing unit of the second set of processing units and receives a corresponding portion of the reduction scattering result from the corresponding processing unit; the first set of processing units performing reduction on portions of the reduction scattering result obtained by performing a reduction scattering sub-operation between the first set of processing units using the corresponding local portion of the reduction scattering result.

[0169] Clause 7: The method according to Clause 1, wherein performing a global aggregate suboperation among a first set of processing units in a first computing node according to a first cluster communication algorithm comprises: receiving a data block from a second processing unit of a first set of processing units at a first processing unit of the first set of processing units according to the first cluster communication algorithm; and reducing the received data block with a local data block at the first processing unit.

[0170] Clause 8: One or more machine-readable media storing machine-readable instructions that, when executed by a first computing node, cause the first computing node to perform actions, said actions including: performing a reduction scatter suboperation among a first set of processing units in the first computing node according to a first cluster communication algorithm; performing a global reduction suboperation among a first set of processing units in the first computing node and a second set of processing units in the second computing node according to a second cluster communication algorithm; and performing a global aggregation suboperation among the first set of processing units in the first computing node according to the first cluster communication algorithm.

[0171] Clause 9: The action further includes, at least in part, selecting a first cluster communication algorithm based on the type or bandwidth of intra-node connections between the first set of processing units in the first computing node, according to one or more machine-readable media as described in Clause 8.

[0172] Clause 10: The action further includes, in accordance with one or more machine-readable media as described in Clause 8, selecting a second cluster communication algorithm based at least in part on the type or bandwidth of the inter-node connection between the first computing node and the other computing nodes, and / or the connection topology between the first computing node and the other computing nodes.

[0173] Clause 11: One or more machine-readable media as described in Clause 8, wherein the first cluster communication algorithm includes a ring-based algorithm or a halving-doubling algorithm.

[0174] Clause 12: One or more machine-readable media according to Clause 8, wherein performing a reduction scatter suboperation among a first set of processing units in a first computing node according to a first cluster communication algorithm comprises: dividing data into multiple data blocks; allocating the multiple data blocks to the first set of processing units; receiving data blocks at a first processing unit of the first set of processing units from a second processing unit of the first set of processing units according to the first cluster communication algorithm; and reducing the received data blocks at the first processing unit using local data blocks.

[0175] Clause 13: One or more machine-readable media according to Clause 8, wherein performing a global reduce suboperation between a first set of processing units in a first computing node and a second set of processing units in a second computing node according to a second clustering algorithm comprises: the first set of processing units receiving portions of a reduce distribution result obtained by the second set of processing units in the second computing node according to the second clustering algorithm; each processing unit of the first set of processing units forming a group with a corresponding processing unit of the second set of processing units and receiving a corresponding portion of the reduce distribution result from the corresponding processing unit; and the first set of processing units performing a reduce on portions of the reduce distribution result obtained by performing a reduce suboperation between the first sets of processing units using the corresponding local portion of the reduce distribution result.

[0176] Clause 14: One or more of the provisions of Clause 8 and their readable media, wherein performing a global aggregate suboperation among a first set of processing units in a first computing node according to a first cluster communication algorithm comprises: receiving a data block from a second processing unit of a first set of processing units at a first processing unit of the first set of processing units according to the first cluster communication algorithm; and reducing the received data block with a local data block at the first processing unit.

[0177] Clause 15: A first computing node includes: a first set of processing units; and a memory storing machine-executable instructions that, when executed by the first set of processing units, cause the first set of processing units to perform actions, said actions including: performing a reduction scatter suboperation among the first set of processing units in the first computing node according to a first cluster communication algorithm; performing a global reduction suboperation among the first set of processing units in the first computing node and a second set of processing units in the second computing node according to a second cluster communication algorithm; and performing a global aggregation suboperation among the first set of processing units in the first computing node according to the first cluster communication algorithm.

[0178] Clause 16: The action of the first computing node as described in Clause 15 further includes: selecting a first cluster communication based at least in part on the type or bandwidth of the intra-node connection between the first set of processing units of the first computing node; selecting a second cluster communication algorithm based at least in part on the type or bandwidth of the inter-node connection between the first computing node and other computing nodes, and / or the connection topology between the first computing node and other computing nodes.

[0179] Clause 17: The first computing node as described in Clause 15, wherein the first cluster communication algorithm includes a ring-based algorithm or a halving-doubling algorithm.

[0180] Clause 18: The first computing node according to Clause 15, wherein performing a reduction scatter suboperation among a first set of processing units in the first computing node according to a first cluster communication algorithm includes: dividing data into multiple data blocks; allocating the multiple data blocks to the first set of processing units; receiving data blocks at a first processing unit of the first set of processing units from a second processing unit of the first set of processing units according to the first cluster communication algorithm; and reducing the received data blocks at the first processing unit using local data blocks.

[0181] Clause 19: The first computing node according to Clause 15, wherein performing a global reduce sub-operation between a first set of processing units in the first computing node and a second set of processing units in the second computing node according to the second clustering algorithm comprises: the first set of processing units receiving portions of the reduce scattering result obtained by the second set of processing units in the second computing node according to the second clustering algorithm; each processing unit of the first set of processing units forming a group with a corresponding processing unit of the second set of processing units and receiving a corresponding portion of the reduce scattering result from the corresponding processing unit; and the second set of processing units performing a reduce on portions of the reduce scattering result obtained by performing a reduce scattering sub-operation between the first set of processing units.

[0182] Clause 20: The first computing node according to Clause 15, wherein performing a global aggregate suboperation among a first set of processing units in the first computing node according to a first cluster communication algorithm comprises: receiving a data block from a second processing unit of the first set of processing units at a first processing unit of the first set of processing units according to the first cluster communication algorithm; and reducing the received data block with a local data block at the first processing unit.

[0183] Clause 21: A method implemented by a first compute node, the method comprising: determining, at least in part, a routing identifier for routing data from the first process to the second process based on whether a network interface controller associated with a first process and a network interface controller associated with a second process are located in the same compute node or linked to the same switch, the first process and the second process belonging to a specific inter-node loop connecting multiple different nodes under a specific network topology; and routing data from the first process to the second process according to the routing identifier.

[0184] Clause 22: The method according to Clause 21, wherein the network interface controller associated with the first process is configured to send data to or receive data from a second compute node in the ring topology only, the second compute node being different from the first compute node.

[0185] Clause 23: The method described in Clause 21, wherein the network interface controller associated with the first process is also associated with one or more processes, wherein all data sent from the first process and the one or more processes is sent through the network interface controller.

[0186] Clause 24: The method described in Clause 21, wherein a particular network topology includes a fat tree topology.

[0187] Clause 25: The method described in Clause 21 further includes: in response to determining that the network interface controller associated with the first process and the network interface controller associated with the second process are located in the same compute node or linked to the same switch, setting the routing identifier to a default identifier.

[0188] Clause 26: The method according to Clause 21 further includes: in response to determining that the network interface controller associated with the first process and the network interface controller associated with the second process are located on different compute nodes or linked to different slice switches, setting the routing identifier to be equal to the identifier of the network interface controller associated with the first process.

[0189] Clause 27: The method according to Clause 26, wherein routing data from a first process to a second process based on a routing identifier comprises: routing data from the first process to the second process at least through a slice switch connected to a network interface controller and an aggregation switch having an identifier corresponding to the identifier of the network interface controller, the network interface controller being associated with the first process.

[0190] Clause 28: One or more machine-readable media storing machine-readable instructions that, when executed by a first compute node, cause the first compute node to perform an action, said action including: determining, at least in part, a routing identifier for routing data from the first process to the second process based on whether a network interface controller associated with the first process and a network interface controller associated with the second process are located in the same compute node or linked to the same switch, wherein the first process and the second process belong to a specific inter-node loop connecting multiple different nodes under a specific network topology; and routing data from the first process to the second process according to the routing identifier.

[0191] Clause 29: One or more machine-readable media as described in Clause 28, wherein a network interface controller associated with the first process is configured to send data to or receive data from a second compute node in a ring topology, the second compute node being different from the first compute node.

[0192] Clause 30: One or more machine-readable media as described in Clause 28, wherein a network interface controller associated with the first process is also associated with one or more processes, wherein all data sent from the first process and one or more processes is sent through the network interface controller.

[0193] Clause 31: One or more machine-readable media as described in Clause 28, wherein a particular network topology includes a fat tree topology.

[0194] Clause 32: One or more machine-readable media as described in Clause 28, wherein said action further includes: in response to determining that the network interface controller associated with the first process and the network interface controller associated with the second process are located in the same compute node or linked to the same switch, setting the routing identifier to a default identifier.

[0195] Clause 33: One or more machine-readable media as described in Clause 28, wherein said action further includes: in response to determining that the network interface controller associated with the first process and the network interface controller associated with the second process are located on different compute nodes or linked to different slice switches, setting the routing identifier to be equal to the identifier of the network interface controller associated with the first process.

[0196] Clause 34: One or more machine-readable media as described in Clause 33, wherein routing data from a first process to a second process based on a routing identifier comprises: routing data from the first process to the second process at least through a slice switch connected to a network interface controller and an aggregation switch having an identifier corresponding to the identifier of the network interface controller, the network interface controller being associated with the first process.

[0197] Clause 35: A first computing node includes: one or more processing units; and a memory storing machine-executable instructions that, when executed by the one or more processing units, cause the one or more processing units to perform actions, said actions including: determining, at least in part, a routing identifier for routing data from the first process to the second process based on whether a network interface controller associated with a first process and a network interface controller associated with a second process are located in the same computing node or linked to the same network switch, the first process and the second process belonging to a specific inter-node loop connecting multiple different nodes under a specific network topology; and routing data from the first process to the second process according to the routing identifier.

[0198] Clause 36: The first computing node as described in Clause 35, wherein the network interface controller associated with the first process is configured to send data to or receive data from a second computing node in the ring topology only, the second computing node being different from the first computing node.

[0199] Clause 37: The first computing node as described in Clause 35, wherein the network interface controller associated with the first process is also associated with one or more processes, wherein all data sent from the first process and the one or more processes is sent through the network interface controller.

[0200] Clause 38: The first compute node as described in Clause 35, wherein the action further includes: in response to determining that the network interface controller associated with the first process and the network interface controller associated with the second process are located in the same compute node or linked to the same switch, setting the routing identifier to the default identifier.

[0201] Clause 39: The first compute node as described in Clause 35, wherein the action further includes: in response to determining that the network interface controller associated with the first process and the network interface controller associated with the second process are located on different compute nodes or linked to different slice switches, setting the routing identifier to be equal to the identifier of the network interface controller associated with the first process.

[0202] Clause 40: The first computing node as described in Clause 39, wherein routing data from the first process to the second process according to a routing identifier comprises: routing data from the first process to the second process at least through a slice switch connected to a network interface controller and an aggregation switch having an identifier corresponding to the identifier of the network interface controller, the network interface controller being associated with the first process.

[0203] Clause 41: A method implemented by a first computing node, the method comprising:

[0204] The aggregation identifier used to send data packets from the first process to the second process is determined based on the node-aware halving and doubling algorithm. The first process and the second process belong to different nodes connected to different slice switches under a specific network topology; and data packets are sent from the first process to the second process through the aggregation switch corresponding to the aggregation identifier.

[0205] Clause 42: The method described in Clause 41 further includes: assigning different aggregation identifiers to packets directed to nodes connected to different slice switches, so that packets can be routed to nodes connected to different slice switches via different aggregation switches.

[0206] Clause 43: The method described in Clause 41 further includes: assigning source and destination ports corresponding to the aggregation switch associated with the aggregation identifier, at least in part based on a predetermined correspondence.

[0207] Clause 44: The method described in Clause 43, wherein the correspondence records the relationship between the aggregation identifiers of multiple aggregation switches and the corresponding source and destination port pairs.

[0208] Clause 45: The method described in Clause 41, wherein a particular network topology includes a fat tree topology.

[0209] Clause 46: The method according to Clause 41 further includes: sending each data packet from a first set of processes included in the first computing node to a second set of processes included in the second computing node via multiple different aggregation switches corresponding to multiple different aggregation identifiers assigned to each data packet.

[0210] Clause 47: The method according to Clause 41 further includes: receiving data packets from a second set of processes included in a second computing node by a first set of processes included in a first computing node through a plurality of different aggregation switches corresponding to a plurality of different aggregation identifiers assigned to each data packet.

[0211] Clause 48: One or more machine-readable media storing machine-readable instructions that, when executed by a first computing node, cause the first computing node to perform an action, said action including: determining an aggregation identifier for sending data packets from a first process to a second process according to a node-aware halving and doubling algorithm, the first process and the second process belonging to different nodes connected to different slice switches under a specific network topology; and sending data packets from the first process to the second process through an aggregation switch corresponding to the aggregation identifier.

[0212] Clause 49: One or more machine-readable media as described in Clause 48, wherein said action further includes: assigning different aggregation identifiers to packets directed to nodes connected to different slice switches, such that packets can be routed to nodes connected to different slice switches via different aggregation switches.

[0213] Clause 50: One or more machine-readable media as described in Clause 48, wherein said action further includes: assigning source and destination ports corresponding to an aggregation switch associated with an aggregation identifier, at least in part based on a predetermined correspondence.

[0214] Clause 51: One or more machine-readable media as described in Clause 50, wherein a correspondence record records the relationship between aggregation identifiers of multiple aggregation switches and corresponding source and destination port pairs.

[0215] Clause 52: One or more machine-readable media as described in Clause 48, wherein a particular network topology includes a fat tree topology.

[0216] Clause 53: One or more machine-readable media as described in Clause 48, wherein said action further includes: sending each data packet from a first set of processes included in a first computing node to a second set of processes included in a second computing node via multiple different aggregation switches corresponding to multiple different aggregation identifiers assigned to each data packet.

[0217] Clause 54: One or more machine-readable media as described in Clause 48, wherein said action further includes: receiving data packets from a second set of processes included in a second computing node by a first set of processes included in a first computing node via a plurality of different aggregation switches corresponding to a plurality of different aggregation identifiers assigned to each data packet.

[0218] Clause 55: A first computing node comprising: one or more processing units; and a memory storing machine-executable instructions that, when executed by the one or more processing units, cause the one or more processing units to perform actions, said actions including: determining an aggregation identifier for sending data packets from a first process to a second process according to a node-aware halving and doubling algorithm, the first process and the second process belonging to different nodes connected to different slice switches under a specific network topology; and sending data packets from the first process to the second process through an aggregation switch corresponding to the aggregation identifier.

[0219] Clause 56: The first computing node as described in Clause 55, wherein the action further includes: assigning different aggregation identifiers to packets directed to nodes connected to different slice switches, so that packets can be routed to nodes connected to different slice switches via different aggregation switches.

[0220] Clause 57: The first computing node as described in Clause 55, wherein said action further includes: allocating source and destination ports corresponding to the aggregation switch associated with the aggregation identifier, at least in part based on a predetermined correspondence.

[0221] Clause 58: The first computing node as described in Clause 57, wherein the correspondence records the relationship between the aggregation identifiers of multiple aggregation switches and the corresponding source and destination port pairs.

[0222] Clause 59: The first computing node as described in Clause 55, wherein the action further includes: sending each data packet from a first set of processes included in the first computing node to a second set of processes included in the second computing node via multiple different aggregation switches corresponding to multiple different aggregation identifiers assigned to each data packet.

[0223] Clause 60: The first computing node as described in Clause 55, wherein the action further includes: receiving data packets from a second set of processes included in the second computing node by a first set of processes included in the first computing node through a plurality of different aggregation switches corresponding to a plurality of different aggregation identifiers assigned to each data packet.

[0224] Clause 61: A method implemented by a first computing node, the method comprising: dividing a data packet allocated to a processing unit into a plurality of data segments, the plurality of data segments including at least a first data segment and a second data segment; allocating the plurality of data segments to a plurality of threads, the plurality of threads including at least a first thread and a second thread; performing intra-node suboperations on a portion of the first data segment using the first thread, and performing inter-node suboperations on a portion of the second data segment in parallel using the second thread.

[0225] Clause 62: The method according to Clause 61, wherein performing an intra-node suboperation on a portion of a first data segment using a first thread comprises: transmitting that portion of the first data segment between a processing unit included in a first computing node and another processing unit via an intra-node connection.

[0226] Clause 63: The method according to Clause 61, wherein performing inter-node suboperations on a portion of the second data segment using a second thread comprises: transmitting that portion of the second data segment between a processing unit and another processing unit included in a second computing node different from the first computing node via an inter-node connection.

[0227] Clause 64: The method according to Clause 61, wherein intra-node suboperations include: a reduction scatter suboperation or a global aggregation suboperation performed within a first computing node, and inter-node suboperations include a global reduction suboperation performed between the first computing node and a second computing node different from the first computing node.

[0228] Clause 65: The method according to Clause 61, wherein intra-node suboperations include global aggregate suboperations or copy suboperations performed within a first compute node, and inter-node suboperations include global aggregate suboperations performed between the first compute node and a second compute node different from the first compute node.

[0229] Clause 66: The method described in accordance with Clause 66 further includes: performing another internode suboperation on a portion of a first data segment using a first thread, and performing another intranode suboperation on a portion of a second data segment in parallel using a second thread.

[0230] Clause 67: The method according to Clause 61, wherein a first thread performs intra-node suboperations on a portion of a first data segment, and a second thread performs inter-node suboperations on a portion of a second data segment in parallel, such that an intra-node join is used to transfer a portion of the first data segment to another processing unit included in the first computing node, and an inter-node join is used concurrently to transfer a portion of the second data segment to another processing unit included in a second computing node different from the first computing node.

[0231] Clause 68: One or more machine-readable media storing machine-readable instructions that, when executed by a first computing node, cause the first computing node to perform actions, said actions including: dividing a data packet allocated to a processing unit into multiple data segments, the multiple data segments including at least a first data segment and a second data segment; allocating the multiple data segments to multiple threads, the multiple threads including at least a first thread and a second thread; using the first thread to perform intra-node sub-operations on a portion of the first data segment, and using the second thread in parallel to perform inter-node sub-operations on a portion of the second data segment.

[0232] Clause 69: One or more machine-readable media as described in Clause 68, wherein performing an intra-node suboperation on a portion of a first data segment using a first thread includes transmitting that portion of the first data segment between another processing unit included in the first computing node and the processing unit via an intra-node connection.

[0233] Clause 70: One or more machine-readable media as described in Clause 68, wherein performing inter-node suboperations on a portion of a second data segment using a second thread comprises: transmitting that portion of the second data segment between a processing unit and another processing unit included in a second computing node different from the first computing node via an inter-node connection.

[0234] Clause 71: One or more machine-readable media as described in Clause 68, wherein intra-node suboperations include reduction scattering suboperations or global aggregation suboperations performed within a first computing node, and inter-node suboperations include global reduction suboperations performed between the first computing node and a second computing node different from the first computing node.

[0235] Clause 72: One or more machine-readable media as described in Clause 68, wherein intra-node suboperations include global aggregation suboperations or copy suboperations performed within a first compute node, and inter-node suboperations include global aggregation suboperations performed between the first compute node and a second compute node other than the first compute node.

[0236] Clause 73: The action further comprises using a first thread to perform another internode suboperation on a portion of a first data segment, and using a second thread in parallel to perform another intranode suboperation on a portion of a second data segment.

[0237] Clause 74: One or more machine-readable media pursuant to Clause 68, wherein a first thread performs intra-node suboperations on a portion of a first data segment, and a second thread performs inter-node suboperations on a portion of a second data segment in parallel, such that an intra-node join is used to transfer a portion of the first data segment to another processing unit included in a first computing node, and an inter-node join is used concurrently to transfer a portion of the second data segment to another processing unit included in a second computing node different from the first computing node.

[0238] Clause 75: A first computing node includes: one or more processing units; and a memory storing machine-executable instructions that, when executed by the one or more processing units, cause the one or more processing units to perform actions, said actions including: dividing a data packet allocated to a processing unit into multiple data segments, the multiple data segments including at least a first data segment and a second data segment; allocating the multiple data segments to multiple threads, the multiple threads including at least a first thread and a second thread; performing intra-node sub-operations on a portion of the first data segment using the first thread, and performing inter-node sub-operations on a portion of the second data segment in parallel using the second thread.

[0239] Clause 76: The first computing node according to Clause 75, wherein performing an intra-node suboperation on a portion of a first data segment using a first thread includes: transmitting that portion of the first data segment between another processing unit included in the first computing node and the processing unit via an intra-node connection.

[0240] Clause 77: The first computing node according to Clause 75, wherein performing inter-node suboperations on a portion of the second data segment using a second thread includes: transmitting that portion of the second data segment between a processing unit and another processing unit included in a second computing node different from the first computing node via an inter-node connection.

[0241] Clause 78: The first computing node as described in Clause 75, wherein intra-node suboperations include reduction scatter suboperations or global aggregation suboperations performed within the first computing node, and inter-node suboperations include global reduction suboperations performed between the first computing node and a second computing node different from the first computing node.

[0242] Clause 79: The first computing node as described in Clause 75, wherein intra-node suboperations include global aggregate suboperations or copy suboperations performed within the first computing node, and inter-node suboperations include global aggregate suboperations performed between the first computing node and a second computing node different from the first computing node.

[0243] Clause 80: A first computing node as described in Clause 75, wherein a first thread performs intra-node suboperations on a portion of a first data segment, and a second thread performs inter-node suboperations on a portion of a second data segment in parallel, such that an intra-node join is used to transfer a portion of the first data segment to another processing unit included in the first computing node, and an inter-node join is used concurrently to transfer a portion of the second data segment to another processing unit included in a second computing node different from the first computing node.

Claims

1. A method implemented by a first computing node, the method comprising: determining, according to a node-aware halving doubling algorithm and based on a rank identifier of a first process, a node identifier of the first computing node, a number of network interface controllers per computing node, and a maximum number of computing nodes per slice switch, a rendezvous identifier for sending a data packet from the first process to a second process, the first process and the second process belonging to different nodes connected to different slice switches under a network topology; and sending the data packet from the first process to the second process through a rendezvous switch corresponding to the rendezvous identifier.

2. The method of claim 1, further comprising: different rendezvous identifiers are assigned to data packets directed to nodes connected to different slice switches to enable the data packets to be routed through different rendezvous switches to the nodes connected to the different slice switches.

3. The method of claim 1, further comprising: source ports and destination ports corresponding to a rendezvous switch associated with the rendezvous identifier are assigned based at least in part on a predetermined correspondence.

4. The method of claim 3, wherein the correspondence records relationships between rendezvous identifiers of a plurality of rendezvous switches and corresponding pairs of source ports and destination ports.

5. The method of claim 1, wherein the network topology comprises a fat tree topology.

6. The method of claim 1, further comprising: different rendezvous switches corresponding to a plurality of different rendezvous identifiers assigned to respective data packets, from a first set of processes included in the first computing node to a second set of processes included in a second computing node.

7. The method of claim 1, further comprising: different rendezvous switches corresponding to a plurality of different rendezvous identifiers assigned to respective data packets, from a first set of processes included in the first computing node to a second set of processes included in a second computing node.

8. One or more machine-readable media storing machine-readable instructions that, when executed by a first computing node, cause the first computing node to perform acts comprising: determining, according to a node-aware halving doubling algorithm and based on a rank identifier of a first process, a node identifier of the first computing node, a number of network interface controllers per computing node, and a maximum number of computing nodes per slice switch, a rendezvous identifier for sending a data packet from the first process to a second process, the first process and the second process belonging to different nodes connected to different slice switches under a network topology; and sending the data packet from the first process to the second process through a rendezvous switch corresponding to the rendezvous identifier.

9. The one or more machine-readable media of claim 8, wherein the acts further comprise: different rendezvous identifiers are assigned to data packets directed to nodes connected to different slice switches to enable the data packets to be routed through different rendezvous switches to the nodes connected to the different slice switches.

10. The one or more machine-readable media of claim 8, wherein the acts further comprise: source ports and destination ports corresponding to a rendezvous switch associated with the rendezvous identifier are assigned based at least in part on a predetermined correspondence.

11. The one or more machine-readable media of claim 10, wherein the correspondence records relationships between rendezvous identifiers of a plurality of rendezvous switches and corresponding pairs of source ports and destination ports.

12. The one or more machine-readable media of claim 8, wherein the network topology comprises a fat-tree topology.

13. The one or more machine-readable media of claim 8, wherein the acts further comprise: transmit, from a first set of processes included in the first computing node, to a second set of processes included in a second computing node, the data packets via a plurality of different aggregation switches corresponding to a plurality of different aggregation identifiers assigned to the data packets.

14. The one or more machine-readable media of claim 8, wherein the acts further comprise: receive, by a first set of processes included in the first computing node, from a second set of processes included in a second computing node, the data packets via a plurality of different aggregation switches corresponding to a plurality of different aggregation identifiers assigned to the data packets.

15. A first computing node, comprising: one or more processing units; and memory storing machine executable instructions that, when executed by the one or more processing units, cause the one or more processing units to perform acts comprising: determining, according to a node-aware halving doubling algorithm and based on a rank identifier of a first process, a node identifier of the first computing node, a number of network interface controllers per computing node, and a maximum number of computing nodes per slice switch, an aggregation identifier for transmitting a data packet from the first process to a second process, the first process and the second process belonging to different nodes connected to different slice switches under a network topology; and transmitting, from the first process to the second process, the data packet via an aggregation switch corresponding to the aggregation identifier.

16. The first compute node of claim 15, wherein the acts further comprise: assigning different aggregation identifiers to data packets directed to nodes connected to different slice switches to enable routing of the data packets to the nodes connected to the different slice switches via different aggregation switches.

17. The first compute node of claim 15, wherein the acts further comprise: assigning, based at least in part on a predetermined correspondence, a source port and a destination port corresponding to an aggregation switch associated with the aggregation identifier.

18. The first computing node of claim 17, wherein the correspondence records relationships between aggregation identifiers of a plurality of aggregation switches and corresponding pairs of source ports and destination ports.

19. The first compute node of claim 15, wherein the acts further comprise: transmit, from a first set of processes included in the first computing node, to a second set of processes included in a second computing node, the data packets via a plurality of different aggregation switches corresponding to a plurality of different aggregation identifiers assigned to the data packets.

20. The first compute node of claim 15, wherein the acts further comprise: receive, by a first set of processes included in the first computing node, from a second set of processes included in a second computing node, the data packets via a plurality of different aggregation switches corresponding to a plurality of different aggregation identifiers assigned to the data packets.

21. A first computing node, comprising: a determining module to determine, according to a node-aware halving doubling algorithm and based on a rank identifier of a first process, a node identifier of the first computing node, a number of network interface controllers per computing node, and a maximum number of computing nodes per slice switch, an aggregation identifier for transmitting a data packet from the first process to a second process, the first process and the second process belonging to different nodes connected to different slice switches under a network topology; and a transmitting module to transmit, from the first process to the second process, the data packet via an aggregation switch corresponding to the aggregation identifier. A sending module is configured to send the data packet from the first process to the second process through a aggregation switch corresponding to the aggregation identifier.

22. A computer program product comprising a computer program which, when executed in a computer, is adapted to implement the method according to any one of claims 1 to 7.

Images