Ensemble communication algorithm supporting network topology awareness

Abstract

The invention discloses a set communication algorithm supporting network topology awareness, which comprises the following steps: a connection relation analysis algorithm: embodying an interconnection relation and a hierarchical relation between nodes through a special two-dimensional matrix to obtain a network topology; a logical topology generation algorithm: after the connection relationship is determined, searching the logical topology based on the connection relationship, searching from inside to outside, and generating an algorithm candidate set; preferentially selecting an overhead model scheme: preferentially selecting an overhead model in the generated algorithm candidate set, and selecting a set communication algorithm with the least time consumption; and executing the scheme. According to the method, for a given physical topology, from an end side link, how links are combined and interconnected is analyzed, a potential implementation algorithm candidate set is searched and constructed, a set of overhead model is proposed, combination of candidate algorithms in multiple sets of candidate sets is traversed, estimated time is calculated, and an optimal scheme is obtained. And the user deploys an optimal scheme and executes set communication.

Description

Technical Field

[0001] The present invention relates to the field of collective communication technology, and in particular to a collective communication algorithm supporting network topology perception. Background Art

[0002] With the rise of large language model technology, research and application in the field of deep learning have gradually focused on training models with large-scale parameters. As the number of model parameters continues to increase, a single GPU cannot support the training of the entire model. Therefore, distributed clusters have become a common choice for training these models. In this process, each GPU card must communicate efficiently to exchange data; collective communication, as a global communication operation, in which each process participates in the operation, is widely used in data exchange in model training. Collective communication is combined into a variety of primitives through basic operations (such as sending, receiving, copying, etc.), including one-to-many broadcast (Broadcast), many-to-one collection (Gather), many-to-many collection (All-Gather), one-to-many divergence (Scatter), many-to-one reduction (Reduce), many-to-many reduction (All-Reduce), combined reduction and divergence (Reduce-Scatter), and many-to-many communication (All-to-All).

[0003] NCCL (NVIDIA Collective Communications Library) is a high-performance communication library provided by NVIDIA, designed for accelerating multi-GPU and multi-node deep learning training tasks. It supports collective communication operations (such as Broadcast, All-Reduce, Reduce-Scatter, All-Gather, and point-to-point communication), and can efficiently transmit data between multiple GPUs in the same server or across multiple GPU clusters of nodes. The performance of collective communication is strongly related to the actual network system and implementation algorithm. In practical scenarios, the actual requirements of the model and the hardware architecture are variable, and collective communication may face heterogeneous, hierarchical, asymmetric, and even fragmented scenarios. However, NCCL only perceives the physical topology and generates the logical topology of collective communication based on a few algorithms and empirical rules. This makes it difficult for NCCL to truly and efficiently utilize network performance in some scenarios.

[0004] In recent years, some new collective communication libraries have tried to optimize the performance of collective communication by optimizing the implementation algorithm of collective communication. For example, SCCL can generate the optimal logical topology on any topology and support multiple communication operators. Its main idea is to convert the topological problem into a mathematical expression and then use the SMT solver to solve it. In theory, it can find the optimal solution, but the computational overhead is too high, and the design architecture does not support switches. TACCL solves the performance problem and the problem of incompatibility with switches based on SCCL. The cost is that the algorithm needs to input experience instructions manually, and the conclusions drawn are not necessarily optimal. It is found that TACCL still faces the problem of too high overhead and is difficult to play a role at the level of thousands or tens of thousands of cards. ForestColl further implements the generation of the optimal Allreduce series algorithm on any topology. Its main idea is to first use graph partitioning to find the theoretical lower bound of time consumption, and then use the tree generation algorithm to generate a strategy with a time consumption equal to the lower bound. Through some simplifying assumptions, ForestColl reduces the NP-hard problem to polynomial complexity, but in fact the complexity is still very high.

[0005] In general, the implementation algorithm of collective communication greatly determines the performance of collective communication, and thus determines the performance of distributed model training; NCCL, as the most mature collective communication library at present, is not flexible enough in its implementation algorithm, resulting in the inability to fully utilize network performance in some topologies; SCCL, TACCL, ForestColl and other related works have implemented perceptual topology based on NCCL, and have improved performance compared to NCCL in some topologies. However, there are problems such as the need for human input of empirical knowledge and high algorithm complexity. Summary of the invention

[0006] The purpose of the present invention is to provide a collective communication algorithm supporting network topology awareness to solve the problems raised in the above background technology.

[0007] To achieve the above object, the present invention provides the following technical solution: a collective communication algorithm supporting network topology awareness, comprising the following steps:

[0008] S1. Connection relationship analysis algorithm: A special two-dimensional matrix is ​​used to reflect the interconnection and hierarchical relationship between nodes to obtain the network topology;

[0009] S2, logical topology generation algorithm: After the connection relationship is determined, the logical topology is searched based on the connection relationship, and the search is performed "from the inside to the outside" to generate an algorithm candidate set;

[0010] S3, cost model scheme selection: among the generated algorithm candidate set, use the cost model to select the optimal solution and select the collective communication algorithm with the least time consumption;

[0011] S4. Solution execution: The output results of the logical topology generation algorithm are processed to generate an XML format file that can be received by MSCCL to actually deploy the optimal set communication implementation algorithm obtained by selection.

[0012] Preferably, the step S1 specifically includes:

[0013] S11. First, find each independent communication node, enclose it in the innermost "bracket", and connect it directly through the switch. This relationship is defined as 1 layer, which is wrapped by a bracket;

[0014] S12, layers are interconnected through a switch to form a larger layer.

[0015] Preferably, any two nodes among the communication nodes can communicate.

[0016] Preferably, the step S3 uses a cost model to select a cost analysis solution and a measurement solution:

[0017] Cost analysis scheme: clarify all factors that constitute the delay in collective communication: fixed network delay, data propagation time in the network, memory read and write, and computing time. The fixed network delay is constant, and the other factors are related to the network bandwidth, memory read and write speed, computing speed, and data volume. Therefore, we record the total delay of a communication as: fixed network delay + data volume / available network bandwidth + data volume / memory read and write speed + data volume / computation speed. When multiple data pass through the network / memory / computation at the same time, resources will be shared, and the speed in different implementation algorithms will be evenly divided according to the degree of parallelism. The result of this formula is used as the cost;

[0018] Measurement scheme: Based on MSCCL, the performance of the aggregate communication is actually measured at a given data volume, which serves as the actual measurement value of the overhead.

[0019] Preferably, the searching logical topology is to search for possible collective communication implementation methods for each layer.

[0020] Preferably, the nodes perform collective communication via a Ring algorithm or a Mesh algorithm, and the two layers are connected via a Ring algorithm or a Mesh algorithm.

[0021] Compared with the prior art, the present invention has the following beneficial effects:

[0022] For a given physical topology, it can analyze how these links are combined and interconnected starting from the end-side links, propose a set of cost models by searching and building a set of potential implementation algorithm candidates, traverse the combinations of candidate algorithms in multiple sets of candidate sets and calculate the estimated time to obtain the optimal solution. Users can then deploy the optimal solution and perform collective communication. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is the overall framework and flow chart;

[0024] Figure 2 This is a sample diagram for connection relationship analysis;

[0025] Figure 3 Generate a sample graph for the logical topology. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] See also Figure 1-3 The present invention provides a technical solution: a collective communication algorithm supporting network topology awareness, comprising the following steps:

[0028] S1. Connection relationship analysis algorithm: A special two-dimensional matrix is ​​used to reflect the interconnection and hierarchical relationship between nodes to obtain the network topology; any topology can be abstracted and expressed through this method. Figure 2 Taking the first physical topology and the connection relationship defined in the present invention as an example, in the topology where four nodes 0, 1, 2, and 3 are connected to the same switch to achieve interconnection, the switch is equivalent to a relay forwarding node and can be ignored. According to our definition of connection relationship, the four nodes 0-4 are independent individuals, so they can be defined as (0), (1), (2), (3). Based on this example, we give the first definition of modeling connection relationship: First, find each independent communication node and enclose it in the innermost "bracket".

[0029] On this basis, these four nodes can be directly connected through switches and can communicate with each other. We define this relationship as 1 layer, wrapped in a bracket, that is, ((0), (1), (2), (3)). Considering that in the ring implementation algorithm, the data flow direction needs to be defined, that is, the data can be implemented through 0->1, 1->2, 2->3, 3->0, or through 0->3, 3->2, 2->1, 1->0. In the layer, it is necessary to additionally define the relationship of full interconnection. For ((0), (1), (2), (3)), if the definition of the reverse order ((3), (2), (1), (0)) is not given, it is fully interconnected, that is, any two nodes between the nodes can communicate. For non-fully interconnected relationships, we need additional rows to express the data flow direction. For example Figure 2 In the second row, 0, 1, 2, and 3 are connected to two other nodes with adjacent serial numbers, respectively, rather than being fully interconnected. For example, 1 and 3 cannot communicate directly. We need to give all possible data flows, so in addition to ((0), (1), (2), (3)), we also need to give a connection relationship such as ((3), (2), (1), (0)). Therefore, the second definition of modeling connection relationships: for the set of nodes that are directly connected, connected through a switch, and connected through other nodes in the range, it is defined as "a layer" and can be wrapped in a "bracket". In addition, if the nodes in the layer cannot be interconnected (they must be connected through other nodes in the layer), all possibilities of forming a ring need to be given.

[0030] Layers can be combined to form a larger layer, for example Figure 2 The topology of the third row. 0, 1, 2 form a layer, i.e. ((0), (1), (2)), and 3, 4, 5 also form ((3), (4), (5)). For these two layers, they are also connected through a switch, so they form a larger layer, which is also the entire network topology. Therefore, this network topology can be defined as (((0), (1), (2)), ((3), (4), (5))).

[0031] Through the above two definitions, we can define all physical topologies.

[0032] S2, logical topology generation algorithm: After determining the connection relationship, search the logical topology based on the connection relationship, search "from the inside to the outside" to generate an algorithm candidate set; for example, Figure 3 The first line shows how the algorithm searches from the inside out. In the topology ((0), (1), (2), (3)), the innermost layer is the four nodes 0-3 that need to communicate collectively. They can be implemented using the Ring algorithm or the Mesh algorithm. Because the topology only contains one layer, only two algorithms are found.

[0033] For a more complex example, as shown in the second row of Figure 3, the topology (((0),(1),(2)),((3),(4),(5))) also starts from the innermost layer. The innermost layer has two "layers" ((0),(1),(2)) and ((3),(4),(5)). These two layers can be connected through Ring or Mesh, corresponding to two situations. In order to perform collective communication globally, the two layers need to communicate further, that is, the two layers composed of 0, 1, 2 and 3, 4, 5 need to communicate, that is, 0-3, 1-4, 2-5 also need to communicate, which can also be achieved through Ring or Mesh. Then there are a total of 2*2=4 situations. In addition, the six nodes from 0 to 5 can form a complete Ring or Mesh, so there are also two implementation algorithms. Then, under this logical topology generation algorithm, 6 algorithms are searched for the topology (((0),(1),(2)),((3),(4),(5))).

[0034] S3, cost model scheme selection: among the generated algorithm candidate set, use the cost model to select the optimal solution and select the collective communication algorithm with the least time consumption;

[0035] S4. Solution execution: The output results of the logical topology generation algorithm are processed to generate an XML format file that can be received by MSCCL to actually deploy the optimal set communication implementation algorithm obtained by selection.

[0036] In the present invention, the step S1 specifically includes:

[0037] S11. First, find each independent communication node, enclose it in the innermost "bracket", and connect it directly through the switch. This relationship is defined as 1 layer, which is wrapped by a bracket;

[0038] S12, layers are interconnected through a switch to form a larger layer.

[0039] In the present invention, any two nodes between the communication nodes can communicate.

[0040] In the present invention, the step S3 uses the cost model to select the cost analysis scheme and the measurement scheme:

[0041] Cost analysis scheme: clarify all factors that constitute the delay in collective communication: fixed network delay, data propagation time in the network, memory read and write, and computing time. The fixed network delay is constant, and the other factors are related to the network bandwidth, memory read and write speed, computing speed, and data volume. Therefore, we record the total delay of a communication as: fixed network delay + data volume / available network bandwidth + data volume / memory read and write speed + data volume / computation speed. When multiple data pass through the network / memory / computation at the same time, resources will be shared, and the speed in different implementation algorithms will be evenly divided according to the degree of parallelism. The result of this formula is used as the cost;

[0042] Measurement scheme: Based on MSCCL, the performance of the aggregate communication is actually measured at a given data volume, which serves as the actual measurement value of the overhead.

[0043] In the present invention, the search logic topology is to search for possible collective communication implementation methods for each layer.

[0044] In the present invention, the nodes perform collective communication through the Ring algorithm or the Mesh algorithm, and the two layers are connected through the Ring algorithm or the Mesh algorithm.

[0045] For a given physical topology, the present invention starts from the end-side links and analyzes how these links are combined and interconnected (connection relationships). A set of connection relationships describes how a link on each end is interconnected. When there are multiple links on an end, multiple sets of connection relationships are obtained, and then potential candidate sets of implementation algorithms are searched and constructed. Each set of connection relationships can generate several collective communication algorithm candidates (logical topologies). The candidate sets generated by multiple sets of connection relationships are independent of each other. Each candidate logical topology tries its best to utilize the end-side links corresponding to its subordinate connection relationships, proposes a set of overhead models, traverses the combinations of candidate algorithms in multiple sets of candidate sets and calculates the estimated time to obtain the optimal solution. The user deploys the optimal solution and executes collective communication.

[0046] The contents not described in detail in this specification belong to the prior art known to professional and technical personnel in the field. Although the embodiments of the present invention have been shown and described, it is understood by ordinary technicians in the field that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principle and spirit of the present invention. The scope of the present invention is defined by the attached claims and their equivalents.

Claims

1. A collective communication algorithm supporting network topology awareness, characterized in that: The steps include: S1. Connection relationship analysis algorithm: A special two-dimensional matrix is ​​used to reflect the interconnection and hierarchical relationship between nodes to obtain the network topology; S2, logical topology generation algorithm: After the connection relationship is determined, the logical topology is searched based on the connection relationship, searching "from the inside to the outside" to generate an algorithm candidate set; S3, cost model scheme selection: among the generated algorithm candidate set, use the cost model to select the optimal solution and select the collective communication algorithm with the least time consumption; S4. Solution execution: The output results of the logical topology generation algorithm are processed to generate an XML format file that can be received by MSCCL to actually deploy the optimal set communication implementation algorithm obtained by selection.

2. The collective communication algorithm supporting network topology awareness according to claim 1, characterized in that: The step S1 specifically includes: S11. First, find each independent communication node, enclose it in the innermost "bracket", and connect it directly through the switch. This relationship is defined as 1 layer, which is wrapped by a bracket; S12, layers are interconnected through a switch to form a larger layer.

3. The collective communication algorithm supporting network topology awareness according to claim 2, characterized in that: Any two nodes between the communication nodes can communicate.

4. The collective communication algorithm supporting network topology awareness according to claim 1, characterized in that: The step S3 uses the cost model to select the best cost analysis solution and the measurement solution: Cost analysis scheme: clarify all factors that constitute the delay in collective communication: fixed network delay, data propagation time in the network, memory read and write, and computing time. The fixed network delay is constant, and the other factors are related to the network bandwidth, memory read and write speed, computing speed, and data volume. Therefore, we record the total delay of a communication as: fixed network delay + data volume / available network bandwidth + data volume / memory read and write speed + data volume / computation speed. When multiple data pass through the network / memory / computation at the same time, resources will be shared, and the speed in different implementation algorithms will be evenly divided according to the degree of parallelism. The result of this formula is used as the cost; Measurement scheme: Based on MSCCL, the performance of the aggregate communication is actually measured at a given data volume, which serves as the actual measurement value of the overhead.

5. The collective communication algorithm supporting network topology awareness according to claim 1, characterized in that: The searching logical topology is to search for possible collective communication implementation methods for each layer.

6. The collective communication algorithm supporting network topology awareness according to claim 1, characterized in that: The nodes communicate collectively through the Ring algorithm or the Mesh algorithm, and the two layers are connected through the Ring algorithm or the Mesh algorithm.

Images