Routing optimization method for distributed training cluster communication

Abstract

The invention discloses a routing optimization method for distributed training cluster communication. The method comprises a flow analysis component and a routing optimization component which are both realized on a centralized control plane of the software defined network. Wherein the flow analysis component needs to perceive a communication mode specifically adopted by the application layer and node distribution set by a network administrator, so that related flow information is obtained, and the related flow information comprises a starting point and a destination point of each flow, a transmitted data volume and a dependency relationship among flows. And the routing optimization component realizes non-blocking routing distribution based on the obtained task flow information and the network topology. In a distributed machine learning-oriented data center training cluster, by sensing the task cluster communication traffic and performing routing optimization, overall non-blocking traffic communication is realized, the communication overhead of training is reduced, the task completion time is further prolonged, and the method has application value in current mainstream large model training and other scenes.

Description

Technical Field

[0001] The present invention belongs to the field of network layer routing optimization of data center networks and relates to a routing optimization method for distributed training cluster communication. Background Art

[0002] Deep learning has achieved remarkable results in various applications. Distributed machine learning has been widely adopted to handle the ever-increasing datasets and model parameters by distributing large tasks in training clusters. Data parallelism is a common approach in distributed training, where each computing node trains a subset of the dataset locally and performs parameter synchronization globally. To achieve parameter synchronization, All-Reduce is a widely used cluster communication mode, which contains multiple stages of communication traffic and involves a large amount of data exchange. In order to meet the huge computing power and intensive communication requirements, data center networks have become key infrastructures, and the topology and transmission protocols have a great impact on the communication cost of distributed training tasks. Among them, Fat-Tree is the main topology used in training clusters, which fully utilizes the capacity of switches and provides multiple equivalent paths for traffic. However, the advantage of multi-path also brings a major challenge: multi-path routing, especially for parameter synchronization tasks with multi-stage communication.

[0003] In data center networks, the routing mechanism determines the transmission path of data traffic. In the case of improper routing, concurrent traffic will be routed to the same link, resulting in load imbalance and bandwidth competition. Tail traffic with unbalanced available bandwidth will seriously affect the completion time of the overall task. Existing routing technologies can make more detailed adjustments through feedback information from the network, but the feedback delay will affect the effect of the adjustment. There are also routing technologies specifically optimized for distributed training traffic, but they are only applicable to specific node allocations and cannot cope with random traffic distribution. Summary of the invention

[0004] In order to solve the problems existing in the background technology, the present invention proposes a routing optimization scheme for distributed training cluster communication, which adopts a task traffic-aware heuristic global routing algorithm, solves the problem that the existing routing algorithms are not optimized for task completion time and cannot be applied to random node allocation, and achieves the goal of task completion time with performance guarantee in distributed cluster communication.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] A routing optimization method for distributed training cluster communication, which is implemented in the control plane of software-defined network (SDN), includes a traffic analysis component and a routing optimization component; wherein the traffic analysis component is responsible for obtaining relevant information of the collective communication traffic from the application layer, including the adopted All-Reduce algorithm and the distribution of nodes; and the routing optimization component is responsible for path planning according to the distribution of traffic and network topology.

[0007] Specifically, the traffic analysis component perceives the communication mode adopted by the application layer and the node allocation set by the network administrator, and then obtains relevant traffic information, including the starting point, destination, amount of data transmitted, and dependencies between flows of each flow; there are two All-Reduce communication modes used in distributed training, namely Ring-based communication and Halving-Doubling mode; which are used to achieve parameter synchronization.

[0008] Specifically, the dependency is that in an All-Reduce communication process, there will be several stages of traffic, and whether the traffic in the current stage can be transmitted needs to consider whether the previous stage has completed the transmission; when the traffic analysis component obtains relevant information, it will be given to the routing optimization component for routing allocation.

[0009] Specifically, the routing optimization component, as a receiving module of the traffic analysis component, adopts an optimization method based on integer linear programming to realize non-blocking routing allocation based on the transmission task traffic and network topology; and uses the output of the traffic analysis component as input to perform unified routing planning for the traffic, and then distributes the final routing solution to each switch in the form of a routing table to complete routing optimization.

[0010] Furthermore, the optimization method based on integer linear programming is used to implement non-blocking routing allocation based on the task flow and network topology of the transmission, which specifically includes the following steps:

[0011] (a) First, the node connections of the Fat-Tree topology are represented in the form of an adjacency matrix, and the communication traffic of All-Reduce is represented as a binary traffic matrix;

[0012] (b) Then, the unique path selection for each flow is set as a constraint, and the independent variable is the routing matrix of all flows;

[0013] (c) An integer linear programming model is constructed with the goal of minimizing the number of conflicting flows on the network path. Based on the flow matrix and constraints, the integer linear programming model is solved to obtain a non-blocking routing solution for each flow.

[0014] Furthermore, the optimization method of integer linear programming also includes using a greedy strategy routing algorithm based on minimum flow priority to perform routing allocation of task traffic to obtain path planning for each traffic, specifically:

[0015] (a) The network topology is a two-layer Fat-Tree of Clos architecture, where half of the ports of each access layer switch are connected to the computing nodes downward, and the other half of the ports are connected to the core layer switches upward. Each core layer switch is connected to all access layers;

[0016] (b) Count the number of flows that each access layer switch needs to transmit, and then sort them in ascending order according to the number of flows to obtain the allocation order of the access layer switches;

[0017] (c) Select a flow from each core layer switch in the sorted order of access layer switches, and obtain the routing allocation of the traffic through a traversal operation.

[0018] Specifically, when the routing optimization component distributes traffic, the traffic under the same access layer switch does not need to be specially routed, and only the traffic across switches needs to be considered; after each core layer switch has completed the distribution, the access layer switches need to be reordered according to the number of remaining flows; for the unallocated traffic that remains after the traversal is completed, it is spread to different core layer switches to ensure that no more than two flows compete for bandwidth.

[0019] The beneficial effects of the present invention are:

[0020] The present invention can be used in a data center training cluster for distributed machine learning. It uses a centralized control plane of a software-defined network to sense the communication traffic of a task cluster, perform routing optimization, and achieve overall non-blocking traffic communication, thereby reducing the communication overhead of training and further improving the task completion time. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a schematic diagram of the overall system architecture of the present invention;

[0022] Figure 2 It is a schematic diagram of the communication patterns involved in the traffic analysis component;

[0023] Figure 3 It is a schematic diagram of the greedy algorithm structure of the routing optimization component. DETAILED DESCRIPTION

[0024] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0025] like Figure 1As shown, the present invention mainly includes two components, a traffic analysis component and a routing optimization component. The traffic analysis component is responsible for obtaining the relevant information of the collective communication traffic from the application layer, including the All-Reduce algorithm adopted and the distribution of nodes; the routing optimization component is responsible for path planning according to the distribution of traffic and the network topology. Both are implemented in the control plane of the software defined network (SDN). Among them, the traffic analysis component needs to obtain the type of All-Reduce synchronization algorithm executed by the current distributed training task and the allocation method of the training task nodes in the network topology from the application layer (i.e., the network manager), from which the relevant information of the collective communication traffic is obtained, including the source point, destination point, data volume and dependency between traffic. There are two All-Reduce communication modes used in distributed training, namely, ring-based communication (Ring-based) and halving-doubling (Halving-Doubling) mode, both of which can achieve the effect of parameter synchronization, but the traffic relationship and changes in the specific communication process are different. The routing optimization component realizes non-blocking routing allocation based on the output information of the traffic analysis component and the topological information of the network to improve the completion time of the overall task and reduce communication overhead.

[0026] The overall network topology used is a two-layer Fat-Tree of Clos architecture, in which half of the ports of each access layer switch are connected to the computing nodes downward, and the other half of the ports are connected to the core layer switches upward. Each core layer switch is connected to all access layers. Fat-Tree has good scalability and excellent properties such as completely equal bandwidth division. It has been widely used in distributed training clusters. Due to its strong symmetric structure, it can be concisely represented by an adjacency matrix. The routing optimization component uses an optimization method based on integer linear programming to allocate routes, represents the All-Reduce communication traffic as a binary traffic matrix, and then sets the unique path selection of each flow as a constraint condition. The independent variable is the routing matrix of all traffic, and the link load can be obtained by matrix multiplication.

[0027] like Figure 2As shown in the figure, the All-Reduce communication process based on Halving-Doubling specifically displays the relevant information of the traffic. Taking the communication process of 8 nodes as an example, in each stage of the two-to-two communication, they exchange part of their own parameter information, and then take the sum and average of each other to achieve overall parameter synchronization. In the Halving-Doubling communication mode, the relationship between the node pairs in each stage of communication and the amount of data communicated will change according to certain rules, and the traffic in the later stage will also depend on the previous flow. In general, as long as the node number and distribution are known, the traffic analysis component can obtain the overall traffic information according to the relationship of the communication process, including the source point, destination point, data volume, etc. of the traffic, as a known quantity, as the specific input information of the subsequent routing load.

[0028] like Figure 3 As shown in the figure, the Fat-Tree network structure can be abstracted. For the traffic under the same access layer switch, it does not need to pass through the core layer switch in the middle, but only needs to pass through the access layer switch to which they are connected to complete the transmission. Under the All-Reduce communication traffic, each node only sends or receives one flow at the same time, so there will be no problems such as routing competition and bandwidth conflict. Therefore, routing only needs to consider the traffic across switches.

[0029] First, the routing optimization component uses an optimization method based on integer linear programming to allocate routes and achieve non-blocking traffic transmission. The specific modeling is to represent the node connection of the Fat-Tree topology in the form of an adjacency matrix, and to represent the All-Reduce communication traffic as a binary traffic matrix. Then, the unique path selection of each flow is set as a constraint condition, and the independent variable is the routing matrix of all flows. Then, an integer linear programming model with the minimum number of conflicting flows on the network path as the target value is constructed. Based on the traffic matrix and the constraints, the integer linear programming model is solved to obtain non-blocking routing solutions for each flow. Although this mathematical modeling can be solved using existing tools such as CPLEX, the cost of solving increases when the network is large. Therefore, a greedy strategy routing algorithm based on minimum flow priority is proposed to allocate the routing of task flows and obtain the path planning of each flow.

[0030] This greedy algorithm needs to count the number of flows that each access layer switch needs to transmit, and then sort them in ascending order according to the number of flows to obtain the allocation order of the access layer switches. When selecting routes according to such priorities, it can ensure that all flows are taken into account as much as possible. Then, for each core layer switch, a flow is selected from the sorted access layer switch order. This operation can ensure that there will be no path conflict between the selected flows. Through a single traversal operation, the routing allocation of the traffic can be obtained.

[0031] In some cases, some flows are not allocated. Theoretically, the number of these unallocated flows must be less than the total number of core switches. For these flows, the routing optimization component spreads them to different core switches to ensure that at most two flows will compete for bandwidth, and no worse situation will occur.

[0032] The time complexity of this greedy algorithm is O(NlogN), where N is the total number of nodes used in the training task. Compared with the solution of integer linear programming modeling, this algorithm obtains suboptimal routing allocation that requires theoretical guarantees with acceptable event complexity, which is more suitable for implementation in the control plane.

[0033] The present invention optimizes routing by sensing the communication traffic of task clusters. It can effectively overcome the performance defects of routing algorithms and is compatible with existing software-defined networks and other features. The above specific implementations are used to explain the present invention, rather than to limit the present invention. Any modifications and changes made to the present invention within the spirit of the present invention and the protection scope of the claims fall within the protection scope of the present invention.

Claims

1. A routing optimization method for distributed training cluster communication, characterized in that: The method is implemented in the control plane of the software defined network SDN, including a traffic analysis component and a routing optimization component; The traffic analysis component is responsible for obtaining relevant information about the collective communication traffic from the application layer, including the All-Reduce algorithm used and the distribution of nodes; the routing optimization component is responsible for path planning based on the distribution of traffic and network topology.

2. A routing optimization method for distributed training cluster communication according to claim 1, characterized in that: The traffic analysis component obtains relevant traffic information by sensing the communication mode adopted by the application layer and the node allocation set by the network administrator, including the starting point, destination point, amount of data transmitted, and dependencies between flows of each flow; there are two All-Reduce communication modes used in distributed training, namely Ring-based communication and Halving-Doubling mode; which are used to achieve parameter synchronization.

3. A routing optimization method for distributed training cluster communication according to claim 2, characterized in that: The dependency relationship is that in an All-Reduce communication process, there will be several stages of traffic, and whether the traffic in the current stage can be transmitted needs to consider whether the previous stage has completed the transmission; when the traffic analysis component obtains relevant information, it will be given to the routing optimization component for routing allocation.

4. A routing optimization method for distributed training cluster communication according to claim 1, characterized in that: The routing optimization component, as a receiving module of the traffic analysis component, uses an optimization method based on integer linear programming to achieve non-blocking routing allocation based on the task traffic and network topology of the transmission; The output of the traffic analysis component is used as input to perform unified routing planning for the traffic, and then the final routing solution is distributed to each switch in the form of a routing table to complete routing optimization.

5. A routing optimization method for distributed training cluster communication according to claim 4, characterized in that: The optimization method based on integer linear programming is used to implement non-blocking routing allocation based on the task flow and network topology of the transmission, which specifically includes the following steps: (a) First, the node connections of the Fat-Tree topology are represented in the form of an adjacency matrix, and the communication traffic of All-Reduce is represented as a binary traffic matrix; (b) Then, the unique path selection for each flow is set as a constraint, and the independent variable is the routing matrix of all flows; (c) An integer linear programming model is constructed with the goal of minimizing the number of conflicting flows on the network path. Based on the flow matrix and constraints, the integer linear programming model is solved to obtain a non-blocking routing solution for each flow.

6. A routing optimization method for distributed training cluster communication according to claim 5, characterized in that: The optimization method of integer linear programming also includes using a greedy strategy routing algorithm based on minimum flow priority to perform routing allocation of task traffic to obtain path planning for each traffic, specifically: (a) The network topology is a two-layer Fat-Tree of Clos architecture, where half of the ports of each access layer switch are connected to the computing nodes downward, and the other half of the ports are connected to the core layer switches upward. Each core layer switch is connected to all access layers; (b) Count the number of flows that each access layer switch needs to transmit, and then sort them in ascending order according to the number of flows to obtain the allocation order of the access layer switches; (c) Select a flow from each core layer switch in the sorted order of access layer switches, and obtain the routing allocation of the traffic through a traversal operation.

7. A routing optimization method for distributed training cluster communication according to claim 6, characterized in that: When the routing optimization component distributes traffic, the traffic under the same access layer switch does not need to be specially routed, and only the traffic across switches needs to be considered; after each core layer switch has been distributed, the access layer switches need to be reordered according to the number of remaining flows; for the unallocated traffic that remains after the traversal is completed, it is spread to different core layer switches to ensure that no more than two flows compete for bandwidth.

Images