A routing optimization method for distributed training cluster communication

Abstract

This invention discloses a routing optimization method for distributed training cluster communication. The method comprises two parts: a traffic analysis component and a routing optimization component, both implemented in the centralized control plane of a software-defined network. The traffic analysis component needs to perceive the specific communication mode adopted by the application layer and the node allocation configured by the network administrator to obtain relevant traffic information, including the origin, destination, and amount of data transmitted for each flow, as well as the dependencies between flows. The routing optimization component then performs non-blocking route allocation based on the obtained task traffic information and network topology. This invention, targeting distributed machine learning data center training clusters, achieves overall non-blocking traffic communication by perceiving task cluster communication traffic and optimizing routes, reducing training communication overhead and further improving task completion time. It has application value in current mainstream large-scale model training scenarios.

Description

Technical Field

[0001] This invention belongs to the field of network layer routing optimization in data center networks, and relates to a routing optimization method for distributed training cluster communication. Background Technology

[0002] Deep learning has achieved remarkable results in various applications. Distributed machine learning, by allocating large tasks across training clusters, is widely adopted to handle ever-increasing datasets and model parameters. Data parallelism is a common approach in distributed training, where each compute node trains a subset of the dataset locally and performs parameter synchronization globally. To achieve parameter synchronization, All-Reduce is a widely used cluster communication pattern involving multiple stages of communication traffic and a large amount of data exchange. To meet the enormous computing power and intensive communication demands, data center networks have become a critical infrastructure, and topology and transport protocols have a significant impact on the communication cost of distributed training tasks. Among these, Fat-Tree is a primary topology used in training clusters, fully utilizing switch capacity and providing multiple equivalent paths for traffic. However, the advantages of multi-path routing also bring a significant challenge, especially for parameter synchronization tasks with multi-stage communication.

[0003] In data center networks, routing mechanisms determine the transmission path of data traffic. Inappropriate routing can lead to concurrent traffic being routed onto the same link, resulting in load imbalance and bandwidth contention. Uneven bandwidth tail traffic can severely impact the overall task completion time. Existing routing technologies can be fine-tuned based on network feedback, but the latency of this feedback can affect the effectiveness of these adjustments. There are also routing technologies specifically optimized for distributed training traffic, but these are only suitable for specific node allocations and cannot handle random traffic distributions. Summary of the Invention

[0004] To address the problems existing in the background technology, this invention proposes a routing optimization scheme for distributed training cluster communication. It adopts a task traffic-aware heuristic global routing algorithm, which solves the problems that existing routing algorithms are not optimized for task completion time and cannot be applied to random node allocation, thus achieving the goal of ensuring task completion time in distributed cluster communication.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] A routing optimization method for distributed training cluster communication is proposed. This method is implemented in the control plane of Software-Defined Network (SDN) and includes a traffic analysis component and a routing optimization component. The traffic analysis component is responsible for obtaining relevant information about the cluster communication traffic from the application layer, including the All-Reduce algorithm used and the distribution of nodes. The routing optimization component is responsible for path planning based on the traffic distribution and network topology.

[0007] Specifically, the traffic analysis component perceives the communication mode adopted by the application layer and the node allocation set by the network administrator to obtain relevant traffic information, including the origin, destination, amount of data transmitted, and dependencies between traffic flows for each flow. There are two All-Reduce communication modes used in distributed training: Ring-based communication and Halving-Doubling mode, which are used to achieve parameter synchronization.

[0008] Specifically, the dependency relationship is that in a single All-Reduce communication process, there will be traffic in several stages, and whether the traffic in the current stage can be transmitted depends on whether the previous stage has completed the transmission; when the traffic analysis component obtains relevant information, it will provide it to the routing optimization component for route allocation.

[0009] Specifically, the routing optimization component, as the receiving module of the traffic analysis component, uses an optimization method based on integer linear programming to achieve non-blocking route allocation based on the transmission task traffic and network topology; and takes the output of the traffic analysis component as input to perform unified route planning for the traffic, and then distributes the final routing scheme to each switch in the form of a routing table to complete the routing optimization.

[0010] Furthermore, the optimization method based on integer linear programming achieves non-blocking route allocation based on the transmitted task traffic and network topology, specifically including the following steps:

[0011] (a) First, the node connections of the Fat-Tree topology are represented in the form of an adjacency matrix, and the communication traffic of All-Reduce is represented as a binary traffic matrix;

[0012] (b) Next, the unique path selection for each flow is set as a constraint, with the routing matrix of all flows as the independent variable;

[0013] (c) Construct an integer linear programming model with the objective of minimizing the number of conflicting traffic on the network path. Based on the traffic matrix and constraints, solve the integer linear programming model to obtain the non-blocking routing scheme for each traffic.

[0014] Furthermore, the optimization method for integer linear programming also includes using a greedy routing algorithm based on the least flow priority strategy to allocate task traffic routes, in order to obtain path planning for each traffic flow, specifically as follows:

[0015] (a) The network topology is a two-layer Fat-Tree of Clos architecture, in which half of the ports of each access layer switch are connected to the compute nodes downwards and the other half of the ports are connected to the core layer switch upwards. Each core layer switch is connected to all access layers.

[0016] (b) Count the number of flows that each access layer switch needs to transmit, and then sort them in ascending order according to the number of flows to obtain the allocation order of the access layer switches.

[0017] (c) For each core layer switch, select one flow from the sorted access layer switches in turn. The traffic routing can be obtained through one traversal operation.

[0018] Specifically, when allocating traffic, the routing optimization component does not require dedicated routing for traffic under the same access layer switch; it only needs to consider traffic across switches. After allocating traffic to each core layer switch, the access layer switches need to be reordered according to the number of remaining flows. Any unallocated traffic after the traversal is completed is distributed to different core layer switches to ensure that no more than two flows compete for bandwidth.

[0019] The beneficial effects of this invention are:

[0020] This invention can be used in data center training clusters for distributed machine learning. By utilizing the centralized control plane of software-defined networking, it can sense the communication traffic of the task cluster, optimize routing, and achieve overall non-blocking traffic communication, thereby reducing the communication overhead of training and further improving the task completion time. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the overall system architecture of the present invention;

[0022] Figure 2 This is a schematic diagram of the communication modes involved in the traffic analysis component;

[0023] Figure 3 This is a schematic diagram of the greedy algorithm structure of the routing optimization component. Detailed Implementation

[0024] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0025] like Figure 1As shown, this invention mainly comprises two components: a traffic analysis component and a route optimization component. The traffic analysis component is responsible for obtaining relevant information about aggregate communication traffic from the application layer, including the All-Reduce algorithm used and the distribution of nodes. The route optimization component is responsible for path planning based on the traffic distribution and network topology. Both are implemented in the control plane of Software-Defined Networking (SDN). Specifically, the traffic analysis component needs to obtain the type of All-Reduce synchronization algorithm executed by the current distributed training task and the allocation method of training task nodes in the network topology from the application layer (i.e., the network administrator). From this, it learns relevant information about aggregate communication traffic, including the source, destination, data volume, and dependencies between traffic flows. There are two All-Reduce communication modes used in distributed training: ring-based and halving-doubling modes. Both can achieve parameter synchronization, but the specific traffic relationships and changes during the communication process differ. The route optimization component, based on the output information of the traffic analysis component and the network topology information, implements non-blocking route allocation to improve the overall task completion time and reduce communication overhead.

[0026] The overall network topology used is a two-layer Fat-Tree with a Clos architecture. Each access layer switch has half of its ports connected downwards to compute nodes and the other half connected upwards to core layer switches. Each core layer switch connects to all access layers. Fat-Trees have excellent scalability and perfectly evenly distributed bandwidth, and are widely used in distributed training clusters. Due to their strong symmetry, they can be concisely represented using an adjacency matrix. The routing optimization component uses an integer linear programming-based optimization method for route allocation. The communication traffic of All-Reduce is represented as a binary traffic matrix. The unique path selection for each flow is then set as a constraint, with the routing matrix of all traffic as the independent variable. The link load can be obtained through matrix multiplication.

[0027] like Figure 2The diagram illustrates the All-Reduce communication process based on Halving-Doubling, showcasing relevant traffic information. Taking an 8-node communication process as an example, each pair of nodes in each stage communicates, exchanging a portion of their own parameter information, and then summing and averaging the results to achieve overall parameter synchronization. In the Halving-Doubling communication mode, the relationship between the node pairs and the amount of data communicated in each stage change according to certain patterns, and the traffic in later stages also depends on the traffic in earlier stages. Overall, as long as the node numbers and distribution are known, the traffic analysis component can obtain overall traffic information, including the source, destination, and data volume, based on the relationships in the communication process. This information serves as known data and is used as specific input information for subsequent routing load balancers.

[0028] like Figure 3 As shown, the Fat-Tree network structure can be abstracted. Traffic within the same access layer switch does not need to pass through intermediate core layer switches; it only needs to pass through the access layer switches to which they are connected to complete the transmission. In All-Reduce communication traffic, each node sends or receives only one stream at a time, thus avoiding routing contention and bandwidth conflicts. Therefore, routing only needs to consider traffic across switches.

[0029] First, the routing optimization component uses an integer linear programming-based optimization method to allocate routes, achieving non-blocking traffic transmission. Specifically, the modeling uses an adjacency matrix to represent the node connections in a Fat-Tree topology, representing the communication traffic of All-Reduce as a binary traffic matrix. Then, the unique path selection for each flow is set as a constraint, with the routing matrix of all flows as the independent variable. Next, an integer linear programming model is constructed with the objective of minimizing the number of conflicting flows on the network path. Based on the traffic matrix and the constraints, the integer linear programming model is solved to obtain the non-blocking routing scheme for each flow. Although this mathematical modeling can be solved using existing tools such as CPLEX, the overhead increases when the network size is large. Therefore, a greedy strategy routing algorithm based on minimum flow priority is subsequently proposed to allocate task traffic routes and obtain path planning for each flow.

[0030] This greedy algorithm requires counting the number of flows that each access layer switch needs to transmit, then sorting them in ascending order based on the number of flows to obtain the allocation order of the access layer switches. Routing according to this priority ensures that as many flows as possible are taken into account. Then, for each core layer switch, one flow is selected from each of the sorted access layer switches. This operation ensures that there are no path conflicts between the selected flows. The traffic routing allocation can be obtained through a single traversal operation.

[0031] In some cases, some flows may remain unassigned. Theoretically, the number of these unassigned flows is always less than the total number of core layer switches. For these flows, the routing optimization component distributes them across different core layer switches to ensure that at most two flows will compete for bandwidth, without any worse scenarios.

[0032] The greedy algorithm has a time complexity of O(NlogN), where N is the total number of nodes used in the training task. Compared to solving integer linear programming models, this algorithm achieves suboptimal route allocations with acceptable event complexity, making it more suitable for implementation in the control plane.

[0033] This invention optimizes routing by sensing the communication traffic of a task cluster. It effectively overcomes the performance limitations of routing algorithms and is compatible with existing software-defined networks. The specific embodiments described above are illustrative of the invention, not restrictive. Any modifications and alterations made to this invention within the spirit and scope of the claims fall within the protection scope of this invention.

Claims

1. A routing optimization method for distributed training cluster communication, characterized in that, This method is implemented in the control plane of software-defined networking (SDN), including traffic analysis components and route optimization components; The traffic analysis component is responsible for obtaining relevant information about aggregated communication traffic from the application layer, including the All-Reduce algorithm used and the distribution of nodes; the routing optimization component is responsible for path planning based on the traffic distribution and network topology. The routing optimization component, as the receiving module of the traffic analysis component, uses an optimization method based on integer linear programming to achieve non-blocking route allocation based on the transmission task traffic and network topology. The output of the traffic analysis component is used as input to perform unified routing planning for the traffic. The final routing scheme is then distributed to each switch in the form of a routing table to complete the routing optimization. The optimization method based on integer linear programming achieves non-blocking route allocation based on the transmitted task traffic and network topology, specifically including the following steps: (a) First, the node connections of the Fat-Tree topology are represented in the form of an adjacency matrix, and the communication traffic of All-Reduce is represented as a binary traffic matrix; (b) Next, the unique path selection for each flow is set as a constraint, with the routing matrix of all flows as the independent variable; (c) Construct an integer linear programming model with the objective of minimizing the number of conflicting traffic on the network path. Based on the traffic matrix and constraints, solve the integer linear programming model to obtain the non-blocking routing scheme for each traffic. The optimization method for integer linear programming also includes using a greedy routing algorithm based on the least flow priority strategy to allocate task traffic routes, in order to obtain path planning for each traffic flow, specifically: (a) The network topology is a two-layer Fat-Tree of Clos architecture, in which half of the ports of each access layer switch are connected downward to the compute nodes and the other half of the ports are connected upward to the core layer switch. Each core layer switch is connected to all access layers. (b) Count the number of flows that each access layer switch needs to transmit, and then sort them in ascending order according to the number of flows to obtain the allocation order of the access layer switches. (c) For each core layer switch, select one flow from the sorted access layer switches in turn, and obtain the traffic routing allocation through one traversal operation.

2. The routing optimization method for distributed training cluster communication according to claim 1, characterized in that, The traffic analysis component obtains relevant traffic information from the communication mode adopted by the application layer and the node allocation set by the network administrator, including the origin, destination, amount of data transmitted, and dependencies between traffic flows. There are two All-Reduce communication modes used in distributed training: ring-based communication and halving-doubling mode, which are used to achieve parameter synchronization.

3. The routing optimization method for distributed training cluster communication according to claim 2, characterized in that, The dependency relationship refers to the fact that in a single All-Reduce communication process, there will be traffic in several stages, and whether the traffic in the current stage can be transmitted depends on whether the previous stage has completed the transmission. When the traffic analysis component obtains relevant information, it will provide it to the routing optimization component for route allocation.

4. The routing optimization method for distributed training cluster communication according to claim 1, characterized in that, When allocating traffic, the routing optimization component does not require dedicated routing for traffic under the same access layer switch; it only needs to consider traffic across switches. After allocating traffic to each core layer switch, the access layer switches need to be reordered according to the number of remaining flows. Any unallocated traffic after the traversal is completed is distributed to different core layer switches to ensure that no more than two flows compete for bandwidth.

Images