A prediction-based load balancing routing algorithm based on algorithm network cooperation

Abstract

This invention discloses a predictive load balancing routing algorithm based on computer network collaboration, belonging to the field of computer technology. This invention is designed for large-scale model training scenarios, fully utilizing the periodicity and predictability of network traffic during large-scale model training. It employs a distributed training traffic prediction algorithm and a global routing planning algorithm to predict and plan network traffic in advance. Compared with similar data center network load balancing algorithms, this invention achieves higher load balancing performance.

Description

Technical Field

[0001] This invention provides a predictive load balancing routing algorithm based on computing network collaboration, belonging to the field of computer technology, relating to intelligent computing system networks, and particularly to a load balancing routing algorithm for intelligent computing system networks. Background Technology

[0002] The rapid iteration of artificial intelligence technology and the explosive growth in the scale of large language model parameters are driving the upgrade of AI training models to large-scale cluster distributed architectures. Large-scale intelligent computing systems are dedicated computing power clusters built for AI models based on distributed training paradigms. Their efficiency in computing power scheduling, data interaction, and cluster collaboration heavily relies on the support of the underlying internet infrastructure. Currently, most intelligent computing networks are optimized from traditional data center networks, employing a CLOS topology (primarily a leaf-spine architecture) and relying on the ECMP (Equivalent Multi-Path Manipulation) algorithm to achieve load balancing routing, making them widely adaptable to general cloud business scenarios.

[0003] However, traditional data center network architectures and routing mechanisms are designed for the traffic characteristics of general cloud services, which are fundamentally different from the business characteristics of distributed training of large AI models, exposing many inherent defects in intelligent computing scenarios. Traditional data center business traffic is characterized by high entropy, low link utilization, and gradual changes. The number of data streams is large, and each stream occupies low bandwidth and has a stable traffic pattern. In contrast, the traffic of distributed training of large models is characterized by low entropy, strong burstiness, and high instantaneous link utilization. The number of data streams is small, but they are transmitted periodically in bursts, which can fill the network card bandwidth in a short period of time. The difference in traffic patterns leads to a significant decrease in the adaptability of traditional networks. The traditional ECMP routing algorithm relies on five-tuple hashing to distribute traffic. It can achieve good load balancing in traditional high-entropy traffic scenarios, but when faced with the low-entropy burst traffic of large model training, it is prone to hash collisions. A large number of training data streams are hashed to the same forwarding path, causing local link congestion. Especially in Layer 3 switching networks, multi-level cascaded hash operations will further exacerbate the traffic distribution imbalance, produce hash polarization, and significantly worsen the overall network load balancing effect. Distributed training of large models relies on a global synchronization communication mechanism, which has strong synchronization and strong dependency business attributes. Network congestion caused by ECMP hash collisions and hash polarization will generate a large number of long-tailed delayed data streams. Affected by the barrel effect, the iteration progress of the entire training cluster is dragged down by high-latency nodes.

[0004] To address the aforementioned issues, existing research has proposed various load balancing schemes to improve data center network performance, among which CONGA and ConWeave are representative examples. CONGA is a distributed load balancing scheme with global congestion awareness. This scheme collects global congestion information by forwarding data packets. After a data packet is sent from the source leaf switch, the maximum link congestion along the path is recorded. Upon reaching the target leaf switch, the congestion information is temporarily stored in a congestion table. The reverse-transmitted data packet then carries this congestion information back to the source leaf switch. The source leaf switch combines local and uplink congestion levels and makes load balancing decisions at the sub-flow scheduling granularity. When the sub-flow interval meets the requirements, out-of-order data packets can be avoided. This scheme has shown good performance in traditional data center networks, but it suffers from drawbacks such as a long link load information collection and control loop and an inability to respond promptly to sudden network traffic spikes. ConWeave is a cautious rerouting load balancing scheme supplemented by a reordering mechanism. This scheme probes the global network congestion status by periodically sending RTT requests through the source leaf switches. When the RTT request times out or a congestion indication is received from the target leaf switch, the cautious rerouting mechanism is triggered. Its rerouting requires three conditions to be met: existing path congestion, the existence of an uncongested feasible path, and the reception of out-of-order packets caused by previous rerouting. It can ensure path optimization and predictability of packet arrival order. However, although this scheme is designed for RDMA traffic, its application scenarios are still limited to general data center cloud computing applications. Its traffic pattern is quite different from that of large-scale intelligent computing system networks and cannot adapt to the special needs of distributed training of large AI models.

[0005] In summary, existing load balancing routing algorithms cannot effectively cope with low-entropy bursts of traffic in large model training scenarios, and urgently need to be optimized and improved through innovative solutions. Summary of the Invention

[0006] This invention provides a predictive load balancing routing algorithm based on computing network collaboration, which consists of two parts: a large-model distributed training traffic prediction algorithm and a global routing planning algorithm. It is deployed in a centralized controller and switches using a combination of centralized and distributed control.

[0007] Firstly, a method for large-scale model distributed training and traffic prediction is provided, specifically:

[0008] The large-scale distributed training traffic prediction algorithm takes a communication task, a parallel strategy, and an ensemble communication algorithm as input, and outputs a set of traffic matrices arranged in time sequence. First, it calculates all communication domains for each parallel dimension according to the parallel strategy. Second, for a communication operation, it selects the appropriate communication domain based on its parallel dimension, and executes the ensemble communication algorithm in each domain to calculate the network traffic matrix. Since the ensemble communication algorithm typically consists of multiple steps, the traffic matrix is ​​calculated independently for each step. Finally, the algorithm outputs a set of traffic matrices arranged in time sequence for each communication operation.

[0009] The first step is to calculate the total communication domain for each parallel dimension according to the parallel strategy. Specifically:

[0010] The global set of compute nodes is denoted as ,in For computing nodes, The total number of compute nodes; the set of parallel dimensions is denoted as . These correspond to tensor parallelism, pipelined parallelism, and data parallelism, respectively; the parallelism dimension is denoted as... The size of each parallel dimension is denoted as The tensor parallelism is The parallelism of the pipeline is Data parallelism is The total number of compute nodes satisfies:

[0011]

[0012] communication domain collection Indicates the parallel dimension as The set of all communication domains; a single communication domain is denoted as . , represents a subset of ordered computation nodes.

[0013] Tensor parallelism uses a single GPU card or a single compute node as the smallest granularity, grouping compute nodes according to the tensor parallelism dimension. Nodes within a group share tensor computation communication. The outer loop traverses the data parallelism dimension. The middle-level loop traverses the pipeline parallel dimension. The inner loop iterates through the tensor in parallel dimensions. The node index mapping rule is as follows:

[0014]

[0015] Each group contains A total of [number] consecutive computing nodes were generated. Independent tensor parallel communication domains:

[0016]

[0017] Pipeline parallelism uses model layers or stages as granularity, grouping computation nodes according to the pipeline parallelism dimension. Nodes within a group are responsible for communication between different stages of the model. The outer loop iterates through the data parallelism dimension. The middle-level loop traverses the tensor in parallel dimensions. The inner loop traverses the pipeline parallel dimension. The node index mapping rule is as follows:

[0018]

[0019] Each group contains A total of [number] consecutive computing nodes were generated. Tensor parallel communication domain:

[0020]

[0021] Data parallelism uses data sharding as its granularity, grouping computing nodes according to the data parallelism dimension. Nodes within a group perform parameter synchronization communication. The outer loop traverses the pipeline's parallelism dimension. The middle-level loop traverses the tensor in parallel dimensions. The inner loop iterates through the data in parallel dimensions. The node index mapping rule is as follows:

[0022]

[0023] Each group contains A total of [number] consecutive computing nodes were generated. Independent tensor parallel communication domains:

[0024]

[0025] The second step, for a single communication operation, is to select the appropriate communication domain based on its parallelism, and execute an ensemble communication algorithm within each communication domain to compute the network traffic matrix. Specifically:

[0026] The set of communication primitives is denoted as , representing no operation, global reduction, global collection, and reduction scattering respectively, and the communication operation is defined as ,in For communication primitives, For the bound parallel communication dimension; the traffic unit is denoted as ,in As the source node, For the target node, To standardize the flow size; the single-stage flow matrix is ​​denoted as... Let represent the set of all traffic within a communication step; the sequence of traffic matrices is denoted as . This indicates a division based on communication timing. Each stage matrix fully describes the entire process of a single group communication.

[0027] The algorithm adopts a decoupled modular architecture, which can support arbitrary communication primitives and communication algorithm extensions by defining the stage number calculation mapping and the flow matrix calculation mapping.

[0028] Phase number calculation mapping is used by communication primitives Collective communication algorithm Parallel Dimension Scale The only factor that determines the total number of stages , is represented as:

[0029]

[0030] Flow matrix calculation mapping is used by set communication algorithms , communication domain Communication progress Generate a single-stage flow matrix , is represented as:

[0031]

[0032] The mapping between the Ring algorithm and the Halving-Doubling algorithm is defined here:

[0033] The Ring algorithm communicates with neighboring nodes in each round, with the communication scale remaining constant. The AllGather and ReduceScatter primitives are each used... Round-robin communication: The AllReduce primitive can be decomposed into first performing ReduceScatter and then AllGather, for a total of [number] rounds. Therefore:

[0034]

[0035] The HalvingDoubling algorithm, for the ReduceScatter primitive, involves communication between adjacent nodes in the first round. In subsequent rounds, the communication distance doubles while the communication scale is halved, for a total of... Round-based communication: For the AllGather primitive, the first round involves communication over the maximum distance; in subsequent rounds, the communication distance is halved while the communication scale is doubled, for a total of... Round-robin communication: The AllReduce primitive can be decomposed into first performing ReduceScatter and then AllGather, totaling [number of rounds]. Round-robin communication, therefore:

[0036]

[0037]

[0038]

[0039] The specific calculation process of the traffic matrix is ​​as follows: First, based on communication primitives... Collective communication algorithm Parallel Dimension Scale Determine the total number of communication stages:

[0040]

[0041] Then, the flow matrix is ​​generated stage by stage. Traverse all communication domains Based on set communication algorithm , generate communication domain In the stage Traffic matrix:

[0042]

[0043] Finally, the traffic matrices of all communication domains in the same phase are merged and added to the traffic matrix sequence. .

[0044] For a single communication operation, the algorithm ultimately outputs a sequence of traffic matrices arranged in the time order of the steps of the set communication algorithm, which is used for subsequent global routing planning.

[0045] Secondly, a global routing planning algorithm based on greedy approximation is provided, specifically:

[0046] For each flow Its set of all candidate paths is .

[0047] Given a flow matrix ,in For any traffic , Define binary decision variables:

[0048]

[0049] If traffic Place in path Above, the decision variables for Otherwise Each traffic flow must choose one and only one path, thus imposing constraints:

[0050]

[0051] Each link Capacity and link load For a certain link Define the indicator parameters:

[0052]

[0053] If path Passed through the link Then the indicator parameter for Otherwise At this point, links can be defined. The total load is:

[0054]

[0055] Link utilization rate:

[0056]

[0057] Define the objective optimization function as the global link utilization function:

[0058]

[0059] To penalize high-load links, select:

[0060]

[0061] The optimization objective is to minimize the global link utilization function, that is:

[0062]

[0063] Considering the large scale of intelligent computing systems, directly using exact optimization methods often leads to high computational complexity and a problem of solution space explosion. To achieve a balance between solution efficiency and solution quality, this invention designs a greedy heuristic method. All traffic is sorted in descending order of volume, prioritizing flows with the greatest impact on link load. When selecting a path for a flow, the overall congestion change caused by incorporating the current network state into each candidate path is evaluated, and the path that minimizes the increment of the objective function is selected as the final transmission path for that flow. Through this process, the algorithm continuously suppresses further growth of high-load links during the local decision-making phase, thereby achieving a better overall load balancing effect.

[0064] First, initialize all link load. Then, according to the flow rate For the flow matrix Sort all traffic in descending order:

[0065]

[0066] For each traffic Let the optimal global link load be positive infinity:

[0067]

[0068] For each candidate path Calculate flow Place in path Global link load under:

[0069]

[0070] like Then this path will be considered the optimal path. Simultaneously update the optimal global link load. Continue until all candidate paths have been traversed; then transfer the traffic. Assigned to path Simultaneously update the load on relevant links:

[0071]

[0072] After traversing all traffic, return the best path for each traffic. . Attached Figure Description

[0073] Figure 1 Overall architecture diagram of a predictive load balancing routing algorithm based on computing network collaboration.

[0074] Figure 2 Centralized controller deployment details.

[0075] Figure 3 Deployment details of distributed controllers (switches). Detailed Implementation

[0076] The overall architecture adopts a deployment approach that combines centralized control with distributed decision-making, such as... Figure 1 As shown. The distributed controller is located in each switch and is responsible for executing the routing decisions issued by the centralized controller. It also monitors local traffic to report communication progress to the centralized controller. The centralized controller is located in a separate node and runs a predictive load balancing routing algorithm based on computer network collaboration. It controls the switches, collects physical topology data, and advances communication progress through control logic. The centralized controller and the distributed controller communicate with each other through a software-defined network control plane protocol.

[0077] Centralized controller internal deployment details such as Figure 2As shown, the centralized controller stores the communication task list, system parallel strategy, ensemble communication algorithm, and physical topology of the cluster network for the current model training load. The communication task list, system parallel strategy, and ensemble communication algorithm need to be manually input and parsed for different training loads. The control logic is responsible for collecting the physical topology from the cluster network, issuing routing decisions generated by the predictive load balancing routing algorithm based on computing-network collaboration through the control plane protocol, and updating the communication progress to enter the next communication stage. Communication tasks are stored in a linked list. Each node represents a communication operation, which consists of a communication primitive and the parallel dimension bound to that operation, indicating that the communication operation is performed within that parallel dimension. A pointer is also maintained to indicate the currently active communication operation, i.e., the communication operation currently occurring in the intelligent computing system network, representing the communication progress in this iteration. Whenever a communication operation is completed, the pointer moves forward, starting the next round of traffic prediction and routing planning. When the pointer reaches the end of the linked list, it indicates that a large-scale distributed training iteration of the model is completed. At this point, the pointer can be moved to the head of the linked list to start a new round of iteration or terminate the operation, depending on the model training needs. The physical topology is stored in the form of an adjacency list. An array is maintained for each vertex to store all its neighbor nodes. When a link changes, the changed node is inserted or deleted from the neighbor array corresponding to the vertex. The physical topology can be generated by the control plane collecting port neighbor information from the data plane and then piecing it together. However, considering that the network topology of the intelligent computing system is pre-designed and not easily changed, the physical topology can also be configured manually in advance. The operation flow of the predictive load balancing routing algorithm based on computing network collaboration running in the centralized controller is as follows: First, the control logic triggers the command to start the algorithm; second, the communication task reads the currently active communication operations. The algorithm takes the communication operations, parallel strategies, communication algorithms, and physical topology as inputs. First, it performs traffic prediction to generate multiple ordered traffic matrices aligned with the time steps of the communication algorithm. Then, it performs global routing planning on the physical topology for each traffic matrix, generating multiple ordered routing tables; then, the control logic sends the routing tables to the distributed controllers in the switches through the control plane protocol; finally, the control logic collects the communication completion signals reported by the distributed controllers. After all switches have completed this round of communication operations, the communication progress pointer is moved forward, triggering the command to start the next round of the algorithm.

[0078] Internal details of the distributed controller (i.e., the switch) are as follows Figure 3As shown. The routing decisions received by the switch from the centralized controller do not overwrite the original routing table (here called the load balancing routing table to distinguish it from the default routing table), but are stored in a separate memory. Unlike the default routing table, which is organized according to the mapping from destination IP address to forwarding port, the load balancing routing table is organized according to the mapping from flow to forwarding port, using the data flow 5-tuple (source IP address, source port number, destination IP address, destination port number, and protocol number) as the lookup entry. After receiving a data packet, the switch first parses its packet header to obtain information such as the source IP address, source port number, destination IP address, destination port number, and protocol number; it then determines whether it is RDMA traffic based on the protocol number. If it is, it forwards the data according to the load balancing routing table issued by the control plane; otherwise, it forwards the data according to the original routing table. For ACK and NACK packets of RDMA traffic, they are transmitted using the same path as the original data. The distributed controller identifies the communication stage to which traffic belongs by the source port number. It internally records the source port numbers of each traffic in the current stage. If the source port number of the data packet is different from the recorded source port number, it is considered to have entered the next stage. The next stage routing table is used to look up the packet and forward it. At the same time, the stage communication completion signal is reported and the controller waits to receive the next stage routing table.

[0079] This invention discloses a predictive load balancing routing algorithm based on computer network collaboration, belonging to the field of computer technology. This invention is designed for large-scale model training scenarios, fully utilizing the periodicity and predictability of network traffic during large-scale model training. It employs a distributed training traffic prediction algorithm and a global routing planning algorithm to predict and plan network traffic in advance. Compared with similar data center network load balancing algorithms, this invention achieves higher load balancing performance.

[0080] A predictive load balancing routing algorithm based on network-computer collaboration, designed for large-scale model training scenarios, leverages the periodicity and predictability of network traffic trained on large models to predict and plan network traffic, including:

[0081] Accurate prediction of traffic in large-scale distributed training networks is achieved by combining three factors: communication tasks, parallel strategies, and ensemble communication algorithms.

[0082] A greedy heuristic is used to perform global route planning on the predicted traffic matrix;

[0083] A load balancing routing algorithm is deployed in the centralized controller and switches by combining centralized decision-making with distributed control.

[0084] The large-model distributed training traffic prediction algorithm takes communication tasks, parallel strategies, and ensemble communication algorithms as inputs and outputs a set of traffic matrices arranged in time sequence. First, it calculates all communication domains for each parallel dimension according to the parallel strategy. Second, for a communication operation, it selects the corresponding communication domain according to its parallel dimension and executes the ensemble communication algorithm in each communication domain to calculate the network traffic matrix. Since the ensemble communication algorithm is usually divided into multiple steps, the traffic matrix is ​​calculated independently for each step. Finally, the algorithm outputs a set of traffic matrices arranged in time sequence for each communication operation.

[0085] The global routing planning algorithm based on greedy approximation selects the sum of squares of the utilization of each link in the network as the optimization objective; sorts all traffic in descending order according to traffic size, so that the flow with greater impact on link load is allocated first; when selecting a path for a flow, it evaluates the overall congestion change caused by each candidate path after adding the current network state, and selects the path that minimizes the increment of the objective function as the final transmission path of the flow.

[0086] The algorithm employs a deployment approach that combines centralized control with distributed decision-making. The distributed controller is located in each switch and is responsible for executing routing decisions issued by the centralized controller, while also detecting local traffic to report communication progress to the centralized controller. The centralized controller is located in a separate node and runs a predictive load balancing routing algorithm based on computer network collaboration. It also controls the switches, collects physical topology data, and advances communication progress through control logic. The centralized controller and the distributed controller communicate with each other through a software-defined network control plane protocol.

Claims

1. A predictive load balancing routing algorithm based on network-computer collaboration, designed for large-scale model training scenarios, utilizes the periodicity and predictability of network traffic trained by the large-scale model to predict and plan network traffic, characterized by: Accurate prediction of traffic in large-scale distributed training networks is achieved by combining three factors: communication tasks, parallel strategies, and ensemble communication algorithms. A greedy heuristic is used to perform global route planning on the predicted traffic matrix; A load balancing routing algorithm is deployed in the centralized controller and switches by combining centralized decision-making with distributed control.

2. The algorithm according to claim 1, characterized in that: The large-model distributed training traffic prediction algorithm takes communication tasks, parallel strategies, and ensemble communication algorithms as inputs and outputs a set of traffic matrices arranged in time sequence. First, it calculates all communication domains for each parallel dimension according to the parallel strategy. Second, for a communication operation, it selects the corresponding communication domain according to its parallel dimension and executes the ensemble communication algorithm in each communication domain to calculate the network traffic matrix. Since the ensemble communication algorithm is usually divided into multiple steps, the traffic matrix is ​​calculated independently for each step. Finally, the algorithm outputs a set of traffic matrices arranged in time sequence for each communication operation.

3. The algorithm according to claim 1, characterized in that: The global routing planning algorithm based on greedy approximation selects the sum of squares of the utilization of each link in the network as the optimization objective; sorts all traffic in descending order according to traffic size, so that the flow with greater impact on link load is allocated first; when selecting a path for a flow, it evaluates the overall congestion change caused by each candidate path after adding the current network state, and selects the path that minimizes the increment of the objective function as the final transmission path of the flow.

4. The algorithm according to claim 1, characterized in that: The algorithm employs a deployment approach that combines centralized control with distributed decision-making. The distributed controller is located in each switch and is responsible for executing routing decisions issued by the centralized controller, while also detecting local traffic to report communication progress to the centralized controller. The centralized controller is located in a separate node and runs a predictive load balancing routing algorithm based on computer network collaboration. It also controls the switches, collects physical topology data, and advances communication progress through control logic. The centralized controller and the distributed controller communicate with each other through a software-defined network control plane protocol.

Images