A large model training load optimization method based on programmable switch
Patent Information
- Application Number
- CN202611227209.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-13
- Publication Date
- 2026-11-03
AI Technical Summary
In heterogeneous GPU clusters, existing methods have failed to effectively unify and optimize computational and network loads, resulting in inconsistent training progress, increased synchronization waiting time, and the problem of falling behind.
Before training begins, a global training status view is built, the initial load is configured based on the node's computing power and link status, and the network transmission control parameters are dynamically updated during training to reduce the differences in training progress between nodes and the spread of falling behind.
By optimizing the training load, reducing synchronization waiting overhead, improving training efficiency, ensuring consistent training progress across nodes, and suppressing the spread of lagging behind.
Smart Images
Figure CN123035202A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of distributed machine learning and programmable data plane technology, specifically to a method for optimizing the training load of large models based on programmable switches. Background Technology
[0002] As the size of deep learning models and the amount of training data continue to grow, distributed training using GPU clusters has become an important way to improve model training efficiency. During synchronous distributed training, each training node needs to complete local computation and data transfer before proceeding to the next round of training after synchronization. Therefore, the overall training efficiency of the cluster is usually affected by the slowest training node.
[0003] In heterogeneous GPU clusters, the computing power and network status of different training nodes vary. Using the same training load and fixed network transmission parameters can easily lead to inconsistent training progress among nodes, increasing synchronization waiting time. Furthermore, the network status changes dynamically during training, further exacerbating the problems of unbalanced training load and node drop-off.
[0004] Existing methods typically focus on computational load allocation or network traffic adjustment, lacking unified optimization for both computational and network loads. Therefore, a method is needed that can configure the initial training load based on node computing power and link status, and dynamically adjust the network load during training to improve the training efficiency of heterogeneous GPU clusters. Summary of the Invention
[0005] To address the shortcomings of the existing technologies, this invention provides a method for optimizing the training load of large models based on programmable switches. Before training starts, the computational load and initial network load of each training node are configured collaboratively based on the node's computing power and link status. During training, the network transmission control parameters are dynamically updated based on the node's operating status to reduce the difference in training progress between nodes, reduce synchronization waiting overhead, and suppress the spread of falling behind.
[0006] A method for optimizing the training load of large models based on programmable switches includes the following steps:
[0007] Step S1: Before training starts, each heterogeneous GPU training node performs preheating training, encapsulating the node computation latency, communication latency, communication data volume, and link bandwidth information into training status messages and sending them to the programmable switch; the programmable switch parses the training status messages and stores the corresponding status information in the status storage unit to construct a global training status view of the heterogeneous GPU cluster.
[0008] Step S2: The programmable switch obtains the computation latency, communication latency, communication data volume and link bandwidth information of each training node according to the global training status view, establishes a node training latency model, and constructs a training load optimization model with the goal of minimizing the training latency of the slowest node; and obtains the batch size allocation result and initial network transmission control parameters corresponding to each training node by solving the training load optimization model.
[0009] Step S3: During model training, each training node periodically collects the current transmission latency and link load status and sends them to the programmable switch. The programmable switch calculates the lag propagation rate and node lag intensity of the current training cycle based on the running status information reported by each training node, so as to characterize the overall lag diffusion trend of the cluster and the local lag degree of each training node.
[0010] Step S4: The programmable switch calculates the network load adjustment amount of the corresponding node based on the transmission latency, link load status, lag propagation rate and lag intensity of each training node, and dynamically updates the network transmission control parameters corresponding to each training node; it controls the training traffic according to the updated network transmission control parameters to keep the training progress of each training node consistent, reduce synchronization waiting overhead and suppress lag propagation.
[0011] The beneficial effects achieved by this invention are as follows:
[0012] (1) Before training starts, both node computing power and access link status should be considered to avoid creating new communication bottlenecks by allocating batch size based solely on GPU computing power.
[0013] (2) Allocate the batch size of each training node with the goal of minimizing the training latency of the slowest node, reduce the difference in training latency between nodes, and reduce the waiting overhead in synchronous training.
[0014] (3) During the training process, the token bucket generation rate and token bucket capacity are dynamically updated based on the transmission delay and link load status, so that the training flow control can adapt to the dynamic network environment.
[0015] (4) The fall-out propagation rate and node fall-out intensity are introduced into the network load optimization process. At the same time, the overall fall-out diffusion trend of the cluster and the local fall-out degree of the nodes are considered. The fall-out diffusion is suppressed and the distributed training efficiency is improved without changing the model parameter update mechanism. Attached Figure Description
[0016] Figure 1 This is a flowchart illustrating the steps of the large model training load optimization method based on a programmable switch in an embodiment of the present invention.
[0017] Figure 2This is a schematic diagram of a heterogeneous GPU cluster training load optimization architecture based on a programmable switch in an embodiment of the present invention.
[0018] Figure 3 The graph shows the experimental results of training throughput of different models on different datasets in the embodiments of the present invention. Detailed Implementation
[0019] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings.
[0020] This invention addresses the scenario of synchronous data parallel large-scale model training, and the optimized training load architecture is as follows: Figure 2 As shown, it includes multiple heterogeneous GPU training nodes, a programmable switch, and a control program. Each training node transmits training data through the programmable switch and sends node running status information to the programmable switch.
[0021] The programmable switch collects and stores node operating status information. The control program calculates the training load and adjusts the network load based on the collected information, and writes the adjustment results into the programmable switch to control the training traffic of each training node.
[0022] A method for optimizing the training load of large models based on programmable switches, referenced Figure 1 It includes the following steps:
[0023] Step S1: Represent the heterogeneous GPU training node set as ,in This represents the total number of training nodes. (Nodes) Perform warm-up training and collect node latency calculations. Communication delay Communication data volume and link bandwidth and construct nodes Training state vector:
[0024]
[0025] Step S1-2: Transfer the training state vector The node identifier, the current training round, the total number of training rounds, and the total number of training nodes are encapsulated into a training status message, and the training status message is sent to the programmable switch.
[0026] Step S1-3: The programmable switch parses the training state message, using the node identifier as an index, and then... Write the corresponding storage location in the state storage unit, and record the node by receiving the state identifier. Whether the training status message has arrived.
[0027] Steps S1-4: When all When all the received state identifiers of the training nodes are in a valid state, the training state vectors are aggregated according to the node identifier order to construct a global training state view:
[0028]
[0029] in, This indicates the state acquisition time before training begins. The global training state view simultaneously includes differences in node computing power and node access link status, and serves as the unified input for the training load optimization model in step S2.
[0030] Step S2: The programmable switch obtains the computation latency, communication latency, communication data volume, and link bandwidth information of each training node based on the global training status view, establishes a node training latency model, and constructs a training load optimization model with the goal of minimizing the training latency of the slowest node. Based on the training load optimization model, the batch size allocation results and initial network transmission control parameters corresponding to each training node are obtained. The specific steps are as follows:
[0031] Step S2-1: Based on the global training state view, adjust the batch size of node i. Modeling computational latency, communication latency, and communication data volume:
[0032]
[0033]
[0034]
[0035] in, and Representing nodes respectively The unit sample computation cost and fixed computation cost, Represents a node Fixed communication overhead, Represents a node The initial token bucket generation rate, and These represent the data growth per unit sample and the scale of fixed communication data, respectively.
[0036] Step S2-2: Determine the minimum link bandwidth from the global training state view, and initialize the node according to the minimum link bandwidth. token bucket generation rate and token bucket capacity :
[0037]
[0038]
[0039] in, Indicates the rate scaling factor. This indicates a preset emergency time window.
[0040] Step S2-3: Transfer the nodes The single-round training latency is defined as the sum of computation latency and communication latency, and a training load optimization model is constructed:
[0041]
[0042] in, This represents the total amount of data that all training nodes need to process in a single round of training. Represents a node The maximum batch size that can be handled. The batch size allocation result is obtained by solving the trained load optimization model. }, and { , , This serves as the initialization result for the training load.
[0043] Step S3: During model training, each training node periodically collects the current transmission latency and link load status and sends them to the programmable switch. The programmable switch calculates the lag propagation rate and node lag intensity for the current training period based on the operational status information reported by each training node, to characterize the overall lag propagation trend of the cluster and the local lag degree of each training node. The specific steps are as follows:
[0044] Step S3-1: In the first Within each training cycle, the node Collect current transmission latency and link load status ,Will and It is encapsulated into an operational status message and sent to the programmable switch.
[0045] Step S3-2: After aggregating the running status messages of all nodes, the programmable switch calculates the lag propagation rate for the current training period based on the token bucket generation rate of each node in the previous training period. :
[0046]
[0047] in, Represents the initial propagation coefficient. Represents a node The token bucket generation rate in the previous training cycle, Indicates the reference rate.
[0048] Step S3-3: Based on the nodes Calculate the node dropout strength based on the current transmission latency and link load status. :
[0049]
[0050] in, Indicates the congestion transmission delay threshold. This represents the load normalization reference value. and These represent the weighting coefficients for the transmission delay term and the link load term, respectively. Lag propagation rate. Used to describe the overall lag diffusion trend of all training nodes, node lag strength Used to describe nodes The degree of local lag relative to the overall state of the cluster.
[0051] Step S4: The programmable switch calculates the network load adjustment for each training node based on its transmission latency, link load status, lag propagation rate, and lag intensity, and dynamically updates the network transmission control parameters for each training node. The updated network transmission control parameters are used to control the training traffic, ensuring consistent training progress across all nodes, reducing synchronization waiting overhead, and suppressing lag propagation. The specific steps are as follows:
[0052] Step S4-1: Based on the node Current transmission delay and link load status The network load adjustment amount is determined according to the following piecewise function. :
[0053]
[0054] in, Indicates the idle transmission delay threshold. Indicates the low load threshold. Indicates the rate boost factor. and This represents the rate reduction coefficient at both stages.
[0055] Step S4-2: Propagate the rate of falling behind Strength of node falling behind The product is used as a correction term for lag propagation, combined with network load adjustment. Calculate the nodes respectively Candidate token bucket generation rate and candidate token bucket capacity:
[0056]
[0057]
[0058] in, This represents the capacity adjustment coefficient.
[0059] Step S4-3: Apply range constraints to the candidate token bucket generation rate and candidate token bucket capacity:
[0060]
[0061]
[0062] in, and These represent the minimum rate constraint and the minimum capacity constraint, respectively. and These represent the maximum rate amplification factor and the maximum capacity amplification factor, respectively.
[0063] Step S4-4: Set the constrained token bucket generation rate and token bucket capacity Write into the programmable switch and node The corresponding token bucket parameter is stored in the following location. The programmable switch reads the corresponding token bucket parameter based on the node identifier of the training data packet. When the remaining number of tokens is not less than the length of the training data packet, the training data packet is forwarded and the corresponding tokens are deducted; otherwise, rate limiting is applied to the training data packet.
[0064] To demonstrate the effectiveness of this invention, preliminary experiments were conducted. The method proposed in this invention was compared with DDP. DDP employs uniform batch size allocation and a fixed communication strategy, with each training node performing a training task of the same scale, and parameter aggregation is completed through synchronous gradient updates. The comparison results are as follows: Figure 3 As shown. Figure 3 The horizontal axis represents the combination of model and dataset, used to characterize different training task scenarios; the vertical axis represents training throughput, used to characterize the ability to process training samples per unit time during training. Higher throughput indicates higher training efficiency. Figure 3 As can be seen, under the conditions of ResNet-101 and ViT-Base-16 models, as well as CIFAR-10, CIFAR-100, and Mini-ImageNet datasets, the training throughput of the PSTLC method proposed in this invention is higher than that of DDP. This indicates that the method proposed in this invention can effectively adjust the training load of heterogeneous GPU clusters, reduce the synchronization waiting overhead between nodes, and improve the efficiency of distributed training.
[0065] The above description is only a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. Any equivalent modifications or changes made by those skilled in the art based on the content disclosed in the present invention should be included within the scope of protection set forth in the claims.
Claims
1. A method for optimizing the training load of large models based on programmable switches, characterized in that: The method includes the following steps: Step S1: Before training starts, each heterogeneous GPU training node performs preheating training, encapsulating the node computation latency, communication latency, communication data volume and link bandwidth information into training status messages and sending them to the programmable switch. The programmable switch parses the training status messages and stores the corresponding status information in the status storage unit to construct a global training status view of the heterogeneous GPU cluster. Step S2: The programmable switch obtains the computation latency, communication latency, communication data volume and link bandwidth information of each training node according to the global training status view, establishes a node training latency model, and constructs a training load optimization model with the goal of minimizing the training latency of the slowest node; and obtains the batch size allocation result and initial network transmission control parameters corresponding to each training node by solving the training load optimization model. Step S3: During model training, each training node periodically collects the current transmission latency and link load status and sends them to the programmable switch. The programmable switch calculates the lag propagation rate and node lag intensity of the current training cycle based on the running status information reported by each training node, so as to characterize the overall lag diffusion trend of the cluster and the local lag degree of each training node. Step S4: The programmable switch calculates the network load adjustment amount of the corresponding node based on the transmission latency, link load status, lag propagation rate and lag intensity of each training node, and dynamically updates the network transmission control parameters corresponding to each training node; it controls the training traffic according to the updated network transmission control parameters to keep the training progress of each training node consistent, reduce synchronization waiting overhead and suppress lag propagation.
2. The method for optimizing large model training load based on a programmable switch according to claim 1, characterized in that: The specific steps in step S1 are as follows: Step S1-1: Represent the heterogeneous GPU training node set as follows ,in The total number of training nodes; nodes Perform warm-up training and collect node latency calculations. Communication delay Communication data volume and link bandwidth and construct nodes Training state vector : Step S1-2: Transfer the training state vector The node identifier, current training round, total training rounds, and total number of training nodes are encapsulated into a training status message, and the training status message is sent to the programmable switch; Step S1-3: The programmable switch parses the training state message, using the node identifier as an index, and then... Write the corresponding storage location in the state storage unit, and record the node by receiving the state identifier. Has the training status message arrived? Steps S1-4: When all When all the received state identifiers of the training nodes are in a valid state, the training state vectors are aggregated according to the node identifier order to construct a global training state view: in, This indicates the state acquisition time before training starts; the global training state view simultaneously includes differences in node computing capabilities and differences in node access link status, and serves as the unified input for the training load optimization model in step S2.
3. The method for optimizing large model training load based on a programmable switch according to claim 1, characterized in that: In step S2, the specific steps are as follows: Step S2-1: Based on the global training state view, adjust the batch size of node i. Modeling computational latency, communication latency, and communication data volume: in, and Representing nodes respectively The unit sample computation cost and fixed computation cost, Represents a node Fixed communication overhead, Represents a node The initial token bucket generation rate, and These represent the data growth per unit sample and the scale of fixed communication data, respectively. Step S2-2: Determine the minimum link bandwidth from the global training state view, and initialize the node according to the minimum link bandwidth. token bucket generation rate and token bucket capacity : in, Indicates the rate scaling factor. Indicates the preset emergency time window; Step S2-3: Transfer the nodes The single-round training latency is defined as the sum of computation latency and communication latency, and a training load optimization model is constructed: in, This represents the total amount of data that all training nodes need to process in a single round of training. Represents a node The maximum batch size that can be handled; the batch size allocation result is obtained by solving the training load optimization model { }, and { , , This serves as the initialization result for the training load.
4. The method for optimizing large model training load based on a programmable switch according to claim 1, characterized in that: The specific steps in step S3 are as follows: Step S3-1: In the first Within each training cycle, the node Collect current transmission latency and link load status ,Will and Encapsulate it into an operational status message and send it to the programmable switch; Step S3-2: After aggregating the running status messages of all nodes, the programmable switch calculates the lag propagation rate for the current training period based on the token bucket generation rate of each node in the previous training period. : in, Represents the initial propagation coefficient. Represents a node The token bucket generation rate in the previous training cycle, Indicates the reference rate; Step S3-3: Based on the nodes Calculate the node dropout strength based on the current transmission latency and link load status. : in, Indicates the congestion transmission delay threshold. This represents the load normalization reference value. and These represent the weighting coefficients for the transmission delay term and the link load term, respectively; the lag propagation rate. Used to describe the overall lag diffusion trend of all training nodes, node lag strength Used to describe nodes The degree of local lag relative to the overall state of the cluster.
5. The method for optimizing large model training load based on a programmable switch according to claim 1, characterized in that: In step S4, the specific steps are as follows: Step S4-1: Based on the node Current transmission delay and link load status The network load adjustment amount is determined according to the following piecewise function. : in, Indicates the idle transmission delay threshold. Indicates the low load threshold. Indicates the rate boost factor. and This represents the rate reduction coefficient at both stages; Step S4-2: Propagate the rate of falling behind Strength of node falling behind The product is used as a correction term for lag propagation, combined with network load adjustment. Calculate the nodes respectively Candidate token bucket generation rate and candidate token bucket capacity: in, Indicates the capacity adjustment coefficient; Step S4-3: Apply range constraints to the candidate token bucket generation rate and candidate token bucket capacity: in, and These represent the minimum rate constraint and the minimum capacity constraint, respectively. and These represent the maximum rate amplification factor and the maximum capacity amplification factor, respectively. Step S4-4: Set the constrained token bucket generation rate and token bucket capacity Write into the programmable switch and node The corresponding token bucket parameter storage location; the programmable switch reads the corresponding token bucket parameter according to the node identifier of the training data packet, and forwards the training data packet and deducts the corresponding token when the remaining number of tokens is not less than the length of the training data packet; otherwise, it performs rate limiting on the training data packet.