An Automatic Parallel Optimization Method for Large Model Training in Hybrid Heterogeneous Clusters

Abstract

This invention discloses an automatic parallel optimization method for large-scale model training in hybrid heterogeneous clusters. This invention requires only real-time collection of a small number of system performance parameters to achieve efficient prediction of training time costs in hybrid heterogeneous cluster environments, while comprehensively considering factors such as GPU performance differences, communication overhead, and resource allocation. This invention formalizes the parallel strategy and parameter optimization problem for large-scale model training in hybrid heterogeneous clusters into a constrained optimization problem and establishes a bandwidth-aware training cost estimation model based on a theoretical model. It decomposes and estimates the time cost during training item by item, achieving rapid and accurate prediction of training costs under different heterogeneous configurations. This invention also designs a parallel strategy search and parameter optimization method based on intelligent optimization algorithms, which significantly reduces the traditional search space size while ensuring near-optimal solutions, improving search efficiency and reducing additional overhead.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and distributed computing technology, and relates to an automatic parallel optimization method for training large models in hybrid heterogeneous clusters. Background Technology

[0002] Traditional distributed training methods typically rely on homogeneous GPU clusters, employing techniques such as data parallelism (DP), pipelined parallelism (PP), tensor parallelism (TP), and expert parallelism (EP) to improve model training efficiency and the scale of trainable models. However, the high price and limited supply of mainstream high-end GPUs present a dual bottleneck of insufficient computing power and excessive cost when performing training tasks on homogeneous GPU clusters. In recent years, with the rapid iteration of intelligent computing hardware, different vendors have continuously launched new, cost-effective AI-specific accelerators, driving the diversification and heterogeneity of computing hardware in intelligent computing clusters. Computing centers are gradually evolving into large-scale heterogeneous clusters containing different generations of GPUs from the same vendor as well as different types of GPUs from different vendors.

[0003] Against this backdrop, fully utilizing large-scale hybrid heterogeneous clusters composed of a limited number of existing GPUs of different types to achieve efficient hybrid parallel training of large models has become an effective way to unleash hardware potential, solve computing power shortages, and improve computing power utilization. However, in this hybrid heterogeneous training scenario, due to the significant differences in computing architecture, computing power, and communication bandwidth of different types of hardware devices, the design and selection of parallel schemes will directly affect the utilization efficiency of hardware resources. Therefore, the optimization of parallel strategies for heterogeneous hybrid training is particularly critical.

[0004] Currently, there are some publicly available studies on techniques for automatically generating parallel strategies on heterogeneous clusters. For example, Metis proposed an automatic parallel strategy search method for heterogeneous GPU clusters. Its core lies in constructing a large-scale candidate parallel search space and achieving a balance between performance and search overhead through heterogeneous-aware pruning and cost models. While this method can generate adaptive parallel schemes based on the computing power, memory, and bandwidth characteristics of different GPUs, it still suffers from drawbacks such as high search costs and limited adaptability in dynamic resource environments.

[0005] HexiScale proposes an optimization framework based on a two-stage hierarchical partitioning and cost model-guided search to adapt to large-scale model training on heterogeneous GPU clusters. Specifically, it first separates intra- and extra-cluster communication through hierarchical partitioning, and then combines theoretically modeled cost estimates to obtain a near-optimal parallel strategy with relatively low search overhead. Unlike methods that rely on large-scale profile data, this approach has advantages in planning time and scalability; however, its static cost prediction model struggles to adapt to dynamic runtime changes and lacks sufficient support for memory optimization and more complex heterogeneous scenarios.

[0006] To address the imbalance between computational and memory resources during large model training, Cephalo proposes an optimization framework that decouples computation from optimizer state. This method effectively alleviates the memory bottleneck and improves training scalability by allocating forward / backward computation and optimizer state storage across heterogeneous GPUs. Cephalo performs well in resource-constrained heterogeneous environments, fully utilizing the computational and storage capabilities of different devices. However, the optimization space of this method is relatively limited, and it does not adequately consider communication topology and global parallelism strategies.

[0007] Zorse proposes an efficient parallel policy planner that addresses the memory and communication efficiency issues in large model training. Its core method combines interleaved pipeline parallelism, ZeRO technology, and a parameter / activation offloading mechanism to significantly reduce memory consumption and increase the overlap between communication and computation on heterogeneous GPU clusters. This approach can significantly improve throughput while maintaining training accuracy, making it particularly suitable for memory-constrained training environments. However, the optimization of this approach mainly focuses on memory and communication, resulting in insufficient coverage of the global search space and significant reliance on underlying system support.

[0008] While the aforementioned existing work has made significant progress in automated parallel scheme generation, communication optimization, and memory utilization, providing strong support for unleashing the potential of heterogeneous clusters, they still have some common limitations in their design concepts and implementation mechanisms, which restrict their practicality and scalability in larger-scale and more complex heterogeneous environments. These limitations mainly include the following aspects:

[0009] 1) Reliance on large-scale profiling: Existing heterogeneous automatic parallel optimization methods typically rely heavily on large amounts of profiling data to obtain the performance of different model layers under various parallel configurations. These methods often require comprehensive experimental sampling of operators or layers in the target environment to generate large-scale profile files as input for cost models and policy searches. However, this reliance leads to two main problems: firstly, profiling overhead grows exponentially with model size, resulting in low search efficiency; secondly, profiling data is highly dependent on the specific environment, often requiring re-collection when changing GPU architectures or cluster configurations, lacking portability and generalization. Therefore, methods based on large-scale profiling struggle to adapt to rapid hardware evolution and diverse heterogeneous cluster scenarios, severely limiting their practicality and scalability.

[0010] 2) High search space complexity: Existing methods often employ brute-force enumeration or simple heuristic pruning for parallel policy search. While brute-force search can guarantee coverage of all candidate solutions, it incurs extremely high computational costs; simple pruning, although reducing some computation, often sacrifices search quality, easily missing the global optimum. More seriously, as the cluster size and model layer number increase, the search space complexity expands exponentially, leading to a sharp increase in search time, making it difficult to apply efficiently in real-world systems.

[0011] 3) Lack of Fully Automated Support: While existing methods offer automated strategy optimization frameworks, in practice, key parameters (such as parallelism configuration, batch partitioning, or load balancing thresholds) still require manual setting or repeated tuning. This not only increases user costs and the professional threshold but also weakens the method's adaptability to different hardware environments and model sizes. When the cluster size increases or the hardware architecture iterates rapidly, manual parameter tuning often needs to be repeated, leading to inefficiency and insufficient strategy stability.

[0012] 4) Insufficient network topology adaptability: Existing methods are mainly designed for heterogeneous GPUs of different generations from the same vendor, thus focusing primarily on optimizing communication between devices within the cluster, without fully considering complex network topologies across nodes, racks, or even data centers. This limitation makes it difficult to achieve optimal performance in large-scale heterogeneous cluster scenarios. Furthermore, some methods employ a two-stage search (first dividing the pipeline into stages, then selecting parallel strategies), which may struggle to find optimal results in complex topology environments. Summary of the Invention

[0013] In view of the technical problems existing in the background art, the purpose of this invention is to provide an automatic parallel optimization method for training large models for hybrid heterogeneous clusters.

[0014] To achieve the above objectives, the present invention adopts the following technical solution:

[0015] An automatic parallel optimization method for training large models on hybrid heterogeneous clusters includes the following steps:

[0016] S1. After receiving the hybrid training task, the system automatically collects the key configuration information of various GPUs in the hybrid heterogeneous cluster environment through the feature acquisition module.

[0017] S2. Based on the collected state information of the hybrid heterogeneous cluster environment and the computational, communication and synchronization overhead of the model to be trained, a bandwidth-aware training time estimation model is constructed, and the training time overhead is estimated.

[0018] S3, Parallel Strategy Search Module integrates a parallel strategy optimizer designed with intelligent optimization algorithm. Based on the intelligent optimization algorithm, it performs real-time screening and iterative optimization of candidate parallel schemes according to the predicted results, and finally generates the best parallel strategy.

[0019] S4. The scheduling and execution module divides the model according to the best parallel construction strategy received and distributes it to the GPU nodes in the specific sub-cluster. Then, it executes the model according to the parallel settings and parameters of the best parallel construction strategy, completing the resource allocation, task scheduling and execution control of the training task on each node of the cluster.

[0020] Preferably, in step S1, the key configuration information includes computing power performance parameters, video memory capacity, intra-node and inter-node bandwidth, network topology, and runtime load status.

[0021] Preferably, in step S2, the estimated training time overhead includes: the total time cost T for heterogeneous hybrid training of a large model M in a specific heterogeneous cluster environment H, using a 1F1B pipelined parallel scheme across clusters of different devices. total , is represented as:

[0022]

[0023] Where B represents the number of Micro Batches; T stage (S i ) indicates stage S i The total execution time is expressed as:

[0024] T stage (S i ) = T comp (S i )+T TPcomm (S i )+T PPcomm (S i(2)

[0025] Among them, T comp (S i ) indicates that this stage S i The computation time is expressed as:

[0026] FLOP(S i )≈6·L i ·Nh 2 ·Seq·MBS i (3)

[0027]

[0028] Among them, FLOP(S i ) indicates stage S i The total computational load on C; eff This indicates the effective computing power after considering actual utilization (0). <C eff Scale factor ≤1; T TPcomm (S i ) indicates stage S i The fine-grained communication time that occurs within the timeframe for achieving TP, i.e., the tensor parallel synchronization time, is expressed as:

[0029]

[0030] Where op represents stage S i Operators that require synchronization in tensor parallelism, α op The communication startup latency for tensor parallelism is the fixed overhead of a single intra-node communication call. For the execution phase S i sub-cluster H k The transmission latency per byte of data in msg_size op The amount of data that the operator op needs to communicate when TP is in parallel is estimated by the following formula:

[0031]

[0032] Where dtype_size is the number of bytes per element (FP16 = 2B, FP32 = 4B); T PPcomm (S i This represents the pipeline communication time between adjacent stages, i.e., the pipeline synchronization time, which is expressed as:

[0033] T PPcomm (S i )≈α pp +β pp ·activation_size (7)

[0034] Where, α pp For pipelined parallel message start delay; β pp =1 / B inter The transmission latency per byte of data between nodes of different types of devices is denoted by activation_size, which is the size of the activation value that needs to be transmitted between the two stages.

[0035] T DPsync This represents the time overhead of synchronizing (all-reduce) the parallel parameters of the data after each Iteration or several Iterations, i.e., the pipeline synchronization time, which is expressed as:

[0036]

[0037] in, The two-stage communication from the ring algorithm of set communication, the number of synchronization times sync_times is the size of the parameter to be synchronized divided by the number of messages in each all-reduce.

[0038] Preferably, in step S2, the construction of a bandwidth-aware training time estimation model based on the collected state information of the hybrid heterogeneous cluster environment and the computational, communication, and synchronization overhead of the model to be trained includes: formalizing the parallel policy search problem into a constrained optimization problem. The decision variables of this problem include data parallelism, tensor parallelism, pipeline parallelism, and model layer partitioning. The optimization objective is the training iteration time or the overall training cost. The constraints include memory capacity limitations, computational resource allocation limitations, and communication topology constraints. Specifically, it includes: first assuming that the parallel policy solution is a vector x:

[0039] x = {pp, dp, tp, L}

[0040] Where pp represents the pipeline parallelism, i.e., the number of stages in the training; dp = {dp1, dp2, K, dp} pp} represents the data parallelism of each stage; tp = {tp1, tp2, K, tp pp Let L = {L1, L2, K, L} represent the tensor parallelism of each stage. pp} represents the number of model layers in each stage;

[0041] The optimization objective is to find the optimal parallel strategy that minimizes the overall training time cost of heterogeneous hybrid training of large models in a heterogeneous cluster environment while satisfying constraints. Specifically, it can be expressed as:

[0042]

[0043] Where, σ * T represents the optimal parallel strategy.total This represents the total time cost of heterogeneous hybrid training;

[0044] The constraints include: (1) The memory of each GPU does not exceed the maximum memory: Mem cosum,di Mem represents the video memory consumed by device di. di (1) The memory capacity of device di is indicated; (2) The number of stages should not exceed the number of sub-clusters: pp≤K; (3) The dp*tp of each sub-cluster should be equal to the total number of devices: Among them, dp i Indicates stage S i Data parallelism, tp i Indicates stage S i Tensor parallelism, N index_Si Indicates stage S i The actual number of nodes in the sub-cluster, G index_Si Indicates stage S i The number of GPU devices on a single node of the actual sub-cluster.

[0045] Preferably, in step S3, the step of using an intelligent optimization algorithm to perform real-time screening and iterative optimization of candidate parallel schemes based on the predicted results, and finally generating the optimal parallel training strategy, includes: first, generating several candidate parallel schemes as an initial solution set; then, using a constraint detection and training time estimation model to determine the feasibility and performance of the candidate parallel schemes, and updating the current optimal solution; then, generating new candidate schemes through iterative update operations of the algorithm; repeating this process until the termination condition is met, and finally outputting the globally near-optimal parallel training scheme.

[0046] Preferably, step S3 specifically includes the following steps:

[0047] S111, Parameter Initialization and Population Initialization: Based on cluster resource information, model size and parallel strategy search space, set relevant parameters of the search algorithm, including population size, number of iterations, crossover probability, mutation probability, and randomly or heuristically generate a batch of feasible initial candidate solutions to form the initial solution set or initial population.

[0048] S112. Feasibility Detection and Training Time Cost Assessment: Constraints are checked for each candidate solution. If a candidate solution does not meet the constraints, it is determined to be an infeasible solution and discarded. If a candidate solution meets the constraints, it is determined to be a feasible solution. For feasible solutions, the training time cost or iteration cost is calculated based on theoretical modeling and communication / computation cost estimator to obtain the corresponding fitness value.

[0049] S113. Update the current optimal solution: Among the evaluated candidate solutions, compare their training time costs, update the solution with the best performance as the current optimal solution, and store it for future comparison.

[0050] S114. Search Algorithm Update Operation: The update operation is performed according to the iterative mechanism of the intelligent optimization algorithm, including: in the genetic algorithm, new individuals are generated through selection, crossover and mutation operators; in the particle swarm optimization algorithm, new solutions are generated through position update and velocity update; in Bayesian optimization, new exploration points are selected through the acquisition function; the newly generated candidate solutions need to undergo feasibility testing and cost evaluation again, and then be added to the population or candidate solution set.

[0051] S115. Termination condition judgment: If the preset termination condition is met, the algorithm stops; otherwise, return to step S112 to continue iterative search.

[0052] S116. Output the final solution: When the algorithm ends, the output record of the globally optimal parallel strategy is used as the final search result. This strategy can significantly reduce training time and cost while meeting resource constraints.

[0053] A second aspect of the present invention provides an automatic parallel optimization system for training large models on hybrid heterogeneous clusters. This system comprises four parts: a feature acquisition module, a cost estimation module, a parallel policy search module, and a scheduling and execution module, with each module connected via a bus communication network.

[0054] Feature Acquisition Module: This module is responsible for automatically acquiring the status information of the hybrid heterogeneous cluster environment, namely the key configuration information of various GPUs in the hybrid heterogeneous cluster, and transmitting it to the cost estimation module. The key configuration information includes computing power performance parameters, memory capacity, intra-node and inter-node bandwidth, network topology, and runtime load status, etc. This module provides basic data support for subsequent modeling and optimization.

[0055] Cost estimation module: This module establishes a bandwidth-aware training time estimation model based on the state information of the hybrid heterogeneous cluster environment collected by the feature acquisition module and the computational, communication and synchronization costs of the model to be trained, and feeds the estimation results back to the parallel policy search module.

[0056] Parallel Strategy Search Module: This module integrates a parallel strategy optimizer designed with intelligent optimization algorithms. Based on the prediction results of the cost prediction module, it performs real-time screening and iterative optimization of candidate solutions, and finally generates the optimal heterogeneous parallel strategy.

[0057] Scheduling and Execution Module: This module receives the optimal parallel construction strategy output from the parallel strategy search module, then divides the model according to the optimal parallel construction strategy and distributes it to the GPU nodes in the specific sub-cluster. It executes the model according to the parallel settings and parameters of the optimal parallel construction strategy, and completes the resource allocation, task scheduling and execution control of the training task on each node of the cluster.

[0058] The beneficial effects of this invention are:

[0059] (1) This invention proposes an automatic parallel optimization method for training large models in hybrid heterogeneous clusters. This method significantly reduces profiling and search overhead: training time prediction can be performed with only a small number of parameters, avoiding large-scale experimental sampling; the proposed parallel strategy search method based on intelligent optimization algorithm significantly reduces the search space and reduces complexity while ensuring near-optimal solution.

[0060] (2) This invention proposes an automatic parallel optimization method for training large models in hybrid heterogeneous clusters. This method realizes the full automation of model segmentation and parallel strategy search parameters without manual intervention. The system can automatically generate the optimal parallel strategy.

[0061] (3) This invention proposes an automatic parallel optimization method and system for training large models for hybrid heterogeneous clusters. It has wide applicability, can be adapted to heterogeneous clusters with multi-generation and multi-vendor GPUs, and supports complex network topologies. Attached Figure Description

[0062] Figure 1 This is a schematic diagram of a hybrid heterogeneous cluster environment.

[0063] Figure 2 This is a diagram illustrating the overall framework of an automatic parallel optimization system for training large models in a hybrid heterogeneous cluster, as proposed in this invention.

[0064] Figure 3 This is a flowchart illustrating an automatic parallel optimization method for training large models in a hybrid heterogeneous cluster, as proposed in this invention.

[0065] Figure 4 This is a flowchart illustrating the parallel strategy optimization algorithm. Detailed Implementation

[0066] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0067] To overcome the shortcomings of existing methods, such as reliance on large-scale profiling, high search space complexity, lack of fully automated support, and limitations in network topology adaptability, this invention proposes an automatic parallel optimization method and system for large-scale model training in heterogeneous cluster environments. By constructing a bandwidth-aware training cost estimation model and combining it with an efficient intelligent search algorithm, it can automatically generate efficient three-dimensional parallel solutions on multi-type heterogeneous (GPU) clusters. Unlike traditional methods that rely on large-scale profiling data and curve fitting, this invention only requires real-time collection of a small number of system performance parameters to achieve efficient prediction of training time costs in heterogeneous cluster environments, and can simultaneously consider factors such as GPU performance differences, communication overhead, and resource allocation. This invention formalizes the parallel strategy and parameter optimization problem for large model training under hybrid heterogeneous (GPU) clusters into a constrained optimization problem and establishes a bandwidth-aware training cost estimation model based on a theoretical model. This model decomposes and estimates the time cost during the training process item by item, enabling rapid and accurate prediction of training costs under different heterogeneous configurations. Furthermore, it designs a parallel strategy search and parameter optimization method based on intelligent optimization algorithms, which significantly reduces the size of the search space while ensuring near-optimal solutions, thereby improving search efficiency and reducing additional overhead.

[0068] Reference Figure 2 This invention provides an automatic parallel optimization system for training large models on hybrid heterogeneous clusters. The system consists of four parts: a feature acquisition module, a cost estimation module, a parallel strategy search module, and a scheduling and execution module, all connected via a bus communication network.

[0069] (1) Feature acquisition module: This module is responsible for automatically acquiring the status information of the hybrid heterogeneous cluster environment, namely the key configuration information of various GPUs in the hybrid heterogeneous cluster, and transmitting it to the cost estimation module. The key configuration information includes computing power performance parameters, memory capacity, intra-node and inter-node bandwidth, network topology and runtime load status, etc. This module provides basic data support for subsequent modeling and optimization.

[0070] (2) Cost estimation module: This module establishes a bandwidth-aware training time estimation model based on the state information of the hybrid heterogeneous cluster environment collected by the feature acquisition module, as well as the computational, communication, and synchronization costs of the model to be trained, and feeds the estimation results back to the parallel strategy search module. This module can quickly and accurately estimate the training cost under different parallel strategies without the need for large-scale profiling data.

[0071] (3) Parallel Strategy Search Module: This module integrates a parallel strategy optimizer designed with intelligent optimization algorithms. Based on the prediction results from the cost estimation module, it performs real-time screening and iterative optimization of candidate parallel solutions, ultimately generating the optimal heterogeneous parallel strategy. This module can significantly reduce the search space and improve search efficiency while ensuring the quality of the solution.

[0072] (4) Scheduling and execution module: This module receives the best parallel strategy output from the parallel strategy search module, then divides the model according to the best parallel strategy and distributes it to the GPU nodes in the specific sub-cluster. It executes the model according to the parallel settings and parameters of the best parallel strategy, and completes the resource allocation, task scheduling and execution control of the training task on each node of the cluster.

[0073] Reference Figure 3 In another aspect, the present invention applies the above system to implement an automatic parallel optimization method for training large models in hybrid heterogeneous clusters, including the following steps:

[0074] S1. After receiving the hybrid training task, the system first automatically collects the key configuration information of various GPUs in the hybrid heterogeneous cluster environment through the feature acquisition module, including computing power performance parameters, memory capacity, intra-node and inter-node bandwidth, network topology and runtime load status, etc.

[0075] S2. The cost estimation module constructs a bandwidth-aware training time estimation model based on the state information of the hybrid heterogeneous cluster environment collected by the feature acquisition module, as well as the computational, communication, and synchronization costs of the model to be trained, and estimates the training time cost.

[0076] S3. The parallel strategy search module integrates a parallel strategy optimizer designed with intelligent optimization algorithms. Based on the prediction results of the cost prediction module, it performs real-time screening and iterative optimization of candidate solutions, and finally generates the optimal heterogeneous parallel strategy.

[0077] S4. The scheduling and execution module receives the optimal parallel scheme output from the parallel strategy search module, divides the model according to the optimal parallel strategy, and distributes it to the GPU nodes in the specific sub-cluster. Then, it executes the model according to the parallel settings and parameters of the optimal strategy, completing the resource allocation, task scheduling, and execution control of the training task on each node of the cluster.

[0078] (1) Heterogeneous cluster environment: Refer to Figure 1 Suppose a heterogeneous cluster environment H consists of K different types of GPU devices, denoted as H = {H...} 1 H 2 H k H K}, where sub-cluster Hk The number of available nodes is N k The number of GPUs in each node is G k The total number of available GPU devices in the heterogeneous cluster environment H can be expressed as: Assume sub-cluster H k The network bandwidth between all GPUs is GBps, different sub-clusters H k The inter-node network bandwidth is B inter GBps, with peak computing power of Comp for each type of device. k FLOPs, the memory size for each type of device is Mem. k GB.

[0079] (2) Hybrid Training Task: Assume a large model M based on the Transformer architecture, consisting of Ln layers, with a total of Pn parameters, Nh hidden dimensions, and Seq_len sequence length. Considering compatibility issues between heterogeneous devices, in a heterogeneous cluster environment, pipelined parallelism is typically used between sub-clusters of different devices during hybrid training. Within sub-clusters of the same type of device, appropriate tensor and data parallelism can be set according to device memory and computing resources. This invention expresses the parallel optimization of heterogeneous hybrid training as follows: First, based on parameters such as the type and number of heterogeneous cluster devices and model structure, the training task of model M is divided into pp training stages, denoted as {S1, S2, ..., S...}. i ,…,S pp The actual execution sub-cluster of each stage is denoted as} Phase S i The number of model layers is denoted as L. i The data parallelism and tensor parallelism are dp and dp, respectively. i tp i The global batch size is GBS, and the local batch size for each stage is MBS. i .

[0080] (3) Training time estimation: For training a large model M in a specific heterogeneous cluster environment H, the total time cost T for heterogeneous hybrid training using a 1F1B pipeline parallel scheme across clusters of different devices is estimated. total , can be represented as:

[0081]

[0082] Among them, T stage (S i ) indicates stage S i The total execution time is calculated using the formula shown in equation (2); B represents the number of Micro Batches; TDPsync This represents the time overhead of synchronizing (all-reduce) the parallel parameters of the data after each Iteration or several Iterations, i.e., the pipeline synchronization time.

[0083] T stage (S i ) = T comp (S i )+T TPcomm (S i )+T PPcomm (S i (2)

[0084] Among them, T comp (S i ) indicates that this stage S i The computation time; T TPcomm (S i ) indicates stage S i The fine-grained communication time that occurs within the timeframe to achieve TP, i.e., the tensor parallel synchronization time; T PPcomm (S i This indicates the pipeline communication time between adjacent stages, i.e., the pipeline synchronization time.

[0085] To ensure the accuracy and reliability of the parallel optimization method proposed in this invention, the time cost of parallel training is modeled and estimated in detail below, where:

[0086] i. Calculation time T comp (S i ):

[0087] Phase S i The computation time for forward and backward propagation of all layers is affected by the computing power of a single GPU and the TP / DP partitioning, stage S i The total computational cost is FLOP(S). i It can be estimated as:

[0088] FLOP(S i )≈6·L i ·Nh 2 ·Seq·MBS i (3)

[0089] Execution Phase S i The peak computing power of each GPU in the sub-cluster is Comp. k (FLOPS), under the TP partitioning, each GPU participating in TP undertakes approximately FLOP(S) i ) / tp i The computational cost; if data-parallel dynamic programming is performed simultaneously, since the parameters are the same, the computational cost in tp is high. i(In parallel operation), the computation of each worker is performed in parallel and independently, therefore the computation time T of each stage is... comp (S i This can be represented as:

[0090]

[0091] Among them, C eff This indicates the effective computing power after considering actual utilization: 0 < C eff A scaling factor ≤ 1.

[0092] ii. Tensor parallel synchronization time T TPcomm (S i ):

[0093] Tensor parallelism (TP) typically occurs within each partitioned layer (e.g., AllGather / Reduce after matrix multiplication partitioning), frequently and with fine granularity. This invention uses an α-β model (startup delay + transmission time) to estimate the tensor parallelism synchronization time T based on the activation size of each operator. TPcomm (S i ), represented as:

[0094]

[0095] Where op represents stage S i Operators that require synchronization in tensor parallelism, α op The communication startup latency for tensor parallelism is the fixed overhead of a single intra-node communication call. For the execution phase S i sub-cluster H k The transmission latency per byte of data in msg_size op The amount of data (in bytes) that the operator op needs to communicate during TP parallelism is estimated using the following formula:

[0096]

[0097] Among them, dtype_size is the number of bytes per element (FP16 = 2B, FP32 = 4B).

[0098] iii. Production line synchronization time T PPcomm (S i ):

[0099] The activation transfer volume across stages is equal to the adjacent boundary activation size (bytes), and the pipeline synchronization time T. PPcomm (S i ) is represented as:

[0100] TPPcomm (S i )≈α pp +β pp ·activation_size (7)

[0101] Where, α pp For pipelined parallel message start delay; β pp =1 / B inter The transmission latency per byte of data between nodes of different types of devices is denoted by 'activation_size', which represents the size of the activation value that needs to be transmitted between the two stages.

[0102] iv. Data parallel parameter synchronization time T DPsync It is represented as:

[0103]

[0104] in, The two-stage communication from the ring algorithm of set communication, the number of synchronization times sync_times is the size of the parameter to be synchronized divided by the number of messages in each all-reduce.

[0105] (4) Parallel optimization problem modeling:

[0106] To automatically find the best parallel solution for training large models in a heterogeneous cluster environment, this invention formalizes the parallel policy search problem as a constrained optimization problem. The decision variables of this problem include data parallelism, tensor parallelism, pipeline parallelism, and model layer partitioning, etc. The optimization objective is the training iteration time or the overall training cost, and the constraints include memory capacity limitations, computational resource allocation limitations, and communication topology constraints. Specifically, it includes: first, assuming the parallel policy solution is a vector x:

[0107] x = {pp, dp, tp, L}

[0108] Where pp represents the pipeline parallelism, i.e., the number of stages in the training; dp = {dp1, dp2, K, dp} pp} represents the data parallelism of each stage; tp = {tp1, tp2, K, tp pp Let L = {L1, L2, K, L} represent the tensor parallelism of each stage. pp} represents the number of model layers in each stage;

[0109] The optimization objective is to find the optimal parallel strategy that minimizes the overall training time cost of heterogeneous hybrid training of large models in a heterogeneous cluster environment while satisfying constraints. Specifically, it can be expressed as:

[0110]

[0111] Where, σ * T represents the optimal parallel strategy. total This represents the total time cost of heterogeneous hybrid training;

[0112] The constraints include: (1) The memory of each GPU does not exceed the maximum memory: Mem cosum,di Mem represents the video memory consumed by device di. di (1) The memory capacity of device di is indicated; (2) The number of stages should not exceed the number of sub-clusters: pp≤K; (3) The dp*tp of each sub-cluster should be equal to the total number of devices: Among them, dp i Indicates stage S i Data parallelism, tp i Indicates stage S i Tensor parallelism, N index_Si Indicates stage S i The actual number of nodes in the sub-cluster, G index_Si Indicates stage S i The number of GPU devices on a single node of the actual sub-cluster.

[0113] (5) Parallel strategy search

[0114] To address the aforementioned optimization problem, this invention designs a parallel strategy search process based on intelligent optimization algorithms (such as genetic algorithms and particle swarm optimization). The core idea is as follows: First, several candidate parallel solutions are generated as an initial solution set; then, constraint detection and training time estimation models are used to determine the feasibility and performance of the candidate parallel solutions, updating the current optimal solution; next, new candidate solutions are generated through iterative updates of the algorithm; this process is repeated until the termination condition is met, ultimately outputting a globally near-optimal parallel training solution. (Refer to...) Figure 4 Specifically, it includes the following steps:

[0115] S111, Parameter Initialization and Population Initialization: Based on cluster resource information, model size and parallel strategy search space, set relevant parameters of the search algorithm (such as population size, number of iterations, crossover probability, mutation probability, etc.), and randomly or heuristically generate a batch of feasible initial candidate solutions to form the initial solution set or initial population.

[0116] S112. Feasibility Detection and Training Time Cost Assessment: Constraint checks are performed on each candidate solution (including memory capacity, GPU topology, bandwidth limitations, etc.). If a candidate solution does not meet the constraints, it is deemed infeasible and discarded; if it meets the constraints, it is deemed feasible. For feasible solutions, the training time cost or iteration overhead is calculated based on theoretical modeling and a communication / computation cost estimator to obtain the corresponding fitness value.

[0117] S113. Update the current optimal solution: Among the evaluated candidate solutions, compare their training time costs, update the solution with the best performance as the current optimal solution, and store it for future comparison.

[0118] S114. Search algorithm update operation:

[0119] Update operations are performed based on the iterative mechanism of intelligent optimization algorithms. For example, in genetic algorithms, new individuals are generated through selection, crossover, and mutation operators; in particle swarm optimization, new solutions are generated through position and velocity updates; and in Bayesian optimization, new exploration points are selected through a collection function. Newly generated candidate solutions must undergo feasibility testing and cost evaluation again before being added to the population or candidate solution set.

[0120] S115. Termination condition judgment: If the preset termination condition is met (such as the number of iterations reaching the upper limit, the optimal solution not being improved for several consecutive generations, the search time exceeding the budget, etc.), the algorithm stops; otherwise, return to step S112 and continue iterative search.

[0121] S116. Output the final solution: When the algorithm ends, the output record of the globally optimal parallel strategy is used as the final search result. This strategy can significantly reduce training time and cost while meeting resource constraints.

[0122] Example 1

[0123] (1) Environment configuration

[0124] This heterogeneous environment contains three different types of GPUs, denoted as Cluster A, Cluster B, and Cluster C. Their hardware configuration parameters are shown in Table 1.

[0125] Table 1

[0126] parameter Cluster A Cluster B Cluster C Single-card computing power (TFLOPS) 280 160 240 Single card memory 64 32 64 Number of nodes 10 16 8 Number of GPUs per node 8 8 8 Intra-node bandwidth GBps 600 400 500

[0127] The three types of GPUs differ significantly in computing power, memory, and quantity, forming a typical heterogeneous hybrid cluster environment with multiple vendors and configurations.

[0128] (2) Training tasks

[0129] The LLAMA-2-13B model, which contains 40 layers, was trained on this cluster.

[0130] The Transformer architecture, with a word vector dimension of 5120 and a large number of parameters, cannot be trained directly using a single type of GPU in this experimental environment. It requires distributed hybrid training through a multi-cluster, multi-dimensional parallel strategy.

[0131] (3) Implementation process

[0132] i. Feature Acquisition: The system collected the following key performance parameters:

[0133] Cluster A has a peak computing power of 280 TFLOPS per card and a network bandwidth of 600 GBps.

[0134] Cluster B has a peak single-card computing power of 160 TFLOPS and a network bandwidth of 400 GBps.

[0135] Cluster C has a peak single-card computing power of 240 TFLOPS and a network bandwidth of 500 GBps.

[0136] GPUs within each node are interconnected via NVLink, and across nodes they are interconnected via InfiniBand, with a unified cross-node bandwidth of 80GBps.

[0137] ii. Cost modeling:

[0138] Cost estimation:

[0139] Single-layer parameter quantity ≈ 13B ÷ 40 ≈ 0.325B ≈ 325M

[0140] Single-layer calculation FLOPs≈2×5120 2 ×batch_size (When batch = 2048, single-layer FLOPs ≈ 1.07e) 13 )

[0141] Calculation time (theoretical value):

[0142] Cluster A single card ≈38ms, Cluster B single card ≈66ms, Cluster C single card ≈45ms

[0143] Communication overhead estimation:

[0144] Assuming that gradient synchronization (AllReduce) requires transferring approximately 0.2GB per cycle (52GB / 256).

[0145] The latency is approximately 0.4ms in the NVLink environment and approximately 1.0ms in the IB environment.

[0146] iii. Parallel scheme generation:

[0147] The search space includes DP (data parallelism), TP (tensor parallelism), and PP (pipeline parallelism).

[0148] Candidate solutions are generated by a searcher based on swarm intelligence optimization algorithms (this embodiment uses genetic algorithms as an example, but is not limited to specific algorithm types) (see Tables 2 and 3 for details).

[0149] iv. Scheme Evaluation and Selection

[0150] The cost model calculates the iteration time, communication overhead, and memory utilization of each scheme.

[0151] Table 2

[0152] parameter Cluster A Cluster B Cluster C Single-card computing power (TFLOPS) 280 160 240 Single card memory 64 32 64 Number of nodes 2 4 2 Number of GPUs per node 8 8 8

[0153] Table 3

[0154]

[0155] The above results indicate that scheme P4 performs best in terms of computing power utilization, communication overhead, and load balancing.

[0156] Predicted iteration time: 158ms / Iteration;

[0157] Communication overhead: Approximately 12% of total duration;

[0158] Memory utilization: Class A / C GPUs ≈ 85%, Class B GPUs ≈ 78%;

[0159] v. Scheduling and Execution: The scheduling module distributes the P4 plan:

[0160] Cluster A: Responsible for the first 15 layers of the model, using 2-way tensor parallelism;

[0161] Cluster C: Responsible for the last 15 layers of the model, using 2-way tensor parallelism;

[0162] Cluster B: Responsible for the last 10 layers of the model, using 2-way tensor parallelism;

[0163] During execution, AllReduce and Pipeline are combined in parallel to ensure that communication and computation overlap.

[0164] (4) Implementation Results

[0165] In this embodiment, compared to the traditional uniform data parallel method: training time is reduced by about 21% (from 205ms to 158ms); cross-cluster communication overhead is reduced by about 30%; and overall GPU utilization is increased to 82% (about 15% higher than the traditional method).

[0166] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and its improved concept, should be covered within the scope of protection of the present invention.

Claims

1. An automatic parallel optimization method for training large models in hybrid heterogeneous clusters, characterized in that: Includes the following steps: S1. After receiving the hybrid training task, the system automatically collects the key configuration information of various GPUs in the hybrid heterogeneous cluster; S2. Based on the collected state information of the hybrid heterogeneous cluster environment and the computational, communication and synchronization overhead of the model to be trained, a bandwidth-aware training time estimation model is constructed, and the training time overhead is estimated. S3. Based on intelligent optimization algorithms, candidate parallel schemes are screened and iteratively optimized in real time according to the predicted results, and finally the best parallel strategy is generated. S4. The model is divided according to the optimal parallel construction strategy and distributed to the GPU nodes in the specific sub-cluster. Then, it is executed according to the parallel settings and parameters of the optimal parallel construction strategy to complete the resource allocation, task scheduling and execution control of the training task on each node of the cluster.

2. The automatic parallel optimization method for training large models in hybrid heterogeneous clusters according to claim 1, characterized in that: In step S1, the key configuration information includes computing power performance parameters, video memory capacity, intra-node and inter-node bandwidth, network topology, and runtime load status.

3. The automatic parallel optimization method for training large models for hybrid heterogeneous clusters according to claim 1, characterized in that: In step S2, the estimated training time overhead includes: the total time cost T for heterogeneous hybrid training of a large model M in a specific heterogeneous cluster environment H, using a 1F1B pipelined parallel scheme across clusters of different devices. total , is represented as: Where B represents the number of Micro Batches; T stage (S i ) indicates stage S i The total execution time is expressed as: T stage (S i )=T comp (S i )+T TPcomm (S i )+T PPcomm (S i ) (2) Among them, T comp (S i ) indicates that this stage S i The computation time is expressed as: FLOP(S i )≈6·L i ·Nh 2 ·Seq·MBS i (3) Among them, FLOP(S i ) indicates stage S i The total computational load on C; eff This indicates the effective computing power after considering actual utilization: 0 < C eff ≤1 scaling factor; TT Pcomm (S i ) indicates stage S i The fine-grained communication time that occurs within the timeframe for implementing TP, i.e., the tensor parallel synchronization time, is expressed as: Where op represents stage S i Operators that require synchronization in tensor parallelism, α op The communication startup latency for tensor parallelism is the fixed overhead of a single intra-node communication call. For the execution phase S i sub-cluster H k The transmission latency per byte of data in msg_size op The amount of data that the operator op needs to communicate when TP is in parallel is estimated by the following formula: Where dtype_size is the number of bytes for each element; T PPcomm (S i This represents the pipeline communication time between adjacent stages, i.e., the pipeline synchronization time, which is expressed as: T PPcomm (S i )≈α pp +β pp ·activation_size (7) Where, α pp For message start delay in pipelined parallelism; β pp =1 / B inter The transmission latency per byte of data between nodes of different types of devices is denoted by activation_size, which is the size of the activation value that needs to be transmitted between the two stages. T DPsync This represents the time overhead of synchronizing (all-reduce) the parallel parameters of the data after each Iteration or several Iterations, i.e., the pipeline synchronization time, which is expressed as: in, The two-stage communication from the ring algorithm of set communication, the number of synchronization times sync_times is the size of the parameter to be synchronized divided by the number of messages in each all-reduce.

4. The automatic parallel optimization method for training large models for hybrid heterogeneous clusters according to claim 1, characterized in that: In step S2, the construction of a bandwidth-aware training time estimation model based on the collected state information of the hybrid heterogeneous cluster environment and the computational, communication, and synchronization overhead of the model to be trained includes: formalizing the parallel policy search problem into a constrained optimization problem. The decision variables of this problem include data parallelism, tensor parallelism, pipeline parallelism, and model layer partitioning. The optimization objective is the training iteration time or the overall training cost. The constraints include memory capacity limitations, computational resource allocation limitations, and communication topology constraints. Specifically, it includes: first assuming that the parallel policy solution is a vector x: x = {pp, dp, tp, L} Where pp represents the pipeline parallelism, i.e., the number of stages in the training; dp = {dp1, dp2, K, dp} pp } represents the data parallelism of each stage; tp = {tp1, tp2, K, tp pp Let L = {L1, L2, K, L} represent the tensor parallelism of each stage. pp } represents the number of model layers in each stage; The optimization objective is to find the optimal parallel strategy that minimizes the overall training time cost of heterogeneous hybrid training of large models in a heterogeneous cluster environment while satisfying constraints. Specifically, it can be expressed as: Where, σ * T represents the optimal parallel strategy. total This represents the total time cost of heterogeneous hybrid training; The constraints include: (1) The memory of each GPU does not exceed the maximum memory: Mem cosum,di Mem represents the video memory consumed by device di. di (1) The memory capacity of device di is indicated; (2) The number of stages should not exceed the number of sub-clusters: pp≤K; (3) The dp*tp of each sub-cluster should be equal to the total number of devices: Among them, dp i Indicates stage S i Data parallelism, tp i Indicates stage S i Tensor parallelism, N index_Si Indicates stage S i The actual number of nodes in the sub-cluster, G index_Si Indicates stage S i The number of GPU devices on a single node of the actual sub-cluster.

5. The automatic parallel optimization method for training large models in hybrid heterogeneous clusters according to claim 1, characterized in that: In step S3, the intelligent optimization algorithm is used to perform real-time screening and iterative optimization of candidate parallel schemes based on the predicted results, and finally generate the optimal parallel strategy. This includes: first, generating several candidate parallel schemes as an initial solution set; then, using a constraint detection and training time estimation model to determine the feasibility and performance of the candidate parallel schemes and updating the current optimal solution; then, generating new candidate schemes through iterative update operations of the algorithm; and so on until the termination condition is met, finally outputting the globally near-optimal parallel training scheme.

6. The automatic parallel optimization method for training large models in hybrid heterogeneous clusters according to claim 5, characterized in that: Step S3 specifically includes the following steps: S111, Parameter Initialization and Population Initialization: Based on cluster resource information, model size and parallel strategy search space, set relevant parameters of the search algorithm, including population size, number of iterations, crossover probability, mutation probability, and randomly or heuristically generate a batch of feasible initial candidate solutions to form the initial solution set or initial population. S112. Feasibility Detection and Training Time Cost Assessment: Constraints are checked for each candidate solution. If a candidate solution does not meet the constraints, it is determined to be an infeasible solution and discarded. If a candidate solution meets the constraints, it is determined to be a feasible solution. For feasible solutions, the training time cost or iteration cost is calculated based on theoretical modeling and communication / computation cost estimator to obtain the corresponding fitness value. S113. Update the current optimal solution: Among the evaluated candidate solutions, compare their training time costs, update the solution with the best performance as the current optimal solution, and store it for future comparison. S114. Search Algorithm Update Operation: The update operation is performed according to the iterative mechanism of the intelligent optimization algorithm, including: in the genetic algorithm, new individuals are generated through selection, crossover and mutation operators; in the particle swarm optimization algorithm, new solutions are generated through position update and velocity update; in Bayesian optimization, new exploration points are selected through the acquisition function; the newly generated candidate solutions need to undergo feasibility testing and cost evaluation again, and then be added to the population or candidate solution set. S115. Termination condition judgment: If the preset termination condition is met, the algorithm stops; otherwise, return to step S112 to continue iterative search. S116. Output the final solution: When the algorithm ends, the output record of the globally optimal parallel strategy is used as the final search result. This strategy can significantly reduce training time and cost while meeting resource constraints.

7. An automatic parallel optimization system for training large models on hybrid heterogeneous clusters, characterized in that: The system consists of four parts: a feature acquisition module, a cost estimation module, a parallel strategy search module, and a scheduling and execution module, all connected via a bus communication system. Feature Acquisition Module: This module is responsible for automatically acquiring the status information of the hybrid heterogeneous cluster environment, namely the key configuration information of various GPUs in the hybrid heterogeneous cluster, and transmitting it to the cost estimation module. The key configuration information includes computing power performance parameters, memory capacity, intra-node and inter-node bandwidth, network topology, and runtime load status, etc. This module provides basic data support for subsequent modeling and optimization. Cost estimation module: This module establishes a bandwidth-aware training time estimation model based on the state information of the hybrid heterogeneous cluster environment collected by the feature acquisition module and the computational, communication and synchronization costs of the model to be trained, and feeds the estimation results back to the parallel policy search module. Parallel Strategy Search Module: This module integrates a parallel strategy optimizer designed with intelligent optimization algorithms. Based on the prediction results of the cost prediction module, it performs real-time screening and iterative optimization of candidate solutions, and finally generates the optimal heterogeneous parallel strategy. Scheduling and Execution Module: This module receives the optimal parallel construction strategy output from the parallel strategy search module, then divides the model according to the optimal parallel construction strategy and distributes it to the GPU nodes in the specific sub-cluster. It executes the model according to the parallel settings and parameters of the optimal parallel construction strategy, and completes the resource allocation, task scheduling and execution control of the training task on each node of the cluster.

Images