Cloud heterogeneous GPU cluster-oriented large model training resource configuration optimization method

Abstract

The invention discloses a large model training resource configuration optimization method for a cloud heterogeneous GPU cluster. The method comprises heterogeneous GPU resource configuration and resource perception stage deployment. A cost tree pruning mechanism is provided for resource configuration, a plan with low cost is searched preferentially, a configuration scheme is expressed as a tree path, meanwhile, a multi-dimensional pruning method is adopted to reduce unnecessary branches, and efficient GPU configuration is achieved. According to stage deployment, through defining a Compute-Memory Ratio (Compute-Memory Ratio, CMR), the requirements of each stage on the GPU computing power and memory capacity are considered in a combined manner, the communication and computing efficiency in the training process is optimized, and the large model training time is remarkably shortened. And finally, a large model training resource configuration optimization method Espresso oriented to the cloud heterogeneous GPU cluster is provided in combination with resource configuration and stage deployment strategies, and the user cost is reduced to the maximum extent while the training performance is ensured.

Description

Technical Field

[0001] The present invention relates to the field of large model resource configuration and optimization, and specifically to a large model training resource configuration optimization method for cloud-based heterogeneous GPU clusters. By reasonably allocating heterogeneous GPU resources and optimizing training stage deployment, the training cost can be significantly reduced while ensuring training performance. Background Art

[0002] With the rapid development of GPU technology, multiple types of GPUs (i.e., resource heterogeneity) are widely deployed in public cloud GPU clusters, including high-performance models (such as A100, H100) and economical models (such as T4, A10). This is due to the investment of many manufacturers in research and development and the rapid upgrading of GPUs. It also provides a certain flexibility for resource allocation and can support diverse training strategies. However, the existing distributed training of large models requires the partitioning of input data and model layers and the allocation of GPUs before training begins. This makes the existing distributed training architecture often rely on homogeneous GPUs and gang-scheduling scheduling, that is, all required GPU resources must be available before training jobs can be started. This traditional mechanism often causes cloud users to face GPU unavailability problems caused by GPU type restrictions or insufficient capacity, which in turn increases the queuing delay of training jobs. In addition, relying on homogeneous GPUs for training can also cause resource waste. For example, when the computing requirements of the model are slightly less than an integer number of GPUs, the configuration plan may allocate excess resources, resulting in increased costs and reduced GPU utilization. Therefore, how to minimize the user's monetary expenditure cost while ensuring the performance of training tasks in a heterogeneous GPU cluster environment through appropriate resource scheduling and stage deployment strategies has become a research topic that has attracted much attention.

[0003] Furthermore, compared with homogeneous GPU clusters, when optimizing resource allocation plans for heterogeneous GPU clusters, there is a larger search space. Determining the most cost-effective configuration plan (the type and corresponding number of allocated GPUs) requires higher computing overhead and affects the real-time response of the cluster. When deploying models for allocated heterogeneous resources, the pipeline parallel training mechanism in heterogeneous clusters needs to divide the model layer into multiple stages and allocate them to different GPUs. However, traditional methods usually consider the computing power or memory capacity of GPUs alone, fail to fully explore the collaborative optimization potential of heterogeneous GPUs, and ignore the diverse needs of computing and memory resources at different stages. Although recent studies have attempted to optimize stage deployment by balancing computing power, they have failed to fully consider the impact of communication time between GPUs. Therefore, how to comprehensively consider the computing power, memory capacity and communication overhead of heterogeneous GPUs and then design a more efficient large model deployment strategy needs to be studied urgently. Summary of the invention

[0004] The purpose of the present invention is to provide a large model training resource configuration optimization method for cloud-based heterogeneous GPU clusters. The method combines GPU resource configuration with stage deployment to minimize training costs while ensuring large model training performance.

[0005] The specific technical solution for achieving the purpose of the present invention is:

[0006] A large model training resource configuration optimization method for cloud heterogeneous GPU clusters includes the following steps:

[0007] Step 1: Set up a deep learning training task, including the Transformer architecture model, data set, training target SLO, and number of training rounds. Use the LLM-analysis analysis tool to analyze the training time and memory usage requirements of different layers in the model on multiple heterogeneous GPUs.

[0008] Step 2: Based on the analysis results in step 1, a cost tree pruning mechanism is proposed to represent the resource allocation plan as a path from the root node to the child node in the tree; the cost model is used to estimate the cost of the resource allocation plan; during the tree construction process, the cost of the resource allocation plan is continuously tracked, and a multi-dimensional pruning method is used to reduce unnecessary branches;

[0009] Step 3: Perform stage deployment according to the GPU resource configuration plan formulated in step 2. The model needs to be divided into multiple stages and the stages are placed on each GPU in the pre-obtained complete configuration plan. To minimize the training time, the time of one iteration is divided into communication time and computing time. Since the GPU computing power, memory capacity and the number of layers allocated in different stages will affect these two times, this step introduces the compute-memory ratio (CMR) to determine the GPU placed in each stage. Based on this, a resource-aware stage deployment strategy is designed to balance computing and communication to ensure the shortest completion time of the slowest stage and improve the overall training speed, thereby optimizing the GPU resource configuration in a heterogeneous environment.

[0010] Further, the step 2 specifically includes:

[0011] (1) Cost tree generation: Each node in the cost tree represents the type of GPU assigned and the number of GPUs determined by the data parallel scale. The path from the root node to the leaf node represents a complete GPU configuration plan, and the other paths are partial configuration plans. The path length of each complete configuration plan is equal to the set pipeline parallel scale. In order to avoid redundancy, the paths of repeated combinations, namely [A, B, C] and [B, A, C], are merged. During the search process, a cost attribute is assigned to each path to guide the search direction. The cost of the partial configuration plan is estimated by assuming that the remaining stages use the same GPU type, which is used as the cost upper bound of the plan and is used to quickly screen the partial configuration plan. For a partial configuration plan N, its cost is estimated by assuming that the remaining stages use homogeneous GPU configurations. Therefore, the cost is defined as:

[0012]

[0013] Where HomoConfig(N,t) calculates the training cost of one iteration when using GPU type t in the remaining stage according to the cost model; a lower value indicates that the probability of some configuration plans N becoming cost-effective plans is higher, so these plans are given priority in the search process, which significantly improves the pruning efficiency;

[0014] (2) Cost model: A cost model is used to estimate the minimum cost of the GPU configuration plan. The model calculates the cost based on the computing power and computing load of the GPU. For an iteration with a total computing load of W, if the sum of the computing power of all GPUs is r and the average GPU utilization is p, the minimum time is: where p depends on the bubble time and the average GPU utilization gp; for k stages and m mini-batches, the bubble time ratio is The working time ratio is Therefore, p is calculated as: According to the minimum time The minimum cost of GPU configuration plan can be predicted, that is, Where ct and nt represent the price of GPU type t and the number of GPU type t in the configuration plan, respectively;

[0015] (3) Multi-dimensional pruning: Boundary pruning (BP): The search space is narrowed by calculating the boundary values ​​of pipeline parallelism (PP Size) and data parallelism (DPSize). PP Size is determined according to the upper and lower limits of GPU memory capacity, while DP Size is set according to the power range of the total number of GPUs. Cost pruning (CP): The minimum training cost of some configuration plans is quickly estimated to ensure that the memory utilization exceeds the threshold, and plans are screened according to the computational load and cost lower bound to eliminate inefficient plans. Time and capacity pruning (TCP): The minimum training time and available memory of some configuration plans are estimated. If the training time of the plan exceeds the SLO or the memory is insufficient to meet the stage requirements, the plan and its derivative paths are discarded.

[0016] Further, the step 3 specifically includes:

[0017] (1) Optimization objective: The objective is to transform the objective from minimizing the maximum value of the completion time of each stage to optimizing the communication time and computing time. The specific formula is as follows:

[0018]

[0019] Among them, each stage i is assigned to GPU t, containing L(t,i) layers; Represents the time required from the start of the current iteration to the completion of the last backpropagation of stage i; is the time required for GPU t to perform one backpropagation at stage i, is the communication time required for gradient synchronization; due to the dependency between stages, the backward calculation of one stage depends on the gradient being passed after the backward calculation of the next stage is completed; therefore, when the first stage completes the backward transfer, the calculation of the entire pipeline is completed, so use represents the computation time of the entire pipeline, which can be optimized by balancing the stages. At the same time, since the communication time of each stage can be masked by the computation of the previous stages, the effective communication time is obtained by subtracting the computation time of the previous stages. Therefore, the effective communication time of the i-th stage is

[0020] (2) Three observations: 1) Increasing pipeline parallelism leads to higher memory requirements for stages, because they need to perform more forward computations when performing backward computations, and therefore need to store more activation values; 2) Communication time tends to increase with the number of layers; therefore, allocating more layers to later stages can mask the computations of the previous stages, thereby effectively reducing communication overhead; 3) Since the communication in the first stage cannot be masked, the communication time of this stage needs to be considered separately, and the communication time of subsequent stages must not exceed that of the first stage; therefore, stage deployment must take into account the differences in GPU computing power, memory capacity, and communication caused by the allocation of layers to different stages. To this end, the computational memory ratio is defined as follows:

[0021]

[0022] Where Gf(t) and Gm(t) represent the computing power and memory capacity of GPU t respectively; the ratio Reflects the strength of GPU computing power relative to memory; ratio The smaller it is, the more remaining memory the GPU has, and the more suitable it is to be placed in the stage with more video memory requirements. In addition, considering the communication constraints may result in fewer layers being placed, that is, insufficient utilization of GPU computing and memory resources. We further introduce a coefficient α(t) to measure memory resource utilization, which is defined as: α(t) = in, represents the number of layers assigned to GPU t under memory constraints, It represents the number of layers under the consideration of both memory and communication constraints. The larger α(t) is, the greater the impact of communication on GPU resource utilization.

[0023] The present invention solves the GPU resource configuration problem for large model training in a cloud heterogeneous environment, the task stage deployment problem under a distributed strategy, and the problem of minimizing training costs. The present invention proposes a new and efficient cost tree pruning mechanism, which prioritizes searching for configuration plans with lower costs, and adopts a multi-dimensional pruning method to reduce unnecessary branches, thereby quickly finding a cost-effective configuration plan. A resource-aware stage deployment strategy is designed, while considering the impact of GPU video memory computing power on training time. Under a given configuration plan, the training time is minimized, thereby reducing user costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A system architecture diagram for implementing the present invention;

[0025] Figure 2 It is a flow chart of the present invention;

[0026] Figure 3 This is a sample diagram of resource allocation plan based on cost tree pruning mechanism. DETAILED DESCRIPTION

[0027] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0028] The present invention proposes a large model training resource configuration optimization method Espresso for cloud-based heterogeneous GPU clusters to ensure large model training performance while minimizing user training costs.

[0029] like Figure 1 As shown, the process of implementing the present invention includes: when a user submits a training task (including a model, a data set, an SLO, and the number of training rounds), the system's LLM-analysis lightweight analysis tool is used to obtain the execution time and GPU memory consumption of different model layers on heterogeneous GPUs. Based on these analysis results, a GPU resource configuration plan is then generated using an efficient cost tree pruning mechanism. Subsequently, the model layer is reasonably divided into multiple stages through a resource-aware stage deployment strategy, and these stages are allocated to heterogeneous GPUs to optimize the training time. Finally, the training task is deployed in a cloud environment according to the generated GPU configuration plan and stage deployment plan.

[0030] See also Figure 2 , the specific steps of the present invention are:

[0031] Set up deep learning training tasks, including Transformer architecture models, data sets, training target SLOs, and number of training rounds. Use the LLM-analysis tool to analyze the training time and memory usage requirements of different layers in the model on multiple heterogeneous GPUs.

[0032] Next, we move on to GPU resource allocation and propose a cost tree pruning mechanism. We represent the resource allocation plan as a path from the root node to the child node in the tree and use the cost model to estimate the cost of the resource allocation plan, such as Figure 3 As shown in Figure 2. During the tree construction process, the cost of the resource allocation plan is continuously tracked, and a multi-dimensional pruning method is used to reduce unnecessary branches. The first step is to generate the cost tree, as shown in Figure 2. Figure 3As shown, each node in the cost tree represents the type of GPU assigned and the number of GPUs determined by the data parallel scale. The path from the root node to the leaf node represents a complete GPU configuration plan, and the other paths are partial configuration plans. The path length of each complete plan is equal to the set pipeline parallel scale. In order to avoid redundancy, paths representing repeated combinations, namely [A, B, C] and [B, A, C], will be merged. During the search process, a cost attribute is assigned to each path to guide the search direction; the cost of the partial configuration plan is estimated by assuming that the remaining stages use the same GPU type, which is used as the upper bound of the cost of the plan and is used to quickly screen the partial configuration plan; for a partial configuration plan N, its cost is estimated by assuming that the remaining stages use homogeneous GPU configurations. Therefore, the cost is defined as:

[0033]

[0034] Where HomoConfig(N,t) calculates the training cost of one iteration when using GPU type t in the remaining stage according to the cost model. Lower representation configuration plans N are more likely to be cost-effective plans, so these plans can be prioritized during the search process, significantly improving pruning efficiency.

[0035] Cost model: A cost model is used to estimate the minimum cost of the GPU configuration plan. The model calculates the cost based on the computing power and computing load of the GPU. For an iteration, the total computing load is W, if the sum of all GPU computing power is r, and the average GPU utilization is p, then the minimum time is: where p depends on the bubble time and the average GPU utilization gp. For k stages and m mini-batches, the bubble time ratio is The working time ratio is Therefore, p is calculated as: According to the above formula, the minimum cost of GPU configuration plan can be predicted, that is, where ct and nt represent the price of GPU t and the number of GPU t in configuration plan N, respectively.

[0036] Multi-dimensional pruning: The cost tree helps identify better plans faster by assigning a cost to each node, guiding the search along lower-cost paths. In addition, the cost model effectively evaluates the cost of each node, enabling faster identification of promising paths. However, despite these improvements, the search space remains large due to the inability to effectively prune unnecessary plans. To address this problem, the present invention further proposes a multi-dimensional pruning method for fine-grained pruning, such as Figure 3 shown.

[0037] (1) Boundary Pruning (BP): BP calculates the reasonable boundary values ​​of pipeline parallelism (PP Size) and data parallelism (DP Size) to reduce the search space. According to the calculation formula of p above, increasing k will reduce the active time ratio, thus requiring more GPU computing resources. In addition, increasing PP Size will increase GPU memory requirements because a large number of activation values ​​need to be stored. Therefore, if memory constraints allow, PP Size should be minimized as much as possible. However, a PPSize that is too small may require the use of a high-memory, high-cost GPU. To determine the optimal PP Size, the lower bound klower and upper bound kupper are analyzed. Specifically, klower is set to the minimum number of stages supported by the GPU with the largest memory, and kupper is set to the minimum number of stages supported by the GPU with the smallest memory. For the configuration of DP Size, the batch size is usually a power of 2, so the power range of DPSize is set to dlower = 0 to dlower = log2(x), where x is the number of GPUs.

[0038] (2) Cost-driven pruning (CP): CP uses a cost model to estimate the minimum training cost of partial configuration plans and filter out configuration plans with excessively high costs. The model evaluates the training cost based on the computational load and produces lower estimates for plans with fewer model layers. Unlike the above cost calculation, CP ensures that the GPU memory utilization remains above a threshold β to determine the minimum computational load that each GPU must handle, thereby estimating the minimum training cost of the partial configuration plan. If this training cost exceeds the cost of the current best complete plan, the partial plan is considered inefficient and is no longer generated.

[0039] (3) Time and Capacity Pruning (TCP): TCP estimates the minimum training time and memory requirements of partially configured plans to eliminate plans that exceed the SLO or cannot be started due to GPU memory limitations. It estimates the minimum training time by allocating the most powerful GPU to the remaining stages. In addition, TCP estimates the maximum available memory by deploying the remaining stages to the GPU with the largest memory. If the minimum training time of a partially configured plan exceeds the SLO, or the maximum memory provided by the plan is less than the minimum memory requirement of the kth stage (i.e., mlower), then the partial plan and all plans derived from it will be discarded.

[0040] For a complete configuration plan, it will be sent to the stage deployment algorithm to divide the model layers into PPsize stages and place the stages on the GPU. The optimization goal is to minimize the training iteration time, which can be converted into the shortest completion time of the slowest stage, thereby improving the overall training speed. The optimization goals are as follows:

[0041]

[0042] Here, each stage i is assigned to GPU t and contains L(t,i) layers. Represents the time required from the start of the current iteration to the completion of the last backpropagation of stage i. is the time required for GPU t to perform one backpropagation at stage i, is the communication time required for gradient synchronization. Due to the dependencies between stages, the backward calculation of one stage depends on the gradient being passed after the backward calculation of the next stage is completed. Therefore, when the first stage completes the backward pass, the calculation of the entire pipeline is completed, so use represents the computation time of the entire pipeline, which can be optimized by balancing the stages. At the same time, since the communication time of each stage can be masked by the computation of the previous stages, we obtain the effective communication time by subtracting the computation time of the previous stages. Therefore, the effective communication time of the i-th stage is

[0043]

[0044] A resource-aware stage deployment strategy is designed based on the following three observations: 1) Increasing pipeline parallelism leads to higher memory requirements for stages, since they need to perform more forward computations when performing backward computations, and therefore need to store more activations; 2) Communication time tends to increase with the number of layers. Therefore, assigning more layers to later stages can mask the computations of the previous stages, thereby effectively reducing communication overhead; 3) Since the communication in the first stage cannot be masked, the communication time of this stage needs to be considered separately, and ensure that the communication time of subsequent stages does not exceed that of the first stage. Therefore, stage deployment must consider the communication differences caused by GPU computing power, memory capacity, and the number of layers allocated to different stages. To this end, the computation-to-memory ratio is defined as follows.

[0045]

[0046] Where Gf(t) and Gm(t) represent the computing power and memory capacity of GPU t respectively. Reflects the strength of GPU computing power relative to memory. The smaller it is, the more spare memory the GPU has, and it is more suitable for being placed in stages that require more video memory. In addition, considering communication constraints may result in fewer layers being placed, which means that GPU computing and memory resources are underutilized. To measure memory resource utilization, α(t) is further introduced, defined as: in, represents the number of layers assigned to GPU t under memory constraints, It represents the number of layers under the consideration of both memory and communication constraints. The larger α(t) is, the greater the impact of communication on GPU resource utilization.

[0047] Based on the above analysis, Espresso designed a resource-aware stage deployment algorithm. Since the communication in the first stage cannot be hidden, Espresso controls the communication volume by enumerating the number of layers in this stage. During the stage deployment process, Espresso traverses each stage in order, selects the GPU with the smallest CMR and determines the corresponding number of layers. Considering that the communication in subsequent stages may overlap with the calculation in the previous stage, the communication constraints in different stages are different. To this end, communication constraints are introduced and updated by accumulating the reverse calculation time of the previous stages. The CMR of the GPU in each stage is adjusted according to the communication constraints. Through the above strategy, Espresso can efficiently allocate model stages in a multi-GPU environment, and make full use of the computing and memory resources of the GPU by balancing the computing and communication overheads, thereby improving training performance.

[0048] Based on the above functions, efficient resource allocation is achieved for large model training in a heterogeneous cloud environment, minimizing user costs while ensuring model training performance.

[0049] Example

[0050] In order to verify the performance of the proposed algorithm in a heterogeneous GPU environment, four representative GPU types are provided for the heterogeneous environment, namely A6000, A30, A4000, and RTX 3090. The cluster supports network isolation to ensure stable bandwidth, and the bandwidth between instances is 2.5Gbps. Considering that various popular models (such as GPT and LLaMA) are based on the Transformer architecture, four representative large Transformer-based models listed in Table 1 are selected to evaluate the performance of Espresso. Specifically, ViT is trained on the ImageNet-1K dataset for image classification. For language modeling, LLaMA-1.3B uses the Wikipedia dataset, while the LLaMA-3B and LLaMA-7B models are trained on the Alpaca dataset for question-answering tasks.

[0051] Evaluation Baselines and Evaluation Metrics: Espresso is compared with three state-of-the-art strategies: (1) Sia + Select the same type of quadratic GPU in a heterogeneous environment. (2) Whale + The GPUs are placed in each stage in descending order of memory capacity, and a model segmentation strategy based on load balancing is combined. (3) HPH + GPUs are placed in order of computing power from small to large at each stage, and a model segmentation strategy based on integer programming is combined. Espresso's GPU allocation algorithm is combined with Whale + and HPH + Combined to support GPU resource configuration in heterogeneous environments. In order to accurately evaluate the gap between the strategy of the present invention and the optimal strategy, Espresso is also compared with the optimal strategy Optimal obtained by exhaustive search.

[0052] Table 1 Description of typical large model loads

[0053]

[0054] Table 2 Comparison of Sia + 、Whale + , HPH + , Optimal and Espresso strategies cost under different training target SLOs (in RMB)

[0055]

[0056]

[0057] Four representative large-scale models listed in Table 1 are evaluated to explore Sia + 、Whale + , HPH + , Optimal, and Espresso. Specifically, each of the four models is trained through 100 iterations and different SLOs are set. As shown in Table 2, compared with other strategies, Espresso successfully meets the SLO while reducing the monetary cost by up to 49.8%. + Although the strategy can effectively reduce training time, it mainly achieves this goal by over-allocating GPUs. This strategy allocates a single type of GPU and ignores the potential benefits of using multiple GPU types in heterogeneous environments. In order to meet the SLO, this strategy often needs to allocate more resources than the model requires, which inevitably leads to increased user costs. The N / A position in Table 2 indicates that Sia is used when resources are limited or computing requirements are high.+ No configuration plan can be provided, resulting in job launch failure. In contrast, Espresso effectively utilizes different GPU types and provides flexible configuration plans to ensure successful job launch. + and HPH + All of them show defects in GPU sequence strategy. + More layers are allocated in the front stage, balancing the pipeline, but increasing the communication time for gradient synchronization. In scenarios with longer computation time, Whale + However, in scenarios with shorter computation time, the increase in communication time leads to a decrease in performance. In contrast, HPH + Selecting a GPU with smaller memory, allocating fewer layers in the front stage, and increasing the number of layers in the back stage is to cover up the communication time. In scenarios where communication time dominates, HPH + Execution is better. However, HPH + It does not fully account for the high memory requirements of the previous stage, which may cause pipeline imbalance and prolong training time. When the stage deployment strategy is consistent, Espresso takes longer to train than HPH. + The growth rate is 0.8% to 2.6%. This is mainly due to HPH + Integer programming is used to determine the optimal model splitting plan. However, for models with more layers, this approach may incur significant runtime overhead.

[0058] The embodiments of the present invention can also provide a resource configuration system that ensures the training performance of large models, and the system includes: a large model analysis module, which predicts video memory and performance based on tasks submitted by users; a resource configuration module, which finds the lowest cost plan among candidate GPU configuration plans for the large model based on an efficient cost tree pruning mechanism and ensures SLO; a stage deployment module, which selects the appropriate GPU type and number of model layers for each stage according to a given GPU configuration plan to minimize training time; and a performance prediction module, which establishes a performance prediction model according to the stage configuration strategy to predict the training performance of the large model.

[0059] When users submit deep learning training tasks, including models, data sets, SLOs, and the number of training rounds, GPU resource configuration and stage deployment can be automatically completed, and user training costs can be minimized while meeting the SLO.

Claims

1. A large model training resource configuration optimization method for cloud heterogeneous GPU clusters, characterized in that: The following steps are involved: Step 1: Set up a deep learning training task, including the Transformer architecture model, data set, training target SLO, and number of training rounds, and use the analysis tool LLM-analysis to analyze the training time and memory usage requirements of different layers in the model on multiple heterogeneous GPUs; Step 2: Based on the analysis results in step 1, a cost tree pruning mechanism is proposed to represent the resource allocation plan as a path from the root node to the child node in the tree; The cost model is used to estimate the cost of the resource allocation plan. During the tree construction process, the cost of the resource allocation plan is continuously tracked, and a multi-dimensional pruning method is used to reduce unnecessary branches. Step 3: According to the GPU resource configuration plan formulated in step 2, stage deployment is performed, the model is divided into multiple stages, and the stages are placed on each GPU in the pre-obtained complete configuration plan; to minimize the training time, the time of one iteration is divided into communication time and computing time, and the computation-memory ratio CMR is introduced to determine the GPU placed in each stage, and a resource-aware stage deployment strategy is designed based on CMR, aiming to optimize the GPU resource configuration in a heterogeneous environment by balancing computing and communication, ensuring the shortest completion time of the slowest stage, and improving the overall training speed; among them: The step 2 specifically includes: (1) Cost tree generation: Each node in the cost tree represents the type of GPU assigned and the number of GPUs determined by the data parallel scale. The path from the root node to the leaf node represents a complete GPU configuration plan, and the other paths are partial configuration plans. The path length of each complete configuration plan is equal to the set pipeline parallel scale. In order to avoid redundancy, the paths of repeated combinations, namely [A, B, C] and [B, A, C], are merged. During the search process, a cost attribute is assigned to each path to guide the search direction. The cost of the partial configuration plan is estimated by assuming that the remaining stages use the same GPU type, which is used as the cost upper bound of the plan and is used to quickly screen the partial configuration plan. For a partial configuration plan N, its cost is estimated by assuming that the remaining stages use homogeneous GPU configurations, which is defined as: Where HomoConfig(N,t) calculates the training cost of one iteration when using GPU type t in the remaining stage according to the cost model; a lower value indicates that the probability of some configuration plans N becoming cost-effective plans is higher, so these plans are given priority in the search process, which significantly improves the pruning efficiency; (2) Cost model: A cost model is used to estimate the minimum cost of the GPU configuration plan. The model calculates the cost based on the computing power and computing load of the GPU. For an iteration with a total computing load of W, if the sum of the computing power of all GPUs is r and the average GPU utilization is p, the minimum time is: where p depends on the bubble time and the average GPU utilization gp; for k stages and m mini-batches, the bubble time ratio is The working time ratio is Therefore, p is calculated as: According to the minimum time The minimum cost of GPU configuration plan can be predicted, that is, Where ct and nt represent the price of GPU type t and the number of GPU type t in the configuration plan, respectively; (3) Multi-dimensional pruning: Boundary pruning (BP): By calculating the boundary values ​​of pipeline parallelism (PP) and data parallelism (DP), the search space is narrowed down. PP is determined according to the upper and lower limits of GPU memory capacity, while DP is set according to the power range of the total number of GPUs. Cost pruning (CP): By quickly estimating the minimum training cost of some configuration plans, the memory utilization rate is ensured to exceed the threshold, and plans are screened according to the computational load and cost lower bound to eliminate inefficient plans. Time and capacity pruning (TCP): Estimate the minimum training time and available memory of some configuration plans. If the training time of the plan exceeds the SLO or the memory is insufficient to meet the stage requirements, the plan and its derivative paths are discarded. The step 3 specifically includes: (1) Optimization objective: The objective is to transform the objective from minimizing the maximum value of the completion time of each stage to optimizing the communication time and computing time. The specific formula is as follows: Among them, each stage i is assigned to GPU t, containing L(t,i) layers; Represents the time required from the start of the current iteration to the completion of the last backpropagation of stage i; is the time required for GPU t to perform one backpropagation at stage i, is the communication time required for gradient synchronization; use represents the computation time of the entire pipeline, which can be optimized by balancing the stages. At the same time, since the communication time of each stage can be masked by the computation of the previous stages, the effective communication time is obtained by subtracting the computation time of the previous stages. Therefore, the effective communication time of the i-th stage is (2) The calculation memory ratio is defined as follows: Where Gf(t) and Gm(t) represent the computing power and memory capacity of GPU t respectively; the ratio Reflects the strength of GPU computing power relative to memory; ratio The smaller it is, the more remaining memory the GPU has, and it is more suitable for being placed in stages with more video memory requirements. In addition, considering communication constraints will result in fewer layers being placed, which means that GPU computing and memory resources are underutilized. We further introduce a coefficient α(t) to measure memory resource utilization, which is defined as: in, represents the number of layers assigned to GPU t under memory constraints, It represents the number of layers under the consideration of both memory and communication constraints. The larger α(t) is, the greater the impact of communication on GPU resource utilization.

Images