Scheduling method and device of training task and electronic equipment

Abstract

This application provides a method, apparatus, and electronic device for scheduling training tasks, relating to the field of artificial intelligence model training technology. The method includes: upon obtaining a training task, determining the layer data information of the model based on the model information corresponding to the training task; obtaining graphics processing unit (GPU) resource information; determining a parallel allocation strategy for the training task based on the GPU resource information and the model's layer data information; dividing the training task into multiple training slices based on the parallel allocation strategy; and executing the training task corresponding to each training slice on the GPU resource corresponding to each training slice according to the communication dependencies. Thus, by generating a parallel allocation strategy for training tasks based on heterogeneous GPU resource information and model layer data information, and performing task slices according to the allocation strategy, the utilization rate of GPUs and the training efficiency of large-scale AI models in multi-cloud environments can be improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence model training technology, and in particular to a method, apparatus and electronic device for scheduling training tasks. Background Technology

[0002] Current artificial intelligence (AI) training platforms typically only support GPU resource management within a single cloud environment or cluster, using a static quota allocation method to allocate GPU resources for training tasks. When submitting a task, users must manually specify the number and model of GPUs, and the scheduling platform allocates resources from a single resource pool using a First Come First Served (FCFS) or priority queue approach.

[0003] However, in multi-cloud business scenarios, computing resources are distributed among multiple cloud service providers. Using the above scheduling method will cause the GPU resources of different cloud service providers to be fragmented, resulting in low GPU utilization and thus low training efficiency of training tasks. Summary of the Invention

[0004] This application provides a method, apparatus, and electronic device for scheduling training tasks to address the problem of low GPU utilization in multi-cloud environments, which leads to low training efficiency of training tasks.

[0005] To solve the above-mentioned technical problems, this application is implemented as follows:

[0006] In a first aspect, embodiments of this application provide a method for scheduling training tasks, the method comprising:

[0007] Once a training task is obtained, the layer data information of the model is determined based on the model information corresponding to the training task.

[0008] Obtain graphics processing unit (GPU) resource information, which includes attribute information of heterogeneous GPU resources located on different cloud platforms and information used to characterize the topological connection relationship between GPU resources;

[0009] Based on the GPU resource information and the layer data information of the model, a parallel allocation strategy for the training task is determined. The parallel allocation strategy includes at least one of the following: data parallelism, pipeline parallelism, and tensor parallelism.

[0010] Based on the parallel allocation strategy, the training task is divided into multiple training slices, each of which includes allocated GPU resources and communication dependencies with other training slices.

[0011] On the GPU resources corresponding to each training slice, the training task corresponding to each training slice is executed according to the communication dependency.

[0012] Secondly, embodiments of this application provide a scheduling device for training tasks, the device comprising:

[0013] The first determining module is used to determine the layer data information of the model based on the model information corresponding to the training task when the training task is obtained.

[0014] The acquisition module is used to acquire graphics processing unit (GPU) resource information, which includes attribute information of heterogeneous GPU resources located on different cloud platforms and information used to characterize the topological connection relationship between GPU resources.

[0015] The second determining module is used to determine the parallel allocation strategy of the training task based on the GPU resource information and the layer data information of the model. The parallel allocation strategy includes at least one of the following: data parallelism, pipeline parallelism, and tensor parallelism.

[0016] The segmentation module is used to segment the training task into multiple training segments based on the parallel allocation strategy, each training segment including allocated GPU resources and communication dependencies with other training segments;

[0017] An execution module is configured to execute the training task corresponding to each training segment on the GPU resources corresponding to each training segment, based on the communication dependency relationship.

[0018] Thirdly, embodiments of this application provide an electronic device, including: a processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, it implements the steps of the training task scheduling method described in the first aspect.

[0019] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the training task scheduling method described in the first aspect.

[0020] Fifthly, a computer program product is provided, including computer instructions that, when executed by a processor, implement the steps of the scheduling method for the training task as described in the first aspect.

[0021] In this embodiment, upon obtaining a training task, the layer data information of the model is determined based on the model information corresponding to the training task; GPU resource information is obtained, including attribute information of heterogeneous GPU resources located on different cloud platforms and information representing the topological connection relationships between GPU resources; based on the GPU resource information and the layer data information of the model, a parallel allocation strategy for the training task is determined, the parallel allocation strategy including at least one of the following: data parallelism, pipeline parallelism, and tensor parallelism; based on the parallel allocation strategy, the training task is divided into multiple training slices, each training slice including allocated GPU resources and communication dependencies with other training slices; on the GPU resources corresponding to each training slice, the training task corresponding to each training slice is executed according to the communication dependencies. Thus, by generating a parallel allocation strategy for the training task based on the information of heterogeneous GPU resources on different cloud platforms and the layer data information of the model, and performing task slices according to the allocation strategy, unified scheduling of heterogeneous GPU resources in a multi-cloud environment can be achieved, improving GPU utilization and training efficiency of large-scale AI models in a multi-cloud environment. Attached Figure Description

[0022] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart of a training task scheduling method provided in an embodiment of this application;

[0024] Figure 2 This is a resource topology diagram provided in an embodiment of this application;

[0025] Figure 3 This is a flowchart of an intelligent training task segmentation method provided in an embodiment of this application;

[0026] Figure 4 This is a schematic diagram of a cross-cloud communication optimization strategy decision tree provided in an embodiment of this application;

[0027] Figure 5 This is a schematic diagram of the scheduling system architecture for training tasks provided in an embodiment of this application;

[0028] Figure 6 This is a schematic diagram of the structure of a training task scheduling device provided in an embodiment of this application;

[0029] Figure 7This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0030] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0031] The following is an explanation of the terms used in this application.

[0032] Training Shard: A standardized training unit formed by dividing the AI ​​model training task according to a parallel strategy. It contains at least four core elements: the list of allocated GPU resources, the scope of the responsible model layer or data shard, the communication dependencies with other shards, and the checkpoint to save the configuration.

[0033] Multi-Cloud Environment: refers to a cloud computing environment provided by two or more different cloud service providers, including any combination of public cloud, private cloud, and hybrid cloud;

[0034] Ultra-Large Scale AI Model: refers to large language models (LLM), multimodal large models, and other artificial intelligence models with 10 billion (10B) or more parameters.

[0035] In large-scale distributed training scenarios of AI models across multiple cloud service providers (public cloud, private cloud, hybrid cloud), especially in ultra-large-scale parameter model training scenarios such as large language models and multimodal large models, there is a massive demand for GPU computing resources. However, current GPU resource scheduling is only applicable to a single resource pool, and the efficiency is low when scheduling resources for the above scenarios.

[0036] The communication layer of training frameworks is typically optimized for high-speed interconnect networks within a single data center (such as InfiniBand), but lacks effective communication optimization methods for wide area network scenarios across clouds and data centers. When training tasks need to be deployed across clouds, gradient synchronization communication is severely constrained by the high latency and low bandwidth of wide area networks, leading to a sharp decline in training efficiency.

[0037] If distributed training frameworks (such as Distributed Data Parallel (PyTorch DDP), DeepSpeed, or Megatron-LM) are used for data parallel processing, the selection and configuration of parallel strategies rely entirely on the manual experience of training engineers. Parallelism planning and parameter configuration must be performed manually based on factors such as model architecture, GPU memory size, and inter-node bandwidth, lacking automated strategy recommendation capabilities.

[0038] In summary, the above scheduling method has the following drawbacks:

[0039] The scheduling system cannot uniformly perceive and manage GPU resources across multiple cloud service providers. GPU resources in each cloud become isolated, and it is impossible to make optimal cross-cloud resource allocation based on factors such as global resource availability, GPU model compatibility, and network topology affinity. This results in low GPU utilization and a prominent contradiction of insufficient resources in one cloud and idle resources in other clouds during peak periods.

[0040] The selection of parallel strategies involves a comprehensive consideration of multiple factors, such as model structure analysis, GPU computing and memory capability assessment, and communication overhead estimation. Manual decision-making is inefficient, has a high error rate, and cannot adapt to the more complex parallelism planning requirements in multi-cloud heterogeneous GPU environments, frequently leading to GPU memory overflow or wasted computing resources.

[0041] When training tasks are deployed across clouds, the latency and bandwidth of gradient synchronization communication are limited by the quality of the wide area network between clouds. The current communication framework lacks adaptive optimization capabilities for multi-cloud scenarios and cannot dynamically adjust the gradient compression ratio, synchronization strategy (synchronous / asynchronous / hybrid) and communication topology according to the real-time network conditions between clouds, resulting in a 40%-70% reduction in cross-cloud training efficiency compared to single-cloud training.

[0042] This application provides a method, apparatus, and electronic device for scheduling training tasks to address the problem of low GPU utilization in multi-cloud environments.

[0043] See Figure 1 , Figure 1 This is a flowchart of a training task scheduling method provided in an embodiment of this application, such as... Figure 1 As shown, the method includes the following steps:

[0044] Step 101: If a training task is obtained, determine the layer data information of the model based on the model information corresponding to the training task;

[0045] Step 102: Obtain graphics processing unit (GPU) resource information, which includes attribute information of heterogeneous GPU resources and information used to characterize the topological connection relationship between GPU resources;

[0046] Step 103: Based on the GPU resource information and the layer data information of the model, determine the parallel allocation strategy for the training task. The parallel allocation strategy includes at least one of the following: data parallelism, pipeline parallelism, and tensor parallelism.

[0047] Step 104: Divide the training task into multiple training slices based on the parallel allocation strategy. Each training slice includes allocated GPU resources and communication dependencies with other training slices.

[0048] Step 105: On the GPU resources corresponding to each training slice, execute the training task corresponding to each training slice according to the communication dependency relationship.

[0049] Once a training task is obtained, the corresponding model information, such as model definition files and model structure information, is retrieved based on the training task. The model can be an artificial intelligence model used to perform the training task, such as a large language model or a multimodal large model.

[0050] In some implementations, when a user-submitted model definition file is obtained, the model definition file (such as PyTorch nn.Module) is parsed to directly extract the model's layer data information, including the number of parameters, computational cost (floating point operations (FLOPs)), memory usage of activation values, and data dependencies between layers, to construct a model computation graph. Here, a model layer can be a collection of neurons / operational units used to perform the same type of computational operation.

[0051] In some implementations, based on the layer data information statically obtained from the model definition file, a small-batch trial run of the model is used, and the obtained layer data information is adjusted based on the results to obtain adjusted layer data information. In other implementations, a simulation simulator can also be used to run the model and adjust the statically obtained layer data information to obtain adjusted layer data information. The adjusted layer data information can be used to determine the parallel allocation strategy for training tasks.

[0052] In addition, it obtains GPU resource information corresponding to heterogeneous GPUs on different cloud platforms. Heterogeneous GPUs can be GPUs with differences in hardware specifications, models, etc., and can come from different cloud service providers or cloud resource pools, such as GPUs in cloud computing environments provided by two or more different cloud service providers.

[0053] Multi-cloud GPU resource information can include stable parameters (GPU model, memory capacity, inter-node network bandwidth, GPU instance pricing, available regions and available quantities) and variable parameters (GPU real-time utilization, network latency and packet loss rate). Stable parameters can be obtained using the periodic probing method of the multi-cloud resource adapter, while variable parameters can be obtained using the high-frequency sampling and real-time updating method of the network probing agent.

[0054] In practical implementation, a multi-cloud resource adapter can be used to connect to the resource management application programming interfaces (APIs) of each cloud service provider. This allows for periodic probing of the available GPU instance types, quantities, and regional distribution information for each cloud, with the probing results uniformly registered to the global GPU resource registry. The information collected by the resource adapter includes: GPU model (e.g., A100, H100, L40S), GPU memory capacity, GPU interconnect topology (high-speed interconnect links / switches / Peripheral Component Interconnect Express (PCIe)), inter-node network bandwidth, GPU instance pricing (on-demand / reserved / spot bidding), available regions, and available quantities.

[0055] The attribute information of GPU resources can include the computing power value and video memory capacity of the GPU resources, and the topology connection relationship can include the communication indicators between GPU resources.

[0056] In some implementations, a resource topology map can be constructed based on the attribute information of GPU resources and information used to characterize the topological relationships between GPU resources, so as to obtain the above resource information more intuitively.

[0057] After acquiring GPU resources and model layer data, a cost model is constructed. Based on the parallelism value corresponding to each candidate parallel strategy (including data parallelism, pipeline parallelism, and tensor parallelism), the estimated training throughput and training cost are calculated. A strategy is then selected based on user preference, choosing either the strategy with the highest training throughput, the strategy with the lowest training cost, or the strategy with the highest combined score for both training throughput and training cost.

[0058] Among them, the data parallelism dimension: the degree of data parallelism is determined based on the total amount of training data, the global batch size, and the single GPU memory capacity, and the training data is evenly distributed to each GPU;

[0059] Pipeline parallelism: Based on the number of model layers, the computational cost of each layer, and the memory usage, the model is divided into several pipeline stages, with each stage assigned to a different GPU node. The selection principle for the split points is to minimize the communication between stages and balance the computational cost of each stage.

[0060] Tensor parallel dimension: For single-layer layers with a very large number of parameters (such as the attention layer and feed-forward network (FFN) layer of Transformer), the weight matrix of the single layer is split into columns or rows and distributed to multiple GPUs within the same node, and tensor fragmentation communication is performed by using the high-speed NVLink interconnect within the node.

[0061] Based on the multiple parallel dimensions in the above parallel strategy, a hierarchical parallel strategy of "intra-cloud tensor parallelism -> intra-cloud pipeline parallelism -> cross-cloud data parallelism" can be adopted. Tensor parallelism is first limited to the same node (high bandwidth interconnection), pipeline parallelism is distributed among different nodes in the same cloud (medium bandwidth), and data parallelism replicas are distributed among different clouds (low bandwidth but lowest communication frequency).

[0062] Using the parallel strategy described above, the overall training task is divided into multiple training shards. Each training shard includes: a list of allocated GPU resources, the range of model layers or data shards it is responsible for, communication dependencies with other shards, and a checkpoint configuration. The training shards are output in a standardized task description format as input for subsequent cross-cloud scheduling and execution.

[0063] Based on the GPU resources corresponding to each training slice, the training task corresponding to the training slice is executed.

[0064] In some implementations, communication strategies can be adjusted based on the communication requirements between training shards and the real-time network conditions between multiple clouds to improve communication performance.

[0065] This application's embodiments generate a parallel allocation strategy for training tasks based on information about heterogeneous GPU resources on different cloud platforms and the layer data information of the model. Task sharding is then performed according to the allocation strategy, enabling unified scheduling of heterogeneous GPU resources in a multi-cloud environment. This improves the utilization rate of large-scale GPUs and the training efficiency of AI models in multi-cloud environments. It is applicable to distributed training scenarios for ultra-large-scale AI models (10 billion or more parameters) across two or more cloud service providers.

[0066] Optionally, after obtaining the graphics processing unit (GPU) resource information, the method further includes:

[0067] The GPU resource information is converted into resource objects with the same resource metrics, and a resource topology graph is constructed based on the resource objects. The resource topology graph includes nodes representing the organizational level of the GPU resources and edges representing the network connection relationships between the nodes. The weights of the edges are used to represent the network characteristics between the nodes. The resource topology graph is used to determine the parallel allocation strategy for the training task.

[0068] After obtaining resource information of heterogeneous GPU resources from various cloud service providers, the heterogeneous GPU resource descriptions are converted into a unified GPU resource object model, which has the same resource indicators, including: computing power (based on GPU model FP16 / FP32 / BF16 TFLOPS), video memory capacity, video memory bandwidth, intra-node GPU interconnect bandwidth, inter-node network bandwidth, and other standardized indicators, in order to eliminate the differences in resource descriptions among various cloud providers and provide a consistent resource view for upper-layer scheduling.

[0069] A resource topology graph is constructed based on unified resource metrics. Taking the construction of a five-level weighted graph data structure as an example, such as... Figure 2 As shown, the resource topology graph includes GPU nodes, edges (used to represent the connection relationships between nodes), and edge weights (used to represent the network characteristics (such as bandwidth attributes) between nodes).

[0070] For example, nodes in a resource topology graph are divided into five organizational levels (such as cloud service provider nodes, regional nodes, availability zone nodes, physical nodes, and GPU nodes). Edges represent the network connections between nodes, and the edge weights between adjacent levels represent the network characteristics (such as bandwidth and latency) between adjacent levels. The edge weights between nodes at the same level represent lateral communication capabilities. The five organizational levels mentioned above are just examples; resource topology graphs containing other numbers of organizational levels can also be constructed.

[0071] The parallel allocation strategy for training tasks is determined based on the resource topology graph and the layer data information of the model.

[0072] Specifically, based on the resource topology graph, a hierarchical partitioning method is employed to transform the cross-cloud resource scheduling problem into a constrained optimization problem on the graph. This allows parallel policy search to leverage the graph's topological affinity for efficient pruning. A real-time edge weight update mechanism (driven by a network probing agent) enables the resource topology graph to reflect dynamically changing cross-cloud network quality, providing real-time data for policy decisions in the communication optimization layer. The resource topology graph is organized in a hierarchical structure of "cloud service provider -> region -> availability zone -> node -> GPU," reflecting the usage status and available capacity of resources at each level in real time, providing a global decision-making basis for training sharding scheduling.

[0073] The system call cost model calculates T_total = T_compute + T_comm + T_bubble, where T_compute can be estimated from the actual TFLOPS and profiling data of the GPU, T_comm can be calculated from the edge weights (actual network bandwidth) and gradient data volume in the five-layer resource topology graph, and T_bubble can be determined by the number of pipeline stages and the number of micro-batches. The model outputs the estimated training throughput and training cost for each parallel allocation strategy.

[0074] Each parallel allocation strategy is evaluated, and the strategy with the lowest training throughput, training cost, or the highest overall score (i.e., the total score that takes into account both training throughput and training cost) can be selected as the final parallel allocation strategy.

[0075] Traditional resource management solutions are limited to a single cloud environment or a single cluster. They allocate GPU cards for training tasks through static quota allocation, which cannot perceive the availability, pricing differences, and interconnection topology characteristics of GPU resources from different cloud service providers in a multi-cloud environment, resulting in isolated GPU resources in each cloud.

[0076] By using a unified pooling mechanism for multi-cloud GPU resources, multi-cloud resource discovery and registration, and a unified abstract model for heterogeneous resources to build a global resource topology, it is possible to obtain GPU resources across clouds and perform unified scheduling, thereby improving task processing efficiency.

[0077] Optionally, after determining the layer data information of the model based on the model information corresponding to the training task, the process includes:

[0078] A model computation graph is constructed based on the layer data information. The model computation graph includes nodes for representing attribute information of each layer of the model, and edges for representing data flow dependencies between the layers of the model.

[0079] The forward and backward propagation of the model are performed using a first GPU, and the actual performance metrics of each level of the model are measured. The actual performance metrics include at least one of the following: computation time and video memory usage data. The first GPU is any GPU in the GPU resources.

[0080] The attribute information of the nodes in the model computation graph is updated using the actual performance metrics; the updated model computation graph is used to determine the parallel allocation strategy for the training task.

[0081] After obtaining the model definition file, traverse the nn.Module layers in the model definition file and create a computation graph node for each computation layer. The node attributes include: layer type, number of parameters, computation amount, activation value, and memory usage. Analyze the data flow dependencies between layers and create directed edges between layers where data is transmitted. The edge attributes include: amount of data transmitted and direction of data transmission. Construct the model computation graph based on the above computation graph node, edge, and edge attributes.

[0082] The model computation graph is topologically sorted to obtain layer groups with dependencies. Furthermore, dependency-free layer groups that can be executed in parallel are identified and marked as candidate regions for parallel computation. In this model computation graph, nodes represent each computation layer or submodule of the model, and the node weights are the number of parameters and computational cost of that layer; directed edges represent data dependencies between layers, and the edge weights are the amount of data transferred between layers.

[0083] Based on the above model calculation diagram, a combination of static analysis and small-batch trial operation was used to obtain accurate calculation time and memory usage data for each layer.

[0084] In the static analysis phase, without executing the model, the system parses the layer definitions and parameter initialization code in the model definition file to statically deduce the number of parameters and the memory usage of activation values ​​for each layer, constructing a coarse-grained computational graph. Specifically, it first iterates through all named parameters of the model and sums the total number of parameters to obtain the total number of model parameters; then it groups by layer and counts the number of parameters (Q / K / V projection matrices and output projection matrices) of the self-attention sublayer and the number of parameters (dimensionality-upgrading and dimensionality-reducing matrices) of the FFN sublayer for each Transformer layer; finally, based on the global batch size and sequence length, it calculates the memory usage of activation values ​​for each layer.

[0085] During the small-batch trial phase, the system performed forward and backward propagation once each on a single GPU with a micro_batch_size=1. The actual performance metrics for each layer were precisely measured using the PyTorch Profiler tool, including at least one of the following: actual forward computation time, backward computation time, peak activation memory usage, and actual GPU TFLOPS utilization. Specifically, the model was first initialized and weights were loaded on the first GPU. A micro_batch of training data was input, and forward propagation was performed, recording the start and end timestamps of each layer and the GPU core utilization. Then, backward propagation was performed, recording the gradient computation time and peak memory usage for each layer. Finally, the profiling data was stored by layer index in the corresponding node attributes of the model's computation graph, serving as the basis for accurately calculating T_compute in the cost model.

[0086] Based on GPU resource information and the updated model computation graph, a parallel allocation strategy for the training task is determined.

[0087] By employing a two-stage approach of static analysis and small-batch trial runs, the time overhead of full-scale training is avoided, while obtaining more accurate actual computational performance data than pure static derivation. The estimation error of T_compute in the cost model is controlled within 10%. A cost model-driven search algorithm based on model computation graph features and the global GPU resource topology graph automatically generates parallel strategy combinations.

[0088] Optionally, determining the parallel allocation strategy for the training task based on the GPU resource information and the layer data information of the model includes:

[0089] Among the multiple pre-acquired allocation strategies, those whose parameter values ​​do not meet the constraints are eliminated to obtain a set of candidate allocation strategies.

[0090] Based on each allocation strategy in the candidate allocation strategy set, a training evaluation metric is calculated according to the GPU resource information and the layer data information of the model. The training evaluation metric includes at least one of training throughput and training cost.

[0091] Based on the training evaluation metrics, a target allocation strategy is determined from the set of candidate allocation strategies.

[0092] Define a search space consisting of three-dimensional parallelism (i.e., allocation strategies). The three-dimensional parallelism includes: data parallelism Dp, pipeline parallelism Pp, and tensor parallelism Tp. This yields multiple allocation strategies, each of which includes the parallelism values ​​corresponding to the three-dimensional parallelism, such as Dp=8, Pp=4, and Tp=2.

[0093] Based on the above three-dimensional parallelism, the following constraints are constructed to eliminate allocation strategies where the parameter values ​​do not meet the constraints:

[0094] Dp×Pp×Tp≤Total number of GPUs N;

[0095] Tp ≤ Number of GPUs per node (Tensor parallelism is limited to the range of high-speed interconnects within a node).

[0096] Pp≤ Total number of layers in the model / Minimum layer grouping granularity (to avoid bubble overhead caused by excessive fineness in the pipeline stage);

[0097] The sum of parameters at each pipeline stage / Tp ≤ single GPU memory capacity (to ensure no OOM occurs).

[0098] After pruning multiple allocation strategies based on the above constraints, a set of candidate allocation strategies is obtained. A cost model is then constructed based on each candidate allocation strategy in the set.

[0099] For each candidate allocation strategy (Dp, Pp, Tp), calculate the estimated training throughput:

[0100] T_total = T_compute + T_comm + T_bubble;

[0101] Where micro_batch_num = Global_batch_size / (Dp×single_GPU_micro_batch_size). Example: When Global_batch_size=1024, Dp=8, and single GPU micro-batch size=32, micro_batch_num=1024 / (8×32)=4.

[0102] T_compute = max (calculation time for each stage): estimated from profiling data and GPU computing power.

[0103] T_comm = T_tp_comm + T_pp_comm + T_dp_comm: Calculated based on the communication volume / intra-node bandwidth of tensor parallel AllReduce, the activation value transmission volume / inter-node bandwidth of pipeline parallel, and the communication volume / cross-cloud bandwidth of data parallel gradient AllReduce, respectively.

[0104] T_tp_comm=2*(Tp-1) / Tp*tensor_comm_size / intra_node_bandwidth, where tensor_comm_size is the size of the single-layer weight matrix, the coefficient 2*(Tp-1) / Tp is the AllReduce communication volume formula, and intra_node_bandwidth is the NVLink interconnection band within the node.

[0105] T_pp_comm = (Pp-1) * activation_size / inter_node_bandwidth * micro_batch_num, where activation_size is the amount of activation data transmitted between pipeline stages, inter_node_bandwidth is the network bandwidth between nodes within the same cloud, and micro_batch_num is the number of micro-batches.

[0106] T_dp_comm = 2×(Dp-1) / Dp×gradient_size / cross_cloud_bandwidth, where gradient_size is determined by the number of model parameters and accuracy (in FP16, the gradient data volume of 10 billion parameters = 10B × 2Bytes = 20GB; the gradient data volume of a model with trillion parameters = 1T × 2Bytes = 2TB), and the coefficient 2×(Dp-1) / Dp is the communication volume formula for Ring-AllReduce.

[0107] T_bubble = (Pp - 1) / (Dp×micro_batch_num) × T_compute: Bubble time in the pipeline.

[0108] Calculate separately for each candidate allocation strategy:

[0109] Training throughput = Global_batch_size / T_total;

[0110] Training cost = N × GPU unit price × estimated total training time;

[0111] After obtaining the training throughput and training cost, you can choose the strategy with the highest training throughput or the lowest training cost. Alternatively, you can consider both training throughput and training cost to calculate the comprehensive score of each strategy and select the strategy with the highest comprehensive score.

[0112] A cost model-driven 3D parallel strategy search is performed on a five-layer resource topology graph. In the cost model, T_compute is bound to the actual GPU computing power (TFLOPS), T_comm is bound to the edge weights (actual network bandwidth) in the five-layer graph, and T_bubble is bound to the number of pipeline stages (Pp) and the number of micro-batches. Training shards are generated according to the hierarchical principle of "intra-cloud tensor parallelism -> intra-cloud pipeline parallelism -> cross-cloud data parallelism" to obtain the distribution strategy.

[0113] Optionally, determining the target allocation strategy based on the training evaluation metric in the candidate allocation strategy set includes:

[0114] Obtain any two candidate allocation strategies from the set of candidate allocation strategies;

[0115] Based on the training evaluation metrics, one candidate allocation strategy is removed from any two candidate allocation strategies to obtain a strategy set.

[0116] The target allocation strategy is determined from the strategy set based on user preferences.

[0117] After evaluating the cost model of all candidate allocation strategies in the pruned search space, the system performs Pareto optimal strategy set screening and user preference selection.

[0118] Specifically, the evaluation result of each candidate allocation strategy is mapped to a two-dimensional objective space (training throughput, training cost). For any two candidate allocation strategies A and B, if A is not inferior to B in all objective dimensions and is strictly superior to B in at least one dimension, then A is said to dominate B. The system iterates through all candidate strategies, eliminates the dominated strategies, and retains the set of strategies corresponding to strategies that are not dominated by any other strategy, which is the Pareto optimal strategy set (Pareto Front).

[0119] Strategy selection is based on the user's pre-set or selected preference mode (speed priority / cost priority / balanced):

[0120] In speed-first mode, the policy with the highest training throughput is selected from the Pareto optimal policy set;

[0121] In the cost-first mode, the strategy with the lowest training cost is selected;

[0122] In equilibrium mode, the normalized comprehensive score of each Pareto optimal policy is calculated (score = α*normalized_throughput + (1-α) * (1-normalized_cost), where α=0.5 is the equilibrium weight, and normalized_throughput and normalized_cost are the normalized values ​​of throughput and cost at the Pareto front, respectively), and the policy with the highest comprehensive score is selected.

[0123] After selecting the recommended strategy, training shards are generated according to the hierarchical principle of "intra-cloud tensor parallelism -> intra-cloud pipeline parallelism -> cross-cloud data parallelism". Each training shard contains a list of allocated GPU resources, the scope of the model layer or data shard it is responsible for, and the communication dependencies with other shards. The training shards are output to the cross-cloud scheduling and execution engine in a standardized task description format.

[0124] As a specific embodiment, a method for scheduling training tasks is as follows: Figure 3 As shown.

[0125] Obtain the model definition file, global GPU resources, and user constraints. Parse the model definition file to extract the model's layer structure information, and obtain layer data information for each computation layer, such as parameter count, FLOPs, memory usage, computation time, and inter-layer dependencies. Construct the model computation graph based on the layer data information.

[0126] By performing small-batch reruns of the model, the precise computation time and memory usage data of each computational layer of the model are collected, and the collected precise data is updated to the model computation graph.

[0127] Based on pre-obtained parallel allocation strategies (including multiple values ​​corresponding to data parallelism, pipeline parallelism, and tensor parallelism), a search space is defined, and parallel strategies that do not meet the constraints are pruned. For the candidate parallel strategies obtained after pruning, a cost model is constructed to evaluate the time consumption, cost, throughput, etc. of each candidate parallel strategy. Combined with pre-obtained user constraints (such as speed priority or cost priority), the Pareto optimal parallel strategy is selected to obtain the optimal strategy combination (including the combination of data parallelism, pipeline parallelism, and tensor parallelism).

[0128] The training task is divided into multiple training slices using a hierarchical parallel allocation principle, and these training slices are then distributed to multiple GPU resources to execute the training task.

[0129] Optionally, executing the training task corresponding to each training segment on the GPU resources corresponding to each training segment according to the communication dependency includes:

[0130] Decision factors are determined based on the information of the training slices and the collected cross-cloud network status information. The decision factors include at least two of the following: cross-cloud network bandwidth, network latency, gradient data volume, and model training stage. The cross-cloud network is the network between different cloud platforms corresponding to the heterogeneous GPU resources.

[0131] A transmission strategy is determined based on at least two of the decision factors, wherein the transmission strategy includes at least one of the following: compression strategy, synchronization mode, and communication topology;

[0132] On the GPU resources corresponding to each training slice, the training task corresponding to each training slice is executed according to the communication dependency and the transmission strategy.

[0133] Traditional training frameworks optimize their communication layers for high-speed interconnect networks within a single data center (such as NVLink and InfiniBand), but lack effective communication optimization methods for wide area network scenarios across clouds and data centers. When training tasks need to be deployed across clouds, gradient synchronization communication is severely constrained by the high latency and low bandwidth of wide area networks, leading to a sharp decline in training efficiency.

[0134] Based on the communication requirements between training shards and the real-time network conditions between multiple clouds, this application adopts a triple optimization approach, namely, gradient compression dynamic selection method, synchronous mode adaptive switching method, and hierarchical AllReduce communication topology construction method, to achieve a leap from "fixed communication strategy" to "network-aware adaptive communication".

[0135] This gradient compression is customized for the sparsity and redundancy of gradients in AI model training. It incorporates an error feedback compensation mechanism during compression, accumulating the discarded gradient residuals to the next round of compensation, ensuring that the convergence accuracy of training in long-term iterations is not affected by compression. In contrast, general data compression (such as gzip / zstd) only focuses on compression ratio and decompression speed, without considering the impact of information loss introduced by compression on downstream tasks (training convergence).

[0136] like Figure 4 As shown, cross-cloud network probing employs an agent to periodically measure cross-cloud network status information, including network latency (Round-Trip Time, RTT), available bandwidth, and packet loss rate, and reports the measurement results to the communication optimization controller. A tiered sampling strategy can be used when collecting this data: high-frequency sampling (every 10 seconds) is used to detect network jitter and sudden congestion, while low-frequency sampling (every 5 minutes) is used to assess long-term trends in network quality.

[0137] For cross-cloud communication, a three-dimensional decision model of "network status - training accuracy - communication overhead" is constructed. Based on this model, the communication optimization controller automatically selects the optimal gradient transmission strategy by comprehensively considering factors such as cross-cloud network bandwidth, network latency, gradient data volume and model training stage (different training stages have different sensitivities to accuracy).

[0138] The decision factors in the aforementioned three-dimensional decision model include at least two of the following: cross-cloud network bandwidth, network latency, gradient data volume, and model training phase. Specifically, network bandwidth determines the choice of compression strategy (high / medium / low bandwidth corresponds to no compression / Top-K / 1-bit), network latency determines the choice of synchronization mode (low / medium / high latency corresponds to Bulk Synchronous Parallel (BSP) / Stale Synchronous Parallel (SSP) / Local Stochastic Gradient Descent (Local SGD)), gradient data volume affects the absolute value of compression gains (the larger the gradient data volume, the more transmission time is saved by compression, and the higher the priority of the compression strategy), and the model training phase affects accuracy tolerance (aggressive compression is possible in the early stages of training when accuracy sensitivity is low, while conservative compression is required in the later stages when accuracy sensitivity is high).

[0139] This application employs a linked decision transmission strategy using at least two of the aforementioned decision factors. The specific decision-making process may include:

[0140] 1. Decision-making compression strategy based on the linkage between network bandwidth and gradient data volume.

[0141] Compression is triggered when the product of bandwidth and gradient data exceeds a threshold. For example, no compression is performed when bandwidth > 10Gbps and gradient < 2GB; Top-K compression is performed when bandwidth is between 1-10Gbps or gradient is between 2-20GB; and 1-bit compression is performed when bandwidth < 1Gbps or gradient > 20GB. For example:

[0142] High bandwidth scenarios (>10Gbps): Uncompressed, direct transmission of full-precision gradient (FP32 / BF16).

[0143] For medium bandwidth scenarios (1-10Gbps): Top-K sparse compression is adopted, which transmits only the K% of elements with the largest absolute gradient value (K=10%, that is, only the top 10% of elements with the largest absolute gradient value are transmitted, and K can be dynamically adjusted according to network bandwidth within the range of 1%-30%), and the compression ratio can reach 10-100 times.

[0144] In low-bandwidth scenarios (<1Gbps): 1-bit quantization compression combined with error feedback compensation is used to compress the gradient to 1-bit representation, achieving a compression ratio of more than 32 times.

[0145] 2. Network latency and training phase linkage decision synchronization mode.

[0146] In the early stages of training (when the rate of loss decreases > the threshold), the latency tolerance is relaxed (T1 is relaxed by 20%), and in the later stages of training, the latency tolerance is tightened (T1 is tightened by 20%). Then, BSP / SSP / Local SGD is selected in combination with the latency threshold.

[0147] Dynamically switch gradient synchronization mode based on cross-cloud network latency and the latency tolerance of the training task:

[0148] Synchronous Mode (BSP): When the cross-cloud network latency is lower than the threshold T1 (T1=50ms), all data parallel replicas perform fully synchronous gradient aggregation to ensure training convergence accuracy. The scenario adaptation rules for T1 / T2 are: T1=50ms and T2=200ms for LLM text model training; for multimodal large model training (text / image / video), T1=80ms and T2=300ms can be relaxed (because the real-time requirements for gradient synchronization in multimodal training are lower than those in pure text LLM). The specific values ​​are dynamically adjusted according to the convergence accuracy requirements of the training task.

[0149] Bounded Asynchronous Mode (SSP): When the network latency is between T1 and T2 (T2=200ms), a gradient version difference of up to S steps (S=5, that is, the fastest data parallel replica is allowed to lead the slowest replica by up to 5 gradient update steps) is allowed between each data parallel replica, achieving a balance between convergence accuracy and training speed.

[0150] Local SGD mode: When the network latency exceeds T2, the parallel data replicas of each cloud execute H steps locally (H=20, based on the convergence theory of Local SGD, when H<=20 and the learning rate is appropriately scaled, the global convergence rate is comparable to that of the fully synchronous mode), and then perform a global gradient synchronization after the gradient update, which greatly reduces the frequency of cross-cloud communication.

[0151] 3. Bandwidth and latency linkage decision-making communication topology.

[0152] When bandwidth is low and latency is high, prioritize using layered AllReduce to reduce the number of cross-cloud communications; when bandwidth is high and latency is low, global AllReduce can be used to reduce aggregation layers.

[0153] The general AllReduce topology is only applicable to homogeneous networks within a single cloud, assuming that all nodes have similar bandwidth, and cannot handle heterogeneous scenarios with large bandwidth differences across clouds. In this application, the hierarchical AllReduce communication topology construction method for multi-cloud scenarios first completes gradient aggregation between nodes within the cloud (utilizing the high-speed network within the cloud), and then synchronizes the aggregation results between clouds (through cross-cloud links), avoiding cross-cloud communication amplification caused by all GPUs directly participating in global AllReduce.

[0154] Based on the training progress, the communication strategy between nodes can be dynamically adjusted. In the early training stage (the first 20% of training steps), the model parameters are changing rapidly, and the information redundancy in the gradients is high. At this time, a higher compression ratio (e.g., K=5% in Top-K) and a more lenient asynchronous tolerance (S=8 in SSP) are used to accelerate training. In the later training stage (the last 20% of training steps), the model is close to convergence, and the fine information in the gradients is crucial to the final accuracy. At this time, the compression ratio is automatically reduced (K=20% or switched to no compression) and the synchronization mode is tightened (switched to BSP or S=2 in SSP) to ensure convergence accuracy. The determination of the training stage is based on the rate of decrease of the training loss: when the rate of decrease of the loss is lower than a preset threshold, it is determined to enter the later stage.

[0155] Furthermore, communication routing optimization can be performed based on cloud bandwidth cost awareness. Outbound traffic costs vary significantly among different cloud service providers. When selecting inter-cloud gradient synchronization paths, the communication routing optimizer comprehensively considers three factors: network latency, available bandwidth, and traffic costs, prioritizing communication paths with "low latency + low cost." When multiple available cross-cloud links exist (such as direct links and links relayed through third-party clouds), a weighted multi-objective evaluation is used to select the optimal route, avoiding the pursuit of low latency at the expense of high traffic costs.

[0156] As a specific embodiment, an intelligent sharding and collaborative scheduling method for AI model training tasks in a multi-cloud environment includes a unified discovery and pooling mechanism for multi-cloud GPU resources, intelligent sharding of training tasks and automatic generation of optimal parallel strategies, and an adaptive optimization strategy for cross-cloud communication.

[0157] A scheduling architecture for training tasks, such as Figure 5 As shown, it includes the following modules:

[0158] The global view building module is used to construct a global resource topology graph.

[0159] The training task intelligent sharding module is used to generate the optimal parallel strategy and intelligently shard the training task.

[0160] The training slice output module is used to output training slices.

[0161] The cross-cloud communication optimization module is used to adaptively optimize cross-cloud communication.

[0162] The multi-source resource adapter module is used to acquire GPU resources.

[0163] Based on such Figure 5 The scheduling architecture shown illustrates an intelligent sharding and collaborative scheduling method for AI model training tasks in a multi-cloud environment, comprising the following steps:

[0164] Step 1: Multi-cloud GPU resource pooling and global resource topology construction.

[0165] The multi-cloud resource adapter interfaces with the GPU resource management APIs of various cloud service providers, automatically discovering, registering, and uniformly abstracting heterogeneous GPU resources, thus building a system such as... Figure 2 The global resource topology diagram shown, consisting of a five-layer directed weighted graph ("cloud service provider -> region -> availability zone -> node -> GPU"), provides a global and standardized resource view for subsequent training sharding. Furthermore, after training tasks are deployed independently on each cloud, model parameters can be merged through federated aggregation.

[0166] Step 2: Intelligent task partitioning and automatic generation of optimal parallel strategy.

[0167] Based on the user-submitted model definition file, the model architecture of the global GPU resource topology map, and user constraints, a cost model was constructed. Based on automatic model architecture analysis and utilizing a three-dimensional parallel strategy search algorithm (data parallelism Dp, pipeline parallelism Pp, tensor parallelism Tp), the training task was divided into several training shards that can be distributed across clouds, following a hierarchical principle of "intra-cloud tensor parallelism -> intra-cloud pipeline parallelism -> cross-cloud data parallelism".

[0168] The intelligent partitioning process for training tasks is as follows: Figure 3 As shown.

[0169] The process of automatically recommending parallel strategies includes:

[0170] Users submit model definition files (such as model code in PyTorch nn.Module format) and training configurations (global batch size, learning rate, etc.). The system automatically calls the model architecture analysis module to parse the model layer structure and build the model computation graph, extracting the number of parameters, computational cost (FLOPs), and memory usage data for each layer.

[0171] The system reads the global GPU resource topology map and obtains parameters such as the model, quantity, computing power (TFLOPS) and inter-cloud network bandwidth of available GPUs in each cloud, which are used as inputs to the cost model.

[0172] Perform search space pruning by constraining the upper limit of tensor parallelism based on the number of GPUs per node, the upper limit of pipeline parallelism based on the number of model layers, and the upper limit of the number of parameters at each stage based on the memory capacity of a single GPU, thus pruning the search space from the original large-scale combination to an effective candidate set.

[0173] For each candidate parallel policy (Dp, Pp, Tp) after pruning, the system calls the cost model to calculate the estimated training throughput and outputs the estimated training throughput and training cost for each candidate policy.

[0174] After evaluating the cost model for all candidate strategies in the trimmed search space, Pareto optimal strategy set selection and user preference selection are performed to obtain the final strategy.

[0175] Step 3: Adaptive Optimization of Cross-Cloud Communication

[0176] like Figure 4 As shown, a cross-cloud network real-time detection mechanism based on the gradient synchronization communication requirements between training shards is constructed to build a three-dimensional decision model of "network status-training accuracy-communication overhead". Based on this decision model, the gradient compression strategy (no compression / Top-K / 1-bit), synchronization mode (BSP / SSP / Local SGD) and communication topology (hierarchical AllReduce) are dynamically selected.

[0177] Taking the intelligent sharding and collaborative scheduling of a large language model with 13 billion parameters trained across three clouds as an example, the method of this application embodiment will be described.

[0178] Training Task Description: An AI R&D center needs to train a large language model (based on the LLaMA architecture) with 13 billion parameters. The model contains 40 Transformer layers, each with Multi-Head Attention and FFN sub-layers, with approximately 325 million parameters per layer, for a total of 13 billion parameters. The training dataset consists of 500 bytes of tokens, with a global batch size of 1024 and a total training step count of approximately 100,000 steps.

[0179] The R&D center has GPU resources from three cloud service providers: Cloud A (East China region): 4 x 8×A100-80G nodes (32 A100s in total), with an intra-cloud NVLink interconnect bandwidth of 600GB / s; Cloud B (South China region): 2 x 8×A100-80G nodes (16 A100s in total), with an intra-cloud NVLink interconnect bandwidth of 600GB / s; Cloud C (Southwest China region): 2 x 8×A100-40G nodes (16 A100s in total), with an intra-cloud NVLink interconnect bandwidth of 300GB / s. Inter-cloud WAN bandwidth: Cloud A-Cloud B: 5Gbps / RTT 60ms; Cloud A-Cloud C: 3Gbps / RTT 90ms; Cloud B-Cloud C: 2Gbps / RTT 120ms. Total available GPUs: 64 A100s.

[0180] Multi-cloud GPU resource pooling and global resource topology construction: The multi-cloud resource adapters connect to the Elastic Cloud Server Application Programming Interface (ECSAPI) of Cloud A, the ECS API of Cloud B, and the Cloud Virtual Machine Application Programming Interface (CVM API) of Cloud C, automatically discovering a total of 8 GPU nodes and 64 A100 graphics cards across the three clouds. The unified GPU resource abstraction module converts heterogeneous resources into a standardized object model: the A100-80G in Cloud A and Cloud B is uniformly abstracted as 312 TFLOPS (FP16) computing power / 80GB of GPU memory / 2TB / s of GPU memory bandwidth, and the A100-40G in Cloud C is uniformly abstracted as 312 TFLOPS (FP16) computing power / 40GB of GPU memory / 1.6TB / s of GPU memory bandwidth. The global resource topology construction module builds a five-layer directed weighted graph: the top layer consists of nodes from the three cloud service providers, the second layer consists of regional nodes (East China / South China / Southwest China), the third layer consists of availability zone nodes, the fourth layer consists of 8 physical nodes, and the fifth layer consists of 64 GPU nodes. Edge weights are updated in real time: the edge weight labeling between Cloud A and Cloud B is 5Gbps / 60ms bandwidth, between Cloud A and Cloud C is 3Gbps / 90ms, and between Cloud B and Cloud C is 2Gbps / 120ms; the edge weight labeling between nodes within the cloud is 600GB / s bandwidth.

[0181] Intelligent training task sharding and automatic generation of optimal parallel strategies: The model architecture analysis module parses the Meta Large Language Model (LLaMA) model definition file, automatically extracts the parameter count and computational cost of 40 Transformer layers, and constructs the model computation graph. The parallel strategy search module performs a cost model-driven three-dimensional strategy search on a five-layer resource topology graph: After search space pruning, the upper limit for Tensor Parallelism (Tp) is 8 (number of GPUs per node), the upper limit for Pipeline Parallelism (Pp) is 8 (40 layers / minimum 5-layer grouping), and the upper limit for Data Parallelism (Dp) is 8 (64 / (Tp×Pp)). The cost model evaluates each candidate strategy: When (Dp=8, Pp=4, Tp=2), T_compute=120ms (estimated from the actual TFLOPS and profiling of A100), T_comm=T_tp_comm(2ms)+T_pp_comm(15ms)+T_dp_comm(180ms)=197ms (T_dp_comm=2×7 / 8×20GB / 5Gbps=180ms, calculated from the edge weights of the five-layer graph), T_bubble=3 / (8×4)×120ms=11.25ms, T_total=328.25ms, training throughput=1024 / 0.328s=3121 samples / s. When (Dp=4, Pp=4, Tp=4), T_compute=60ms, T_comm=T_tp_comm(5ms)+T_pp_comm(15ms)+T_dp_comm(360ms)=380ms, T_total=445ms, and throughput=2301 samples / s. The system automatically selects (Dp=8, Pp=4, Tp=2) as the optimal strategy and generates training shards according to the hierarchical principle of "intra-cloud TP→intra-cloud PP→cross-cloud DP": 4 pipeline stages within each cloud (2 GPUs for tensor parallelism in each stage), and 8 data parallel replicas distributed across the three clouds (4 replicas allocated to cloud A, 2 replicas allocated to cloud B, and 2 replicas allocated to cloud C), generating a total of 8×4=32 training shards.

[0182] The training sharding described above differs from file sharding, logical block merging, or transmission sharding for file transfer or data movement scenarios. Its sharding object is not a list of files to be transferred, nor is the number of shards determined solely by the total data volume, storage performance metrics, and transmission network bandwidth. The sharding is based on a joint search result of the model computation graph, global GPU resource topology, and data parallelism Dp, pipeline parallelism Pp, and tensor parallelism Tp. The sharding result includes not only the data sharding range but also the GPU resource list, model layer range, inter-shard communication dependencies, and checkpoint storage configuration, directly driving cross-cloud distributed training execution.

[0183] Cross-cloud communication adaptive optimization: The cross-cloud network detection agent measures the network status between the three clouds in real time and reports the bandwidth of 5Gbps / RTT 60ms between Cloud A and Cloud B. Three-dimensional decision model linkage decision: (a) Gradient compression strategy - 5Gbps is a medium bandwidth scenario, so Top-K sparse compression (K=10%) is selected, and the gradient data volume is compressed from 20GB to 2GB, with a compression ratio of 10 times; (b) Synchronous mode - RTT 60ms is between T1 (50ms) and T2 (200ms), so bounded asynchronous mode SSP (S=5 steps) is selected, allowing a maximum of 5 steps of gradient version difference between parallel replicas of data in each cloud; (c) Communication topology - A hierarchical AllReduce topology is constructed. Gradient aggregation is first completed between 4 replicas in Cloud A (using Link 600GB / s), then between 2 replicas in Cloud B, and then between 2 replicas in Cloud C. Finally, only 3 aggregated gradients are transmitted between the three clouds (instead of 8 original gradients), reducing cross-cloud communication volume by 62.5%. Once training enters the later stages (when the loss rate falls below the threshold), the system automatically adjusts the Top-K compression ratio from K=10% to K=20% and tightens the synchronization mode from SSP (S=5) to SSP (S=2) to ensure final convergence accuracy.

[0184] By incorporating heterogeneous GPU resources from multiple cloud service providers into a unified resource view, the system achieves an improvement from static allocation in a single cloud to global optimal scheduling across multiple clouds. GPU utilization is increased by more than 30%. In a three-cloud environment with 64 A100 GPUs, training a 13 billion parameter LLaMA model, GPU utilization reached 78.5% (a 50.1% improvement compared to 52.3% in a single-cloud static allocation scheme). Cross-cloud training throughput was restored to 81% of that in a single cloud (a 153% improvement compared to 32% in an unoptimized scheme).

[0185] By using a cost-model-driven automated search algorithm, parallelism configuration driven by human experience is replaced, achieving a shift from manual trial and error to automated optimal recommendation. The configuration time for parallel strategies is reduced from 3-4 hours in manual solutions to 8 minutes, and the complete training cycle is shortened by 42% compared to single-cloud solutions.

[0186] By employing a triple optimization approach of gradient compression dynamic selection, adaptive switching of synchronization mode, and hierarchical AllReduce topology, a leap from fixed communication strategy to network-aware adaptive communication is achieved, resulting in a more than 50% improvement in cross-cloud training efficiency compared to the unoptimized approach.

[0187] See Figure 6 , Figure 6 This is a schematic diagram of the structure of a training task scheduling device provided in an embodiment of this application, as shown below. Figure 6 As shown, the training task scheduling device 600 includes:

[0188] The first determining module 601 is used to determine the layer data information of the model based on the model information corresponding to the training task when the training task is obtained.

[0189] The acquisition module 602 is used to acquire graphics processor (GPU) resource information, which includes attribute information of heterogeneous GPU resources located on different cloud platforms and information used to characterize the topological connection relationship between GPU resources.

[0190] The second determining module 603 is used to determine the parallel allocation strategy of the training task based on the GPU resource information and the layer data information of the model. The parallel allocation strategy includes at least one of the following: data parallelism, pipeline parallelism, and tensor parallelism.

[0191] The segmentation module 604 is used to segment the training task into multiple training segments based on the parallel allocation strategy, each training segment including allocated GPU resources and communication dependencies with other training segments;

[0192] The execution module 605 is used to execute the training task corresponding to each training segment on the GPU resources corresponding to each training segment according to the communication dependency relationship.

[0193] Optionally, the device further includes:

[0194] The conversion module is used to convert the GPU resource information into resource objects with the same resource indicators, and to construct a resource topology graph based on the resource objects. The resource topology graph includes nodes representing the organizational level of the GPU resources and edges representing the network connection relationships between the nodes. The weight of the edges is used to represent the network characteristics between the nodes.

[0195] The resource topology map is used to determine the parallel allocation strategy for the training task.

[0196] Optionally, the device further includes:

[0197] A construction module is used to construct a model computation graph based on the layer data information. The model computation graph includes nodes for representing attribute information of each layer of the model, and edges for representing data flow dependencies between the layers of the model.

[0198] The measurement module is used to perform forward and backward propagation of the model using the first GPU and measure the actual performance indicators of each level of the model. The actual performance indicators include at least one of the following: computation time and video memory usage data. The first GPU is any GPU in the GPU resources.

[0199] An update module is used to update the attribute information of nodes in the model computation graph using the actual performance metrics; the updated model computation graph is used to determine the parallel allocation strategy for the training task.

[0200] Optionally, the second determining module is specifically used for:

[0201] Among the multiple pre-acquired allocation strategies, those whose parameter values ​​do not meet the constraints are eliminated to obtain a set of candidate allocation strategies.

[0202] Based on each allocation strategy in the candidate allocation strategy set, a training evaluation metric is calculated according to the GPU resource information and the layer data information of the model. The training evaluation metric includes at least one of training throughput and training cost.

[0203] Based on the training evaluation metrics, a target allocation strategy is determined from the set of candidate allocation strategies.

[0204] Optionally, the second determining module is specifically used for:

[0205] Obtain any two candidate allocation strategies from the set of candidate allocation strategies;

[0206] Based on the training evaluation metrics, one candidate allocation strategy is removed from any two candidate allocation strategies to obtain a strategy set.

[0207] The target allocation strategy is determined from the strategy set based on user preferences.

[0208] Optionally, the execution module is specifically used for:

[0209] Decision factors are determined based on the information of the training slices and the collected cross-cloud network status information. The decision factors include at least two of the following: cross-cloud network bandwidth, network latency, gradient data volume, and model training stage. The cross-cloud network is the network between different cloud platforms corresponding to the heterogeneous GPU resources.

[0210] A transmission strategy is determined based on at least two of the decision factors, wherein the transmission strategy includes at least one of the following: compression strategy, synchronization mode, and communication topology;

[0211] On the GPU resources corresponding to each training slice, the training task corresponding to each training slice is executed according to the communication dependency and the transmission strategy.

[0212] The training task scheduling device can achieve Figure 1 The various processes implemented in the method embodiments can achieve the same technical effect, and will not be described again here to avoid repetition.

[0213] It should be noted that the electronic device provided in this application embodiment is a device capable of scheduling the above-described training tasks. Therefore, all implementation methods of the above-described training task scheduling method embodiment are applicable to this electronic device and can achieve the same or similar beneficial effects. To avoid repetition, this embodiment will not elaborate further.

[0214] See Figure 7 As shown in the figure, this application embodiment also provides an electronic device, including a bus 601, a transceiver 602, an antenna 603, a bus interface 604, a processor 605, and a memory 606.

[0215] Processor 605, used for:

[0216] Once a training task is obtained, the layer data information of the model is determined based on the model information corresponding to the training task.

[0217] Obtain graphics processing unit (GPU) resource information, which includes attribute information of heterogeneous GPU resources located on different cloud platforms and information used to characterize the topological connection relationship between GPU resources;

[0218] Based on the GPU resource information and the layer data information of the model, a parallel allocation strategy for the training task is determined. The parallel allocation strategy includes at least one of the following: data parallelism, pipeline parallelism, and tensor parallelism.

[0219] Based on the parallel allocation strategy, the training task is divided into multiple training slices, each of which includes allocated GPU resources and communication dependencies with other training slices.

[0220] On the GPU resources corresponding to each training slice, the training task corresponding to each training slice is executed according to the communication dependency.

[0221] exist Figure 7In this document, the bus architecture is represented by bus 601. Bus 601 may include any number of interconnected buses and bridges, linking various circuits including one or more processors represented by processor 605 and memory represented by memory 606. Bus 601 may also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. Bus interface 604 provides an interface between bus 601 and transceiver 602. Transceiver 602 may be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by processor 605 is transmitted over a wireless medium via antenna 603, which further receives data and transmits data to processor 605.

[0222] Processor 605 manages bus 601 and general processing, and also provides various functions, including timing, peripheral interface, voltage regulation, power management, and other control functions. Memory 606 can be used to store data used by processor 605 during operation.

[0223] Alternatively, the processor 605 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a complex programmable logic device (CPLD).

[0224] Optionally, after the processor 605 performs the step of obtaining the graphics processing unit (GPU) resource information, it is further configured to:

[0225] The GPU resource information is converted into resource objects with the same resource metrics, and a resource topology graph is constructed based on the resource objects. The resource topology graph includes nodes representing the organizational level of the GPU resources and edges representing the network connection relationships between the nodes. The weights of the edges are used to represent the network characteristics between the nodes. The resource topology graph is used to determine the parallel allocation strategy for the training task.

[0226] The step of determining the parallel allocation strategy for the training task based on the GPU resource information and the model's layer data information includes:

[0227] Based on the resource topology map and the layer data information of the model, a parallel allocation strategy for the training task is determined.

[0228] Optionally, after the processor 605 executes the step of determining the layer data information of the model based on the model information corresponding to the training task, it is further configured to:

[0229] A model computation graph is constructed based on the layer data information. The model computation graph includes nodes for representing attribute information of each layer of the model, and edges for representing data flow dependencies between the layers of the model.

[0230] The forward and backward propagation of the model are performed using a first GPU, and the actual performance metrics of each level of the model are measured. The actual performance metrics include at least one of the following: computation time and video memory usage data. The first GPU is any GPU in the GPU resources.

[0231] The attribute information of the nodes in the model computation graph is updated using the actual performance metrics; the updated model computation graph is used to determine the parallel allocation strategy for the training task.

[0232] Optionally, the processor 605 executes the step of determining the parallel allocation strategy for the training task based on the GPU resource information and the layer data information of the model, including:

[0233] Among the multiple pre-acquired allocation strategies, those whose parameter values ​​do not meet the constraints are eliminated to obtain a set of candidate allocation strategies.

[0234] Based on each allocation strategy in the candidate allocation strategy set, a training evaluation metric is calculated according to the GPU resource information and the layer data information of the model. The training evaluation metric includes at least one of training throughput and training cost.

[0235] Based on the training evaluation metrics, a target allocation strategy is determined from the set of candidate allocation strategies.

[0236] Optionally, the processor 605 performs the step of determining a target allocation strategy based on the training evaluation metric in the candidate allocation strategy set, including:

[0237] Obtain any two candidate allocation strategies from the set of candidate allocation strategies;

[0238] Based on the training evaluation metrics, one candidate allocation strategy is removed from any two candidate allocation strategies to obtain a strategy set.

[0239] The target allocation strategy is determined from the strategy set based on user preferences.

[0240] Optionally, the processor 605 executes the training task corresponding to each training segment on the GPU resources corresponding to each training segment, according to the communication dependency, including:

[0241] Decision factors are determined based on the information of the training slices and the collected cross-cloud network status information. The decision factors include at least two of the following: cross-cloud network bandwidth, network latency, gradient data volume, and model training stage. The cross-cloud network is the network between different cloud platforms corresponding to the heterogeneous GPU resources.

[0242] A transmission strategy is determined based on at least two of the decision factors, wherein the transmission strategy includes at least one of the following: compression strategy, synchronization mode, and communication topology;

[0243] On the GPU resources corresponding to each training slice, the training task corresponding to each training slice is executed according to the communication dependency and the transmission strategy.

[0244] It should be noted that the electronic device provided in this application embodiment is a device capable of executing the scheduling method for the above-described training task. Therefore, all implementation methods of the scheduling method for the above-described training task embodiment are applicable to this electronic device and can achieve the same or similar beneficial effects. To avoid repetition, this embodiment will not elaborate further.

[0245] This application also provides a computer-readable storage medium storing a computer program. When executed by a processor, this computer program implements the various processes of the above-described training task scheduling method embodiment and achieves the same technical effect. To avoid repetition, it will not be described again here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0246] This application also provides a computer program product, including computer instructions, which, when executed by a processor, implement the above-described... Figure 1 The various processes of the method embodiments shown can achieve the same technical effect, and will not be described again here to avoid repetition.

[0247] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0248] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0249] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A method for scheduling a training task, characterized in that, include: Once a training task is obtained, the layer data information of the model is determined based on the model information corresponding to the training task. Obtain graphics processing unit (GPU) resource information, which includes attribute information of heterogeneous GPU resources located on different cloud platforms and information used to characterize the topological connection relationship between GPU resources; Based on the GPU resource information and the layer data information of the model, a parallel allocation strategy for the training task is determined. The parallel allocation strategy includes at least one of the following: data parallelism, pipeline parallelism, and tensor parallelism. Based on the parallel allocation strategy, the training task is divided into multiple training slices, each of which includes allocated GPU resources and communication dependencies with other training slices. On the GPU resources corresponding to each training slice, the training task corresponding to each training slice is executed according to the communication dependency.

2. The method of claim 1, wherein, After obtaining the graphics processing unit (GPU) resource information, the method further includes: The GPU resource information is converted into resource objects with the same resource metrics, and a resource topology graph is constructed based on the resource objects. The resource topology graph includes nodes representing the organizational level of the GPU resources and edges representing the network connection relationships between the nodes. The weights of the edges are used to represent the network characteristics between the nodes. The resource topology graph is used to determine the parallel allocation strategy for the training task.

3. The method of claim 1, wherein, After determining the layer data information of the model based on the model information corresponding to the training task, the method further includes: A model computation graph is constructed based on the layer data information. The model computation graph includes nodes for representing attribute information of each layer of the model, and edges for representing data flow dependencies between the layers of the model. The forward and backward propagation of the model are performed using a first GPU, and the actual performance metrics of each level of the model are measured. The actual performance metrics include at least one of the following: computation time and video memory usage data. The first GPU is any GPU in the GPU resources. The attribute information of the nodes in the model computation graph is updated using the actual performance metrics; the updated model computation graph is used to determine the parallel allocation strategy for the training task.

4. The method according to any one of claims 1 to 3, characterized in that, The step of determining the parallel allocation strategy for the training task based on the GPU resource information and the model's layer data information includes: Among the multiple pre-acquired allocation strategies, those whose parameter values ​​do not meet the constraints are eliminated to obtain a set of candidate allocation strategies. Based on each allocation strategy in the candidate allocation strategy set, a training evaluation metric is calculated according to the GPU resource information and the layer data information of the model. The training evaluation metric includes at least one of training throughput and training cost. Based on the training evaluation metrics, a target allocation strategy is determined from the set of candidate allocation strategies.

5. The method of claim 4, wherein, The step of determining the target allocation strategy based on the training evaluation metric in the candidate allocation strategy set includes: Obtain any two candidate allocation strategies from the set of candidate allocation strategies; Based on the training evaluation metrics, one candidate allocation strategy is removed from any two candidate allocation strategies to obtain a strategy set. The target allocation strategy is determined from the strategy set based on user preferences.

6. The method according to any one of claims 1 to 3, characterized in that, The step of executing the training task corresponding to each training segment on the GPU resources corresponding to each training segment according to the communication dependency includes: Decision factors are determined based on the information of the training slices and the collected cross-cloud network status information. The decision factors include at least two of the following: cross-cloud network bandwidth, network latency, gradient data volume, and model training stage. The cross-cloud network is the network between different cloud platforms corresponding to the heterogeneous GPU resources. A transmission strategy is determined based on at least two of the decision factors, wherein the transmission strategy includes at least one of the following: compression strategy, synchronization mode, and communication topology; On the GPU resources corresponding to each training slice, the training task corresponding to each training slice is executed according to the communication dependency and the transmission strategy.

7. A device for scheduling training tasks, characterized in that include: The first determining module is used to determine the layer data information of the model based on the model information corresponding to the training task when the training task is obtained. The acquisition module is used to acquire graphics processing unit (GPU) resource information, which includes attribute information of heterogeneous GPU resources located on different cloud platforms and information used to characterize the topological connection relationship between GPU resources. The second determining module is used to determine the parallel allocation strategy of the training task based on the GPU resource information and the layer data information of the model. The parallel allocation strategy includes at least one of the following: data parallelism, pipeline parallelism, and tensor parallelism. The segmentation module is used to segment the training task into multiple training segments based on the parallel allocation strategy, each training segment including allocated GPU resources and communication dependencies with other training segments; An execution module is configured to execute the training task corresponding to each training segment on the GPU resources corresponding to each training segment, based on the communication dependency relationship.

8. An electronic device, comprising: include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the scheduling method for the training task as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the scheduling method for the training task as described in any one of claims 1 to 6.

10. A computer program product, characterised in that, It includes computer instructions that, when executed by a processor, implement the steps of the scheduling method for the training task as described in any one of claims 1 to 6.

Images