A computing power scheduling method for large model training, an electronic device, a storage medium and a program product
Patent Information
- Application Number
- CN202610811403.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-06-05
- Publication Date
- 2026-08-28
AI Technical Summary
[0003]当前,主流的算力调度平台(如Kubernetes结合Volcano/Scheduler-plugins、Slurm等)主要面向通用计算任务或传统深度学习训练,其调度策略(如基于资源请求量的Binpack、Spread策略)存在模型特征不感知、资源视图静态化、异构环境适配差、弹性效率低下等缺陷
[0019] In the above implementation process, the problem of resource supply and demand mismatch caused by model errors, changes in task behavior, or training phase switching is solved, enabling resource supply to keep pace with the dynamic needs of tasks in real time. At the same time, the status changes of heterogeneous computing power clusters are monitored, and adjustment operations are triggered in a timely manner when events such as node failure, resource addition, or high-priority task preemption occur, enabling the system to respond quickly to environmental changes and optimize resource allocation.
Smart Images

Figure CN122653836A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer technology, and more specifically, to a computing power scheduling method, electronic device, storage medium, and program product for large model training. Background Technology
[0002] With the rapid development of domestically developed large language models such as DeepSeek, Tongyi Qianwen, Wenxin Yiyan, and Zhipu GLM, their model scale has increased from hundreds of billions of parameters to trillions. Training such large models requires continuous consumption of massive computing power, and usually relies on large-scale heterogeneous computing clusters composed of thousands or even tens of thousands of high-end accelerator cards (such as NVIDIA B-series and H-series GPUs, domestic Ascend AI processors, Hygon DCUs, etc.).
[0003] Currently, mainstream computing scheduling platforms (such as Kubernetes combined with Volcano / Scheduler-plugins, Slurm, etc.) are mainly geared towards general computing tasks or traditional deep learning training. Their scheduling strategies (such as Binpack and Spread strategies based on resource request volume) have shortcomings such as lack of awareness of model features, static resource views, poor adaptation to heterogeneous environments, and low elasticity efficiency.
[0004] Therefore, a new intelligent computing power scheduling method is needed that can deeply perceive the training characteristics and dynamic needs of domestic large-scale models, and achieve accurate, flexible and efficient resource scheduling on complex heterogeneous computing power pools, so as to reduce the training cost of large models and accelerate the research and development iteration cycle. Summary of the Invention
[0005] The purpose of this application is to provide a computing power scheduling method, electronic device, storage medium, and program product for large model training, so as to achieve the technical effect of improving the accuracy of scheduling decisions.
[0006] The first aspect of this application provides a computational power scheduling method for training large models, the method comprising: A global resource view is constructed based on the real-time status information of heterogeneous computing resources; Obtain the task characteristics of the training task of the large model to be scheduled, and predict the resource demand trend of the training task in the preset future time window based on the performance data of the training task in the preset historical time window. Based on the global resource view and the trend of resource demand changes, a constrained bipartite graph maximum weight matching problem is constructed between task units and resource units to determine the matching relationship between the task units and the resource units. Task placement is performed based on the matching relationship.
[0007] In the above implementation process, the system realizes the perception of heterogeneous computing resources and the prediction of the dynamic requirements of large model training tasks. The scheduling decision is modeled as a constrained graph optimization problem, thereby obtaining the globally optimal task placement scheme under the premise of satisfying multiple constraints. This improves the accuracy of scheduling decisions and the utilization rate of cluster resources, and solves the problems of traditional schedulers being unaware of task characteristics, having static resource views, and having poor adaptability to heterogeneous environments.
[0008] Furthermore, the global resource view is constructed through the following steps: A resource descriptor is defined for each computing node to determine the global resource view; the resource descriptor includes hardware type, computing power, storage configuration, network interconnection topology, and real-time utilization metrics.
[0009] In the above implementation process, resource descriptors enable unified abstraction and dynamic monitoring of heterogeneous hardware resources. The constructed global resource view not only includes static attributes but also integrates real-time status and interconnection topology, which provides a comprehensive basis for scheduling decisions, effectively shields the heterogeneity of the underlying hardware, and allows the upper-level scheduling algorithm to focus on optimizing matching.
[0010] Furthermore, the resource demand change trend includes the computational load change trend, communication sensitivity change trend, and memory usage change trend of the task unit; predicting the resource demand change trend of the training task in a preset future time window based on the performance data of the training task within a preset historical time window includes: Obtain the task characteristics of the training task; the task characteristics include model type, training stage, and parallel strategy; Using a time series analysis model, based on the iteration time and resource utilization of the training task within the historical time window, the trends of computational load, communication sensitivity, and memory usage of the training task within the future time window are predicted.
[0011] In the above implementation process, static features such as model type, training stage, and parallel strategy are combined with dynamic time-series data such as iteration time and resource utilization collected in the early stage of task operation. The time-series analysis model is used to predict the trend of changes in computing load, communication sensitivity, and memory usage within the future time window. This enables the scheduling system to have predictive capabilities, allowing it to anticipate risks and prepare in advance before resource bottlenecks actually occur, providing a reliable foundation for subsequent resource matching and elastic scheduling.
[0012] Furthermore, the step of constructing a constrained bipartite graph maximum weight matching problem between task units and resource units based on the global resource view and the resource demand change trend, and determining the matching relationship between the task units and the resource units, includes: The training tasks to be scheduled are divided into multiple task units, and the heterogeneous computing resources are divided into multiple resource units. Based on the global resource view and the trend of resource demand changes, calculate the matching weight between each task unit and each resource unit; The multiple task units are used as the first set of nodes in the bipartite graph, the multiple resource units are used as the second set of nodes in the bipartite graph, and the matching weight is used as the weight of the edge connecting the first set of nodes and the second set of nodes to construct the constrained maximum weight matching problem of the bipartite graph. Solve the constrained bipartite graph maximum weight matching problem to obtain the matching relationship between the task unit and the resource unit.
[0013] In the above implementation process, by transforming the complex heterogeneous computing power scheduling problem into a constrained bipartite graph maximum weight matching problem, the modeling and global optimization of scheduling decisions are realized. This solves the problem that traditional schedulers are unable to take into account multi-dimensional optimization objectives and complex constraints in heterogeneous environments, and improves the accuracy of computing power scheduling and resource utilization efficiency in large-scale heterogeneous clusters.
[0014] Furthermore, the global resource view includes the computing power, video memory capacity, network performance, and elastic support capabilities of the resource unit; the resource demand change trend includes the computing load change trend, communication sensitivity change trend, and memory usage change trend of the task unit; the calculation of the matching weight between each task unit and each resource unit based on the global resource view and the resource demand change trend includes: Obtain the elastic requirements of task units; The computing matching degree is determined based on the computing load change trend and the computing capacity; The matching degree of video memory is determined based on the trend of memory usage change and the video memory capacity; Based on the communication sensitivity change trend and the network performance, network affinity is determined; Determine the elasticity matching degree based on the elasticity requirements and the elasticity support capabilities; The matching weight is obtained by weighted summing of the matching degrees of at least two dimensions among the calculated matching degree, the memory matching degree, the network affinity, and the elastic matching degree.
[0015] In the above implementation process, firstly, the computing power, memory capacity, network performance, and elastic support capabilities of resource units are extracted from the global resource view. Then, the computing load, communication sensitivity, and memory usage trends of task units are extracted from resource demand change trends, and the elastic requirements of task units are obtained. Next, matching values for four dimensions—matching degree, memory matching degree, network affinity, and elastic matching degree—are calculated, corresponding to four objectives: computing power supply and demand balance, memory capacity adaptation, communication efficiency optimization, and elastic scaling reservation, respectively. Finally, the matching degrees of at least two dimensions are merged into a comprehensive matching weight through weighted summation. This achieves the quantification of the task-resource matching relationship, enabling scheduling decisions to simultaneously consider computing performance, memory safety, communication efficiency, and elasticity.
[0016] Furthermore, the constraints of the constrained bipartite graph maximum weight matching problem are determined through the following steps: Set locality constraints to limit the placement of task units with communication dependencies on resource units that satisfy low-latency network interconnection conditions; and / or, Set an exclusivity constraint, which limits a resource unit to being allocated to at most one task unit at any given time; and / or, Set architectural constraints, which are used to limit the dependencies of tasks on hardware architecture or software stack that must be met.
[0017] In the above implementation process, locality constraints force task units with high-frequency communication dependencies to be placed on resource units that meet the low-latency network interconnection conditions, thus avoiding performance degradation caused by cross-node communication; exclusivity constraints ensure that each resource unit is allocated to at most one task unit at any given time, preventing resource contention and conflicts; and architectural constraints ensure that the dependencies of tasks on specific hardware architectures or software stacks are strictly satisfied, thus solving the technical problem of incompatibility between different architectures in domestically produced heterogeneous clusters.
[0018] Furthermore, the method also includes: During the execution of the training task, adjustments are made to the training task based on the deviation between the actual resource utilization rate and the trend of resource demand changes and / or the state changes of the heterogeneous computing power cluster.
[0019] In the above implementation process, the problem of resource supply and demand mismatch caused by model errors, changes in task behavior, or training phase switching is solved, enabling resource supply to keep pace with the dynamic needs of tasks in real time. At the same time, the status changes of heterogeneous computing power clusters are monitored, and adjustment operations are triggered in a timely manner when events such as node failure, resource addition, or high-priority task preemption occur, enabling the system to respond quickly to environmental changes and optimize resource allocation.
[0020] Furthermore, adjusting the training task based on the deviation between the actual resource utilization rate and the trend of resource demand changes and / or the state changes of the heterogeneous computing power cluster includes: When the deviation exceeds a preset threshold, and / or when the state of the heterogeneous computing power cluster changes, the current state data of the training task is obtained; Based on the current state data, the execution of the training task is resumed on the target resource unit.
[0021] In the above implementation process, the migration process is triggered when the deviation exceeds the threshold or the cluster state changes, ensuring that the migration timing considers both changes in the task's own requirements and changes in the cluster environment. Then, task execution is resumed on the target resource unit based on the state data, achieving seamless continuation from the breakpoint. This enables the system to proactively migrate tasks to better resource units in scenarios such as resource performance degradation, node failure, the addition of new resources, and intensified resource competition.
[0022] Furthermore, adjusting the training task based on the deviation between the actual resource utilization rate and the trend of resource demand changes and / or the state changes of the heterogeneous computing power cluster includes: When the deviation exceeds a preset threshold, and / or when the state of the heterogeneous computing power cluster changes, obtain the elastic parallel configuration information of the training task; Based on the elastic parallel configuration information, the data parallelism of the training task is increased or decreased to obtain the adjusted data parallelism. Adjust the number of task units corresponding to the training task based on the adjusted data parallelism; The newly added task units are allocated to the available resource units in the heterogeneous computing power cluster, or the resource units occupied by the reduced task units are released.
[0023] In the above implementation process, when the deviation exceeds a threshold or the cluster state changes, the elastic parallel configuration information of the task is obtained to ensure that the scaling up and down operations are performed within the range supported by the task. The data parallelism is increased or decreased according to the configuration information to dynamically adjust the computational scale of the task. Then, the number of task units is adjusted according to the adjusted parallelism, transforming the change in parallelism into operable task unit management. Finally, available resource units are allocated to newly added task units, or resources occupied by reduced task units are released. This achieves real-time adjustment of the parallel scale of the training task, accelerating training through scaling up when resources are sufficient and releasing resources for other tasks through scaling down when resources are scarce.
[0024] A second aspect of this application provides an electronic device, the electronic device comprising: processor; Memory used to store processor-executable instructions; Wherein, when the processor invokes the executable instructions, it implements any of the methods described in the first aspect.
[0025] A third aspect of this application provides a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of any of the methods described in the first aspect.
[0026] A fourth aspect of this application provides a computer program product, the computer program product including a computer program, which, when executed by a processor, implements any of the methods described in the first aspect. Attached Figure Description
[0027] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.
[0028] Figure 1 A flowchart illustrating a computing power scheduling method for large model training provided in this application embodiment; Figure 2 A system overall architecture block diagram provided for embodiments of this application; Figure 3 A flowchart of task feature extraction and prediction provided in an embodiment of this application; Figure 4 This is a structural block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation
[0029] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.
[0030] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.
[0031] In related technologies, 1. Model feature unawareness: The scheduler treats accelerator cards as homogeneous computing resources, ignoring the significant differences in computing density, communication mode, memory usage and growth mode, checkpoint I / O characteristics of different large model training tasks (such as pre-training, supervised fine-tuning, and reinforcement learning based on human feedback), resulting in a mismatch between resource allocation and real needs.
[0032] 2. Static resource view: Scheduling decisions heavily rely on the statically declared resource request amount when the task is submitted, which cannot effectively respond to the dynamic fluctuations in computing power demand caused by changes in data batches and model states during training, resulting in low resource utilization or task performance bottlenecks.
[0033] 3. Poor adaptability to heterogeneous environments: Domestic computing infrastructure often features multiple computing architectures (such as x86 / ARM CPUs, NVIDIA / Ascend / Hygon DCUs) coexisting. Existing schedulers lack a fine-grained awareness of hardware architecture characteristics, high-speed interconnect topologies (such as NVLink, HCCS, RoCE networks), and computing library compatibility, making it difficult to achieve optimal task placement across architectures.
[0034] 4. Low elasticity efficiency: For the dynamic scaling requirements of training tasks, most related technologies are based on simple global resource thresholds for reactive adjustment. The decision-making is slow and it is not coordinated with the inherent stages of training tasks (such as gradient synchronization, validation and evaluation), which can easily lead to resource waste or training interruption.
[0035] To address any of the aforementioned problems, this application provides an elastic computing power scheduling method that can dynamically optimize resource allocation based on model characteristics and runtime state. This method achieves intelligent scheduling by uniformly abstracting heterogeneous resources, statically extracting and dynamically predicting task characteristics, modeling the core scheduling problem as a constrained bipartite graph maximum weight matching problem and solving it, implementing elastic rescheduling at runtime, and periodically optimizing the global strategy.
[0036] Based on this, embodiments of this application provide a computing power scheduling method for large model training, referring to... Figure 1 , Figure 1 This is a flowchart illustrating a computing power scheduling method for large model training provided in an embodiment of this application.
[0037] In this embodiment, the method includes: Step S10: Construct a global resource view based on the real-time status information of heterogeneous computing resources; It should be noted that heterogeneous computing resources include computing resources with various hardware architectures, such as NVIDIA GPUs, Huawei Ascend, and Hygon DCUs. Different architectures differ in terms of computing power, memory capacity, and interconnect topology.
[0038] Real-time status information: Current resource utilization data collected by the monitoring agent, including dynamic indicators such as GPU utilization, video memory usage, and network latency.
[0039] Global resource view: A description of the resource status of the entire heterogeneous cluster, including the static attributes (such as hardware type and computing power) and dynamic indicators (such as real-time utilization) of each node.
[0040] Unified abstraction and real-time perception of heterogeneous computing resources can eliminate the differences in underlying hardware and provide an accurate resource status basis for subsequent scheduling.
[0041] Step S20: Obtain the task characteristics of the training task of the large model to be scheduled, and predict the resource demand change trend of the training task in the preset future time window based on the performance data of the training task in the preset historical time window. It should be noted that in related technologies, traditional schedulers rely solely on static task declarations, while the embodiments of this application can predict future changes in resource requirements by analyzing the actual running data of tasks in the early stages.
[0042] Task features describe the static features of a task. These features include model type, training phase (pre-training phase / fine-tuning phase / reinforcement learning phase), and parallel strategies (data parallelism / tensor parallelism / pipeline parallelism), etc.
[0043] Preset historical time window: The time range for collecting historical performance data of the task, such as the first few iterations after the task starts (e.g., the first 10 iterations). This window can be dynamically adjusted.
[0044] Performance data includes iteration time, GPU / DCU utilization, and video memory usage.
[0045] Preset future time window: The time range used to predict the future resource requirements of a task. The length of this window can be dynamically set based on factors such as the task's training phase, the rate of change in resource requirements, the scheduling decision cycle, and the effective prediction range of the prediction model. For example, for tasks in the early stages of training, a shorter time window (such as the next iteration or tens of seconds in the future) can be set to quickly respond to changes in requirements; for tasks in a stable training phase, a longer time window (such as several minutes or several iteration cycles in the future) can be set to match the granularity of the scheduling strategy adjustment.
[0046] Resource Demand Trends: This section outlines the dynamic changes in the demand for various resources during the training of large models over time. Large model training exhibits significant phased characteristics. For example, there may be a warm-up phase in the early stages of training, during which the computational load gradually increases; the gradient synchronization phase sees a surge in communication demands, with communication sensitivity peaking; and the checkpoint saving phase may experience a temporary increase in memory usage.
[0047] Step S30: Based on the global resource view and the resource demand change trend, construct a constrained bipartite graph maximum weight matching problem between task units and resource units, and determine the matching relationship between the task units and the resource units; It should be noted that the task unit is the smallest scheduling unit after the large model training task to be scheduled is dynamically divided, such as a process group divided according to tensor parallelism.
[0048] Resource unit: The smallest unit of allocation after dividing heterogeneous computing resources, such as a single GPU / DCU card, or computing resources divided according to virtualization granularity.
[0049] Maximum weight matching in a bipartite graph: The left node is the task unit, the right node is the resource unit, and the edge weight is the matching degree. The goal is to find the matching scheme with the maximum total weight.
[0050] With constraints: Constraints that must be met during the matching process, such as locality constraints (task units that communicate closely must be placed on the same node), exclusivity constraints (resource units cannot be shared), and architectural constraints (specific tasks must be adapted to specific hardware).
[0051] Matching relationship: The correspondence between each task unit and which resource unit it is bound to.
[0052] Step S40: Perform task placement according to the matching relationship.
[0053] It should be noted that task placement refers to actually deploying the task unit to the corresponding resource unit based on the matching relationship obtained from the solution, and starting task execution.
[0054] In this embodiment, the system realizes the perception of heterogeneous computing resources and the prediction of the dynamic requirements of large model training tasks. The scheduling decision is modeled as a constrained graph optimization problem, thereby obtaining the globally optimal task placement scheme under the premise of satisfying multiple constraints. This improves the accuracy of scheduling decisions and the utilization rate of cluster resources, and solves the problems of traditional schedulers being unaware of task characteristics, having static resource views, and having poor adaptation to heterogeneous environments.
[0055] Based on any of the above embodiments, the global resource view is constructed through the following steps: A resource descriptor is defined for each computing node to determine the global resource view; the resource descriptor includes hardware type, computing power, storage configuration, network interconnection topology, and real-time utilization metrics.
[0056] It should be noted that a resource descriptor is defined for each computing node. This descriptor should include at least the hardware type, computing power, storage configuration, network interconnection topology, and real-time utilization metrics.
[0057] Hardware type is used to identify different architectures (such as x86, ARM, NVIDIA, Ascend); computing power may include floating-point operation power (TFLOPS). Optionally, the computing power of different types of computing nodes can be normalized and quantized.
[0058] Storage configuration includes memory and video memory capacity; network interconnection topology records the connection topology and performance parameters within nodes (such as NVLink) and between nodes; real-time utilization metrics are continuously collected by a monitoring agent to reflect the current load status of resources.
[0059] In this embodiment, resource descriptors enable unified abstraction and dynamic monitoring of heterogeneous hardware resources. The constructed global resource view not only includes static attributes but also integrates real-time status and interconnection topology, which provides a comprehensive basis for scheduling decisions, effectively shields the heterogeneity of the underlying hardware, and allows the upper-level scheduling algorithm to focus on optimizing matching.
[0060] Based on any of the above embodiments, the resource demand change trend includes the computational load change trend, communication sensitivity change trend, and memory usage change trend of the task unit; predicting the resource demand change trend of the training task in a preset future time window based on the performance data of the training task within a preset historical time window includes: Obtain the task characteristics of the training task; the task characteristics include model type, training stage, and parallel strategy; Using a time series analysis model, based on the iteration time and resource utilization of the training task within the historical time window, the trends of computational load, communication sensitivity, and memory usage of the training task within the future time window are predicted.
[0061] It should be noted that the computing load change trend—the demand for computing resources (such as GPU cores and floating-point computing power) by a task—may vary over time, exhibiting a stable, increasing, decreasing, or periodic fluctuation. The purpose of predicting the computing load change trend in this embodiment is to anticipate the emergence of computing bottlenecks and adjust the allocation of computing resources in advance.
[0062] Communication sensitivity trends: The sensitivity of a task to network bandwidth and latency changes over time, generally increasing significantly during the gradient synchronization phase. The purpose of predicting communication sensitivity trends is to anticipate communication peaks and reserve network bandwidth in advance or schedule relevant task units to high-bandwidth interconnect domains.
[0063] Memory usage trends: The changing pattern of a task's demand for video memory or system memory over time, which may increase with model parameter updates and the accumulation of activation values. The purpose of predicting memory usage trends is to anticipate the risk of video memory overflow and adjust or reserve video memory resources in advance.
[0064] Task characteristics: Static information describing the inherent attributes of the training task, which does not change over time.
[0065] Model type: Model architecture identifier, such as Qwen2-72B, DeepSeek-V2, Llama3-70B, etc. Different model architectures have different operator structures, computation modes, and communication modes. Model type information can help predict model behavior.
[0066] Training phase: This refers to the training phase of the task, including pre-training, supervised fine-tuning, and reinforcement learning. The computational density, memory usage patterns, and communication requirements differ across these phases.
[0067] Parallelism strategy: The distributed training strategy employed by the task, such as data parallelism, tensor parallelism, pipelined parallelism, and combinations thereof. The parallelism strategy directly affects how the task is divided into multiple task units and the communication dependencies between these units.
[0068] The aforementioned static features constitute the prior knowledge of the prediction model. For example, tasks using tensor parallelism are generally more communication-sensitive than tasks using only data parallelism; the computational load during the pre-training phase is generally more stable than that during the fine-tuning phase; and the memory growth curves of different model architectures differ.
[0069] Time series analysis model: A machine learning model that can process time series data to learn patterns from historical data and predict future numerical trends.
[0070] Optionally, a time series analysis model such as LSTM (Long Short-Term Memory Network) can be used to take the iteration time and resource utilization rate of the task within the historical time window as input, learn the changing patterns of these indicators over time, and then output the changing trends of resource demand in various dimensions within the future time window.
[0071] In this embodiment, static features such as model type, training stage, and parallel strategy are combined with dynamic time-series data such as iteration time and resource utilization collected in the early stage of task operation. The time-series analysis model is used to predict the trend of changes in computing load, communication sensitivity, and memory usage within the future time window. This enables the scheduling system to have predictive capabilities, allowing it to anticipate risks and prepare in advance before resource bottlenecks actually occur, providing a reliable foundation for subsequent resource matching and elastic scheduling.
[0072] Based on any of the above embodiments, the step of constructing a constrained bipartite graph maximum weight matching problem between task units and resource units based on the global resource view and the resource demand change trend, and determining the matching relationship between the task units and the resource units, includes: The training tasks to be scheduled are divided into multiple task units, and the heterogeneous computing resources are divided into multiple resource units. Based on the global resource view and the trend of resource demand changes, calculate the matching weight between each task unit and each resource unit; The multiple task units are used as the first set of nodes in the bipartite graph, the multiple resource units are used as the second set of nodes in the bipartite graph, and the matching weight is used as the weight of the edge connecting the first set of nodes and the second set of nodes to construct the constrained maximum weight matching problem of the bipartite graph. Solve the constrained bipartite graph maximum weight matching problem to obtain the matching relationship between the task unit and the resource unit.
[0073] It should be noted that a task unit is a smaller granular schedulable subtask obtained by decomposing a large model training task to be scheduled according to its parallel strategy (such as data parallelism, tensor parallelism, pipeline parallelism).
[0074] For example, task units are obtained through the following division method: Data parallelism dimension: The training data is divided into shards, and each shard processes a portion of the data in parallel, corresponding to a task unit; Tensor parallel dimension: The model parameter matrix is divided, and each tensor parallel process is responsible for a part of the parameter calculation; Pipeline parallel dimension: The model layer is divided into stages, with each stage corresponding to a task unit.
[0075] By breaking down large tasks into multiple task units, the scheduling granularity becomes finer, allowing for full utilization of fragmented resources within the cluster and avoiding large resource wait times caused by overall scheduling. For example, a task requiring 64 GPUs can be started as 16 task units when resources are scarce, and then elastically scaled up later.
[0076] Resource unit: The smallest allocatable unit obtained by dividing the physical resources in a heterogeneous computing power cluster according to a certain granularity.
[0077] Alternatively, resource units can be divided in the following manner: Single-card granularity: A single GPU, DCU, or CPU core is treated as an independent resource unit; Node granularity: The entire server is treated as a resource unit (suitable for communication-intensive tasks); Virtualization granularity: Through GPU virtualization technology, a physical card is divided into multiple virtual resource units.
[0078] Heterogeneous physical resources (different models of GPUs, different architectures of CPUs, and network links of different speeds) are abstracted into unified resource units, enabling scheduling algorithms to process heterogeneous resources in a consistent manner. The granularity of the resource unit partitioning should match the granularity of the task unit to ensure the feasibility of the matching.
[0079] Matching weight: A numerical value that quantifies the degree of matching between a specific task unit and a specific resource unit. The higher the value, the more suitable the resource unit is for executing the task unit.
[0080] The goal of the maximum weight matching problem in a bipartite graph is to find a set of edges in a bipartite graph (where vertices can be divided into two disjoint sets and edges exist only between the two sets) such that no two edges in this set have a common vertex (i.e., each vertex is matched at most once) and the sum of the weights of all matched edges is maximized.
[0081] Constrained maximum weight matching problem in bipartite graphs: Based on the standard maximum weight matching problem in bipartite graphs, additional constraints are added to limit the feasibility of matching solutions.
[0082] Specifically, based on the parallel strategy of the task, the training task is divided into multiple independently schedulable small-granularity units. For example, a task with TP=4 and PP=2 can be divided into 8 task units (4 units within each TP group, for a total of 2 PP stages). Furthermore, based on the state of the resource pool and the granularity requirements of scheduling, physical resources (GPU, CPU, memory, network bandwidth) are divided into the smallest allocatable units. For example, each GPU card can be considered a resource unit, or a single card can be further subdivided into multiple resource units using virtualization technology. The computational load, communication sensitivity, and memory usage trends of the task units are obtained from resource demand trends; elastic requirements are obtained from static configurations. The computational power, memory capacity, network performance, and elastic support capabilities of the resource units are obtained from the global resource view. The requirements and capabilities are compared to determine the computational matching degree (the ratio of requirements to capabilities), memory matching degree (the degree of matching between requirements and capacity), network affinity (the adaptation of communication requirements to network performance), and elasticity matching degree (the matching between elastic requirements and elastic support capabilities). Next, the matching scores of each dimension are weighted and summed according to preset weight coefficients to obtain the comprehensive matching weight. All task units are treated as the left-side node set U = {u1, u2, ..., u...} of the bipartite graph. m All resource units are represented as the right-hand node set V = {v1, v2, …, v} n For each pair of task units u i and resource unit v j Establish a route from u i to v j The edges, with weights equal to the matching weights W mentioned above. ijBased on locality constraints, exclusivity constraints, and architectural constraints, the graph is pruned or marked. For example, edges with mismatched architectures are deleted; groups of nodes that need to be matched (locality constraints) are marked; constraints that each node can be matched at most once are marked (naturally satisfying the bipartite graph matching definition). An appropriate algorithm is selected based on the problem size; for example, the Hungarian algorithm is suitable for small to medium-sized problems, while the auction algorithm is suitable for large-scale distributed scenarios. Then, the algorithm is run to find the matching scheme with the largest total weight under the constraints. The algorithm will output a set of matching pairs {(u1, v...} j1 ),(u2,v j2 ), …,(u k ,v jk This represents which resource unit each task unit is assigned to. In other words, it obtains the globally optimal task placement scheme.
[0083] In this embodiment, by transforming the complex heterogeneous computing power scheduling problem into a constrained bipartite graph maximum weight matching problem, the modeling and global optimization of scheduling decisions are realized. This solves the problem that traditional schedulers have difficulty taking into account multi-dimensional optimization objectives and complex constraints in heterogeneous environments, and improves the accuracy of computing power scheduling and resource utilization efficiency in large-scale heterogeneous clusters.
[0084] Based on any of the above embodiments, the global resource view includes the computing power, video memory capacity, network performance, and elastic support capability of the resource unit; the resource demand change trend includes the computing load change trend, communication sensitivity change trend, and memory usage change trend of the task unit; the step of calculating the matching weight between each task unit and each resource unit according to the global resource view and the resource demand change trend includes: Obtain the elastic requirements of task units; The computing matching degree is determined based on the computing load change trend and the computing capacity; The matching degree of video memory is determined based on the trend of memory usage change and the video memory capacity; Based on the communication sensitivity change trend and the network performance, network affinity is determined; Determine the elasticity matching degree based on the elasticity requirements and the elasticity support capabilities; The matching weight is obtained by weighted summing of the matching degrees of at least two dimensions among the calculated matching degree, the memory matching degree, the network affinity, and the elastic matching degree.
[0085] It should be noted that computing power refers to the number of floating-point operations that a resource unit can perform per second.
[0086] Video memory capacity: The total video memory capacity of a resource unit.
[0087] Network performance: The network transmission capability of a resource unit, including metrics such as bandwidth (Gbps) and latency (μs).
[0088] Elasticity support capability is used to characterize whether a resource unit supports elastic parallelism, dynamic scaling, and the maximum degree of parallelism it supports.
[0089] Computational load variation trend: The pattern of how the task's demand for computing resources changes over time within a future time window.
[0090] Communication sensitivity change trend: The pattern of how the sensitivity of a task to network resources changes over time within a future time window.
[0091] Memory usage trend: The pattern of how the task's demand for video memory changes over time within a future time window.
[0092] Elasticity requirements: The degree to which a task depends on elastic parallelism and its expansion requirements, including whether dynamic scaling is supported, the expected maximum parallelism, and data synchronization requirements during scaling.
[0093] The trend of resource demand changes reflects the changing pattern of resource demand during task operation, while elasticity demand reflects the degree of support for elastic parallel capabilities and the willingness to expand.
[0094] Computational matching degree: A numerical value that quantifies the degree of matching between the computational requirements of a task unit and the computational capabilities of a resource unit. It is based on a comparison between the computational load change trend of the task unit and the computational capabilities of the resource unit. For example, if the computational load trend indicates a computationally intensive task, the matching degree tends to favor resource units with strong computational capabilities; if the computational load trend indicates a computationally sparse task, the matching degree tends to favor resource units with moderate computational capabilities but better energy consumption. It is determined comprehensively by combining the ratio of computational load intensity to computational capability with characteristics such as the peak, mean, and variance of load changes.
[0095] Memory matching: A numerical value that quantitatively evaluates the degree of matching between the memory requirements of a task unit and the memory capacity of a resource unit. It is based on a comparison between the memory usage trend of a task unit and the memory capacity of a resource unit.
[0096] Network affinity: A numerical value that quantifies the degree of match between the communication needs of a task unit and the network performance of a resource unit. It is based on a comparison of the changing trends in the communication sensitivity of a task unit with the network performance (bandwidth, latency) of the resource unit.
[0097] Elasticity Matching Degree: A numerical value that quantifies the degree of matching between the elasticity requirements of a task unit and the elasticity support capabilities of a resource unit. It compares the elasticity requirements of the task (whether it supports scaling, expected maximum parallelism, etc.) with the elasticity support capabilities of the resource unit (whether it supports virtualization, dynamic allocation, remaining scalable capacity, etc.): If the task requires support for elastic scaling, resource units that support elastic parallelism and have sufficient scalability are prioritized for matching; if the task has no elasticity requirements, the elasticity matching degree can be set to a constant or a lower weight; it is determined comprehensively through indicators such as the degree of elasticity requirement satisfaction and the size of the scalability.
[0098] Specifically, at least two dimensions of matching degree are selected (all four dimensions or some dimensions can be selected depending on the scenario), and a weighted summation operation is performed on each selected dimension to obtain the comprehensive matching weight. This integrates multi-dimensional and potentially conflicting matching objectives into a single value, enabling the scheduling algorithm to make optimization decisions based on a unified standard. Furthermore, by adjusting the weight coefficients, the scheduler's preferences can be flexibly controlled to adapt to different optimization objectives.
[0099] In this embodiment, the computing power, memory capacity, network performance, and elastic support capability of resource units are first extracted from the global resource view. The computing load, communication sensitivity, and memory usage trends of task units are extracted from resource demand change trends, and the elastic requirements of task units are obtained. Then, matching values for four dimensions—matching degree, memory matching degree, network affinity, and elastic matching degree—are calculated, corresponding to four objectives: computing power supply and demand balance, memory capacity adaptation, communication efficiency optimization, and elastic scaling reservation. Finally, the matching degrees of at least two dimensions are merged into a comprehensive matching weight through weighted summation. This quantifies the matching relationship between tasks and resources, enabling scheduling decisions to simultaneously consider computing performance, memory safety, communication efficiency, and elasticity.
[0100] Based on any of the above embodiments, the constraints of the constrained bipartite graph maximum weight matching problem are determined through the following steps: Set locality constraints to limit the placement of task units with communication dependencies on resource units that satisfy low-latency network interconnection conditions; and / or, Set an exclusivity constraint, which limits a resource unit to being allocated to at most one task unit at any given time; and / or, Set architectural constraints, which are used to limit the dependencies of tasks on hardware architecture or software stack that must be met.
[0101] It should be noted that locality constraints handle communication dependencies between task units. Exclusivity constraints handle resource unit usage conflicts. Architectural constraints handle specific hardware / software requirements of a task.
[0102] Constraints: These are the restrictive conditions that must be satisfied during the solution of the maximum weighted matching problem in a bipartite graph. These conditions define which matchings are allowed and which are prohibited, thus ensuring the feasibility of the solution in actual execution. Without constraints, the maximum weighted matching problem in a bipartite graph might distribute task units with close communication dependencies across different nodes, leading to a surge in communication latency; it might also assign a single resource unit to multiple task units simultaneously, causing resource conflicts; and it might assign tasks requiring Ascend architecture to NVIDIA GPUs, causing the tasks to fail. Constraints are set precisely to mitigate these risks.
[0103] Locality constraint: A mandatory placement restriction that requires groups of task units with communication dependencies to be placed on resource units that meet the conditions for low-latency network interconnection.
[0104] Task unit groups with communication dependencies: After task partitioning, these are sets of task units that require data exchange due to parallel strategies (such as tensor parallelism and pipelined parallelism). These units need to frequently exchange intermediate computation results, and their communication volume is much greater than that of typical data-parallel tasks.
[0105] Low-latency network interconnection conditions: Network connections between resource units meet specific latency and bandwidth requirements. These include: Intra-node interconnects: such as NVLink (NVIDIA), HCCS (Ascend), and Infinity Fabric (AMD), with nanosecond-level latency and hundreds of GB / s bandwidth; High-speed inter-node networks: such as RoCE and InfiniBand, with microsecond-level latency and tens to hundreds of Gbps bandwidth; Within the same switch: Avoid the additional latency caused by routing across switches.
[0106] Exclusivity constraint: A restriction on resource usage that stipulates that a resource unit can be allocated to at most one task unit at any given time.
[0107] At the time of the scheduling decision, a resource unit can only be allocated to one task unit. However, within different time windows, a resource unit can be allocated to different task units (e.g., resources are released for use by subsequent tasks after a task is completed).
[0108] A resource unit being allocated to at most one task unit at a time means that the resource unit is allowed to be idle (allocated to zero task units), but is not allowed to be allocated to multiple task units.
[0109] Architectural constraints: A type of compatibility restriction that requires a task to satisfy its dependencies on a specific hardware architecture or software stack.
[0110] Hardware architecture: The underlying hardware design of the computing unit, such as NVIDIA's Ampere / Hopper architecture, Huawei Ascend's Da Vinci architecture, and Hygon's GCN / CDNA architecture. Different architectures have different instruction sets, memory access modes, and parallel computing models.
[0111] Software stack: The software environment that supports task execution, including drivers, runtime libraries, computing frameworks, etc. For example, CUDA is the programming model for NVIDIA GPUs, and CANN is the heterogeneous computing architecture of the Ascend AI processor.
[0112] Specifically, locality constraints are set as follows: Based on the task's parallel strategy, identify which task units have high-frequency communication dependencies. For example, task units within the same tensor parallel group need to frequently synchronize activation values and gradients; task units within the same pipeline stage have data transfer relationships. Obtain the network interconnection topology between resource units from the global resource view, and mark which resource units satisfy the "low-latency network interconnection condition" (e.g., being connected via NVLink within the same node, or being under the same switch). In the bipartite graph maximum weight matching problem, treat task units within communication dependency groups as a whole for matching constraints, requiring them to be assigned to the set of resource units that satisfy the interconnection condition.
[0113] Exclusivity constraints can be set as follows: Identify from the global resource view which resource units do not support sharing (e.g., physical GPUs are typically exclusive) and which support sharing (e.g., CPU cores can be shared through virtualization). In the bipartite graph maximum weight matching problem, set a capacity limit for each resource unit, meaning each resource unit can match at most one task unit.
[0114] Architectural constraints are set as follows: Obtain the hardware architecture and software stack requirements from the task's task characteristics. For example, a task might specify the need for Ascend 910B hardware and CANN software stack. Obtain the hardware type (e.g., nvidia_h100, ascend_910b) and software stack label (e.g., cuda12.4, CANN7.0) for each resource unit from the global resource view. In the bipartite graph maximum weight matching problem, for each task unit, only edges are allowed to be established with resource units that satisfy the architectural dependencies; edges that do not satisfy the dependencies are directly deleted or have their weights set to negative infinity.
[0115] In this embodiment, locality constraints force task units with high-frequency communication dependencies to be placed on resource units that meet low-latency network interconnection conditions, thus avoiding performance degradation caused by cross-node communication; exclusivity constraints ensure that each resource unit is allocated to at most one task unit at any given time, preventing resource contention and conflicts; and architectural constraints ensure that the dependencies of tasks on specific hardware architectures or software stacks are strictly satisfied, solving the technical problem of incompatibility between different architectures in domestically produced heterogeneous clusters.
[0116] Based on any of the above embodiments, the method further includes: During the execution of the training task, adjustments are made to the training task based on the deviation between the actual resource utilization rate and the trend of resource demand changes and / or the state changes of the heterogeneous computing power cluster.
[0117] It should be noted that the actual resource utilization rate refers to the actual usage of each resource unit collected in real time by the monitoring agent during the training task.
[0118] Actual resource utilization includes computing resource utilization (GPU / DCU core utilization, floating-point unit utilization), video memory utilization (the percentage of video memory used out of the total video memory), network utilization (the percentage of network interface throughput out of the total bandwidth), and CPU / memory utilization (CPU core utilization and memory utilization).
[0119] State changes in heterogeneous computing clusters refer to various events occurring in the cluster environment that may affect task execution. For example, types of state changes include: Resource changes: node addition, node removal, node failure recovery; Resource competition: vying for high-priority tasks; Task phase switching: Training enters the gradient synchronization phase and then the checkpoint saving phase; Performance degradation: Overheating of hardware and increased network packet loss rate.
[0120] Adjusting the training task refers to performing resource allocation changes on the running training task based on triggering conditions, so as to adapt it to new requirements or environmental changes.
[0121] Specifically, after the training task is placed and starts running, the scheduling system continuously monitors two types of information: the actual resource utilization of each resource unit (multi-dimensional, including computation, memory, and network), and the real-time status of the heterogeneous computing cluster (node health, resource availability, network topology changes, etc.). Then, it calculates the deviation between the actual resource utilization and the trend of changing resource demands, determining whether it exceeds a preset threshold (different thresholds can be set for different dimensions); and detects whether the cluster status has changed (node failure, new resource addition, task phase switching, etc.). If the deviation exceeds the threshold, it indicates that the actual task demand does not match the prediction, or the task behavior has changed, requiring a reassessment of resource allocation. If the cluster status changes, it indicates that the environment has changed, potentially presenting optimization opportunities or risks, requiring a readjustment of resource layout. Finally, based on the specific situation, an appropriate adjustment method is selected: if the deviation manifests as insufficient resources, horizontal scaling (increasing parallelism) can be prioritized; if the deviation manifests as resource waste, horizontal scaling (reducing parallelism) can be prioritized; if triggered by node failure or performance degradation, task migration can be prioritized; if triggered by the addition of new resources, migrating some tasks to new resources can be considered to achieve load balancing.
[0122] This embodiment addresses the resource supply-demand mismatch caused by model errors, changes in task behavior, or training phase switching, enabling resource supply to keep pace with the dynamic needs of the task in real time. Simultaneously, it monitors the state changes of the heterogeneous computing cluster, triggering timely adjustment operations when events such as node failures, resource additions, or high-priority task preemption occur, allowing the system to quickly respond to environmental changes and optimize resource allocation.
[0123] Based on any of the above embodiments, adjusting the training task according to the deviation between the actual resource utilization rate and the trend of resource demand changes and / or the state changes of the heterogeneous computing power cluster includes: When the deviation exceeds a preset threshold, and / or when the state of the heterogeneous computing power cluster changes, the current state data of the training task is obtained; Based on the current state data, the execution of the training task is resumed on the target resource unit.
[0124] It should be noted that a deviation exceeding a preset threshold indicates that the difference between the actual resource utilization rate and the changing trend of resource demand exceeds a pre-defined acceptable range. When the deviation exceeds the threshold, it means that the actual operation of the task is seriously inconsistent with expectations, possibly due to factors such as prediction errors, changes in task behavior, and fluctuations in hardware performance, leading to an imbalance between resource supply and demand. In such cases, it is necessary to migrate the task to a more suitable resource unit and release the resources occupied by the training task in the original resource unit.
[0125] Thresholds can be set for different dimensions such as computing load, communication sensitivity, and memory usage, or a comprehensive deviation score can be used. This embodiment does not limit the preset threshold.
[0126] Current state data: The runtime state information of the training task at the moment the transfer is triggered. Specifically, the current state data is obtained by triggering the checkpoint saving operation of the training framework. Checkpoints are a commonly used fault-tolerance mechanism in large model training, which periodically saves the training state to persistent storage. When a transfer is needed, the most recently saved checkpoint can be used directly, or a checkpoint saving can be triggered temporarily before the transfer.
[0127] Target resource unit: The target location for task migration, i.e., the newly allocated resource unit for performing the training task.
[0128] Optionally, a resource unit that meets the task requirements (computing power, video memory capacity, network performance, architecture compatibility) can be selected from the currently available idle resources as the target resource unit, or a resource unit that optimizes task performance (such as strong computing power and good network location) can be selected as the target resource unit.
[0129] Resuming the execution of the training task means restarting the training task on the target resource unit based on the acquired current state data and continuing execution from the saved state.
[0130] In this embodiment, the migration process is triggered when the deviation exceeds a threshold or the cluster state changes, ensuring that the migration timing considers both changes in the task's own requirements and changes in the cluster environment. Then, task execution is resumed on the target resource unit based on state data, achieving seamless continuation from the breakpoint. This enables the system to proactively migrate tasks to better resource units in scenarios such as resource performance degradation, node failure, the addition of new resources, and increased resource contention.
[0131] Based on any of the above embodiments, adjusting the training task according to the deviation between the actual resource utilization rate and the trend of resource demand changes and / or the state changes of the heterogeneous computing power cluster includes: When the deviation exceeds a preset threshold, and / or when the state of the heterogeneous computing power cluster changes, obtain the elastic parallel configuration information of the training task; Based on the elastic parallel configuration information, the data parallelism of the training task is increased or decreased to obtain the adjusted data parallelism. Adjust the number of task units corresponding to the training task based on the adjusted data parallelism; The newly added task units are allocated to the available resource units in the heterogeneous computing power cluster, or the resource units occupied by the reduced task units are released.
[0132] It should be noted that elastic parallel configuration information describes the degree of support for elastic parallelism in the training task and the configuration parameters. This information determines whether and how to scale up or down. Elastic parallel configuration information includes whether elastic parallelism is supported, the supported range of data parallelism, scaling granularity, and scaling strategies.
[0133] Data parallelism: The number of replicas of the training task in the data parallel dimension, that is, the number of task units that process different data partitions at the same time.
[0134] Number of task units: The number of task units corresponding to the data parallelism. In data-parallel training, each data-parallel replica typically corresponds to one task unit, therefore the number of task units is equal to the data parallelism.
[0135] Newly added task units refer to new task units generated due to increased data parallelism, and computing resources need to be allocated to them.
[0136] Reduced task units refer to task units that need to be stopped due to reduced data parallelism, and the resources they occupy need to be released.
[0137] It should be understood that when the deviation exceeds a preset threshold, and / or when the state of the heterogeneous computing cluster changes, different adjustment methods can be selected according to the specific scenario. For example: When the deviation manifests as insufficient resources and the cluster has idle resources, horizontal scaling (increasing parallelism) should be prioritized. When the deviation manifests as a waste of resources, horizontal scaling down (reducing parallelism) should be prioritized. When deviations are accompanied by deterioration in node performance, task migration can be selected.
[0138] Specifically, the system continuously monitors the deviation between actual resource utilization and resource demand trends, as well as cluster state change events. When the deviation exceeds a preset threshold, or a cluster state change is detected, a scaling assessment process is triggered. This process reads elastic parallel configuration information from task submission information or the runtime environment, including whether elastic parallelism is supported, the parallelism range, and the scaling granularity. When the deviation indicates insufficient resources and the cluster has idle resources, or when new resources are added, the data parallelism is increased within the configured range (e.g., from 8 to 16). When the deviation indicates resource waste, or when resources are scarce and need to be released, the data parallelism is decreased within the configured range (e.g., from 16 to 8). Parallelism adjustment is performed through the training framework's elastic API, synchronizing the state of each task unit. Based on the adjusted data parallelism and the task's parallelism strategy (e.g., tensor parallelism, pipeline parallelism depth), the total number of new task units is calculated, along with the number of newly added or removed task units, to determine which task units need to be created and which need to be stopped. Next, regarding resource allocation during expansion: from the currently available resources, select resource units that meet the requirements (computing power, GPU memory capacity, network performance, architecture compatibility) for the new task units, create task unit processes, initialize the communication environment, synchronize states, and start training; regarding resource release during scaling down: notify the task units to be stopped to complete their current work, stop the task unit processes, release the occupied computing, GPU memory, and network resources, and mark the released resources as free for use by other tasks.
[0139] In this embodiment, when the deviation exceeds a threshold or the cluster state changes, the elastic parallel configuration information of the task is obtained to ensure that scaling up and down operations are performed within the range supported by the task. The data parallelism is increased or decreased according to the configuration information to dynamically adjust the computational scale of the task. Then, the number of task units is adjusted according to the adjusted parallelism, transforming the change in parallelism into operable task unit management. Finally, available resource units are allocated to newly added task units, or resources occupied by reduced task units are released. This achieves real-time adjustment of the parallel scale of the training task, accelerating training through scaling up when resources are sufficient and releasing resources for other tasks through scaling down when resources are scarce.
[0140] Furthermore, based on the same concept as the aforementioned computational power scheduling method for large model training, this application also provides an elastic computational power scheduling method and system for heterogeneous domestically produced large models. (Refer to...) Figure 2 , Figure 2 This is a system architecture block diagram provided for an embodiment of this application.
[0141] In this embodiment, the elastic computing power scheduling system for heterogeneous domestic large-scale models consists of four core layers: 1. Resource Abstraction and Monitoring Layer (Underlying Data Foundation): Function: This is the system's "sensory" layer. It collects the lowest-level hardware metrics (GPU utilization, video memory, network I / O, etc.) in real time through monitoring agents deployed on all computing nodes.
[0142] Data Flow: All agents converge their data to a central resource manager. The manager standardizes this heterogeneous data, integrating it into a unified, global, dynamic resource graph that includes real-time status and topological relationships. This graph is the sole source of facts for all higher-level intelligent decisions.
[0143] 2. Instruction Execution Layer (Execution Arm): Function: This is the system's "execution arm." The elastic actuator receives specific instructions from the scheduler.
[0144] Data flow: It translates abstract scheduling instructions (such as "expand by 2 workers") into a series of specific calls to the underlying infrastructure (such as Kubernetes API, or the elastic control API of the DeepSpeed training framework) to complete the actual Pod lifecycle management, task scaling and migration operations.
[0145] 3. Real-time scheduling decision-making layer (intelligent brain): Function: This is the system's online "intelligent decision-making core," handling all real-time scheduling requests.
[0146] Internal collaboration process: The task analyzer receives tasks submitted by users, combines historical data with performance samples from the early stages of real-time operation, performs feature extraction and dynamic prediction, and forms a "digital profile" of the task.
[0147] The intelligent scheduler is the core algorithm carrier. It simultaneously monitors two types of events: 1) new task submissions; 2) elastic events triggered by reports from the monitoring layer (such as resource utilization exceeding limits or node failures).
[0148] When an event is triggered, the scheduler synchronously obtains the "task profile" from the task analyzer and the "global resource graph" from the resource manager.
[0149] Based on these two types of information, the scheduler runs an elastic scheduling algorithm based on bipartite graph matching to calculate the optimal resource allocation or adjustment scheme and then issues the scheme to the elastic executor.
[0150] 4. Global Strategy Optimization Layer (Experience Brain): Function: This is the system's offline "experience learning and optimization center," responsible for medium- to long-term overall optimization.
[0151] Data Flow: The strategy optimizer periodically (e.g., hourly) retrieves historical operational data of the cluster from the central resource manager, including: historical resource utilization (GPU, CPU, memory, etc.), historical task data (e.g., task arrival time, completion time, resources used, task type, etc.), energy consumption data, and network usage.
[0152] These data are used to calculate the various indices in the global objective function. To evaluate the effectiveness of the current scheduling strategy.
[0153] Control flow: Obtain parameters of the current scheduling policy (such as weight coefficients) from the intelligent scheduler. The system calculates the parameters (including various trigger thresholds) and the scheduler's performance under these parameters (such as scheduling success rate and scheduling time). The optimized parameters are then sent to the intelligent scheduler, enabling it to make scheduling decisions using these new parameters. This is a slow but crucial feedback loop that allows the system to adapt to changes in load patterns and continuously approach the optimal scheduling strategy.
[0154] Elastic computing power scheduling methods for heterogeneous domestic large-scale models include: Step S1: Perform unified abstraction and real-time monitoring of hardware resources in the heterogeneous computing cluster, and construct a global dynamic resource topology graph; Step S2: Extract the static features of the large model training task and combine them with runtime monitoring data to dynamically predict its computing power requirements. Step S3: Based on the dynamic resource topology and the predicted computing power demand characteristics, the matching problem between the task unit to be scheduled and the available resource unit is formalized into a constrained bipartite graph maximum weight matching problem. By calculating and solving the comprehensive weight matrix, the optimal task placement scheme is obtained. Step S4: During task execution, dynamically trigger rescheduling evaluation and elastic scaling operations based on preset performance indicators and event triggers; Step S5: Periodically collect historical operating data of the cluster, evaluate and adjust scheduling strategy parameters based on multi-objective optimization functions.
[0155] In step S1, the unified abstraction includes defining a structured resource descriptor for each computing node. The resource descriptor includes at least a hardware type identifier, the number and model of computing units, memory capacity, storage configuration, network interconnection topology within and between nodes, and real-time utilization indicators collected by a monitoring agent.
[0156] In step S2, the static features include at least model type, training stage, and parallel strategy; the dynamic prediction adopts a prediction model based on time series analysis, which predicts future computational load intensity, communication sensitivity, and memory usage trends based on the iteration time and resource utilization history in the early stage of task operation.
[0157] In step S3, the comprehensive weight Calculate the matching weight Memory matching weight Network affinity weight And elasticity matching weight The weighted summation is obtained, and the calculation formula is: ,in, These are adjustable non-negative weighting coefficients, and The calculation of network affinity weights depends on the global dynamic resource topology graph.
[0158] In step S3, the constraints include: the locality constraint that task units with close communication dependencies must be placed on resource units that meet the low-latency network interconnection conditions; the exclusivity constraint that any resource unit can be allocated to at most one task unit at any given time; and the architecture constraint that the task's dependency on a specific hardware architecture or software stack must be satisfied.
[0159] In step S4, the triggers include an indicator trigger based on the deviation between resource utilization and predicted values, and an event trigger based on cluster resource change events or switching of training phases within a task; the elastic scaling operation includes horizontal scaling of tasks based on elastic data parallelism and task migration based on checkpoint saving and recovery.
[0160] In step S5, the multi-objective optimization function aims to maximize the overall utility value. Its expression is ,in For cluster weighted average resource utilization, The average time to complete the task. The total power consumption of the cluster; the optimization process uses a metaheuristic algorithm to adjust the weight coefficients in step S3. Perform iterative optimization.
[0161] Specifically, for step S1: Unified abstraction and feature profiling of heterogeneous resources.
[0162] Principle Explanation: To shield the heterogeneity of underlying hardware, a unified resource abstraction layer encompassing computing, storage, and networking is constructed. It not only records static attributes (such as GPU type, video memory size, CPU core count, and RAM size), but also collects dynamic features (such as SM utilization, video memory usage, inter-GPU interconnect bandwidth utilization, and intra- / inter-node network latency and packet loss rate) through monitoring agents deployed on each node, forming a global real-time resource view.
[0163] Technical Implementation: 1. Deploy a lightweight monitoring agent on each computing node to periodically collect metrics such as GPU / DCU utilization, video memory usage, CPU / memory usage, network bandwidth and latency.
[0164] 2. Define standardized resource descriptors to describe the attributes, status, and interconnection relationships of nodes and their internal accelerator cards.
[0165] 3. The central resource manager aggregates the descriptors of all nodes, constructs and maintains a global dynamic resource topology graph, where vertices represent computing units and edges represent the network interconnection relationships and performance attributes between them.
[0166] For step S2: feature extraction and dynamic prediction for large model training task, see [link to relevant documentation]. Figure 3 , Figure 3 This is a flowchart of a task feature extraction and prediction process provided in an embodiment of this application.
[0167] Principle Explanation: Resource requirement characteristics are acquired and inferred at different stages of a task's lifecycle. These characteristics are categorized into user-declared static configurations and dynamic behavioral patterns inferred from initial operational observations.
[0168] Technical Implementation: 1. Static Feature Extraction: When a user submits a task, in addition to specifying basic resource requirements, they need to declare model features through configuration files or annotations. model_type: Model type (e.g., "Qwen2-72B", "DeepSeek-V2").
[0169] training_stage: Training stage ("pretrain", "sft", "rlhf").
[0170] parallel_strategy: Parallel strategy preference ("data parallel", "tensor parallel=4, pipeline parallel=2").
[0171] checkpoint_freq: Checkpoint frequency and expected I / O load.
[0172] 2. Dynamic Feature Learning and Prediction: In the early stages of the task (such as the first 10 iterations), its performance data (such as iteration time, the proportion of computation and communication of each accelerator card, and changes in video memory usage) are collected intensively.
[0173] The historical performance sequence and the static configuration of the task are input into a lightweight prediction module (e.g., based on LSTM or temporal convolutional network) to output predictions of the task's computational demand fluctuations, communication pressure cycles, and memory growth curves within a short-term window.
[0174] For step S3: an elastic scheduling algorithm based on maximum weight matching of bipartite graphs.
[0175] Principle Explanation: The scheduling decision is modeled as a maximum weight matching problem in a bipartite graph. The left vertex set of the bipartite graph represents the atomic task units to be scheduled (derived from the decomposition of a large task), and the right vertex set represents the currently available resource units. The weights of the connecting edges are calculated based on the results of steps S1 and S2. The goal is to find a matching scheme that maximizes the total weight while satisfying all constraints.
[0176] Technical Implementation: 1. Dynamic Task Unit Sharding: Based on the current cluster resource status and task characteristics (such as tensor parallelism requirements), a large training task is dynamically decomposed into several smaller "task units". For example, a task requiring 64 cards may initially request 16 cards to start when resources are scarce, and then elastically expand it later.
[0177] 2. Weight matrix calculation: For each (task unit) Resource Unit Yes, calculate the matching weight. The weights are derived by linearly weighting multiple sub-weights: Formula 1; in, This indicates the calculation of matching degree weights, used to measure task units. Required computing power and resource units The degree of matching between the provided computing power and the resource unit's hardware architecture and driver environment. A higher score is awarded if the computing libraries required by the task unit (such as CUDA, CANN, etc.) match the resource unit's hardware architecture and driver environment. This represents the memory matching weight, used to measure the performance of a task unit. Video memory requirements and resource units The degree of matching between available video memory and available video memory. The closer the ratio of peak video memory usage to peak available video memory for a task is to 1 (neither wasting nor overloading), the higher the score; Represents network affinity weights, used to measure the affinity of task units. Dispatch to resource unit At the same time, it determines the communication performance between itself and other units in the same task. If the communication requirements between subtasks within a task unit are high, they will be scheduled to resource units with low network latency (such as via NVLink or under the same switch); This represents the elasticity matching weight, used to measure resource units. The resource pool contains the task unit. The ability to meet future flexible and scalable demands; , , , These are adjustable non-negative weight coefficients, whose values are dynamically adjusted by the policy optimizer according to the global multi-objective optimization algorithm, and satisfy the following conditions: .
[0178] 3. Constraint Injection: Locality constraint: Tensor parallel task groups must be placed on high-bandwidth interconnect GPUs within the same node; Exclusivity constraint: A resource unit can only be allocated to one task unit; Architecture constraints: Specific tasks (such as Ascend adapted versions) must be scheduled to the corresponding architecture hardware.
[0179] Model Solving and Decision Making: The maximum weight matching of this constrained bipartite graph is solved using the improved Hungarian Algorithm or Auction Algorithm to obtain the optimal or near-optimal scheduling and placement scheme.
[0180] For step S4: runtime elastic rescheduling and resource adjustment.
[0181] Explanation of the principle: The scheduling strategy is not static. The system continuously monitors the actual running status of tasks and changes in the cluster environment, and triggers rescheduling when necessary to achieve dynamic resource optimization.
[0182] Technical Implementation: 1. Set up an elastic trigger: Metric trigger: The actual resource utilization of the monitoring task deviates significantly from the predicted value (e.g., the GPU utilization is consistently lower than the expected threshold).
[0183] Event triggers: changes in cluster size (addition or deletion of nodes), preemption of high-priority tasks, and tasks entering a specific computation phase (such as dense gradient synchronization or checkpoint saving).
[0184] 2. Triggering Rescheduling Evaluation: When the trigger is activated, the scheduler re-executes lightweight feature prediction (S2) and matching calculation (S3) for the affected task and resource subset, performing local optimization to avoid the oscillation caused by global rescheduling.
[0185] 3. Perform elastic scaling: Scaling up and down: For training frameworks that support elastic parallelism, the data parallelism can be dynamically increased or decreased through the framework API, and the underlying Pod resources can be adjusted by the elastic executor.
[0186] Migration: For tasks whose performance deteriorates due to hardware failure or resource contention, the entire task is migrated to a better resource set by combining the model checkpoint mechanism.
[0187] For step S5: Global multi-objective optimization and fault tolerance.
[0188] Principle explanation: From the perspective of the entire cluster, the overall effect of the current scheduling strategy is periodically evaluated, and the strategy parameters are automatically adjusted to seek the long-term optimal balance among multiple objectives.
[0189] Technical Implementation: 1. Construct a cluster-level objective function: Set the optimization objective to maximize overall utility. : Formula 2; in, : Represents the weighted average resource utilization of the cluster, calculated using the following formula: , This represents the total number of resource types (such as GPU, CPU, video memory, etc.). For the first Global average utilization rate of resource class Assign a weight to its importance in the optimization objective.
[0190] : Represents the average task completion time, which is the average of the job completion time (JCT) of all completed tasks within a certain time window.
[0191] : Indicates the total power consumption of the cluster during the corresponding time period.
[0192] , , A positive normalization coefficient is used to adjust the relative importance among the three optimization objectives (improving utilization, shortening task time, and reducing energy consumption), and usually satisfies... .
[0193] 2. Parameter Tuning: The strategy optimizer periodically collects historical running data and calculates the parameters under the current conditions. Value. Metaheuristic algorithms such as genetic algorithms and simulated annealing are employed to achieve the desired result. For the fitness function, in the parameter space (e.g. , , , Search for a better configuration among various trigger thresholds and push it to the scheduler application.
[0194] 3. Fault Tolerance: Monitor agent heartbeats to detect node and device failures. Once a failure is detected, the relevant resources are immediately marked as unavailable, and an emergency rescheduling (S4) of the affected tasks is triggered. Training is resumed using the most recently saved checkpoints to ensure task reliability.
[0195] In an elastic computing power scheduling system for heterogeneous, domestically-developed large-scale models: The resource abstraction and monitoring layer is used to execute the above step S1 to achieve unified management and state synchronization of heterogeneous resources; The task analyzer is used to perform step S2 above to extract and dynamically predict task features. The intelligent scheduler is used to execute the decision logic of steps S3 and S4 above, and has a built-in core scheduling algorithm. The flexible executor is used to execute the specific resource operation instructions generated in step S4 above. The policy optimizer is used to perform step S5 above to achieve periodic self-optimization of the scheduling policy.
[0196] The following specific embodiments illustrate a method and system for elastic computing power scheduling for heterogeneous, domestically-developed large-scale models: This embodiment considers a hybrid heterogeneous computing cluster containing NVIDIA H100 GPUs and Huawei Ascend 910B AI processors, which needs to simultaneously schedule a Qwen2-72B pre-training task (T1) and a DeepSeek-V2 instruction fine-tuning task (T2).
[0197] 1. Environment Deployment and Resource Abstraction (S1): Deploy monitoring agents on all nodes in the cluster and configure them with a unified resource descriptor template. After collecting information, the central resource manager forms the following topology fragment: Node #1 to Node #10: Each is equipped with 8 H100 GPUs, fully interconnected via NVLink.
[0198] Nodes #11 to #20: Each is equipped with 8 Ascend 910B GPUs, interconnected via HCCS.
[0199] The nodes are interconnected via a 200G RoCE network.
[0200] 2. Task Submission and Feature Analysis (S2): Submit task T1, declared as Qwen2-72B pre-training, using hybrid parallelism with TP=4 and PP=2: model_type:Qwen2-72B, stage: pretrain, parallel_strategy: tp=4, pp=2, dp=auto.
[0201] Task T2 is submitted, declared as DeepSeek-V2 SFT, using pure data parallelism: model_type: DeepSeek-V2, stage: sft, parallel_strategy: dp=16.
[0202] The task analyzer learned from the historical database that Qwen2 has extremely high communication overhead in the early stages of pre-training and is sensitive to NVLink bandwidth; DeepSeek-V2's memory usage increases slowly during the SFT stage, and its computational requirements are stable.
[0203] 3. Initial scheduling decision (S3): Assume there are currently 16 H100s and 16 Ascend 910Bs available.
[0204] For T1 (Qwen2 pre-trained): Due to its high communication requirements, the scheduler prioritizes placing its tensor parallel groups on the same node of the NVLink fully interconnected GPU. Through graph matching algorithm, it is found that placing the 8 task units of T1 (each unit corresponds to a process group with TP=4 and PP=1) on Node#1 and Node#2 (a total of 16 H100s) has the highest weight.
[0205] For T2 (DeepSeek-V2 SFT): This task has no special architectural requirements, but H100 is currently occupied by T1. The scheduler calculation shows that scheduling it onto the Ascend 910B cluster, although... The weight is slightly lower (potentially due to performance loss caused by software stack differences), but the overall weight is still acceptable because it meets resource requirements and avoids fragmentation. Furthermore, H100 resources are reserved for tasks that may become more reliant on the CUDA ecosystem in the future. Therefore, it was decided to schedule T2 to Nodes #11 to #14 (a total of 16 Ascend 910Bs).
[0206] 4. Runtime Flexible Adjustment (S4): After T1 had been running for 2 hours, monitoring revealed that the communication time during its gradient synchronization phase had increased from the predicted 25% to 35%, becoming a bottleneck. At the same time, four new H100 nodes (Node#21 to Node#24) were added to the cluster.
[0207] Elastic trigger activation: Communication latency deviation exceeds the threshold, and new resources are added.
[0208] Rescheduling Evaluation: The scheduler considers expanding the data parallelism dimension of T1 from the current dp=2 to dp=4 to reduce the batch size within each data parallel group, potentially lowering synchronization overhead. Fast graph matching calculations confirm that scheduling the newly added task units to the new nodes Node#21 and Node#22 is the most performance-optimal.
[0209] Perform elastic scaling: The elastic executor creates new Pods through the Kubernetes API and notifies the T1 training process through DeepSpeed's elastic API to dynamically add new data parallel workers, seamlessly scaling the training task to 24 H100 sheets.
[0210] 5. Global strategy optimization (S5): At night, the cluster enters a low-load period, and the policy optimizer analysis reveals that the current weight settings... Overemphasis This has led to some lightweight inference tasks also competing for high-end GPUs.
[0211] Optimization and Adjustment: The strategy optimizer initiates a round of simulated annealing optimization, with slight downgrades. (Calculate matching weight), adjust upwards (Flexibility matching weight).
[0212] Strategy in effect: After the new weight parameters take effect, subsequent batch inference tasks are more likely to be scheduled to the Ascend cluster with lower utilization. The fragmentation of the H100 cluster is reduced, reserving more complete resource blocks for the large-scale pre-training tasks that may come the next day, thus achieving better planning of global resources.
[0213] In this embodiment, computing resources encompassing various architectures are uniformly abstracted and monitored in real time, constructing a global dynamic resource topology graph. Static configuration features of large-scale model training tasks are extracted, and their computing power demand patterns are dynamically predicted based on initial performance. The scheduling problem is modeled as a constrained bipartite graph maximum weight matching problem, and optimal decisions are made by comprehensively calculating the weights of hardware matching degree, memory matching degree, network affinity, and elastic matching degree. During task execution, elastic rescheduling and resource adjustments are triggered based on performance deviations and cluster events. Scheduling strategy parameters are periodically adjusted based on a multi-objective optimization function. This achieves deep perception of the training characteristics of different domestic large-scale models, enabling accurate, elastic, and efficient resource scheduling in heterogeneous computing environments, improving overall cluster utilization and training task performance, and reducing overall operating costs.
[0214] Based on the methods described in any of the above embodiments, this application also provides, as follows: Figure 4 The diagram shows the structure of an electronic device. Figure 4 At the hardware level, the electronic device includes a processor, an internal bus, a network interface, memory, and non-volatile memory, and may also include other hardware required for business operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to implement the methods described in any of the above embodiments.
[0215] Based on the methods described in any of the above embodiments, this application also provides a computer storage medium storing a computer program, which, when executed by a processor, can be used to perform the methods described in any of the above embodiments.
[0216] Based on the methods described in any of the above embodiments, this application also provides a computer program product, which includes one or more computer programs or instructions. The computer program or instructions may be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. When executed by a processor, the computer program implements the methods described in any of the above embodiments.
[0217] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.
[0218] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.
[0219] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0220] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application. It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.
[0221] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
[0222] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
Claims
1. A computational power scheduling method for training large models, characterized in that, The method includes: A global resource view is constructed based on the real-time status information of heterogeneous computing resources; Obtain the task characteristics of the training task of the large model to be scheduled, and predict the resource demand trend of the training task in the preset future time window based on the performance data of the training task in the preset historical time window. Based on the global resource view and the trend of resource demand changes, a constrained bipartite graph maximum weight matching problem is constructed between task units and resource units to determine the matching relationship between the task units and the resource units. Task placement is performed based on the matching relationship.
2. The method according to claim 1, characterized in that, The global resource view is constructed using the following steps: A resource descriptor is defined for each computing node to determine the global resource view; the resource descriptor includes hardware type, computing power, storage configuration, network interconnection topology, and real-time utilization metrics.
3. The method according to claim 1, characterized in that, The resource demand change trend includes the computational load change trend, communication sensitivity change trend, and memory usage change trend of the task unit; predicting the resource demand change trend of the training task in a preset future time window based on the performance data of the training task within a preset historical time window includes: Obtain the task characteristics of the training task; the task characteristics include model type, training stage, and parallel strategy; Using a time series analysis model, based on the iteration time and resource utilization of the training task within the historical time window, the trends of computational load, communication sensitivity, and memory usage of the training task within the future time window are predicted.
4. The method according to claim 1, characterized in that, The step of constructing a constrained bipartite graph maximum weight matching problem between task units and resource units based on the global resource view and the resource demand change trend, and determining the matching relationship between the task units and the resource units, includes: The training tasks to be scheduled are divided into multiple task units, and the heterogeneous computing resources are divided into multiple resource units. Based on the global resource view and the trend of resource demand changes, calculate the matching weight between each task unit and each resource unit; The multiple task units are used as the first set of nodes in the bipartite graph, the multiple resource units are used as the second set of nodes in the bipartite graph, and the matching weight is used as the weight of the edge connecting the first set of nodes and the second set of nodes to construct the constrained maximum weight matching problem of the bipartite graph. Solve the constrained bipartite graph maximum weight matching problem to obtain the matching relationship between the task unit and the resource unit.
5. The method according to claim 4, characterized in that, The global resource view includes the computing power, video memory capacity, network performance, and elastic support capabilities of resource units; the resource demand change trends include the computing load change trends, communication sensitivity change trends, and memory usage change trends of task units. The step of calculating the matching weight between each task unit and each resource unit based on the global resource view and the resource demand change trend includes: Obtain the elastic requirements of task units; The computing matching degree is determined based on the computing load change trend and the computing capacity; The matching degree of video memory is determined based on the trend of memory usage change and the video memory capacity; Based on the communication sensitivity change trend and the network performance, network affinity is determined; Determine the elasticity matching degree based on the elasticity requirements and the elasticity support capabilities; The matching weight is obtained by weighted summing of the matching degrees of at least two dimensions among the calculated matching degree, the memory matching degree, the network affinity, and the elastic matching degree.
6. The method according to claim 4, characterized in that, The constraints of the constrained bipartite graph maximum weight matching problem are determined by the following steps: Set locality constraints to limit the placement of task units with communication dependencies on resource units that satisfy low-latency network interconnection conditions; and / or, Set an exclusivity constraint, which limits a resource unit to being allocated to at most one task unit at any given time; and / or, Set architectural constraints, which are used to limit the dependencies of tasks on hardware architecture or software stack that must be met.
7. The method according to claim 1, characterized in that, The method further includes: During the execution of the training task, adjustments are made to the training task based on the deviation between the actual resource utilization rate and the trend of resource demand changes and / or the state changes of the heterogeneous computing power cluster.
8. The method according to claim 7, characterized in that, The step of adjusting the training task based on the deviation between the actual resource utilization rate and the trend of resource demand changes and / or the state changes of the heterogeneous computing power cluster includes: When the deviation exceeds a preset threshold, and / or when the state of the heterogeneous computing power cluster changes, the current state data of the training task is obtained; Based on the current state data, the execution of the training task is resumed on the target resource unit.
9. The method according to claim 7, characterized in that, The step of adjusting the training task based on the deviation between the actual resource utilization rate and the trend of resource demand changes and / or the state changes of the heterogeneous computing power cluster includes: When the deviation exceeds a preset threshold, and / or when the state of the heterogeneous computing power cluster changes, obtain the elastic parallel configuration information of the training task; Based on the elastic parallel configuration information, the data parallelism of the training task is increased or decreased to obtain the adjusted data parallelism. Adjust the number of task units corresponding to the training task based on the adjusted data parallelism; The newly added task units are allocated to the available resource units in the heterogeneous computing power cluster, or the resource units occupied by the reduced task units are released.
10. An electronic device, characterized in that, The electronic device includes: processor; Memory used to store processor-executable instructions; Wherein, when the processor invokes the executable instructions, it implements the method according to any one of claims 1-9.
11. A computer-readable storage medium, characterized in that, It stores computer instructions that, when executed by a processor, implement the steps of any of the methods described in claims 1-9.
12. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method described in any one of claims 1-9.
