Computing power resource pool scheduling method supporting distributed training

Abstract

The invention relates to a computing power resource pool scheduling method supporting distributed training. The computing power resource pool scheduling method comprises the following steps: step 1, constructing a computing power resource pool; step 2, analyzing a distributed training task; step 3, resource allocation based on a hierarchical scheduling strategy; and 4, performing task execution and dynamic scheduling. Compared with an existing computing power resource pool scheduling method, the method has the advantages that the unified computing power resource pool is constructed, the heterogeneous computing resources are managed and monitored in a standardized mode, the characteristic requirements of distributed training tasks are combined, and the resources are intelligently distributed through a multi-level scheduling strategy. According to the method, resource load and task demand changes can be dynamically perceived, the resource utilization rate is optimized, the completion time of distributed training tasks is shortened, the stability and expansibility of the system are improved, and the method is suitable for distributed training scenes of large-scale machine learning models.

Description

Technical Field

[0001] This invention relates to the fields of distributed computing, resource scheduling and machine learning technology, and in particular to a method for scheduling computing resource pools that supports distributed training. It is suitable for distributed training scenarios of large-scale machine learning and deep learning models, and can be applied to computing-intensive application environments such as cloud computing platforms, high-performance computing centers, and artificial intelligence training platforms. Background Technology

[0002] With the rapid development of artificial intelligence technology, the scale and complexity of machine learning models are constantly increasing. The computing power of a single node can no longer meet the training needs of large-scale models, making distributed training an inevitable choice. Distributed training, by allocating training tasks to multiple computing nodes for parallel processing, can significantly shorten training time and improve model training efficiency.

[0003] However, distributed training places higher demands on the scheduling of computing resources: First, distributed training tasks typically require multiple computing nodes to work together, with frequent data interactions between nodes, which places high demands on network communication performance; second, different types of training tasks (such as computer vision and natural language processing) have different hardware resource requirements, requiring targeted resource allocation; third, computing resources are often composed of various heterogeneous computing devices (such as different models of CPUs and GPUs), resulting in high resource management complexity; and finally, the concurrent execution of large-scale training tasks can easily lead to resource contention, reducing resource utilization and task execution efficiency.

[0004] In existing technologies, resource scheduling methods are mostly designed for general computing tasks and do not fully consider the characteristics and requirements of distributed training, resulting in the following shortcomings: (1) The resource allocation strategy is simple and difficult to match the complex resource requirements of distributed training; (2) There is a lack of optimization for the communication performance between nodes, which affects the efficiency of distributed training; (3) The dynamic adjustment capability of resources is weak and it is difficult to cope with node failures or sudden load changes; (4) The resource utilization rate is low, which easily leads to resource fragmentation and waste. Therefore, there is an urgent need for a computing power resource pool scheduling method specifically for distributed training to improve resource utilization, optimize training efficiency, and enhance the stability and scalability of the system. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a computing resource pool scheduling method that supports distributed training, so as to solve the problems of low resource scheduling efficiency, low resource utilization and difficulty in adapting to dynamic changes in distributed training.

[0006] The technical solution adopted by the present invention to achieve the above objectives is as follows:

[0007] A method for scheduling computing resource pools to support distributed training includes the following steps:

[0008] 1) Collect heterogeneous computing resources, standardize and manage them in a unified manner, and build a computing resource pool that includes resource attribute information and status monitoring;

[0009] 2) Based on the distributed training tasks submitted by the user, parse task information including the task's computing power requirements, data distribution, communication mode, and priority;

[0010] 3) Based on the task parsing results, a scheduling strategy combining global and local scheduling is adopted to allocate appropriate computing resources to the task from the computing resource pool;

[0011] 4) Monitor the task execution process and resource status, and make dynamic adjustments based on real-time data to achieve load balancing and fault recovery.

[0012] Step 1) includes the following steps:

[0013] 1.1) Standardize the access of heterogeneous acceleration devices and discover and register the computing resources of the devices through a unified agent program;

[0014] 1.2) Extract hardware characteristics of various computing resources, and quantify the extracted characteristics by running standardized benchmark tests to generate unified performance metrics;

[0015] 1.3) Perform dynamic registration or deregistration, status update, performance data analysis and resource health assessment on the processed computing resources, and query and manage computing resource data based on RESTful API and graphical interface.

[0016] Step 2) includes the following steps:

[0017] 2.1) Based on the size of the training model and the amount of training data from external access, estimate the total computational load and phased computing power requirements of the task according to expert experience;

[0018] 2.2) Identify the data source, data sharding method, total data volume, and shard size for the task, and set the data localization priority level;

[0019] 2.3) Determine the communication topology, communication frequency, and data transmission volume based on the type of distributed training framework for the task;

[0020] 2.4) Determine task priority, time constraints, fault tolerance level and interruptibility based on user configuration and task type, resolve the dependencies between tasks, and construct the task execution topology.

[0021] Step 3) includes the following steps:

[0022] 3.1) Construct a resource priority scheduling algorithm based on genetic algorithm, with the goal of comprehensively optimizing task priority, resource demand matching degree, resource load balancing degree and task deadline, and select resource clusters from computing power resource pool;

[0023] 3.2) A communication-aware scheduling algorithm is adopted to perform global scheduling and allocation within the resource cluster, and task nodes with communication frequencies higher than the threshold are allocated to physical nodes with shorter network paths;

[0024] 3.3) Reserve a set proportion of redundant resources for tasks with priority higher than the threshold, and support a resource preemption mechanism;

[0025] 3.4) Based on historical scheduling data and task execution records, reinforcement learning algorithms are used to perform task-oriented local optimization scheduling. The optimization objectives are to maximize resource utilization, minimize task completion time, and meet task deadlines. The scheduling strategy is dynamically optimized by continuously learning the task-resource matching pattern.

[0026] Step 4) includes the following steps:

[0027] 4.1) Dynamically collect metrics from the computing resource pool, including task execution progress, task execution efficiency, resource utilization of each node, inter-node communication latency, and data transmission rate;

[0028] 4.2) When the difference in node resource utilization is detected to exceed the threshold and the duration exceeds the set value, the task is dynamically fragmented and the fragmentation granularity and allocation strategy are adaptively adjusted to redistribute the computing load in a task fragmentation processing mode.

[0029] 4.3) Dynamically adjust resource allocation based on task execution efficiency, increase computing resources for computationally intensive phases, and optimize network resource allocation for communication-intensive phases;

[0030] 4.4) Automatically increase or decrease the amount of allocated resources based on changes in task execution progress and resource requirements.

[0031] A computing resource pool scheduling system supporting distributed training includes:

[0032] The computing resource pool construction module is used to collect heterogeneous computing resources, standardize and manage them in a unified manner, and build a computing resource pool that includes resource attribute information and status monitoring.

[0033] The distributed training task parsing module is used to parse task information, including the task's computing power requirements, data distribution, communication mode, and priority, based on the distributed training task submitted by the user.

[0034] The resource allocation module based on the hierarchical scheduling strategy is used to allocate appropriate computing resources to tasks from the computing resource pool according to the task parsing results, using a scheduling strategy that combines global scheduling and local scheduling.

[0035] The task execution and dynamic scheduling module is used to monitor the task execution process and resource status, and make dynamic adjustments based on real-time data to achieve load balancing and fault recovery.

[0036] A computing resource pool scheduling device supporting distributed training includes a memory and a processor; the memory is used to store a computer program; the processor is used to implement the computing resource pool scheduling method supporting distributed training when executing the computer program.

[0037] A computer-readable storage medium storing a computer program, which, when executed by a processor, implements a computing resource pool scheduling method supporting distributed training.

[0038] The present invention has the following beneficial effects and advantages:

[0039] 1. This invention achieves centralized management and optimized allocation of resources by constructing a unified computing resource pool and a hierarchical scheduling strategy, thereby reducing resource fragmentation and waste and improving overall resource utilization.

[0040] 2. This invention adopts a hierarchical scheduling strategy that combines global and local approaches, taking into account both global resource balance and local execution efficiency, reducing communication overhead, and shortening task completion time.

[0041] 3. This invention supports a variety of hardware architectures, virtualization technologies, and training frameworks, and can be applied to various environments such as cloud platforms, data centers, and edge computing, exhibiting strong versatility and scalability. Attached Figure Description

[0042] Figure 1 This is a schematic diagram of a computing resource pool scheduling method supporting distributed training provided in an embodiment of the present invention. Detailed Implementation

[0043] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments.

[0044] like Figure 1 As shown, the present invention includes the following steps:

[0045] Step 1: Construct a computing resource pool. The specific method is as follows:

[0046] 1) Supports standardized access for heterogeneous acceleration devices such as x86 / ARM architecture physical servers, Kubernetes containers, GPU / TPU / NPU, etc., and realizes resource discovery and registration through a unified agent program;

[0047] 2) Extract hardware characteristics of various resources, including the number of CPU cores, clock speed, memory capacity, GPU computing power, etc., and generate unified performance metrics by running standardized benchmark tests;

[0048] 3) Implement dynamic registration / deregistration, status updates, performance data analysis, and resource health assessment for resources, providing resource query and management functions based on RESTful API and a graphical interface. Resource health assessment includes hardware failure history, performance stability indicators, and resource availability statistics, classifying resources into three levels: available, limited availability, and unavailable based on the assessment results.

[0049] Step 2: Distributed training task analysis, the specific method is as follows:

[0050] 1) Estimate the total computational load and phased computing power requirements of the task based on the model size and the amount of training data;

[0051] 2) Identify the data source, data sharding method, total data volume, and shard size of the task to determine the priority level of data localization;

[0052] 3) Analyze the type of distributed training framework for the task (such as TensorFlow, PyTorch, MXNet) and determine the communication topology, communication frequency, and data transmission volume;

[0053] 4) Determine task priority, time constraints, fault tolerance level, and interruptibility based on user configuration and task type, and parse the dependencies between tasks to construct the task execution topology. Task priority supports dynamic adjustment; the system can adjust the priority level during task execution based on changes in task progress, resource utilization efficiency, and user needs, with adjustments not exceeding two levels.

[0054] Step 3: Resource allocation based on a hierarchical scheduling strategy, specifically as follows:

[0055] 1) Construct a resource priority scheduling algorithm based on genetic algorithm, which comprehensively considers task priority, resource demand matching degree, resource load balancing degree and task deadline, and selects a suitable resource cluster from the resource pool;

[0056] 2) Within the resource cluster allocated by global scheduling, a communication-aware scheduling algorithm is used to assign frequently communicating task nodes to physical nodes with close network distance, thereby achieving proximity matching between computing tasks and data;

[0057] 3) Reserve a certain percentage (5%-20%) of redundant resources for high-priority tasks and critical phases of tasks to support resource preemption mechanisms;

[0058] 4) Based on historical scheduling data and task execution records, a reinforcement learning algorithm is used to optimize the scheduling decision model. The optimization objectives are to maximize resource utilization, minimize task completion time, and meet task deadlines. By continuously learning the task-resource matching pattern, the scheduling strategy is dynamically optimized to improve the accuracy of resource allocation and the efficiency of task execution.

[0059] Step 4: Task execution and dynamic scheduling, the specific method is as follows:

[0060] 1) Dynamically collect indicators such as task execution progress, task execution efficiency, resource utilization of each node, inter-node communication latency, and data transmission rate through a distributed monitoring system;

[0061] 2) When the difference in node resource utilization is detected to exceed the threshold and the duration exceeds the set value, the computing load is redistributed in a task fragmentation processing mode through an adaptive load-aware fragmentation rescheduling algorithm.

[0062] 3) Dynamically adjust resource allocation based on task execution efficiency, increase computing resources for computationally intensive phases, and optimize network resource allocation for communication-intensive phases.

[0063] 4) Automatically increase or decrease the amount of allocated resources based on changes in task execution progress and resource requirements, supporting flexible resource adjustments to match the needs of each stage of the task.

[0064] Figure 1 The flowchart of the computing resource pool scheduling method supporting distributed training provided by this invention mainly includes the following four steps:

[0065] 1) Construction of computing power resource pool

[0066] The system comprises a GPU cluster environment with 20 compute nodes, including different models of GPUs (such as NVIDIA A100 and L40) and CPU servers, connected via 10 Gigabit Ethernet. A resource acquisition agent is deployed on each compute node, employing an adaptive sampling mechanism: data is collected every 10 seconds when resource load is stable, increasing to once per second when load fluctuates drastically.

[0067] 2) Analysis of Distributed Training Tasks

[0068] A user submits a distributed training task based on the PyTorch framework. The task information is as follows: the task ID is T001, the required number of computing nodes is 8, each node needs to be configured with 2 NVIDIA L40 GPUs, the memory requirement of a single node is ≥32GB, the task priority is high, and the maximum tolerable latency is 24 hours.

[0069] 3) Resource allocation based on hierarchical scheduling strategy

[0070] Global resource matching: Based on resource requirement parameters, nodes with 2 or more NVIDIA L40 GPUs and ≥64GB of memory are selected from 20 computing nodes to obtain 10 candidate nodes; a greedy algorithm is used to select 8 low-load nodes with CPU utilization <30% and GPU utilization <20% from the candidate nodes; through network distance matrix analysis, 8 nodes in the same subnet are selected, whose average communication latency is 1.2ms, which is lower than the preset threshold (2ms) and meets the communication requirements.

[0071] Local resource optimization: The selected 8 nodes are defragmented, unnecessary background processes are closed, and about 5GB of memory resources are freed up; Since the task has a high priority, the 4 nodes with better performance (such as higher GPU core frequency) are assigned as master nodes; According to the task characteristics, the nodes are set to high-performance mode to ensure training efficiency.

[0072] 4) Task execution and dynamic scheduling

[0073] Local resource optimization: The selected 8 nodes are defragmented, unnecessary background processes are closed, and approximately 5GB of memory resources are freed up. Due to the high task priority, the 4 nodes with better performance (e.g., higher GPU core frequency) are assigned as master nodes. PyTorch version 1.12 and the corresponding CUDA driver are pre-installed on these 8 nodes. Based on the task characteristics, the nodes are set to high-performance mode to ensure training efficiency.

[0074] During training, the monitoring system collects node status at an adaptive frequency. In the third hour of task execution, predictive scheduling analysis revealed a significant decline in the GPU performance of node Node005. Its task process was proactively migrated to the backup node Node020, preventing task interruption due to potential failures. The migration process took approximately one minute.

[0075] By the 6th hour of task execution, the CPU utilization of the 8 nodes varied significantly (highest 75%, lowest 20%), exceeding the preset threshold (30%). A multi-dimensional load balancing strategy was employed, and analysis revealed that this was due to uneven distribution of data preprocessing load. Adjusting the data sharding strategy, distributing some data preprocessing tasks from high-load nodes to low-load nodes, reduced the CPU utilization difference between nodes to within 20%.

[0076] The above implementation cases demonstrate that the computing resource pool scheduling method supporting distributed training has significant advantages in application running efficiency and node resource utilization.

Claims

1. A method for scheduling computing resource pools to support distributed training, characterized in that, Includes the following steps: 1) Collect heterogeneous computing resources, standardize and manage them in a unified manner, and build a computing resource pool that includes resource attribute information and status monitoring; 2) Based on the distributed training tasks submitted by the user, parse task information including the task's computing power requirements, data distribution, communication mode, and priority; 3) Based on the task parsing results, a scheduling strategy combining global and local scheduling is adopted to allocate appropriate computing resources to the task from the computing resource pool; 4) Monitor the task execution process and resource status, and make dynamic adjustments based on real-time data to achieve load balancing and fault recovery.

2. The computing resource pool scheduling method supporting distributed training according to claim 1, characterized in that, Step 1) includes the following steps: 1.1) Standardize the access of heterogeneous acceleration devices and discover and register the computing resources of the devices through a unified agent program; 1.2) Extract hardware characteristics of various computing resources, and quantify the extracted characteristics by running standardized benchmark tests to generate unified performance metrics; 1.3) Perform dynamic registration or deregistration, status update, performance data analysis and resource health assessment on the processed computing resources, and query and manage computing resource data based on RESTful API and graphical interface.

3. The method for scheduling computing resource pools to support distributed training according to claim 1, characterized in that, Step 2) includes the following steps: 2.1) Based on the size of the training model and the amount of training data from external access, estimate the total computational load and phased computing power requirements of the task according to expert experience; 2.2) Identify the data source, data sharding method, total data volume, and shard size for the task, and set the data localization priority level; 2.3) Determine the communication topology, communication frequency, and data transmission volume based on the distributed training framework type of the task; 2.4) Determine task priority, time constraints, fault tolerance level and interruptibility based on user configuration and task type, resolve the dependencies between tasks, and construct the task execution topology.

4. The computing resource pool scheduling method supporting distributed training according to claim 1, characterized in that, Step 3) includes the following steps: 3.1) Construct a resource priority scheduling algorithm based on genetic algorithm, with the goal of comprehensively optimizing task priority, resource demand matching degree, resource load balancing degree and task deadline, and select resource clusters from computing power resource pool; 3.2) A communication-aware scheduling algorithm is adopted to perform global scheduling and allocation within the resource cluster, and task nodes with communication frequencies higher than the threshold are allocated to physical nodes with shorter network paths; 3.3) Reserve a set proportion of redundant resources for tasks with priority higher than the threshold, and support a resource preemption mechanism; 3.4) Based on historical scheduling data and task execution records, reinforcement learning algorithms are used to perform task-oriented local optimization scheduling. The optimization objectives are to maximize resource utilization, minimize task completion time, and meet task deadlines. The scheduling strategy is dynamically optimized by continuously learning the task-resource matching pattern.

5. The method for scheduling computing resource pools supporting distributed training according to claim 1, characterized in that, Step 4) includes the following steps: 4.1) Dynamically collect metrics from the computing resource pool, including task execution progress, task execution efficiency, resource utilization of each node, inter-node communication latency, and data transmission rate; 4.2) When the difference in node resource utilization is detected to exceed the threshold and the duration exceeds the set value, the task is dynamically fragmented and the fragmentation granularity and allocation strategy are adaptively adjusted to redistribute the computing load in a task fragmentation processing mode. 4.3) Dynamically adjust resource allocation based on task execution efficiency, increase computing resources for computationally intensive phases, and optimize network resource allocation for communication-intensive phases; 4.4) Automatically increase or decrease the amount of allocated resources based on changes in task execution progress and resource requirements.

6. A computing resource pool scheduling system supporting distributed training, characterized in that, include: The computing resource pool construction module is used to collect heterogeneous computing resources, standardize and manage them in a unified manner, and build a computing resource pool that includes resource attribute information and status monitoring. The distributed training task parsing module is used to parse task information, including the task's computing power requirements, data distribution, communication mode, and priority, based on the distributed training task submitted by the user. The resource allocation module based on the hierarchical scheduling strategy is used to allocate appropriate computing resources to tasks from the computing resource pool according to the task parsing results, using a scheduling strategy that combines global scheduling and local scheduling. The task execution and dynamic scheduling module is used to monitor the task execution process and resource status, and make dynamic adjustments based on real-time data to achieve load balancing and fault recovery.

7. A computing resource pool scheduling device supporting distributed training, characterized in that, It includes a memory and a processor; the memory is used to store a computer program; the processor is used to implement, when executing the computer program, a computing resource pool scheduling method for supporting distributed training as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements a computing resource pool scheduling method for supporting distributed training as described in any one of claims 1-5.

Images