Large-model heterogeneous cluster scheduling system and method based on adaptive parallel co-optimization
By using an adaptive parallel co-optimization large-model heterogeneous cluster scheduling system, resource allocation and parallel strategies are dynamically adjusted, solving the problem of resource waste in training large models in heterogeneous clusters and achieving high training efficiency and reduced training time.
Patent Information
- Application Number
- CN202410946115.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-15
- Publication Date
- 2024-11-08
Smart Images

Figure CN118916156A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a technology in a certain field, specifically a large-model heterogeneous cluster scheduling system and method based on adaptive parallel co-optimization. Background Art
[0002] With the rapid development of deep learning, large models have shown excellent performance in natural language processing, computer vision, recommendation systems and other fields. Large models refer to machine learning models with large parameters and high computational complexity. These models usually have hundreds of millions of parameters and require a large amount of data and computing resources for training. Due to their powerful representation learning capabilities, large models such as BERT and GPT have achieved outstanding results in various tasks in recent years and have become a research hotspot in the field of artificial intelligence. However, the training of large models requires a lot of time and computing resources, so how to efficiently train and deploy large models has become an important issue in current research and applications. Since training large models requires a lot of parallel resources, existing large-scale clusters continue to integrate new-generation GPUs to meet their computing needs. However, this also leads to heterogeneity in terms of GPU types, GPU interconnection topologies, etc. In order to deal with this heterogeneity, adaptive parallelism automatically explores the best parallel strategy for training tasks on allocated resources by combining data, tensor and pipeline parallelism. Through exhaustive search, adaptive parallelism can always maximize the utilization of allocated resources. Summary of the invention
[0003] In view of the above-mentioned deficiencies in the prior art, the present invention proposes a large model heterogeneous cluster scheduling system and method based on adaptive parallel co-optimization, which ensures efficient performance analysis of multiple large model training tasks, improves cluster-level throughput performance and resource utilization, and reduces task completion time at the task level.
[0004] The present invention is achieved through the following technical solutions:
[0005] The present invention relates to a large-model heterogeneous cluster scheduling system based on adaptive parallel co-optimization, comprising: a performance analyzer based on execution flow modeling, a cluster elastic preemptive task scheduler and a task parallel execution engine, wherein: the cluster elastic preemptive task scheduler dynamically changes the computing resources of tasks according to task queues to obtain approximately optimal task resource allocation under different cluster task loads; the performance analyzer measures the parallel block operator delay on a single device based on model parallel blocks from the cluster elastic preemptive task scheduler and hardware information from a heterogeneous GPU cluster, and estimates the iteration time of the optimal parallel strategy in the subspace defined by each parallel block; the task parallel execution engine receives the scheduled parallel blocks, performs parallel search based on the pruned search space, generates tasks and their optimal parallel strategies, and then outputs them to the heterogeneous GPU cluster.
[0006] The parallel block refers to a subspace of the scheduling space bound to the deep learning task, and each parallel block corresponds to a different number of pipeline parallel stages and resource allocation.
[0007] The resources include: GPU type, number and connection topology.
[0008] The operators include but are not limited to matrix multiplication calculations.
[0009] The present invention relates to a large model heterogeneous cluster scheduling method based on the above system, which performs equivalent compilation on the model to be processed to analyze its operator composition, measures the performance of each operator on a sample device and estimates the end-to-end iteration time, performs sampling in multiple parallel blocks, generates performance evaluation of the parallel blocks, performs task elastic preemptive scheduling and allocates resources based on the performance evaluation, performs pruning-based optimal parallel strategy search based on hardware resources and optimal parallel blocks, and trains the model to be processed according to the obtained approximately optimal parallel strategy. Technical Effects
[0010] The present invention performs latency measurement on a parallel block operator on a single device and adopts a cluster elastic preemptive task scheduling operation that takes into account adaptive parallelism. Compared with the prior art, the present invention performs performance modeling and analysis of large-scale distributed training of large models and multiple devices on a single device; in GPU cluster scheduling, scheduling decisions are made based on the performance data obtained by the model under adaptive parallelism. BRIEF DESCRIPTION OF THE DRAWINGS
[0011] Figure 1 It is a schematic diagram of the system of the present invention;
[0012] Figure 2 The following is a schematic diagram of the definition of parallel blocks and the division of the scheduling space;
[0013] Figure 3 Execution flow diagram for performance analysis based on execution flow modeling;
[0014] Figure 4 A schematic diagram for batch acquisition of performance data based on a small number of measurements;
[0015] Figure 5 Schematic diagram of the mapping of hardware resources to pipeline stages to minimize communication;
[0016] Figure 6 This is a schematic diagram of the elastic preemptive scheduling strategy;
[0017] Figure 7 Schematic diagram of search subspace pruning based on performance analysis;
[0018] Figure 8 It is a flow chart of the present invention. DETAILED DESCRIPTION
[0019] like Figure 1 As shown, this embodiment involves a large-model heterogeneous cluster scheduling system based on scheduling-adaptive parallel co-optimization, including: a performance analyzer based on execution flow modeling, a cluster elastic preemptive task scheduler and a task parallel execution engine, wherein: the cluster elastic preemptive task scheduler dynamically changes the computing resources of the task according to the task queue, and obtains the approximately optimal task resource allocation under different cluster task loads; the performance analyzer measures the parallel block operator delay on a single device based on the parallel blocks from the cluster elastic preemptive task scheduler and the hardware information from the heterogeneous GPU cluster, and estimates the iteration time of the optimal parallel strategy in the subspace defined by each parallel block; the task parallel execution engine receives the scheduled parallel blocks and performs parallel search based on the pruned search space, generates tasks and their optimal parallel strategies, and then outputs them to the heterogeneous GPU cluster.
[0020] The performance analyzer includes: a communication operator delay estimation unit, a computing operator delay measurement unit and a pipeline end-to-end time modeling unit, wherein: the communication operator delay estimation unit obtains the delay estimation of each communication operator based on the communication volume difference according to the target model to be measured, the communication operator composition of the device after parallel division and compilation, and the communication delay data measured offline; the computing operator delay measurement unit performs delay measurement on a single device according to the computing operator composition of the target model to be measured after parallel division and compilation, and obtains the delay data of each computing operator; the pipeline end-to-end time modeling unit performs modeling based on the execution flow of the pipeline according to the delay estimation and delay measurement, and obtains the end-to-end time performance estimation under the execution of the pipeline.
[0021] The cluster elastic preemptive task scheduler includes: a task queue unit, a heuristic parallel block allocation optimization unit and a chance-retaining execution unit, wherein: the task queue unit queues each task in a first-come-first-served manner according to the user's task submission order for multiple parallel blocks of multiple tasks; the heuristic parallel block allocation optimization unit performs heuristic multi-task resource scheduling according to the task information data in the task queue and the current cluster load level to obtain a resource allocation decision with better cluster throughput; the chance-retaining execution unit performs opportunistic temporary preemptive execution of tasks according to the task queuing status in the task queue and the status of cluster resources to obtain a task execution decision with better cluster throughput.
[0022] The heuristic multi-task resource scheduling refers to: heuristically selecting tasks for upward expansion when the load is low, increasing the number of allocated GPUs or switching GPU types, and performing the opposite downward expansion when the load is high.
[0023] The opportunistic task temporary preemptive execution means that when the execution resources of the large task at the front of the queue are insufficient, the small task at the back temporarily uses part of the resources reserved for the large task for training.
[0024] The task parallel execution engine includes: a parallel space pruning search unit and a training task execution unit, wherein: the parallel search space pruning unit performs parallel search pruning in the subspace defined by the parallel block for the parallel block allocated by the cluster elastic preemptive task scheduler according to the task performance data obtained by the performance analyzer module, and obtains the optimal parallel strategy after searching in the pruned parallel subspace; the training task execution unit deploys the target model on the hardware corresponding to the parallel block based on the optimal parallel strategy obtained by the parallel space pruning search unit, and obtains an executable training task consisting of the model, hardware and parallel method.
[0025] like Figure 2 As shown, the parallel block refers to: a subspace of the scheduling space bound to the deep learning task, each parallel block corresponds to a different number of pipeline parallel stages and resource allocation; when a user submits a task, the performance analyzer will first perform equivalent compilation and latency measurement on the model corresponding to the task on each single GPU of all available GPU types in the heterogeneous GPU cluster on different parallel blocks, perform performance analysis based on end-to-end iteration time modeling based on the execution flow, and perform heuristic elastic preemptive rescheduling of the task and the previously submitted tasks based on the analysis results, so as to optimize the overall performance of the cluster while ensuring that the task will not starve to death. When the submitted task is actually allocated to the hardware resources, the execution engine will search for parallel strategies based on pruning in the divided parallel subspace based on the parallel blocks corresponding to these resources, and start executing the task with the optimal parallel strategy found.
[0026] The equivalent compilation means that the performance analyzer performs equivalent compilation equivalent to that of multiple GPUs on a single GPU and obtains optimized (eg operator fusion) computing operators and communication operators.
[0027] The delay measurement refers to: estimating the delay of each communication operator by adding the pre-measured communication operator delay and the interpolation based on the communication volume, measuring the delay of each calculation operator in real time on a single card, and then combining them in the form of a pipeline to obtain the end-to-end iteration time of the entire model on a parallel block (resources + number of pipeline stages).
[0028] like Figure 3As shown in the figure, it is the execution flow of the performance analyzer. Given a specific resource allocation and parallel mode, (a) in the figure is a data parallel model, which is a mixture of tensor parallelism and pipeline parallelism. Every two blocks are a pipeline stage, with a total of four stages). After equivalent compilation, the performance analyzer splits the model into two operator subsets: computing operators and communication operators. For the computing operators shown in (b) in the figure, the delay measurement is performed on a single device with the pipeline stage as the granularity, and then the delay of each computing operator is obtained. For the communication operator in (c) in the figure, the communication volume of each communication operator is obtained, and then combined with the pre-measured delay of operators of the same type, interpolation based on the communication volume is performed to obtain the delay estimate of each communication operator. Finally, the performance analyzer performs end-to-end model iteration delay modeling based on the pipeline strategy used, as shown in (d) in the figure, providing performance data for subsequent task scheduling as a scheduling basis.
[0029] like Figure 4 As shown, the delay measurement preferably divides the parallel block specified model into 4 pipeline stages and performs 2 measurements. Then, a total of 2 4 = 16 parallel methods. Therefore, through two measurements, the performance analyzer can estimate the performance of the 16 parallel strategies in the parallel block, so that the performance analyzer can efficiently obtain performance data in batches through a small number of measurements on a single GPU.
[0030] like Figure 5 As shown, the optimal parallel strategy preferably determines in which hardware subset each model pipeline stage after parallelization should be placed based on the mapping relationship between hardware and parallel stages. Specifically, given the hardware resources and the number of pipeline stages, the analyzer aggregates the model operators into pipeline stages based on minimizing the amount of communication between stages, and maps them to a certain amount of resources based on the principle of balanced operator computation.
[0031] For example, operator 1 has a large computational load, so it is assigned 0.5 shares of the GPU, and since the communication volume between operators 2 and 3 is small, operators 1 and 2 are divided into the same pipeline stage. Then, the analyzer applies the aforementioned performance measurement method to obtain a large amount of performance data.
[0032] like Figure 7 As shown in Figure 1, stage 0 tends to be data parallel, while stage 1 tends to be tensor parallel. Therefore, when the execution engine searches for the optimal parallel strategy in the subspace, it only searches for the part from full data parallelism to half data parallelism and half tensor parallelism for stage 0, while stage 1 only searches for the part from half data parallelism and half tensor parallelism to full tensor parallelism.
[0033] The dynamically changing computing resources of the task refers to: for the training task, the dynamic switching between different parallel blocks, that is, the optimization problem of multi-task parallel block allocation.
[0034] The optimization problem of multi-task parallel block allocation is solved by a low-overhead heuristic algorithm.
[0035] The amount of resources for training tasks can be elastically expanded, including dynamic changes in the number of GPUs and switching of GPU types. Based on the characteristics of elastic expansion, the scheduler allows preemptive scheduling of tasks and adopts an opportunity retention mechanism to avoid task starvation.
[0036] like Figure 6 As shown in (a), the first-come-first-served means that when the cluster load increases (new tasks arrive), the newly arrived tasks cannot be directly allocated enough resources. Therefore, the server 1 of a running task is expanded downward, that is, the number of GPUs is changed from 2 to 1, and another task is expanded downward, that is, the GPU type is changed from B to A with weaker computing power and migrated to server 1, and then a new task that requires 2 GPUs is preemptively scheduled to server 2. For tasks that require 4 GPUs, the scheduler queues it into the pause queue and opportunistically reserves two GPUs for it (actually being used by tasks with lower priority), and suspends the low-priority tasks when the other two GPUs are idle, and resumes the operation of the 4-GPU task. In this way, the task scheduler can not only serve the training tasks on a first-come-first-served basis by setting priorities to avoid starvation, but also opportunistically run low-priority tasks to improve the overall performance of the cluster.
[0037] like Figure 6 As shown in (c), when the cluster load decreases (some tasks end), the scheduler will first check whether there are currently queued tasks that can be restarted. If so, it will try to restart them according to the priority (treating them as newly arrived tasks for scheduling). Since 4 GPUs are required, the queued tasks cannot be restarted temporarily. Subsequently, the scheduler will check whether the currently running tasks can be scaled up, increasing the number of GPUs or switching to a GPU type with better computing power to improve task performance. In the figure, the tasks on server 1 are expanded from 1-GPU to 2-GPU.
[0038] like Figure 8As shown in the figure, after specific practical experiments, in a heterogeneous cluster of 16 A40 servers (32 A40 GPUs) and 16 A10 servers (32 A10 GPUs), using various industrial-level task load data sets such as Microsoft as input, the performance analyzer based on execution flow modeling is called for the model corresponding to the submitted task to measure the performance data considering adaptive parallelism; according to the task load submitted by the current user, the cluster elastic preemptive task scheduler is called in the heterogeneous cluster to dynamically schedule the resources of the task in real time; after allocating resources to each task, the task parallel execution engine is called on the allocated hardware to search for the optimal parallel strategy, as well as the final training task deployment, and the task throughput is monitored in real time. In the end, the average task completion time of the cluster is reduced by up to 48.9%, the cluster task throughput is increased by up to 1.49 times, and the average queuing time of the task is reduced by up to 71.0%.
[0039] Compared with the prior art, the present invention performs efficient task performance analysis by calling a performance analyzer based on execution flow modeling, and calls a cluster elastic preemptive task scheduler to perform dynamic resource scheduling of tasks in real time.
[0040] The above-mentioned specific implementation can be partially adjusted in different ways by those skilled in the art without departing from the principle and purpose of the present invention. The protection scope of the present invention shall be based on the claims and shall not be limited by the above-mentioned specific implementation. Each implementation scheme within its scope shall be subject to the constraints of the present invention.
Claims
1. A large-model heterogeneous cluster scheduling system based on adaptive parallel co-optimization, characterized in that: include: A performance analyzer based on execution flow modeling, a cluster elastic preemptive task scheduler, and a task parallel execution engine, where: the cluster elastic preemptive task scheduler dynamically changes the computing resources of the task according to the task queue to obtain the approximately optimal task resource allocation under different cluster task loads; the performance analyzer measures the parallel block operator delay on a single device based on the model parallel blocks from the cluster elastic preemptive task scheduler and the hardware information from the heterogeneous GPU cluster, and estimates the iteration time of the optimal parallel strategy in the subspace defined by each parallel block; The task parallel execution engine receives the scheduled parallel blocks, performs parallel search based on the pruned search space, generates tasks and their optimal parallel strategies, and then outputs them to the heterogeneous GPU cluster; The parallel block refers to: a subspace of the scheduling space bound to the deep learning task, each parallel block corresponds to a different number of pipeline parallel stages and resource allocation; The dynamically changing computing resources of the tasks refers to: for training tasks, dynamically switching between different parallel blocks, and solving the optimization problem of multi-task parallel block allocation through a low-overhead heuristic algorithm.
2. The large-model heterogeneous cluster scheduling system based on adaptive parallel co-optimization according to claim 1 is characterized in that: The performance analyzer includes: a communication operator delay estimation unit, a computing operator delay measurement unit and a pipeline end-to-end time modeling unit, wherein: the communication operator delay estimation unit obtains the delay estimation of each communication operator based on the communication volume difference according to the target model to be measured, the communication operator composition of the device after parallel division and compilation, and the communication delay data measured offline; the computing operator delay measurement unit performs delay measurement on a single device according to the computing operator composition of the target model to be measured after parallel division and compilation, and obtains the delay data of each computing operator; the pipeline end-to-end time modeling unit performs modeling based on the execution flow of the pipeline according to the delay estimation and delay measurement, and obtains the end-to-end time performance estimation under the execution of the pipeline.
3. The large-model heterogeneous cluster scheduling system based on adaptive parallel co-optimization according to claim 1 is characterized in that: The cluster elastic preemptive task scheduler includes: a task queue unit, a heuristic parallel block allocation optimization unit and a chance-retaining execution unit, wherein: the task queue unit queues each task in a first-come-first-served manner according to the user's task submission order for multiple parallel blocks of multiple tasks; the heuristic parallel block allocation optimization unit performs heuristic multi-task resource scheduling according to the task information data in the task queue and the current cluster load level to obtain a resource allocation decision with better cluster throughput; the chance-retaining execution unit performs opportunistic temporary preemptive execution of tasks according to the task queuing status in the task queue and the status of cluster resources to obtain a task execution decision with better cluster throughput.
4. The large-model heterogeneous cluster scheduling system based on adaptive parallel co-optimization according to claim 3 is characterized in that: The heuristic multi-task resource scheduling refers to: heuristically selecting tasks for upward expansion when the load is low, increasing the number of allocated GPUs or switching GPU types, and performing the opposite downward expansion when the load is high.
5. The large-model heterogeneous cluster scheduling system based on adaptive parallel co-optimization according to claim 3 is characterized in that: The opportunistic task temporary preemptive execution means that when the execution resources of the large task at the front of the queue are insufficient, the small task at the back temporarily uses part of the resources reserved for the large task for training.
6. The large-model heterogeneous cluster scheduling system based on adaptive parallel co-optimization according to claim 1 is characterized in that: The task parallel execution engine includes: a parallel space pruning search unit and a training task execution unit, wherein: the parallel search space pruning unit performs parallel search pruning in the subspace defined by the parallel block for the parallel block allocated by the cluster elastic preemptive task scheduler according to the task performance data obtained by the performance analyzer module, and obtains the optimal parallel strategy after searching in the pruned parallel subspace; the training task execution unit deploys the target model on the hardware corresponding to the parallel block based on the optimal parallel strategy obtained by the parallel space pruning search unit, and obtains an executable training task consisting of the model, hardware and parallel method.
7. The large-model heterogeneous cluster scheduling system based on adaptive parallel co-optimization according to claim 1 or 2 is characterized in that: The parallel block refers to: a subspace of the scheduling space bound to the deep learning task, each parallel block corresponds to a different number of pipeline parallel stages and resource allocation; when a user submits a task, the performance analyzer will first perform equivalent compilation and latency measurement on the model corresponding to the task on each single GPU of all available GPU types in the heterogeneous GPU cluster on different parallel blocks, perform performance analysis based on end-to-end iteration time modeling based on the execution flow, and perform heuristic elastic preemptive rescheduling of the task and previously submitted tasks based on the analysis results to optimize the overall performance of the cluster while ensuring that the task will not starve to death. When the submitted task is actually allocated to the hardware resources, the execution engine will search for parallel strategies based on pruning in the divided parallel subspace based on the parallel blocks corresponding to these resources, and start executing the task with the optimal parallel strategy found; The equivalent compilation means that the performance analyzer performs equivalent compilation equivalent to multiple GPUs on a single GPU and obtains optimized computing operators and communication operators; The delay measurement refers to: estimating the delay of each communication operator by adding the pre-measured communication operator delay and the interpolation based on the communication volume, measuring the delay of each calculation operator in real time on a single card, and then combining them in the form of a pipeline to obtain the end-to-end iteration time of the entire model on a parallel block.
8. The large-model heterogeneous cluster scheduling system based on adaptive parallel co-optimization according to claim 1 or 6, characterized in that: The optimal parallel strategy determines which hardware subset each model pipeline stage after parallelization should be placed in based on the mapping relationship between hardware and parallel stages. Specifically, given the hardware resources and the number of pipeline stages, the analyzer aggregates the model operators into pipeline stages based on minimizing the amount of communication between stages, and maps them to the amount of resources based on the principle of balanced operator computation.
9. A large model heterogeneous cluster scheduling method based on the system described in any one of claims 1 to 8, characterized in that: The model to be processed is equivalently compiled to analyze its operator composition, the performance of each operator is measured on the sample device and the end-to-end iteration time is estimated, sampling is performed in multiple parallel blocks, and performance evaluation of the parallel blocks is generated. Task elastic preemptive scheduling and resource allocation are performed based on the performance evaluation, and the optimal parallel strategy based on pruning is searched based on hardware resources and the optimal parallel block. The model to be processed is trained according to the obtained approximate optimal parallel strategy.
Citation Information
Cited By
Heterogeneous cluster parallel training method, device and system for large model
CN119719949A
Large model computing cluster service reliability evaluation and optimization method and device
CN119806973A
Large model adaptive parallel training method capable of being used for heterogeneous cluster
CN119938327A
Performance tuning method of cluster operating system, electronic equipment and storage medium
CN120276865A
Automatic tuning method and device for distributed parallel strategy, medium and program product
CN120449991A
