Large model parallel task scheduling method based on space-time two-dimensional segmentation and intelligent sharing

Abstract

The invention discloses a large model parallel task scheduling method based on space-time two-dimensional segmentation and intelligent sharing, and particularly provides a model segmentation strategy based on space-time two dimensions, and based on significant differences of training and reasoning tasks in calculation properties, targeted optimization strategies are adopted respectively, so that the task execution efficiency is improved; the invention further provides a model placement strategy based on the heterogeneous resources and the interference model, the defects that a traditional method lacks intelligent sharing judgment and cannot be dynamically adjusted are overcome, the flexibility and adaptability of scheduling are improved, and therefore the maximum benefit of the shared resources is guaranteed. The method solves the problems that when an existing heterogeneous intelligent computing cluster scheduling method is used for processing large model training and reasoning tasks, the model placement optimization technology of model segmentation and heterogeneous resource adaptation is not fully utilized, intelligent sharing scheduling of computing resources in a cluster is lacked, system task scheduling is unreasonable, and the scheduling efficiency is high. And the utilization rate of computing resources is low.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology and relates to a large model parallel task scheduling method based on two-dimensional space-time segmentation and intelligent sharing. Background Art

[0002] With the rapid development of artificial intelligence technology, large language models (LLM) have been widely used in various fields. At the same time, the scale of model parameters and training data has also increased exponentially, which has made the training and reasoning process more complex and time-consuming, posing a severe challenge to computing resources, making tasks such as large model training and reasoning need to rely on large-scale computing power clusters. Therefore, more and more data centers are building heterogeneous intelligent computing clusters including multiple accelerators such as GPUs and XPUs to provide efficient large model training and reasoning services.

[0003] For multi-tenant computing clusters, the system will continuously receive different types of LLM tasks submitted by each tenant. The task scheduler, as the core component of the cluster, is responsible for automatically adapting appropriate resources and configurations according to task requirements to ensure that tasks can be executed smoothly. The task scheduler plays a key role in coordinating task execution in the cluster, and is responsible for effectively managing and scheduling various computing resources to ensure that tasks submitted by tenants can be completed efficiently. If task scheduling is inefficient, it will not only affect the tenant's user experience, but may also lead to resource contention, thereby significantly reducing the training speed and increasing the cluster's energy consumption and operating costs. Therefore, in this system, flexible resource allocation and efficient task scheduling are crucial to maximizing resource utilization and improving overall performance.

[0004] The existing heterogeneous intelligent computing cluster task scheduling methods mainly include: preemptive and exclusive scheduling methods and sharing-based non-preemptive scheduling methods.

[0005] Preemptive scheduling: When a task is occupying computing resources, other high-priority tasks can preempt these resources and interrupt the executing task. This scheduling method can ensure that high-priority tasks are processed in a timely manner, thereby improving the system's response speed and the dynamic scheduling ability of resources. It is usually suitable for scenarios with obvious differences in task priorities and strict response time requirements. However, this method has the following disadvantages:

[0006] 1) High switching overhead: Preemptive scheduling requires the system to frequently switch tasks, and each switch requires saving and restoring the context, which increases the complexity and overhead of scheduling and management.

[0007] 2) Resource fragmentation: Preemptive scheduling may lead to discontinuous resource allocation, and idleness and over-allocation of computing resources are inevitable, resulting in inefficient utilization of resources.

[0008] 3) Unstable task delay: Another disadvantage of preemptive scheduling is that it affects the delay of low-priority tasks, especially when the system load is high. Low-priority tasks may be frequently interrupted by high-priority tasks, making the completion time of tasks unpredictable.

[0009] Exclusive scheduling: refers to requiring tasks to exclusively use the allocated computing resources during execution until the task is completed. This method can avoid resource competition between tasks and ensure the stability and execution efficiency of tasks. Exclusive scheduling is suitable for computing-intensive tasks, especially when training large models. Exclusive resources can help improve performance and ensure efficient completion of tasks. However, this method has the following disadvantages:

[0010] 1) Resource waste: In exclusive scheduling, since each task needs to completely monopolize resources when executing, some resources may be idle when the task is not completed, resulting in a decrease in the overall resource utilization efficiency.

[0011] 2) Load imbalance: Exclusive scheduling usually allocates fixed resources to each task without considering the differences in resource requirements of different tasks, which may lead to over-allocation of resources on some nodes. When multiple tasks request computing resources, exclusive scheduling may cause some computing nodes to be overloaded while other computing nodes are idle, resulting in uneven resource allocation, affecting the load balance and overall performance of the cluster.

[0012] Non-preemptive shared scheduling: refers to multiple tasks sharing the same computing resources, but the resources will not be preempted or interrupted due to the arrival of new tasks. Each task is scheduled according to a fixed time slice or priority, and the task is not interfered with by other tasks during execution. This scheduling method is suitable for scenarios with light loads or less dependencies between tasks, where sharing resources between tasks is less likely to cause conflicts or performance bottlenecks. For example, some inference tasks or low-priority training tasks can use this type of scheduling to avoid frequent context switching and resource preemption, and improve the overall throughput and resource utilization of the system. However, this method has the following disadvantages:

[0013] 1) Performance degradation: Since multiple tasks share the same computing resources, there will be competition between tasks, which may lead to performance degradation, especially when the task has a large demand for resources, resource contention will become more serious;

[0014] 2) Inflexible task scheduling: Due to the use of fixed time slices or sequential execution strategies and the lack of intelligent sharing judgment, shared resources cannot be adjusted dynamically. When a task occupies too many resources, it cannot be immediately preempted or rescheduled, making the system scheduling relatively rigid and unable to flexibly adapt to the needs of different tasks, thereby ensuring the maximum benefit of shared resources.

[0015] 3) Lack of heterogeneous resource awareness: Most existing methods are based on homogeneous clusters (the underlying devices are all GPUs of the same type), lack optimization for heterogeneous scenarios, and fail to combine model segmentation technology to optimize scheduling strategies in large model tasks. Summary of the invention

[0016] In view of the technical problems existing in the background technology, the purpose of the present invention is to provide a large model parallel task scheduling method based on two-dimensional space-time segmentation and intelligent sharing. This method solves the problems that the existing scheduling methods of heterogeneous intelligent computing clusters fail to fully utilize the model placement optimization technology of model segmentation and heterogeneous resource adaptation when processing large model training and reasoning tasks, lack intelligent sharing scheduling of computing resources in the cluster, resulting in unreasonable system task scheduling and low computing resource utilization.

[0017] In order to achieve the above object, the present invention adopts the following technical solutions:

[0018] The large model parallel task scheduling method based on time-space two-dimensional segmentation and intelligent sharing includes the following steps:

[0019] S1. Accept large model tasks submitted by users;

[0020] S2, putting the large model task into the task priority queue and waiting for scheduling by the system scheduler; the scheduler includes a task segmentation module, a task combination module and a task placement module;

[0021] S3, the task segmentation module uses a two-dimensional spatiotemporal model segmentation strategy to segment the large model task into several subtasks according to the task type and characteristics;

[0022] S4, the task combination module combines the subtasks according to the characteristics of the subtasks and computing resources;

[0023] S5, the task placement module generates a placement strategy and sends it to the instruction executor;

[0024] S6. Placement strategy: Based on the task type, a search algorithm is used to search and select the corresponding computing resources from the cluster. During the search process, the linear programming / dynamic programming algorithm is used to optimize resource allocation with the goal of minimizing transmission overhead and idle resources.

[0025] S7. Allocate computing resources to the task and place the task, then execute it.

[0026] Preferably, in step S3, the task segmentation module adopts a spatiotemporal two-dimensional model segmentation strategy to segment the large model task into several subtasks according to the task type and characteristics. The large model task is divided into a training task and an inference task, specifically including:

[0027] For large model training tasks, the spatial dimension segmentation method is used to segment them in order to improve parallelism and resource utilization. Specifically, for large model training tasks, the spatial dimension segmentation method is used for segmentation. This process uses inter-operator parallelism and intra-operator parallelism to segment the large model training tasks on a cluster of heterogeneous computing resources. The tasks are segmented by operators to achieve inter-operator parallelism, and further segmented within operators to achieve intra-operator parallelism. Among them, the inter-operator parallelism distributes multiple different computing processes to different computing devices, and executes computing tasks in parallel through pipelines, and completes the calculation in the order of dependencies. The intra-operator parallelism splits the same computing process into multiple sub-tasks, which are distributed to multiple computing devices for processing, and then the calculation results are aggregated through a synchronization mechanism to complete the final calculation.

[0028] For large model inference tasks, different time segmentation strategies are adopted according to different stages of the task to improve inference efficiency and reduce system latency. Specifically, large model inference tasks are segmented from the time dimension to optimize computing efficiency. Each user request will go through two stages: Prefill stage and Decode stage. For the Prefill stage, tensor cutting is used to implement tensor parallelism. The model is divided into tensors and distributed to different devices for parallel computing, thereby accelerating the overall calculation. In the Decode stage, pipeline parallelism is used to overlap the computing process and make full use of memory access time to improve computing efficiency.

[0029] Preferably, in the Prefill stage, the scheduler will dynamically select multiple GPU instances for calculation according to the request to make the best use of computing resources; whenever the calculation of the Prefill stage is completed, the generated KVCache will be transmitted to the Decode stage cluster; at this time, the scheduler of the Decode stage will select the appropriate GPU instance for subsequent calculation according to the needs.

[0030] Preferably, the calculation process of the Prefill stage overlaps with the transmission of KVCache. When the Prefill stage generates a part of KVCache, the scheduler immediately starts to transmit this data, and the Prefill cluster continues to calculate the subsequent KVCache. In the Decode stage, the scheduler will try to package multiple requests into larger batches for calculation.

[0031] Preferably, in step S6, the linear programming is to search for node selection under the constraint of whether some resources are available to achieve the minimum cost; the objective function of minimizing communication overhead is shown in formula (1):

[0032] Minimize t=Σ ij c ij xij (1)

[0033] Where cij represents the communication overhead from node i to node j, and xij represents the amount of data from node i to node j;

[0034] The dynamic programming is based on the previously searched local minimum cost, i.e., f(n-1), f(n-2), and the current single point cost v n To analyze and obtain the current local minimum cost f(n), after searching all resources, the local minimum cost is the global minimum cost, which is specifically expressed as shown in formula (2):

[0035]

[0036] Preferably, the placement strategy uses a search algorithm to search and select corresponding computing resources from the cluster according to the task type (including training tasks / inference tasks, intra-operator parallel tasks / inter-operator parallel tasks, Prefill tasks / Decode tasks). In the search process, the linear programming / dynamic programming algorithm is used to optimize resource allocation with the goal of minimizing transmission overhead and idle resources, including: enumerating all possible segmentation strategies of operators, then calculating the communication cost, and finally searching in the search space defined in the model segmentation with the goal of minimizing the cost, so as to find the optimal solution. For the intra-operator parallel placement strategy, due to the huge communication volume, the model is placed on the GPU connected to the high-speed bandwidth in the same machine as much as possible, and the transmission overhead is minimized by linear programming; for the inter-operator parallel placement strategy, considering that the delay of this part is mainly composed of two items, the first is the warm-up stage, and the subsequent is the stabilization stage, which is dependent on each other, resulting in idle time between devices, and dynamic programming is used to minimize idle resources and reduce delays; and by mapping the results after model segmentation with heterogeneous GPU resources, separated reasoning calculations are realized and service indicators are met as much as possible.

[0037] Preferably, the placement strategy follows the following principles:

[0038] (1) When there are idle computing resources, they are used first rather than overlapping jobs. When the same task occupies multiple computing resources, they are allocated to the computing resources of the same machine first.

[0039] (2) Normally, the priority of work is to prioritize the shortest work first;

[0040] (3) When computing resources are insufficient, find the occupied task scheduler to run new tasks.

[0041] Beneficial effects of the present invention:

[0042] (1) The present invention proposes a large-model parallel task scheduling method based on two-dimensional space-time segmentation and intelligent sharing for heterogeneous intelligent computing systems, which solves the shortcomings of existing methods that lack heterogeneous resource perception, and achieves the technical effect of reducing latency and operating costs, and improving the completion time of the overall workload and the utilization efficiency of computing resources.

[0043] (2) In the present invention, a (fine-grained) model segmentation strategy based on two dimensions of space and time is proposed. Based on the significant differences in the computational properties of the two types of tasks, training and reasoning, targeted optimization strategies are adopted to improve the efficiency of task execution.

[0044] (3) In the present invention, a model placement strategy based on heterogeneous resources and interference models is proposed to solve the shortcomings of traditional methods that lack intelligent sharing judgment and cannot be adjusted dynamically, improve the flexibility and adaptability of scheduling, and thus ensure the maximum benefit of shared resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 This is a large model parallel task scheduling framework diagram based on two-dimensional space-time segmentation and intelligent sharing proposed by the present invention;

[0046] Figure 2 Schematic diagram of the two meta-parallel methods for training large models and mapping them to a two-dimensional device grid;

[0047] Figure 3 This is an example diagram of two phases of large model reasoning separated and placed on different computing resources;

[0048] Figure 4 It is a flow chart of segmentation strategy and processing tasks;

[0049] Figure 5 Schematic diagram of the process of placing a strategy. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] In the design of scheduling frameworks, the existing technologies often ignore the characteristic differences of computing resources in heterogeneous computing systems, which leads to challenges in fair scheduling and efficient scheduling of multi-tasks. This paper designs functions to solve this problem and proposes a new scheduling framework, which is improved and optimized mainly in the following aspects:

[0052] 1) Performance heterogeneity: There are significant performance differences among different hardware computing devices in heterogeneous intelligent computing systems. Therefore, the scheduler framework needs to consider how to balance the performance of different hardware accelerators to achieve unified resource management and optimize the completion time and resource utilization of the overall workload.

[0053] 2) Resource allocation flexibility: In heterogeneous intelligent computing systems, the demand for hardware resources by tasks often changes dynamically, so resource allocation needs to be adjusted in real time according to the needs of the tasks. However, the traditional scheduling framework cannot be adjusted dynamically. The present invention improves the flexibility and adaptability of scheduling, effectively reducing latency and operating costs.

[0054] 3) Task sharing and location allocation sensitivity: In heterogeneous intelligent computing systems, the scheduling framework should adopt resource sharing and location-aware optimization strategies. For example, by explicitly modeling spatial sharing, multiple tasks can be executed simultaneously on the same accelerator; subtasks of the same task should be deployed on the same server as much as possible to reduce cross-node communication overhead.

[0055] 4) Multi-level scheduling strategy requirements: Heterogeneous intelligent computing systems usually involve multiple computing resources and task types, so the scheduling framework needs to support complex multi-level scheduling strategies. The scheduling system of the present invention takes into account multiple factors such as task performance, cost, and throughput, and performs comprehensive optimization.

[0056] The present invention provides a large model parallel task scheduling method based on time-space two-dimensional segmentation and intelligent sharing, comprising the following steps: after the system accepts the large model task submitted by the user, the system first puts the task into a task priority queue to wait for scheduling by the scheduler, and then the scheduler uses the time-space two-dimensional model segmentation technology to divide the task into a plurality of subtasks according to the task type and characteristics, and the task combination module combines the subtasks according to the characteristics of the subtasks and computing resources, and then the task placement module generates a placement strategy and sends it to the instruction executor. The placement strategy uses a search algorithm to search and select corresponding computing resources from a cluster according to the task type (training task / inference task, intra-operator parallel task / inter-operator parallel task, Prefill task / Decode task), and in the search process, minimizes transmission overhead and idle resources as the goal, and uses a linear programming / dynamic programming algorithm to optimize resource allocation; the instruction executor allocates computing resources to the task and places the task, and then executes it.

[0057] Reference Figure 1 For heterogeneous computing systems, the present invention designs a framework of a large model parallel task scheduling system based on two-dimensional space-time segmentation and intelligent sharing, and designs a dedicated scheduler based on the two-dimensional space-time model segmentation technology and the interference model-based placement technology. The scheduler includes a task segmentation module (device), a task combination module (device) and a task placement module (device).

[0058] (1) Task segmentation module

[0059] See also Figure 4 , large model tasks can usually be divided into training tasks and reasoning tasks, among which the parallelism of training tasks is significantly stronger than that of reasoning tasks. Based on the significant differences in the computational properties of the two types of tasks, this paper proposes a (fine-grained) model task partitioning strategy based on two dimensions of space and time: for training tasks, a spatial partitioning method is used to improve parallelism and resource utilization; for reasoning tasks, different time partitioning strategies are used according to different stages of the task to improve reasoning efficiency and reduce system latency.

[0060] For large model training tasks: Thanks to the powerful parallel computing capabilities in existing large model training, the tasks will be mainly segmented from the spatial dimension. This process requires model segmentation on a cluster of heterogeneous computing resources to achieve efficient parallel computing and automatically generate the optimal parallelization strategy. The tasks are segmented into operators to achieve inter-operator parallelism, and further segmented within operators to achieve intra-operator parallelism. Figure 2 As shown in the figure, inter-operator parallelism distributes multiple different computing processes to different computing devices, and executes computing tasks in parallel through pipelines, completing the calculation in the order of dependencies. Intra-operator parallelism is to split the same computing process into multiple subtasks, distribute them to multiple computing devices for processing, and then aggregate the calculation results through a synchronization mechanism to complete the final calculation. By combining these two meta-parallel methods, all existing parallelization strategies can be covered, improving the parallel efficiency of large model training tasks and the utilization of computing resources.

[0061] However, these two meta-parallelism methods have different characteristics, so a trade-off needs to be made between efficiency and overhead. Intra-operator parallelism breaks down the same task and distributes it to different computing devices. Although this brings about higher communication overhead, each device can independently complete the complete task, and its resource utilization is high. In contrast, inter-operator parallelism is completed independently by different computing resources at different stages of the task phase. Although the communication overhead is small, it may cause idle waiting due to dependencies, thus affecting computing efficiency.

[0062] In view of this, the present invention comprehensively considers the characteristics of the two parallel modes and adopts corresponding optimization strategies when splitting model training tasks. Since intra-operator parallelism requires a lot of communication, it is deployed between high-bandwidth, low-latency devices as much as possible to reduce communication bottlenecks. For inter-operator parallelism, in order to avoid idle waiting caused by dependencies, the dependency structure between tasks should be optimized as much as possible during scheduling to avoid unnecessary synchronization and maximize resource utilization efficiency.

[0063] For large model inference tasks: Although the inference process does not have the good parallelism as large model training tasks, it can be divided into two stages with significant differences in characteristics. Therefore, the present invention chooses to split the inference task from the time dimension to optimize the computing efficiency. For general large model inference services, each user request will go through two stages: Prefill stage and Decode stage, which have different computing characteristics:

[0064] 1) The Prefill phase is a compute-bound process that usually requires a very small batch or a single long request to fully utilize computing resources. It is mainly used to optimize the first token latency.

[0065] 2) The decoding stage is a memory-bound process. Each calculation requires access to a large amount of KVCache. Usually a larger batch is required to effectively utilize computing resources, mainly to meet the requirements of the token generation speed.

[0066] In view of the characteristics of these two stages and their service requirements, the present invention adopts different parallel strategies to split the model. Common model parallel methods include tensor parallelism and pipeline parallelism. For the Prefill stage, the present invention uses tensor cutting to achieve tensor parallelism, splits the model into tensors and distributes them to different devices for parallel calculation, thereby accelerating the overall calculation; in the Decode stage, pipeline parallelism is used to overlap the calculation process, making full use of memory access time to improve calculation efficiency. Due to the different communication overheads brought by the two parallel methods, the present invention comprehensively considers computing resources and memory requirements, allocates resources for different stages, and constructs a unified search space to optimize overall performance.

[0067] In addition, the different characteristics of the Prefill stage and the Decode stage provide opportunities for optimizing resource utilization on heterogeneous clusters. Specifically, in the Prefill stage, the scheduler dynamically selects multiple GPU instances for calculation based on requests to make the best use of computing resources. Whenever the calculation in the Prefill stage is completed, the generated KVCache is transferred to the Decode stage cluster. At this time, the Decode stage scheduler selects the appropriate GPU instance for subsequent calculations as needed.

[0068] To further improve efficiency, the calculation process of the Prefill phase can partially overlap with the transmission of KVCache. When the Prefill phase generates a part of KVCache, the scheduler can immediately start transmitting this data, while the Prefill cluster continues to calculate the subsequent KVCache (such as Figure 3 This design reduces waiting time and improves resource utilization.

[0069] In the decode phase, since the computation process is usually limited by memory bandwidth, the scheduler will try to package multiple requests into larger batches for computation. This approach can significantly improve hardware utilization and ensure full utilization of GPU resources. Through this heterogeneous cluster design, the different computational characteristics of the two phases are fully utilized, ultimately achieving efficient inference performance.

[0070] (2) Task placement module

[0071] In the training and inference tasks of large models, the subtasks after model segmentation usually require data communication between different devices, and the size of the communication overhead is closely related to the selected device location and the amount of data transmitted between devices. Therefore, the present invention adopts placement optimization and interference model technology to design a new task placement module.

[0072] Specifically, there may be a small number of very large tensors in the model, which contain most of the parameters in the model. Therefore, it is easy to cause performance bottlenecks due to network bandwidth contention during transmission. At the same time, the transmission of smaller tensors is usually relatively scattered and may be interleaved with other transmission tasks, further affecting the utilization efficiency of network resources. In order to reduce the problem of insufficient network bandwidth, the present invention adopts row aggregation and column aggregation strategies, selects a certain dimension according to the size of the tensor to slice and aggregate the tensor, thereby optimizing communication overhead, reducing the impact of network contention on transmission performance, and improving the efficiency of data transmission.

[0073] See also Figure 5 , combined with different model segmentation strategies, the present invention adopts linear programming and dynamic programming methods to optimize the specific model placement problem, and its basic principles are shown in formulas (1) and (2).

[0074] Linear programming: Its function is to search for node selection under the constraints of whether some resources are available to achieve the minimum cost; the objective function of minimizing communication overhead is shown in Formula 1.

[0075] Minimize t=Σ ij c ij x ij (1)

[0076] Among them, cij represents the communication overhead from node i to node j, and xij represents the amount of data from node i to node j.

[0077] Dynamic programming: through the previously searched local minimum cost, that is, f(n-1), f(n-2), and the current single point cost v nTo analyze and obtain the current local minimum cost f(n), after searching all resources, the local minimum cost is the global minimum cost.

[0078]

[0079] Specifically, it includes: enumerating all possible segmentation strategies of operators, then calculating the communication cost, and finally searching in the search space defined in the model segmentation with the goal of minimizing the cost, so as to find the optimal solution. More specifically, for the placement strategy of intra-operator parallelism (such as tensor parallelism), due to the huge amount of communication, the present invention places the model on a GPU connected by high-speed bandwidth in the same machine as much as possible, and minimizes the transmission overhead through linear programming. For the placement strategy of inter-operator parallelism (such as pipeline parallelism), considering that the delay of this part is mainly composed of two items, the first is the warm-up stage, and the subsequent is the stable stage, and there is a dependency between the two, resulting in idle time between devices. The present invention further minimizes idle resources and reduces delays through dynamic programming. In addition, the present invention also takes into account the properties of different stages during model reasoning and makes full use of heterogeneous GPU resources. By mapping the results of model segmentation with heterogeneous GPU resources, separated reasoning calculations are achieved and service indicators are met as much as possible.

[0080] Considering that frequent job preemption and migration will lead to a decrease in cluster resource utilization and a significant increase in the overall task completion time, even if the amount of resources allocated to the task has been determined, there may be interference and preemption between tasks for the specific placement strategy under multiple tasks. For multiple tasks under the same computing resources, there are three possible parallelizations: fully serial, fully parallel, and partially parallel. Longer parallelization time can make better use of computing resources, but may lead to stronger resource competition and interference. At the same time, for different stages of different tasks, the interference between tasks will vary.

[0081] Since both full serial and full parallel are special cases of partial parallelism, the strategy of the present invention uses parallel scheduling, and focuses on the parallel task insertion point and scheduling task selection, fully considering the interference between tasks, so as to decide whether tasks need to be placed separately or can efficiently share the same computing resources. Therefore, the present invention uses this as a constraint or as an optimization item in the optimization target to formulate a better placement strategy, making sharing decisions more reasonable and timely.

[0082] Specifically, the placement strategy of the present invention follows the following principles:

[0083] (1) When there are idle computing resources, they are used first rather than overlapping jobs. When the same task occupies multiple computing resources, they are allocated to the computing resources of the same machine first. This is because overlapping jobs will cause performance degradation due to resource competition, and high-speed interconnection between computing resources on the same machine can improve communication efficiency.

[0084] (2) In general, the priority of tasks follows the shortest task first. This is because the algorithm (SJF) performs well in most cases, and giving priority to short tasks helps avoid head-of-line blocking and reduce the average task completion time.

[0085] (3) When computing resources are insufficient, find the occupied task scheduling to run new tasks. The basis for this selection is the shortest completion time of the new job team on the computing resources after scheduling. This is because frequently scheduled shared tasks will affect the performance of the original tasks.

[0086] Example 1

[0087] Assume that a certain center has two computers, C1 has GPU1, GPU2 and TPU, C2 has GPU3, GPU4, GPU5, and C3 has GPU6 and GPU7. G1 and G7 have a training task in progress, G2, G3, G4, G5, G6 are idle, and TPU is idle. The tenant currently has 3 tasks to be executed {Task 1, Task 2, Task 3}.

[0088] First, the task selector selects task 1 (training) with the shortest running time, first determines the task type as a training task, then divides the parallel tasks between operators, and dynamically allocates them to idle devices: TPU of C1 and G3 of C2; then divides the parallel tasks within operators, and linear programming prioritizes the high-bandwidth, low-latency device G2 on the same host as TPU and the high-bandwidth, low-latency device G4 on the same host as G3, because the computing resources of C1 and C2 have been occupied, to minimize the cost. Then take out the shortest tasks in turn and send them to TPU, G2, G3, and G4 for calculation.

[0089] Then, the task selector selects Task 2 (inference). Task 2-Prefill performs tensor segmentation and linear programming after segmentation and combination to calculate the minimum cost of allocating to idle devices G5 and G6. Then the computing resources are allocated to G5 and G6. Task 2-Decode performs pipeline parallel division and then performs dynamic programming after sub-task combination. At this time, there are no idle resources. Next, the computing resources with the minimum cost for multi-task parallel execution are searched in existing devices. It is calculated that the cost of Task 2-Decode is the minimum when it is allocated to G3, G4 and G7. Then it is calculated that directly preempting G7 brings higher parallel efficiency, so G7 is directly preempted. Preempting G3 and G4 will affect the efficiency of Task 1. After executing Task 2-Prefill, G5 and G6 transfer part of KVcache to G3, G4 and G7, and then issue Task 2-Decode batch processing.

[0090] Finally, the task selector selects task 3 (training), which also adopts inter-algorithm and intra-operator partitioning. However, after the partitioning, due to its tensor characteristics, row aggregation and column aggregation strategies can be performed, and combined with linear programming to minimize transmission overhead and dynamic programming to minimize idle resources, priority allocation is given to: C1's TPU; linear programming selects high-bandwidth, low-latency devices G1 and G2 on the same host as TPU to minimize the cost. Dynamic programming selects C2's G6 because G6 is an earlier reasoning task and will end the fastest. Then the shortest tasks are taken out in turn and sent to TPU, G1, G2, and G6 for calculation. Preempting TPU, G2, and G6 will affect the efficiency of tasks 1 and 2. Directly preempting G1 will bring higher parallel efficiency, so G1 is directly preempted.

[0091] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and improved concepts of the present invention within the technical scope disclosed by the present invention, and they should be covered by the protection scope of the present invention.

Claims

1. A large model parallel task scheduling method based on two-dimensional space-time segmentation and intelligent sharing, characterized by: The following steps are involved: S1. Accept large model tasks submitted by users; S2, putting the large model task into the task priority queue and waiting for scheduling by the system scheduler; the scheduler includes a task segmentation module, a task combination module and a task placement module; S3, the task segmentation module uses a two-dimensional spatiotemporal model segmentation strategy to segment the large model task into several subtasks according to the task type and characteristics; S4, the task combination module combines the subtasks according to the characteristics of the subtasks and computing resources; S5, the task placement module generates a placement strategy and sends it to the instruction executor; S6. Placement strategy: Based on the task type, a search algorithm is used to search and select the corresponding computing resources from the cluster. During the search process, the linear programming / dynamic programming algorithm is used to optimize resource allocation with the goal of minimizing transmission overhead and idle resources. S7. Allocate computing resources to the task and place the task, then execute it.

2. The large model parallel task scheduling method based on spatiotemporal two-dimensional segmentation and intelligent sharing according to claim 1 is characterized by: In step S3, the task segmentation module uses a spatiotemporal two-dimensional model segmentation strategy to segment the large model task into several subtasks according to the task type and characteristics. The large model task is divided into a training task and an inference task, specifically including: For large model training tasks, the spatial dimension segmentation method is used to segment them to improve parallelism and resource utilization. Specifically, for large model training tasks, the spatial dimension segmentation method is used for segmentation. This process uses two methods, inter-operator parallelism and intra-operator parallelism, to segment large model training tasks on a cluster of heterogeneous computing resources. The tasks are segmented by operators to achieve inter-operator parallelism, and further segmented within operators to achieve intra-operator parallelism. Among them, the inter-operator parallelism distributes multiple different computing processes to different computing devices, and executes computing tasks in parallel in a pipeline manner, and completes the calculation in the order of dependencies. The intra-operator parallelism is to split the same computing process into multiple subtasks, which are distributed to multiple computing devices for processing, and then the calculation results are aggregated through a synchronization mechanism to complete the final calculation. For large model inference tasks, different time segmentation strategies are adopted according to different stages of the task to improve inference efficiency and reduce system latency. Specifically, large model inference tasks are segmented from the time dimension to optimize computing efficiency. Each user request will go through two stages: Prefill stage and Decode stage. For the Prefill stage, tensor cutting is used to implement tensor parallelism. The model is divided into tensors and distributed to different devices for parallel computing, thereby accelerating the overall calculation. In the Decode stage, pipeline parallelism is used to overlap the computing process and make full use of memory access time to improve computing efficiency.

3. The large model parallel task scheduling method based on spatiotemporal two-dimensional segmentation and intelligent sharing according to claim 2 is characterized by: In the Prefill stage, the scheduler will dynamically select multiple GPU instances for calculation according to the request to make the best use of computing resources. Whenever the calculation of the Prefill stage is completed, the generated KVCache will be transmitted to the Decode stage cluster. At this time, the scheduler of the Decode stage will select the appropriate GPU instance for subsequent calculation according to the needs.

4. The large model parallel task scheduling method based on spatiotemporal two-dimensional segmentation and intelligent sharing according to claim 3 is characterized by: The calculation process of the Prefill phase overlaps with the transmission of KVCache. When a part of KVCache is generated in the Prefill phase, the scheduler immediately starts to transmit this data, while the Prefill cluster continues to calculate the subsequent KVCache. During the Decode phase, the scheduler will try to package multiple requests into larger batches for calculation.

5. The large model parallel task scheduling method based on spatiotemporal two-dimensional segmentation and intelligent sharing according to claim 4 is characterized by: In step S6, the linear programming is to search for node selection under the constraint of whether some resources are available to achieve the minimum cost; the objective function of minimizing communication overhead is shown in formula (1): Minimizet=∑ ij c ij x ij (1) Where cij represents the communication overhead from node i to node j, and xij represents the amount of data from node i to node j; The dynamic programming is to analyze the current local minimum cost f(n) through the previously searched local minimum cost, i.e., f(n-1), f(n-2), and the current single point cost vn. After searching all resources, the local minimum cost is the global minimum cost, which is specifically expressed as shown in formula (2):

6. The large model parallel task scheduling method based on spatiotemporal two-dimensional segmentation and intelligent sharing according to claim 5 is characterized by: In step S6, the placement strategy uses a search algorithm to search and select corresponding computing resources from the cluster according to the task type. During the search process, the linear programming / dynamic programming algorithm is used to optimize resource allocation with the goal of minimizing transmission overhead and idle resources. The task types include training tasks / inference tasks, intra-operator parallel tasks / inter-operator parallel tasks, and Prefill tasks / Decode tasks. Specifically, it includes: enumerating all possible segmentation strategies of operators, then calculating the communication cost, and finally searching in the search space defined in the model segmentation with the goal of minimizing the cost to find the optimal solution. For the intra-operator parallel placement strategy, due to the huge communication volume, the model is placed on a GPU connected by high-speed bandwidth in the same machine as much as possible, and the transmission overhead is minimized through linear programming; for the inter-operator parallel placement strategy, dynamic programming is used to minimize idle resources and reduce delays.

7. The large model parallel task scheduling method based on spatiotemporal two-dimensional segmentation and intelligent sharing according to claim 6 is characterized by: The placement strategy follows the following principles: (1) When there are idle computing resources, they are used first rather than overlapping jobs. When the same task occupies multiple computing resources, they are allocated to the computing resources of the same machine first. (2) Normally, the priority of work is to prioritize the shortest work first; (3) When computing resources are insufficient, find the occupied task scheduler to run new tasks.

Images