Large model task intelligent scheduling and distribution method and system
Patent Information
- Application Number
- CN202611041057.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-14
- Publication Date
- 2026-10-23
AI Technical Summary
[0006]本发明提供了大模型任务智能调度分配方法及系统，解决了上述背景技术中提出的现有方案无法识别节点连续显存分布状态，并缺乏对排队任务显存需求分布的感知，导致大小任务混合时资源分配失衡的问题
[0026]1、该大模型任务智能调度分配方法及系统，通过对集群算力节点进行分类，精准区分不同算力类群的显存承载能力与资源状态，结合各算力类群实时显存占用、碎片分布的动态感知能力，配合任务优先级排序与显存需求分布拐点智能切分，区分大规模任务的适配边界，实现算力资源精细化分层管理，从源头抑制显存碎片累积，并实现任务与算力类群的精准匹配，提升大模型任务的调度分配速度。
Smart Images
Figure CN122963775A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence technology, specifically to a method and system for intelligent scheduling and allocation of large-scale model tasks. Background Technology
[0002] With the rapid iteration of generative artificial intelligence technology, large language models and multimodal large models have been widely applied in various business scenarios such as intelligent question answering, content generation, code development, intelligent reasoning, and model fine-tuning training. Large model operation relies on the high-performance computing power and large-capacity video memory resources of GPUs. Compared with traditional deep learning tasks, large model tasks have significant characteristics such as large parameter scale, high video memory consumption, sensitivity to inference latency, dynamic fluctuations in concurrency, variable context length, and dynamic addition and removal of key-value caches. Currently, the industry has mature large model computing power scheduling solutions that can achieve basic resource pooling and task allocation. However, in actual large-scale deployments, there are still shortcomings such as prominent video memory fragmentation, low resource utilization, unreasonable scheduling decisions, and redundant communication overhead, which seriously limit the overall throughput and operational stability of the cluster.
[0003] However, in terms of memory management, traditional frameworks statically pre-allocate memory by requesting a whole block of contiguous memory space at once according to the maximum sequence length supported by the model. But most users' actual requests require far less memory than the pre-allocated space, and the unused memory space cannot be split and reused, resulting in serious internal memory waste. To improve this problem, existing mainstream inference engines such as vLLM have introduced a page-based attention mechanism, which splits the original whole KV cache into multiple fixed-size memory blocks and uses page table mapping to access non-contiguous memory space, effectively solving the problem of external memory fragmentation during KV cache use and significantly improving memory utilization efficiency. However, this optimization method only optimizes fragmentation for dynamic KV cache and does not address the memory fragmentation problem caused by static model weight occupancy and the underlying memory allocation mechanism of CUDA. When the cluster needs to load a new model or allocate a large block of computational memory for a new task, even if the total remaining memory of the graphics card is sufficient, there will be frequent memory overflow and task startup failures due to insufficient contiguous memory address space.
[0004] At the cluster scheduling level, existing schedulers typically only perform matching and verification based on the total amount of video memory, and cannot identify the distribution status of contiguous video memory on nodes. At the same time, existing fragmentation optimization and backfill scheduling are all local greedy strategies, lacking global queue awareness and the ability to predict future tasks. They are prone to causing small tasks to fill up scattered spaces and block contiguous video memory blocks, leading to blocking and deadlock of large tasks.
[0005] Based on this, this application proposes a method and system for intelligent scheduling and allocation of large-scale model tasks. Summary of the Invention
[0006] This invention provides a method and system for intelligent scheduling and allocation of large model tasks, which solves the problem that the existing solutions proposed in the background art cannot identify the continuous distribution of node memory and lack awareness of the distribution of memory demand of queued tasks, resulting in unbalanced resource allocation when large and small tasks are mixed.
[0007] This invention provides the following technical solution: a method for intelligent scheduling and allocation of large-scale model tasks, comprising the following steps:
[0008] S1. Implement unified management of all computing power nodes within the system, and classify the computing power nodes into several groups based on their characteristic data.
[0009] S2. Real-time detection of various groups, reception and analysis of large model tasks, determination of task type, priority, minimum continuous video memory required for the task and task execution time, and pre-sorting of queued tasks within a preset time according to priority.
[0010] S3. Divide the pre-sorted tasks into several task subsets;
[0011] S4. Calculate the amount of contiguous memory required for tasks of the same type within each task subset.
[0012] S5. Based on task scheduling and allocation rules, intelligent scheduling and allocation of large model tasks are performed. When the remaining video memory of the corresponding group meets the needs of the same type of task within the task subset, the corresponding task is scheduled and allocated to the same computing power group. Otherwise, some tasks with high video memory requirements are allocated to another computing power group of the same type, reserving resources for tasks with low video memory requirements to be allocated in the next task subset.
[0013] Preferably, in step S2, the memory usage, memory fragmentation distribution, maximum continuous memory, and amount of free memory within each computing node in the group are collected in real time.
[0014] Preferably, in step S5, the number of high memory-demand tasks to be migrated is the minimum migration amount, so that the remaining contiguous memory in the class group can be adapted to the memory requirements of the low memory-demand tasks to be allocated in the next task subset.
[0015] Preferably, the specific operation of S3 is as follows: based on the task analysis results, identify the video memory required by each task, and calculate the difference in video memory required by adjacent tasks. When the difference exceeds a set threshold, determine that the high video memory task is an inflection point, perform segmentation at the inflection point, and classify the inflection point task and the tasks to be classified before it into the same task subset.
[0016] Preferably, step S4 further calculates the estimated release time of occupied video memory between adjacent free video memory in the same category.
[0017] The large-scale model task intelligent scheduling and allocation system includes a group management module, a status acquisition module, a task profiling module, a queue splitting module, and a scheduling and allocation module. The group management module is used to uniformly manage all computing power nodes in the system and classify the computing power nodes into several groups based on their characteristic data.
[0018] The status acquisition module performs real-time detection on various groups, and collects the memory occupancy, memory fragmentation distribution, maximum contiguous memory, and amount of free memory in each computing node within the group.
[0019] The task profiling module is used to receive and analyze large model tasks, determine the task type, priority, minimum continuous video memory required by the task, and task execution time, and pre-sort the queued tasks within a preset time according to task priority.
[0020] The queue segmentation module divides the pre-sorted tasks into several task subsets based on the analysis results of the task profiling module.
[0021] The scheduling and allocation module calculates the continuous GPU memory required by tasks of the same type within each task subset based on feedback from the status acquisition module, task profiling module, and queue splitting module. When the remaining GPU memory of the corresponding group meets the requirements, the corresponding task is scheduled and allocated to the same computing power group. Otherwise, based on the task scheduling and allocation rules, the task is scheduled and allocated, and some tasks with high GPU memory requirements are allocated to another computing power group of the same type, reserving resources for tasks with low GPU memory requirements to be allocated in the next task subset.
[0022] Preferably, the queue segmentation module divides the queued tasks according to the following rule: the division is based on the inflection point of the continuous video memory required by the task.
[0023] Preferably, the task scheduling and allocation rule is as follows: when the remaining video memory of the computing power group meets the needs of the same type of task within the same task subset, the minimum fragmentation rule is activated; otherwise, the minimum fragmentation rule is paused, and some tasks with high video memory requirements are allocated to another group with sufficient video memory, reserving resources for tasks with low video memory requirements in the next task subset.
[0024] Preferably, the migration amount of high memory-demand tasks is determined by the memory threshold of low memory-demand tasks to be allocated in the next task subset.
[0025] Compared with the prior art, the present invention has the following beneficial effects:
[0026] 1. This intelligent scheduling and allocation method and system for large-scale model tasks classifies cluster computing power nodes, accurately distinguishes the memory carrying capacity and resource status of different computing power groups, combines the dynamic perception capability of real-time memory occupation and fragmentation distribution of each computing power group, and coordinates task priority sorting and intelligent segmentation of memory demand distribution inflection points to distinguish the adaptation boundaries of large-scale tasks, realizes refined hierarchical management of computing power resources, suppresses the accumulation of memory fragmentation from the source, and achieves accurate matching between tasks and computing power groups, thereby improving the scheduling and allocation speed of large-scale model tasks.
[0027] 2. This large-scale model task intelligent scheduling and allocation method and system, based on task scheduling and allocation rules, dynamically adjusts the scheduling strategy according to the real-time GPU memory availability and task queue distribution status of different computing power groups. In scenarios with sufficient GPU memory resources, it rationally utilizes the minimum fragmentation rule to fill scattered GPU memory resources, avoiding idle and wasted computing power. When GPU memory resources are insufficient, it promptly adjusts the allocation logic and coordinates with cross-group task migration and resource reservation mechanisms to prioritize resource space for tasks with large GPU memory requirements, achieving a dynamic balance between fragmentation management and resource utilization, improving allocation speed while increasing computing power resource utilization. Attached Figure Description
[0028] Figure 1 This is a schematic diagram of the intelligent scheduling and allocation method for large-scale model tasks proposed in this invention;
[0029] Figure 2 This is a block diagram of the intelligent scheduling and allocation system for large-scale model tasks proposed in this invention. Detailed Implementation
[0030] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0031] This invention provides embodiments: Please refer to Figure 1 The intelligent scheduling and allocation method for large-scale model tasks includes the following steps:
[0032] S1. All computing power nodes within the system are managed uniformly. Based on their characteristic data, the nodes are classified, grouping nodes of the same type into a single cluster, with the number of nodes within each cluster not exceeding a set threshold. All computing power nodes in the system are then divided into several clusters. The characteristic data of the computing power nodes includes multi-dimensional data such as hardware configuration, total video memory, and available memory. This clustering of computing power nodes avoids invalid attempts at cross-specification allocation during task assignment, improving task allocation efficiency.
[0033] S2. Real-time detection of various clusters, real-time collection of memory usage, memory fragmentation distribution, maximum contiguous memory, and free memory within each cluster; receiving and analyzing large model tasks, determining task type, priority, minimum contiguous memory required, and execution time, and pre-sorting queued tasks within a preset time according to priority. By pre-sorting queued tasks, tasks can be directly allocated according to task priority during scheduling, improving the efficiency of subsequent task allocation.
[0034] S3. Divide the pre-sorted tasks into several task subsets;
[0035] The specific operation is as follows: Based on the task analysis results, identify the video memory required by each task, and calculate the difference in video memory required by adjacent tasks. When the difference exceeds the set threshold, determine the high video memory task as the inflection point, perform segmentation at the inflection point, and classify the inflection point task and the tasks to be classified before it into the same task subset.
[0036] S4. Calculate the amount of contiguous video memory required for tasks of the same type within each task subset; and calculate the estimated release time of occupied video memory between adjacent free video memory in the same group. Calculate the time required to merge adjacent free video memory intervals into a complete large video memory interval. This allows the system to determine whether to delay the occupation of free video memory intervals based on the waiting time during task allocation, so that adjacent small video memory spaces can be merged into large video memory spaces for the allocation of tasks with high video memory requirements. This task scheduling and allocation method can ensure the execution of large tasks while taking into account the response latency of small tasks, thereby improving the system's adaptability to large model tasks.
[0037] S5. Based on task scheduling and allocation rules, intelligent scheduling and allocation are performed on large model tasks. When the remaining GPU memory of the corresponding group meets the needs of the same type of tasks within the task subset, the corresponding tasks are scheduled and allocated to the same computing power group, and the idle computing power is filled using the minimum fragmentation rule. Otherwise, some tasks with high GPU memory requirements are allocated to another computing power group of the same type, reserving resources for tasks with low GPU memory requirements to be allocated in the next task subset, thus achieving a dynamic balance between fragmentation management and resource utilization. In this step, the number of high GPU memory requirement tasks migrated is the minimum migration amount, ensuring that the remaining contiguous GPU memory in the group is adapted to the GPU memory requirements of tasks with low GPU memory requirements to be allocated in the next task subset.
[0038] As described above, this application classifies cluster computing power nodes to accurately distinguish the memory carrying capacity and resource status of different computing power groups. Combined with the dynamic perception capability of real-time memory occupation and fragmentation distribution of each computing power group, and with the intelligent segmentation of task priority ranking and memory demand distribution inflection points, it distinguishes the adaptation boundaries of large-scale tasks, realizes refined hierarchical management of computing power resources, suppresses the accumulation of memory fragmentation from the source, and achieves accurate matching between tasks and computing power groups, thereby improving the scheduling and allocation speed of large model tasks.
[0039] Based on task scheduling and allocation rules, the scheduling strategy is dynamically adjusted according to the real-time GPU memory availability and task queue distribution status of different computing power groups. In scenarios with sufficient GPU memory resources, the fragmentation minimum rule is used to fill scattered GPU memory resources, avoiding idle and wasted computing power. When GPU memory resources are insufficient, the allocation logic is adjusted in a timely manner and combined with cross-group task migration and resource reservation mechanisms to prioritize the resource space of tasks with large GPU memory requirements. This achieves a dynamic balance between fragmentation management and resource utilization, improving allocation speed while increasing computing power resource utilization.
[0040] like Figure 2 As shown, this invention provides a large-scale model task intelligent scheduling and allocation system, including a group management module, a status acquisition module, a task profiling module, a queue segmentation module, and a scheduling and allocation module. The scheduling and allocation module interacts with the group management module, the status acquisition module, the task profiling module, and the queue segmentation module. The group management module interacts with the status acquisition module, and the task profiling module interacts with the queue segmentation module.
[0041] Among them, the cluster management module is used to manage all computing power nodes in the system in a unified manner. Based on the characteristic data of the computing power nodes (such as hardware configuration, software environment, total amount of video memory and hollow amount of other multi-dimensional data), the computing power nodes are classified into several clusters. Computing power nodes of the same type are assigned to the same cluster, and the number of computing power nodes in the same cluster does not exceed the threshold. Moreover, the computing power nodes in the system are divided into at least two clusters.
[0042] The status acquisition module is used to perform real-time detection of various groups, and to collect the memory usage, memory fragmentation distribution, maximum contiguous memory, and amount of free memory in each computing node within the group.
[0043] The task profiling module is used to receive and analyze large model tasks, determine the task type, priority, minimum contiguous memory required for the task, and task execution time, and pre-sort the queued tasks within a preset time according to task priority.
[0044] Based on the analysis results from the task profiling module, the queue segmentation module divides the pre-sorted tasks into several task subsets. The segmentation rule adopted by the queue segmentation module is: to divide according to the inflection point of the continuous video memory required by the task.
[0045] The scheduling and allocation module calculates the required contiguous GPU memory for similar tasks within each task subset based on feedback from the status acquisition module, task profiling module, and queue partitioning module. When the remaining GPU memory in the corresponding group meets the requirements, the corresponding tasks are scheduled and allocated to the same computing power group. Otherwise, based on the task scheduling and allocation rules, tasks with high GPU memory requirements are allocated to another computing power group of the same type, reserving resources for tasks with low GPU memory requirements to be allocated in the next task subset. The task scheduling and allocation rules are as follows: when the remaining GPU memory in the computing power group meets the requirements for similar tasks within the same task subset, the minimum fragmentation rule is activated; otherwise, the minimum fragmentation rule is paused, and some tasks with high GPU memory requirements are allocated to another group with sufficient GPU memory, reserving resources for tasks with low GPU memory requirements in the next task subset. Furthermore, the migration amount of tasks with high GPU memory requirements is determined by the GPU memory threshold of tasks with low GPU memory requirements to be allocated in the next task subset. The criteria for determining a task with low video memory requirements is that the minimum contiguous video memory range required by the task is not greater than a preset contiguous video memory threshold, while the criteria for determining a task with high video memory requirements is that the minimum contiguous video memory range required by the task is greater than a preset contiguous video memory threshold.
[0046] In summary, this intelligent scheduling and allocation method and system for large-scale tasks can identify and retain potentially large memory intervals that can be recovered from short-term releases, reducing allocation failures caused by insufficient contiguous space for large tasks, improving the task allocation success rate, and suppressing memory fragmentation accumulation and improving system computing power utilization through inflection point partitioning and task scheduling and allocation rules.
Claims
1. A method for intelligent scheduling and allocation of large-scale model tasks, characterized in that, Includes the following steps: S1. Implement unified management of all computing power nodes within the system, and classify the computing power nodes into several groups based on their characteristic data. S2. Real-time detection of various groups, reception and analysis of large model tasks, determination of task type, priority, minimum continuous video memory required for the task and task execution time, and pre-sorting of queued tasks within a preset time according to priority. S3. Divide the pre-sorted tasks into several task subsets; S4. Calculate the amount of contiguous memory required for tasks of the same type within each task subset. S5. Based on task scheduling and allocation rules, intelligent scheduling and allocation of large model tasks are performed. When the remaining video memory of the corresponding group meets the needs of the same type of task within the task subset, the corresponding task is scheduled and allocated to the same computing power group. Otherwise, some tasks with high video memory requirements are allocated to another computing power group of the same type, reserving resources for tasks with low video memory requirements to be allocated in the next task subset.
2. The intelligent scheduling and allocation method for large model tasks according to claim 1, characterized in that: In S2, the memory usage, memory fragmentation distribution, maximum contiguous memory, and amount of free memory within the group are collected in real time for each computing node.
3. The intelligent scheduling and allocation method for large model tasks according to claim 1, characterized in that: In step S5, the number of high memory-demand tasks to be migrated is the minimum migration amount, which is sufficient to make the remaining contiguous memory in the class group fit the memory requirements of the low memory-demand tasks to be allocated in the next task subset.
4. The intelligent scheduling and allocation method for large model tasks according to claim 1, characterized in that: The specific operation of S3 is as follows: based on the task analysis results, identify the video memory required by each task, and calculate the difference in video memory required by adjacent tasks. When the difference exceeds a set threshold, determine that the high video memory task is an inflection point, perform segmentation at the inflection point, and classify the inflection point task and the tasks to be classified before it into the same task subset.
5. The intelligent scheduling and allocation method for large model tasks according to claim 1, characterized in that: Step S4 also calculates the estimated release time of occupied video memory between adjacent free video memory in the same group.
6. A large-scale model task intelligent scheduling and allocation system, comprising a group management module, a status acquisition module, a task profiling module, a queue segmentation module, and a scheduling and allocation module, characterized in that: The category management module is used to manage all computing power nodes in the system in a unified manner. Based on the characteristic data of the computing power nodes, the computing power nodes are classified into several categories. The status acquisition module performs real-time detection on various groups, and collects the memory occupancy, memory fragmentation distribution, maximum contiguous memory, and amount of free memory in each computing node within the group. The task profiling module is used to receive and analyze large model tasks, determine the task type, priority, minimum continuous video memory required by the task, and task execution time, and pre-sort the queued tasks within a preset time according to task priority. The queue segmentation module divides the pre-sorted tasks into several task subsets based on the analysis results of the task profiling module. The scheduling and allocation module calculates the continuous GPU memory required by tasks of the same type within each task subset based on feedback from the status acquisition module, task profiling module, and queue splitting module. When the remaining GPU memory of the corresponding group meets the requirements, the corresponding task is scheduled and allocated to the same computing power group. Otherwise, based on the task scheduling and allocation rules, the task is scheduled and allocated, and some tasks with high GPU memory requirements are allocated to another computing power group of the same type, reserving resources for tasks with low GPU memory requirements to be allocated in the next task subset.
7. The intelligent scheduling and allocation system for large-scale model tasks according to claim 6, characterized in that: The queue segmentation module divides the queued tasks according to the following rule: the division is based on the inflection point of the continuous video memory required by the task.
8. The intelligent scheduling and allocation system for large-scale model tasks according to claim 6, characterized in that: The task scheduling and allocation rule is as follows: when the remaining video memory of the computing power group meets the needs of the same type of tasks within the same task subset, the minimum fragmentation rule is activated; otherwise, the minimum fragmentation rule is paused, and some tasks with high video memory requirements are allocated to another group with sufficient video memory, reserving resources for tasks with low video memory requirements in the next task subset.
9. The intelligent scheduling and allocation system for large-scale model tasks according to claim 8, characterized in that: The migration amount of high memory-demand tasks is determined by the memory threshold of low memory-demand tasks to be allocated in the next task subset.