A resource scheduling method and system for large model inference services

Abstract

This invention discloses a resource scheduling method and system for large-model inference services. Deployed at the gateway layer of the large-model inference service, the method maintains waiting queues and decoding queues to uniformly schedule and manage requests entering the pre-filling stage and executing decoding steps. The method constructs request priorities by combining request length characteristics, waiting time, and memory usage information, and introduces a memory pressure awareness mechanism to dynamically adjust resource allocation strategies based on the system's memory status. When memory pressure increases, mechanisms such as stage budget adjustment, high-water mark protection, and preemption recovery are used to suppress new cache usage and promote the release of resources from existing requests, thereby achieving layered control of memory congestion. Compared with existing scheduling methods, this invention can reduce first-word response latency, alleviate high memory pressure and decoding blocking problems, and improve the resource utilization, operational stability, and continuous service capability of large-model inference services.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence infrastructure and large language model reasoning service technology, and in particular to a resource scheduling method and system for large model reasoning services. Background Technology

[0002] With the widespread application of large language models in scenarios such as intelligent question answering, code generation, content creation, and intelligent agent systems, online inference services for large language models are gradually becoming an important part of artificial intelligence infrastructure. To improve model service capabilities, existing large language model inference systems typically adopt an architecture combining a service gateway and an inference runtime. The service gateway is responsible for receiving user requests, maintaining request status, and scheduling request execution; the inference runtime is responsible for performing model computations and managing cached resources generated during the inference process.

[0003] In existing large language model inference processes, a request typically goes through a pre-filling stage and a decoding stage. The pre-filling stage processes input prompts and generates corresponding caches; the computational cost is usually related to the input length. The decoding stage uses an autoregressive approach to gradually generate output content and continuously expands cache usage. Since the cached resources are only released after a request is completed, a large number of active requests at different stages exist simultaneously in the system.

[0004] However, in real-world online service scenarios, the input length, output length, and resource requirements of different requests vary significantly. Some requests can be completed quickly, while others require a longer time. If the system schedules requests solely based on their arrival order, long requests can easily block short requests, preventing subsequent requests from receiving timely service, thus increasing the initial response time and overall completion time.

[0005] On the other hand, caching resources typically consume a significant amount of video memory. As requests continuously enter the system and output content is constantly generated, cache usage will continue to increase. When the number of concurrent requests increases, the system is prone to a continuous rise in video memory pressure. Excessive video memory pressure not only reduces the system's ability to accept new requests, but may also cause the decoding process to stall due to insufficient video memory, and in severe cases, may even lead to the system being unable to generate effective service progress for an extended period.

[0006] Existing technologies primarily improve system performance through batch processing optimization, cache organization and management, multi-instance load balancing, and underlying inference performance optimization. However, for request scheduling within a single inference instance, there is still a lack of a unified decision-making method that can simultaneously consider request length characteristics, stage differences, and memory pressure. Existing scheduling methods often only focus on request length or computational overhead, lacking the ability to perceive changes in cache resource usage and memory pressure; while existing memory management mechanisms typically lack coordinated control with the request scheduling process, making it difficult to balance response performance, memory utilization, and system stability.

[0007] Therefore, there is an urgent need to propose a resource scheduling method and system for large model inference services, which can dynamically adjust the request scheduling order and resource allocation strategy according to request characteristics, execution stage and memory pressure, so as to reduce the first response latency, improve the utilization efficiency of cache resources, and alleviate the memory congestion problem in high-concurrency scenarios while ensuring the stable operation of the system. Summary of the Invention

[0008] To overcome the shortcomings of existing technologies, this invention proposes a resource scheduling method and system for large-scale model inference services. The method and system of this invention are deployed to the service gateway layer of large-scale language model inference service instances to realize online scheduling of concurrent inference requests, admission control during the pre-filling / decoding stage, and control of GPU cache pressure.

[0009] The technical solution of the present invention is as follows:

[0010] This invention first provides a resource scheduling method for large model inference services, the method comprising:

[0011] S1: Receive user inference requests, parse the request metadata of user inference requests, and add the requests to the waiting queue;

[0012] S2: Obtain the current request status, queue status, video memory cache usage, and GPU computing resource status during the underlying large language model inference runtime, and calculate the current video memory pressure;

[0013] S3: Calculate the memory shadow price based on the current memory pressure. The memory shadow price increases when the memory pressure increases. Dynamically allocate the pre-filling stage budget and the decoding stage budget based on the current memory pressure. When the memory pressure increases, compress the computing power budget of the pre-filling stage and increase the computing power budget of the decoding stage.

[0014] S4: Calculate the waiting queue priority based on the prompt word length, waiting time, and memory cost of the requests in the waiting queue, and select the requests allowed to enter the pre-filling stage in combination with the admission conditions of the pre-filling stage.

[0015] S5: Add the pre-filled requests to the decoding queue. Calculate the decoding queue priority based on the estimated remaining generation length of the requests in the decoding queue, the decoding wait time, and the benefit of releasing the video memory cache. Combine this with the admission criteria for the decoding stage to select the requests that are allowed to execute the decoding step.

[0016] S6: Submit the selected pre-filled requests and decoding requests to the underlying large language model inference runtime for execution, and update the request status, the number of generated tokens, and the memory usage based on the execution results;

[0017] S7: Release the video memory cache occupied by completed requests, and perform high-water level protection, pre-filling during operation, or preemption recovery on the gateway side when the system reaches high video memory pressure or a continuous state of no progress.

[0018] According to a preferred embodiment of the present invention, S1 can be implemented as follows: The service gateway receives inference requests from external users and records request metadata for each request (request arrival time, input prompt word length, maximum output length upper bound, estimated output length, current request stage, and number of generated tokens). Newly arrived requests first enter the waiting queue. At this time, the request has not yet entered the pre-filling stage and has not yet been allocated memory cache.

[0019] In S3 of the present invention, the price of the video memory shadow increases when the video memory pressure increases; when the video memory pressure increases, the computing power budget of the pre-filling stage is compressed and the computing power budget of the decoding stage is increased.

[0020] According to one preferred embodiment of the present invention, the step of calculating the memory shadow price based on the current memory pressure includes:

[0021] Set the low watermark threshold and high watermark threshold for video memory;

[0022] When the memory pressure is lower than the memory low water level threshold, the memory shadow price is set to the base value.

[0023] When the memory pressure is between the low memory water level threshold and the high memory water level threshold, the memory shadow price increases as the current memory pressure increases.

[0024] When the memory pressure reaches or exceeds the memory high water level threshold, the memory shadow price takes the maximum value.

[0025] The dynamic allocation of pre-filling stage budget and decoding stage budget based on current memory pressure includes:

[0026] When memory pressure is below the low-water mark threshold, a higher proportion of the computational budget is allocated to the pre-filling phase to accelerate new requests entering the model inference process. When memory pressure is between the low-water mark threshold and the high-water mark threshold, the pre-filling phase budget is gradually compressed as the current memory pressure increases, while the decoding phase budget is increased accordingly to facilitate the completion of existing active requests and release of memory cache. When memory pressure reaches or exceeds the high-water mark threshold, new pre-filling admissions are disabled. If there are running pre-filling requests that have started but not yet completed, a small amount of minimum budget is reserved to complete the current pre-filling work; if there are no running pre-filling requests, the pre-filling phase budget is reduced to zero.

[0027] The present invention also provides a resource scheduling system for large model inference services, for implementing the aforementioned resource scheduling method, which includes a waiting queue module, a decoding queue module, a lexical awareness scheduling module, a stage admission control module, a resource management module, and a low-level inference runtime interface module;

[0028] The waiting queue module is configured to store requests that have arrived but have not yet entered the pre-filling stage;

[0029] The decoding queue module is configured to store active requests that have completed the pre-filling phase and are waiting to continue executing the decoding steps.

[0030] The word-aware scheduling module is configured to calculate the priority of the waiting queue and the priority of the decoding queue, respectively.

[0031] The stage admission control module is configured to determine whether a candidate request can enter the pre-filling stage or execute the decoding step based on the current resource status.

[0032] The resource management module is configured to maintain GPU computing budget, memory usage, memory pressure, pre-fill / decoding stage budget, number of active requests, decoding batch processing usage, memory level status, and request completion feedback.

[0033] The underlying inference runtime interface module is configured to submit pre-filling requests and decoding requests to the large language model inference runtime, and receive the pre-filling completion status, decoding generation result, request completion status and video memory cache usage update information returned by the inference runtime.

[0034] Furthermore, the system can be deployed within a single-machine large language model inference service instance, or it can be used as a local scheduler within each service instance of a multi-instance large language model inference service system, in conjunction with the upper-layer load balancing module.

[0035] The present invention has the following beneficial effects:

[0036] (1) This invention takes into account the characteristics of request length, request waiting time, differences in execution stages and memory resource status, and realizes the coordinated scheduling of requests in the pre-filling stage and the decoding stage, which helps to reduce the queuing waiting time of requests and improve the service response capability.

[0037] (2) This invention introduces a memory pressure sensing mechanism, which dynamically reflects the current memory resource shortage through the memory shadow price, so that the scheduling process can consider both computing resources and memory resources at the same time, thereby improving the memory resource utilization efficiency.

[0038] (3) The present invention constructs a hierarchical memory congestion control mechanism. With memory pressure as the feedback signal, it achieves multi-level control from congestion prevention, congestion isolation to congestion recovery through measures such as dynamic adjustment of stage budget, high water level protection and preemption recovery. While ensuring memory security, it improves system operation stability and continuous service capability under high load scenarios.

[0039] (4) This invention is applicable to dynamic video memory cache management scenarios. It can dynamically manage video memory usage according to the actual execution process of the request, reduce resource waste, and improve the effective video memory utilization rate and request admission flexibility.

[0040] In summary, this invention can balance request response performance, memory resource utilization efficiency, and system operation stability, and is suitable for online resource scheduling and memory pressure control in single-machine large model inference service scenarios. Attached Figure Description

[0041] Figure 1 is a diagram of the resource scheduling system architecture for large model inference services according to the present invention.

[0042] Figure 2 shows the state changes of a request from waiting to pre-filling, decoding, and completion, as well as the flowchart of cache allocation, growth, and release.

[0043] Figure 3 is a flowchart of the word perception process of the resource scheduling method for large model inference services according to the present invention.

[0044] Figure 4 is a diagram of a hierarchical congestion control mechanism. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the invention.

[0046] In one embodiment, the present invention can be deployed in the service layer of a large language model inference service system, running independently as a service gateway, or integrated with the underlying inference engine. The service gateway can be deployed at the front end of a single service instance in a single-machine single-card, single-machine multi-card, or distributed inference cluster, for unified scheduling and memory pressure control of inference requests entering that service instance. The service gateway can interact with the underlying large language model inference runtime through remote call interfaces, shared memory, inter-process communication interfaces, or other communication methods. The underlying large language model inference runtime can be implemented using any inference framework that supports pre-filling stages, decoding stages, and memory cache management; the present invention does not limit specific hardware models, programming languages, or underlying inference frameworks.

[0047] Figure 1 This diagram illustrates the architecture of a resource scheduling system for large-model inference services provided by an embodiment of the present invention. The system mainly includes: a waiting queue module, a decoding queue module, a lexical awareness scheduling module, a stage admission control module, a resource management module, and a low-level inference runtime interface module.

[0048] The waiting queue module stores requests that have arrived but have not yet entered the pre-filling stage. Requests in this queue have not yet been allocated to GPU memory cache and are used as candidate requests for priority calculation and scheduling selection by the lexical awareness scheduling module.

[0049] The decoding queue module is used to store active requests that have completed the pre-filling stage and are waiting to continue the decoding step. Preferably, the decoding queue module maintains the number of generated tokens, decoding wait time, current video memory cache usage, and request completion status for active requests.

[0050] The lexical awareness scheduling module is used to calculate the priority of the waiting queue and the priority of the decoding queue, respectively. In this invention, for the waiting queue, the module comprehensively considers the pre-filling computation cost, future decoding cost, memory cache holding cost, waiting time compensation, and memory shadow price; for the decoding queue, the module comprehensively considers the estimated remaining generation length, decoding waiting time, memory cache release benefit, and memory shadow price.

[0051] The stage admission control module determines whether a candidate request can enter the pre-filling stage or execute the decoding step. Preferably, for the pre-filling stage, this module checks the pre-filling gating status, the pre-filling stage budget, the video memory safety threshold, and the maximum concurrent pre-filling quantity; for the decoding stage, this module checks the decoding stage budget, the decoding batch size, and whether the video memory cache added after the decoding step meets the video memory safety constraints. The underlying inference runtime interface module submits pre-filling and decoding requests to the underlying large language model inference runtime and receives the request execution status and video memory cache usage update information returned by the runtime.

[0052] The resource management module is used to maintain the GPU computing budget, memory cache usage, memory pressure, pre-filling stage budget, and decoding stage budget, and triggers high-water level protection, pre-filling completion, or gateway-side preemption recovery based on memory pressure. In this embodiment, as a preferred embodiment, the module is also configured to calculate the memory shadow price based on memory pressure, dynamically adjust the pre-filling / decoding stage budget based on memory pressure, and trigger corresponding control mechanisms under high-water level or critical pressure conditions.

[0053] In one embodiment, multiple clients can concurrently send inference requests to the service gateway. The service gateway operates cyclically according to a scheduling cycle. In each scheduling cycle, it receives newly arriving requests, updates resource status, calculates request priorities, performs phase admission control, and submits selected requests to the underlying inference runtime for execution. Figure 2 This invention illustrates the request state transition process, including a waiting state, a pre-filling state, a decoding state, and a completion state. Upon arrival at the service gateway, the request first enters the waiting state and is added to the waiting queue. In the waiting state, the request has not yet been allocated video memory cache. If the request meets the pre-filling stage admission criteria, it enters the pre-filling state, where the underlying inference runtime processes the input prompts and generates a prompt word video memory cache. After pre-filling is complete, the request enters the decoding state and is added to the decoding queue. In the decoding state, the request generates output tokens incrementally in an autoregressive manner. With each decoding step, the number of generated tokens increases, and the corresponding video memory cache usage also increases. When the request reaches the actual output length or meets the stopping condition, the request enters the completion state and releases its occupied prompt word video memory cache and the video memory cache used during the generation process.

[0054] The service gateway adopts a single-step loop scheduling method, and executes the resource scheduling method process of this invention in its entirety in each cycle to serve the continuous influx of requests from clients.

[0055] Figure 3 The present invention illustrates a resource scheduling method for large-model inference services, which specifically includes the following steps:

[0056] S1: Receive newly arrived user requests, parse the request metadata, and add requests that have not yet entered the pre-filling stage to the waiting queue.

[0057] The request metadata mentioned in this step includes the request arrival time, input prompt word length, maximum output length upper bound, estimated output length, current request stage, and number of generated tokens.

[0058] For newly arrived requests, they first enter a waiting state and are added to the waiting queue. Requests in the waiting state have not yet been allocated video memory cache and do not occupy decoding batch processing resources.

[0059] S2: Obtain the current request status, queue status, GPU cache usage, and computing resource status from the underlying large language model during inference runtime, and calculate the current GPU memory pressure.

[0060] In a specific embodiment of the present invention, S2 is implemented as follows:

[0061] The service gateway obtains the currently executing pre-filled request set, the active request set in the decoding queue, the completed request set, and the corresponding video memory cache usage from the underlying large language model inference runtime; it also calculates the current system's used video memory cache, GPU computing budget, pre-filling stage budget, decoding stage budget, current pre-filling concurrency, and decoding batch processing usage; it calculates the video memory pressure based on the current used video memory cache and the total video memory cache capacity, and classifies the system state into low-pressure, medium-pressure, high-pressure, or critical-pressure states based on the range of video memory pressure.

[0062] S3: Calculate the memory shadow price based on the current memory pressure. The memory shadow price increases when the memory pressure increases. Dynamically allocate the pre-filling stage budget and the decoding stage budget. When the memory pressure increases, compress the computing power budget of the pre-filling stage and increase the computing power budget of the decoding stage.

[0063] In a preferred embodiment of the present invention, the video memory shadow price The formula for dynamically measuring the value of a unit of video memory resource in the current state is as follows:

[0064]

[0065] in Due to current memory pressure, and This is the water level threshold. This is a parameter for the increase in video memory price, which can be used to adjust the intensity of the impact of video memory pressure on request scheduling. When video memory pressure is low, the video memory shadow price remains at a base value of 1; as video memory pressure increases, its value gradually increases, reaching its maximum value when it reaches a high-water mark threshold. ). The larger the value, the faster the price of video memory shadows increases, and the more the scheduler prioritizes the cost of requesting video memory usage and the benefits of releasing video memory.

[0066] S3 dynamically allocates the pre-filling stage budget and the decoding stage budget based on the current memory pressure, including:

[0067] When the memory pressure is below the memory low watermark threshold, a higher proportion of the computational budget is allocated to the pre-filling stage.

[0068] When the memory pressure is between the low memory watermark threshold and the high memory watermark threshold, the prefill stage budget is gradually reduced and the decoding stage budget is increased as the current memory pressure increases.

[0069] When memory pressure reaches or exceeds the memory high-water mark threshold, new prefilling admissions are disabled, and computing resources are allocated to the decoding phase. If there are ongoing prefilling requests that have started but not yet completed, a minimum prefilling budget is reserved for these requests to complete their current prefilling work and enter the decoding queue. If there are no ongoing prefilling requests, the computing budget for the prefilling phase is reduced to zero.

[0070] S4: First, calculate the waiting queue priority for each request in the waiting queue. Then, traverse the candidate requests from highest to lowest priority and perform pre-filling admission control. If a candidate request passes all the restrictions of the pre-filling admission control, it is allowed to enter the pre-filling stage; otherwise, the request remains in the waiting queue.

[0071] In this embodiment, the waiting queue priority can be set to be positively correlated with the waiting time and negatively correlated with the pre-filling computation cost, the estimated decoding cost, and the memory cache holding cost weighted by the memory shadow price. The decoding queue priority can be set to be positively correlated with the decoding waiting time and the memory cache release benefit, and negatively correlated with the estimated remaining generation length. The above priority calculation method is only one optional implementation method, and those skilled in the art can also adjust the priority function according to business priority, service level, tenant weight, or request type.

[0072] In a specific embodiment of the present invention, S4 is implemented as follows:

[0073] For requests in the waiting queue, the service gateway calculates the waiting queue priority based on the length of the input prompt, the estimated output length, the waiting time, and the video memory shadow price; the calculation formula is as follows:

[0074]

[0075] Among them, service computing cost This is used to characterize the service computing resources that may be occupied after a request enters the system, and is related to the length of the request prompt and the estimated output length; video memory usage cost. The video memory shadow price is used to measure the video memory resources held after a request enters the system and the duration of that holding. Used to amplify memory cache holding costs under high memory pressure; latency Used to compensate for requests that have not been scheduled for a long time; α is the waiting compensation base, used to ensure that newly arriving requests have an initial priority and adjust the intensity of waiting time compensation; ε is a positive number smoothing term to prevent the denominator from being zero.

[0076] S5: Add requests that have completed the pre-filling phase to the decoding queue, and calculate the decoding queue priority for the requests in the decoding queue. Traverse the candidate requests in descending order of decoding queue priority, and perform decoding phase admission control. If a candidate request passes the decoding admission control, it is allowed to perform the decoding step; otherwise, the request remains in the decoding queue.

[0077] In a specific embodiment of the present invention, S5 is implemented as follows:

[0078] For requests in the decoding queue, the service gateway calculates the decoding queue priority based on the estimated remaining generation length of the request, the decoding wait time, and the benefit of releasing the video memory cache. The calculation formula is as follows:

[0079]

[0080] Among them, the estimated remaining generation length Used to characterize how close the request is to completion; decoding wait time Used to compensate for requests that have not received a decoding step for a long time; benefits from memory cache release. This is used to characterize the value of the video memory cache that can be released after a request is completed, and is expressed through the video memory shadow price. Weighting is applied. α is the waiting compensation base, used to ensure that newly entered requests in the decoding queue have an initial priority and to adjust the strength of waiting time compensation; ε is a smoothing term to prevent positive numbers with zero denominators.

[0081] The service gateway iterates through candidate requests in descending order of priority in the decoding queue and performs admission control for the decoding phase. If the current decoding phase has sufficient budget, the decoding batch size does not exceed the upper limit, and the addition of video memory cache after executing this decoding step will not cause the system's video memory usage to exceed the safety threshold, then the request is allowed to execute the decoding step; otherwise, the request remains in the decoding queue waiting for the next scheduling cycle.

[0082] S6: Submit the selected pre-filled requests and decoding requests to the underlying large language model inference runtime for execution, and update the request status, number of generated lexical units, GPU memory cache usage, queue status, and stage budget usage based on the execution results.

[0083] In a specific embodiment of the present invention, S6 is implemented as follows:

[0084] The service gateway submits the selected pre-filled requests and decoding requests to the underlying large language model inference runtime for execution;

[0085] For pre-filled requests, the underlying runtime processes the input prompts and generates corresponding prompt memory caches. For decoding requests, the underlying runtime generates one or more output tokens in an autoregressive manner and expands the memory cache corresponding to the request. The service gateway updates the number of tokens generated for the request, the current request stage, memory cache usage, stage budget usage, and queue status based on feedback from the underlying runtime. When a request reaches the actual output length or meets the stopping condition, the request is marked as completed and the memory cache it occupies is released.

[0086] S7: Release the video memory cache occupied by completed requests, and check whether the system has reached a critical pressure state and whether there has been no effective progress for several consecutive scheduling cycles. Effective progress includes one or more of the following: request completion, release of video memory cache, successful execution of the decoding step, decrease in video memory pressure, or effective change in queue status. If the triggering condition is met, the service gateway selects one or more low-priority or high-video-memory-occupancy requests as preempted requests, releases their occupied video memory cache, and puts them back into the waiting queue, into the restart queue, or marks them as needing to be re-executed; if the triggering condition is not met, the system status is updated and the next scheduling cycle begins.

[0087] Furthermore, as memory pressure continues to increase, this example employs a hierarchical congestion control mechanism (corresponding to step S3 of this invention), such as... Figure 4 The structure of a specific hierarchical congestion control mechanism is shown:

[0088] When memory pressure is low, the system allocates a higher proportion of the computational budget to the pre-filling stage, enabling new requests to enter the model inference process more quickly. When memory pressure increases, the system gradually compresses the pre-filling stage budget and increases the decoding stage budget to encourage existing active requests to continue generating and releasing memory cache. When memory pressure reaches the high watermark threshold, the system disables new pre-filling admissions to prevent new prompt words from being injected into the memory cache.

[0089] Under high-water level protection, if there are ongoing pre-filling requests in the system that have started but not yet completed, the resource management module reserves a small amount of backup budget for them to complete the current pre-filling work and enter the decoding queue. By completing the pre-filling in progress, it is possible to avoid pre-filling requests that have started remaining in a half-finished state for a long time, thereby reducing resource stagnation.

[0090] When the system's video memory pressure reaches a critical threshold and no effective progress is achieved for several consecutive scheduling cycles, gateway-side preemption recovery is triggered. Effective progress includes one or more of the following: request completion, release of video memory cache, successful execution of the decoding step, decrease in video memory pressure, or a valid change in queue status. After gateway-side preemption recovery is triggered, the service gateway selects the preempted request based on one or more factors: request priority, current video memory cache usage, number of generated tokens, expected restart cost, or business priority. The video memory cache occupied by the preempted request is released, and the request can be re-added to the waiting queue, added to the restart queue, or marked as needing to be re-executed. In this way, the system can release part of the video memory cache under extreme video memory pressure and continuous no-progress conditions, restoring execution space for subsequent requests.

[0091] In this embodiment, the memory shadow price also makes a significant contribution to controlling memory pressure. On one hand, it amplifies the memory cache holding cost of requests in the waiting queue, making high memory usage requests less likely to be accepted under high pressure. On the other hand, it amplifies the memory cache release benefit after requests in the decoding queue are completed, making requests that can release more memory cache more likely to get decoding opportunities under high pressure. The memory shadow price can be determined based on the current memory pressure relative to the low-water mark threshold and the high-water mark threshold. For example, when the memory pressure is below the low-water mark threshold, the memory shadow price takes the base value; when the memory pressure is between the low-water mark threshold and the high-water mark threshold, the memory shadow price increases as the memory pressure increases; when the memory pressure is close to the high-water mark threshold, the memory shadow price reaches a relatively high level.

[0092] To verify the technical effectiveness of the resource scheduling method described in this invention, a discrete-time simulation environment for large language model inference services was constructed to simulate the processes of request arrival, pre-filling execution, decoding execution, and dynamic growth and release of video memory cache.

[0093] Experiment 1 was used to verify the effect of request scheduling. Simulation experiments were conducted using the First-Come, First-Served (FIFO) method, the Shortest Task First (SJF) method, and the method of this invention. The experimental results are shown in Table 1. The average TTFT (first-term delay) and P95 TTFT of the method of this invention are 346.39 and 1953.90 scheduling cycles, respectively, which are reduced by approximately 69.66% and 19.12% compared with the FIFO method, and by approximately 24.03% and 8.25% compared with the SJF method. The average completion delay and P95 completion delay of the method of this invention are 759.73 and 2683.20 scheduling cycles, respectively, which are reduced by approximately 47.57% and 9.91% compared with the FIFO method, and by approximately 2.55% and 2.22% compared with the SJF method. The above results show that the present invention can effectively alleviate head-of-line blocking caused by long requests and improve the first response performance and overall completion delay of requests.

[0094] Table 1: Comparison of performance metrics for different scheduling methods

[0095]

[0096] Experiment 2 was used to verify the effectiveness of hierarchical memory congestion control. Simulation experiments were conducted using a scheduling scheme without hierarchical memory control and the method described in this invention. The experimental results are shown in Table 2. This invention reduced the cumulative time (number of high-water level cycles) when memory utilization exceeded the high-water level threshold from 1407 to 326, and reduced the number of decoding blockages due to insufficient memory from 20779 to 3591. Without memory congestion control, the system could only complete 36.4% of requests; with it enabled, all requests were completed. These results demonstrate that dynamic adjustment of the stage budget, high-water level fencing, and preemption recovery mechanism can form a progressive memory congestion control, improving system stability.

[0097] Table 2: Comparison of memory pressure indicators with and without hierarchical congestion control enabled

[0098]

[0099] In summary, simulation results show that the resource scheduling method described in this invention can improve the performance of the first response to a request, reduce continuous high pressure on the video memory and decoding blocking, and improve the efficiency of video memory resource utilization and the continuous service capability under high load conditions.

[0100] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made by those skilled in the art to the technical solutions of the present invention without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A resource scheduling method for large model inference services, characterized in that, include: S1: Receive user inference requests, parse the request metadata of the inference requests, and add the requests to the waiting queue; S2: Obtain the current request status, queue status, video memory cache usage, and GPU computing resource status during the runtime of the large language model inference, and calculate the current video memory pressure; S3: Calculate the memory shadow price based on the current memory pressure; dynamically allocate the pre-filling stage budget and the decoding stage budget based on the current memory pressure; S4: Calculate the waiting queue priority based on the prompt word length, waiting time and memory cost of the requests in the waiting queue, and select the requests allowed to enter the pre-filling stage in combination with the admission conditions of the pre-filling stage; S5: Add requests that have completed the pre-filling phase to the decoding queue. Calculate the decoding queue priority based on the estimated remaining generation length, decoding wait time, and VRAM cache release benefits of the requests in the decoding queue. Combine this with the decoding phase admission criteria to select requests that are allowed to execute the decoding step. S6: Submit the selected pre-filled requests and decoding requests to the underlying large language model inference runtime for execution, and update the request status, the number of generated lexical units, and the memory cache usage based on the execution results; S7: Release the video memory cache occupied by completed requests, and perform high watermark protection, prefilling completion, or gateway-side preemption recovery when the system reaches high video memory pressure or is in a state of continuous no progress.

2. The method according to claim 1, characterized in that, The request metadata in S1 includes the request arrival time, input prompt word length, maximum output length upper bound, estimated output length, current request stage, and number of generated lexical units; Newly arrived requests first enter a waiting state and are added to the waiting queue. Requests in the waiting state have not yet been allocated video memory cache and do not occupy decoding batch processing resources.

3. The method according to claim 1, characterized in that, Specifically, S2 is: The service gateway obtains the currently executing pre-filled request set, the active request set in the decoding queue, the completed request set, and the GPU cache usage corresponding to each request from the underlying large language model inference runtime. The system calculates the current amount of video memory cache used, GPU computing budget, pre-filling stage budget, decoding stage budget, current pre-filling concurrency, and decoding batch processing usage. Based on the current amount of video memory cache used and the total video memory cache capacity, the system state is divided into low-pressure, medium-pressure, high-pressure, or critical-pressure states according to the range of video memory pressure.

4. The method according to claim 1, characterized in that, In S3, the calculation of the video memory shadow price based on the current video memory pressure includes: Set the low watermark threshold and high watermark threshold for video memory; When the memory pressure is lower than the memory low water level threshold, the memory shadow price is set to the base value. When the memory pressure is between the low memory water level threshold and the high memory water level threshold, the memory shadow price increases as the current memory pressure increases. When the memory pressure reaches or exceeds the memory high water level threshold, the memory shadow price takes the maximum value.

5. The method according to claim 1, characterized in that, In S3, the dynamic allocation of the pre-filling stage budget and the decoding stage budget based on the current memory pressure includes: When the memory pressure is below the memory low watermark threshold, a higher proportion of the computational budget is allocated to the pre-filling stage. When the memory pressure is between the low memory watermark threshold and the high memory watermark threshold, the prefill stage budget is gradually reduced and the decoding stage budget is increased as the current memory pressure increases. When the memory pressure reaches or exceeds the memory high watermark threshold, new prefilling admissions are disabled, and computing resources are tilted toward the decoding stage. If there are running prefilling requests that have started but not yet completed, a prefilling reserve budget is reserved for the running prefilling requests so that they can complete the current prefilling work and enter the decoding queue. If there are no running prefilling requests, the computing budget for the prefilling stage is reduced to zero.

6. The method according to claim 1, characterized in that, Specifically, S4 is: For requests in the waiting queue, the service gateway calculates the waiting queue priority based on the length of the input prompt, the estimated output length, the waiting time, and the video memory shadow price. The waiting queue priority is positively correlated with the waiting time of the request, negatively correlated with the pre-filling computation cost and decoding cost of the request, and negatively correlated with the video memory cache holding cost weighted by the video memory shadow price. The service gateway iterates through the candidate requests in descending order of priority in the waiting queue and performs pre-population phase admission control on the candidate requests. If new prefill admissions are allowed, the prefill phase budget is sufficient, the memory cache usage after accepting the request does not exceed the memory safety threshold, and the current prefill concurrency does not exceed the maximum concurrency limit, then the request is allowed to enter the prefill phase; otherwise, the request remains in the waiting queue awaiting the next scheduling cycle.

7. The method according to claim 1, characterized in that, Specifically, S5 is: For requests in the decoding queue, the service gateway calculates the decoding queue priority based on the estimated remaining generation length of the request, the decoding wait time, and the benefit of releasing the video memory cache. The decoding queue priority is positively correlated with the decoding wait time and the benefit of releasing the video memory cache, and negatively correlated with the estimated remaining generation length of the request. The benefit of releasing the video memory cache is weighted by the video memory shadow price. The service gateway traverses candidate requests from high to low priority in the decoding queue and performs admission control for the candidate requests during the decoding phase. If the current decoding stage has sufficient budget, the decoding batch size does not exceed the upper limit, and the addition of video memory cache after executing this decoding step will not cause the system video memory usage to exceed the safe threshold, then the request is allowed to execute the decoding step. Otherwise, the request remains in the decoding queue awaiting the next scheduling cycle.

8. The method according to claim 1, characterized in that, Specifically, S6 is: The service gateway submits the selected pre-filled requests and decoding requests to the underlying large language model inference runtime for execution; For pre-filled requests, the underlying runtime processes the input prompts and generates the corresponding prompt memory cache; for decoded requests, the underlying runtime generates one or more output tokens in an autoregressive manner and expands the memory cache corresponding to the request. Based on feedback from the underlying runtime, the service gateway updates the number of generated tokens, the current request stage, the amount of video memory cache used, the stage budget usage, and the queue status. When a request reaches the actual output length or meets the stopping conditions, it marks the request as completed and releases the video memory cache it occupies.

9. The method according to claim 1, characterized in that, Specifically, S7 is: When the system memory pressure reaches the high watermark threshold, the high watermark protection mechanism is activated, and new prefilling admission is disabled, so that new requests in the waiting queue cannot enter the prefilling stage temporarily, thereby preventing new prompt words from being injected into the memory cache. Under the high watermark protection state, for running prefilling requests that have started but have not yet been completed, the system reserves a small amount of backup budget to enable them to complete the current prefilling work and enter the decoding queue, so as to avoid running prefilling requests being in a half-completed state for a long time. When the system memory pressure reaches a critical threshold and no effective progress is made in multiple consecutive scheduling cycles, the gateway-side preemption recovery is triggered. The effective progress includes one or more of the following: request completion, memory cache release, successful execution of the decoding step, memory pressure reduction, or effective change in queue status. The service gateway selects the preempted request based on one or more of the following factors: request priority, current memory cache usage, number of generated tokens, expected restart cost, or business priority. It releases the memory cache occupied by the preempted request and puts the preempted request back into the waiting queue, the restart queue, or marks it as needing to be re-executed.

10. A resource scheduling system for large language model inference services, used to implement the method according to any one of claims 1-9, characterized in that, include: The waiting queue module is used to store requests that have arrived but have not yet entered the pre-population stage; The decoding queue module is used to store active requests that have completed the pre-filling stage and are waiting to continue the decoding step; The lexical awareness scheduling module is used to calculate the priority of the waiting queue and the priority of the decoding queue, respectively. The phase admission control module is used to determine whether a candidate request can enter the pre-filling phase or execute the decoding step based on the current resource status. The resource management module is used to maintain GPU computing budget, memory cache usage, memory pressure, pre-filling stage budget, decoding stage budget, number of active requests, decoding batch processing usage, memory level status, and request completion feedback. The underlying inference runtime interface module is used to submit pre-filling requests and decoding requests to the large language model inference runtime, and to receive the request execution status and memory cache usage update information returned by the large language model inference runtime.

Images