A deep learning inference request scheduling method and system

Abstract

The application discloses a kind of deep learning inference request scheduling method and system, belong to deep learning inference scheduling technical field.The application is to solve the technical problems that long and short request mix in shared GPU deep learning inference service causes short request head mark delay to rise, fixed pre-filling block is difficult to adapt to runtime competition state, default scheduling rule cannot effectively utilize return scheduling boundary, by obtaining the runtime state data of active inference request set, determine target long pre-filling request and runtime load pressure signal, determine next continuous execution upper limit in combination with inference execution cost estimation data corresponding to candidate continuous execution upper limit, and select benefit request to next execution window after scheduler returns control right.The application can reduce the tail of short request and overall request Head mark delay.

Description

Technical Field

[0001] This invention belongs to the field of deep learning inference scheduling technology, specifically relating to a deep learning inference request scheduling method and system. Background Technology

[0002] In recent years, deep learning models, especially large language models, have been widely applied to intelligent question answering, text generation, code assistance, search recommendation, and multimodal services. These inference services typically run on accelerated devices such as GPUs. To reduce deployment costs and improve hardware utilization, multiple user requests, multiple model variants, or multiple adapter tasks are often placed in the same inference runtime for shared execution. In practical services, inference requests exhibit significant heterogeneity. One type of request has short inputs, and users expect to quickly obtain the first output result; another type of request carries long context or long prompts, and its pre-filling stage requires processing a large number of input tokens, potentially occupying the GPU execution window for a considerable period. For autoregressive generative models, the arrival time of the first output token directly affects the user's perceived response speed. Therefore, in a shared GPU inference environment, how to reduce the waiting time of short requests after long pre-filling requests is a crucial issue affecting service quality and resource efficiency.

[0003] Existing inference runtimes typically employ mechanisms such as continuous batch processing, iterative scheduling, key-value caching, and fixed-size pre-filled chunks to improve GPU utilization and increase the scheduler's re-decision opportunities. The default continuous batch processing strategy merges newly arriving requests into running batches to improve GPU throughput. However, when long and short requests are mixed, batch composition is often determined by request arrival and runtime default rules, lacking targeted control over the waiting state of short requests. Fixed-size pre-filled chunks divide long prompt processing into several fixed-length intervals, allowing the runtime to reschedule other requests between intervals. However, chunk size is usually configured as a static parameter, making it difficult to dynamically adjust based on runtime states such as current queue length, short request waiting time, and remaining workload of long requests. Preemption, task priority, or resource separation strategies can alleviate blocking issues through stronger execution control, but may require modifications to the underlying execution path, the introduction of additional resource pools, or the incurred overhead of cache migration, state recovery, and additional synchronization. For cost-sensitive scenarios such as single-GPU or small-GPU deployments, departmental private services, and hybrid adapter inference, direct adoption is not always suitable.

[0004] Existing technologies still have shortcomings in scheduling deep learning inference requests on shared GPUs. First, the timing of scheduling boundaries is difficult to dynamically determine based on the current contention state. Returning control too late can cause long pre-filled requests to continuously occupy the execution window, while returning control too early may lead to excessive block splitting and additional scheduling overhead. Second, while fixed block splitting or default batching rules can provide more scheduling boundaries, they cannot determine which requests should be placed in the next execution window after returning scheduling control. This results in short requests, requests with long waiting times, and requests with remaining pre-filling workload suitable for the current boundary not being consistently prioritized, potentially leading to significant tail-heading delays. Third, the decision-making time available for each scheduling operation during online inference is limited. Scheduling methods should not rely on complex global searches or future request predictions but should instead make rapid decisions based on the current active request state, lightweight execution cost estimation, and a bounded candidate set. Therefore, existing methods struggle to simultaneously achieve runtime adaptive control of the continuous execution boundaries of long pre-filled requests, selection of beneficial requests after returning to the scheduling boundary, and low online scheduling overhead. Summary of the Invention

[0005] The purpose of this invention is to propose a deep learning inference request scheduling method and system to solve the technical problems in shared GPU deep learning inference services, such as the increased latency of short request first mark caused by the mixture of long and short requests, the difficulty of fixed pre-filled blocks adapting to runtime competition, and the inability of default scheduling rules to effectively utilize the return scheduling boundary. The invention aims to achieve the effect of dynamically controlling the execution boundary of long pre-filled requests and optimizing the placement of requests in the next execution window.

[0006] To achieve the above objectives, the present invention adopts the following technical solution.

[0007] A deep learning inference request scheduling method includes the following steps: At the scheduling boundary of the deep learning inference runtime, obtain runtime state data of the active inference request set; Based on the runtime state data, a target long pre-filled request is determined from the active inference request set, and a runtime load pressure signal is generated; Based on the runtime load pressure signal and the inference execution cost estimation data corresponding to the candidate continuous execution limit, determine the next continuous execution limit of the target long pre-filled request; Write the next consecutive execution limit as a temporary control limit into the scheduler state, so that the target long pre-fill request returns to the scheduler after execution reaches the temporary control limit; After the scheduler returns control, a set of candidate benefit requests is constructed from the set of active inference requests based on the scheduling boundary formed by the most recent continuous execution limit. Based on the runtime state data and the adaptation relationship between each candidate benefit request and the most recent continuous execution limit, the benefit score of each candidate benefit request is determined. Based on the benefit score, a benefit request is selected from the set of candidate benefit requests. The benefit request is then placed in the next execution window during the deep learning inference runtime. Unselected requests are then placed with bounded delay or bounded supplementation.

[0008] Furthermore, at the scheduling boundary of the deep learning inference runtime, runtime state data of the active inference request set is obtained, including: When preparing to form an execution window during deep learning inference runtime, read the request status information of each active inference request in the active inference request set; The request status information is collected to obtain the runtime status data, which includes the remaining pre-filled workload, waiting status data, and queue load status data.

[0009] Furthermore, the request status information of each active inference request in the active inference request set is read, including: Read the arrival time, waiting time, remaining pre-filled workload, and scheduled workload of each active inference request; The request status information is obtained by reading the request stage status and request attribute information of each active inference request.

[0010] Further, based on the runtime state data, a target long pre-filled request is determined from the active inference request set, and a runtime load pressure signal is generated, including: Based on the remaining pre-filling workload in the runtime state data, determine the long pre-filling request set from the active inference request set; The target long pre-fill request is determined based on the remaining pre-fill workload and waiting status data of each long pre-fill request in the long pre-fill request set. The runtime load pressure signal is generated based on the runtime state data, the long pre-fill request set, and the target long pre-fill request.

[0011] Further, based on the runtime state data, the long pre-filled request set, and the target long pre-filled request, the runtime load pressure signal is generated, including: The urgency of short requests is determined based on the number of waiting short requests, the maximum waiting time, and the workload of short requests in the runtime status data. The execution load pressure is determined based on the queue length in the runtime status data and the remaining pre-filling workload of the target length pre-filling request; The runtime load pressure signal is generated based on the urgency of the short request and the execution load pressure.

[0012] Further, based on the runtime state data, the long pre-filled request set, and the target long pre-filled request, the runtime load pressure signal is generated, including: Based on the request attribute information in the runtime state data, determine the heterogeneous load status of the active inference request set within the execution window; The execution load pressure is determined based on the heterogeneous load status, the queue load status data in the runtime status data, and the remaining pre-filling workload of the target long pre-filling request. The runtime load pressure signal is generated based on the execution load pressure.

[0013] Further, based on the runtime load pressure signal and the inference execution cost estimation data corresponding to the candidate consecutive execution upper limit, the next consecutive execution upper limit of the target long pre-filled request is determined, including: Based on the runtime load pressure signal, the target block size corresponding to the target length pre-fill request is generated; The candidate continuous execution limit is determined based on the target block size, and the inference execution cost estimation data corresponding to the candidate continuous execution limit is obtained; Based on the inference execution cost estimation data, the next consecutive execution limit for the target long pre-filled request is determined from the candidate consecutive execution limits.

[0014] Further, obtaining the inference execution cost estimation data corresponding to the candidate continuous execution upper limit includes: Based on the candidate continuous execution limit and the runtime state data, match the corresponding execution cost data from a predetermined cost estimation source; Based on the execution cost data, inference execution cost estimation data corresponding to the candidate continuous execution upper limit is generated. The cost estimation source includes at least one of the following: bucket profile table, online sliding statistics table, regression prediction model, or hardware counter estimation model.

[0015] Further, based on the runtime load pressure signal and the inference execution cost estimation data corresponding to the candidate consecutive execution upper limit, the next consecutive execution upper limit of the target long pre-filled request is determined, including: Based on the runtime load pressure signal, determine the candidate block size from the candidate block size set; The upper limit for continuous execution of candidates is determined based on the candidate block size and the remaining pre-filling workload of the target long pre-filling request; The next consecutive execution limit is determined based on the inference execution cost estimation data corresponding to the candidate consecutive execution limit.

[0016] Furthermore, writing the next consecutive execution limit as a temporary control limit into the scheduler state, so that the target long pre-fill request returns to the scheduler after reaching the temporary control limit, includes: Associate the next consecutive execution limit with the request identifier of the target long pre-filled request to generate a temporary control state; The temporary control state limits the amount of pre-filling work that the target long pre-filling request can continuously advance within the execution window, so that the target long pre-filling request returns to the scheduler after being executed to the temporary control limit.

[0017] Further, after the scheduler returns control, a candidate benefit request set is constructed from the active inference request set based on the scheduling boundary formed by the most recent continuous execution limit, including: The boundary adaptation range is determined based on the scheduling boundary formed by the most recent continuous execution limit. The target long pre-filled request and requests that do not meet the pre-filled state conditions are excluded from the active inference request set to obtain the request set to be filtered. The set of requests to be filtered is filtered according to the boundary adaptation range to obtain the set of candidate benefit requests.

[0018] Furthermore, based on the scheduling boundary formed by the most recent consecutive execution limit, the boundary adaptation range is determined, including: The size of the most recent long pre-filled block is determined based on the most recent consecutive execution limit. The boundary adaptation range is determined based on the most recent long pre-filled block size and the preset boundary expansion conditions.

[0019] Further, based on the runtime state data and the adaptation relationship between each candidate benefit request and the most recent continuous execution limit, the benefit score of each candidate benefit request is determined, including: Based on the waiting status data in the runtime status data, determine the waiting priority of each candidate benefit request; The boundary fit of each candidate benefit request is determined based on the fit relationship between the remaining pre-filled workload of each candidate benefit request and the upper limit of the most recent continuous execution. Based on the waiting priority and the boundary fit, the benefit score of each candidate benefit request is determined.

[0020] Further, based on the waiting priority and the boundary fit, a benefit score is determined for each candidate benefit request, including: Based on the runtime load pressure signal, determine the scoring weights corresponding to the waiting priority and the boundary adaptability; Based on the scoring weights, the waiting priority and the boundary fit are fused to obtain the benefit score for each candidate benefit request.

[0021] Further, based on the benefit score, a benefit request is selected from the set of candidate benefit requests, and the benefit request is preferentially placed in the next execution window during the deep learning inference runtime. Bounded delay or bounded supplementary placement is performed on unselected requests, including: Based on the benefit score, the candidate benefit requests in the candidate benefit request set are sorted, and the candidate benefit requests that meet the benefit selection criteria are selected as benefit requests. The benefit request will be written first into the next execution window during the deep learning inference runtime; Based on the available capacity of the next execution window, bounded deferral or bounded supplementary placement is performed on unselected requests.

[0022] Furthermore, selecting candidate benefit requests that meet the benefit selection criteria as benefit requests includes: Adjust the benefit selection criteria based on the runtime load pressure signal; Based on the adjusted benefit selection criteria and the benefit score, the benefit request is selected from the set of candidate benefit requests.

[0023] Further, based on the available capacity of the next execution window, bounded deferral or bounded supplementary placement is performed on unselected requests, including: When the next execution window has available capacity, a supplementary placement request is determined from the requests that have never been selected, based on the supplementary placement conditions; Write the supplementary placement request into the next execution window, and determine the remaining unselected requests as temporarily postponed requests; After the next execution window is completed, clear the temporary preference state corresponding to the temporary postponement request.

[0024] A deep learning inference request scheduling system includes: The request status acquisition module is used to acquire runtime status data of the active inference request set at the scheduling boundary during deep learning inference runtime. The load pressure generation module is used to determine the target long pre-fill request from the active inference request set based on the runtime state data, and generate a runtime load pressure signal. The block upper limit determination module is used to determine the next continuous execution upper limit of the target long pre-filled request based on the runtime load pressure signal and the inference execution cost estimation data corresponding to the candidate continuous execution upper limit; The scheduling status writing module is used to write the next consecutive execution limit as a temporary control limit into the scheduler status, so that the target long pre-fill request returns to the scheduler after execution reaches the temporary control limit; The candidate set construction module is used to construct a candidate benefit request set from the active inference request set based on the scheduling boundary formed by the most recent continuous execution limit after the scheduler returns control. The benefit score determination module is used to determine the benefit score of each candidate benefit request based on the runtime state data and the adaptation relationship between each candidate benefit request and the most recent continuous execution limit. The execution window placement module is used to select a benefit request from the candidate benefit request set according to the benefit score, place the benefit request in the next execution window during the deep learning inference runtime, and perform bounded delay or bounded supplementary placement for the unselected requests.

[0025] The present invention has achieved the following beneficial effects.

[0026] 1. This invention generates runtime load pressure signals based on runtime status data and determines the next consecutive execution limit of the target long pre-filled request by combining inference execution cost estimation data. It can dynamically adjust the consecutive execution boundary of the long pre-filled request based on the current queue length, the waiting status of short requests, and the remaining workload of long requests, thereby reducing the waiting time of short requests caused by long requests continuously occupying the execution window.

[0027] 2. This invention divides the scheduling process into boundary timing control for long pre-filled requests and boundary usage control after the scheduler returns. First, the target long pre-filled request is returned to the scheduler at the temporary control upper limit, and then the request placement is performed based on the scheduling boundary after the return. This can avoid the problem that the fixed block method is difficult to match the runtime competition state.

[0028] 3. After the scheduler returns to control, the present invention determines the benefit score based on runtime state data and the adaptation relationship between candidate benefit requests and the most recent continuous execution limit, and prioritizes the placement of benefit requests in the next execution window. This can identify requests that are more likely to generate first-mark benefits and improve the utilization efficiency of returning to the scheduling boundary.

[0029] 4. This invention performs bounded postponement or bounded supplementary placement on unselected requests, so that low-benefit requests are only temporarily postponed within the current execution window, and limited supplementary placement is allowed when the execution window still has capacity. This can optimize the delay of short request first marking while controlling scheduling bias and additional execution overhead.

[0030] 5. This invention can be used as a scheduler-side solution to access continuous batch processing inference runtime without changing the underlying model execution operators, key-value cache management, adapter execution, and GPU kernel execution paths. It can reduce system integration costs and is applicable to shared GPU, long and short request mixed, and mixed adapter inference scenarios. Attached Figure Description

[0031] Figure 1 This is a flowchart of the deep learning inference request scheduling method in an embodiment of the present invention; Figure 2 This is a flowchart illustrating the offline portrait table construction process in an embodiment of the present invention. Figure 3 This is a flowchart illustrating the overall online scheduling process in this embodiment of the invention. Figure 4 This is a schematic diagram of the parameter selection process for long pre-filled blocks in an embodiment of the present invention; Figure 5 This is a schematic diagram illustrating the process of selecting the benefit request and placing the next execution window in an embodiment of the present invention; Figure 6 This is an architecture diagram of a deep learning inference request scheduling system in an embodiment of the present invention. Detailed Implementation

[0032] To make the various technical features, advantages, or effects of the present invention more apparent and understandable, detailed descriptions are provided below through embodiments.

[0033] This invention provides a deep learning inference request scheduling method. This method can be applied to shared GPU inference services, cloud inference services, edge inference services, private large model services, or multi-tenant adapter services, and can also be applied to single-GPU, small-scale GPU, or multi-instance GPU services. The inference service can run large language models (LLM), visual language models, or other autoregressive generative models. This method can be integrated into the scheduler side of continuous batch inference runtime, with the scheduling boundary being the scheduling decision point before the next batch or the next execution window is formed during inference runtime. In large language model inference tasks, the remaining pre-filling workload can be the number of remaining pre-filled tags. This method involves processing steps such as request status acquisition, cost estimation, runtime load pressure calculation, long pre-filling block control, benefit request selection, and execution window placement, etc. Figure 1 As shown.

[0034] Before performing online scheduling, a cost estimation source can be determined based on the target inference model, target GPU model, inference runtime version, and candidate request length buckets. The cost estimation source can be a bucket profile table, the offline preparation process of which is as follows: Figure 2As shown, the bucketing profile table includes at least a table of pre-filled time for long requests, a table of service time for short requests under candidate block sizes, and a table of overhead for long pre-filled blocks. Specifically, pre-filling profiles are performed on the target inference model, recording the time spent executing long requests of different lengths individually to form a table of pre-filled time for long requests; short requests are measured in combination with different candidate block sizes, recording the serviceable time for short requests at different candidate block boundaries to form a table of service time for short requests; the additional cost when long requests are segmented or truncated into different consecutive execution limits is measured to form a table of overhead for long pre-filled blocks; and the request length, candidate block size, and execution time are mapped to discrete buckets and saved as the bucketing profile table. The bucketing profile table can be reconstructed for different GPU models or different inference runtime versions.

[0035] The online scheduling process in this embodiment is as follows: Figure 3 As shown. When the inference runtime is preparing to form the next execution window, if there are fewer than two active inference requests, or if there are no requests with remaining pre-filled workload greater than the preset long request judgment threshold, the original inference runtime scheduling strategy can be maintained; if the entry condition is met, steps S1 to S7 are executed.

[0036] Step S1: At the scheduling boundary of the deep learning inference runtime, obtain the runtime state data of the active inference request set.

[0037] Specifically, runtime state data may include the remaining pre-filled workload, waiting time, arrival time and current queue length for each active inference request, and may also include the number of short requests waiting, maximum waiting time, total number of short request tags and remaining workload for long requests.

[0038] In an optional embodiment of the present invention, step S1 may include: step S11, when preparing to form an execution window during deep learning inference runtime, reading the request status information of each active inference request in the active inference request set.

[0039] Specifically, the request status information may include the request identifier, arrival time, number of input tokens, number of remaining pre-filled tokens, whether it is a short request, adapter information, and number of scheduled tokens.

[0040] Step S12: Collect the request status information to obtain the runtime status data, which includes the remaining pre-filled workload, waiting status data, and queue load status data.

[0041] Specifically, waiting status data can include waiting time, arrival time, number of short requests waiting, and maximum waiting time. Queue load status data can include current queue length, total number of short request tags, and remaining workload for long requests.

[0042] In an optional embodiment of the present invention, step S11 may include: step S111, reading the arrival time, waiting time, remaining pre-filled workload, and scheduled workload of each active inference request.

[0043] Step S112: Read the request stage status and request attribute information of each active inference request to obtain the request status information.

[0044] Specifically, the request stage status can be used to distinguish whether a request is in the pre-filling stage or the decoding stage. Request attribute information may include the number of input tokens, whether it is a short request, and adapter information. In a hybrid adapter inference scenario, adapter information may include adapter rank, adapter type, or tenant identifier.

[0045] Step S2: Based on the runtime state data, determine the target long pre-fill request from the active inference request set and generate a runtime load pressure signal.

[0046] Specifically, the target long pre-filled request is one of the requests whose remaining pre-filled workload exceeds a preset long request threshold. Runtime load pressure signals can include short request urgency and execution load pressure.

[0047] In an optional embodiment of the present invention, step S2 may include: step S21, determining a long pre-fill request set from the active inference request set based on the remaining pre-fill workload in the runtime state data.

[0048] Specifically, the set of long pre-filled requests can consist of requests whose remaining pre-filled workload is greater than a preset long request threshold. The representative set of short requests can consist of requests whose remaining pre-filled workload is less than a certain proportion of the remaining pre-filled workload of long pre-filled requests.

[0049] Step S22: Determine the target long pre-fill request based on the remaining pre-fill workload and waiting status data of each long pre-fill request in the long pre-fill request set.

[0050] Specifically, when there are multiple requests that meet the conditions in the long pre-fill request set, the request with the largest remaining pre-fill workload, the longest waiting time, or the highest weighted value of the two can be selected as the target long pre-fill request.

[0051] Step S23: Generate the runtime load pressure signal based on the runtime state data, the long pre-fill request set, and the target long pre-fill request.

[0052] In an optional embodiment of the present invention, step S23 may include: step S231, determining the urgency of short requests based on the number of waiting short requests, the maximum waiting time, and the workload of short requests in the runtime state data.

[0053] Specifically, the urgency of short requests can be obtained by weighting a factor of the number of waiting short requests, a factor of the maximum waiting time, and a factor of the total number of short request tags, and is limited to a preset range. In one specific implementation, the urgency of short requests can be expressed as: in, Indicates the urgency of short requests; This represents the normalization factor for the number of short requests waiting; This represents the normalization factor for the maximum waiting time; This represents the normalized ratio between the total number of short request tags and the remaining pre-filling workload of the target long pre-filled request; , , Indicates the preset weight; This means that the results will be restricted to a preset range.

[0054] Step S232: Determine the execution load pressure based on the queue length in the runtime status data and the remaining pre-filling workload of the target length pre-filling request.

[0055] Specifically, the execution load pressure can be obtained by weighting the queue length factor and the remaining pre-filling workload factor of the target length pre-filling request. In one specific implementation, the execution load pressure can be expressed as: in, Indicates the execution load pressure; This represents the queue length normalization factor; The normalization factor representing the remaining pre-filling workload of the target long pre-filling request; , This indicates the preset weight.

[0056] Step S233: Generate the runtime load pressure signal based on the short request urgency and the execution load pressure.

[0057] Specifically, the execution load pressure can be reduced based on the urgency of short requests to obtain the overall scheduling pressure, and the short request urgency, execution load pressure, and overall scheduling pressure can be used as the runtime load pressure signal. In one specific implementation, the overall scheduling pressure can be expressed as: in, This indicates overall scheduling pressure; Indicates the execution load pressure; Indicates the urgency of short requests; This represents the reduction factor.

[0058] In another optional embodiment of the present invention, step S23 may also include: step S231, determining the heterogeneous load state of the active inference request set within the execution window based on the request attribute information in the runtime state data.

[0059] Specifically, the heterogeneous load status can be determined based on the differences in input length, adapter rank, adapter type, or tenant identifier of the request within the execution window.

[0060] Step S232: Determine the execution load pressure based on the heterogeneous load status, the queue load status data in the runtime status data, and the remaining pre-filling workload of the target long pre-filling request.

[0061] Step S233: Generate the runtime load pressure signal based on the execution load pressure.

[0062] Step S3: Determine the next consecutive execution limit of the target long pre-filled request based on the runtime load pressure signal and the inference execution cost estimation data corresponding to the candidate consecutive execution limit.

[0063] Specifically, step S3 corresponds to Figure 4 The diagram illustrates the long pre-filled block parameter selection process. Inference execution cost estimation data can include latency benefit estimates and block cost estimates corresponding to the candidate consecutive execution cap.

[0064] In an optional embodiment of the present invention, step S3 may include: step S31, generating the target block size corresponding to the target length prefill request based on the runtime load pressure signal.

[0065] Specifically, the target block size can be determined between small block parameters and large block parameters based on the overall scheduling pressure. In one specific implementation, the target block size can be expressed as: in, Indicates the target block size; Indicates the parameters of the small blocks; Indicates the parameters of the large block; This indicates the overall scheduling pressure.

[0066] Step S32: Determine the candidate continuous execution limit based on the target block size, and obtain the inference execution cost estimation data corresponding to the candidate continuous execution limit.

[0067] Specifically, the target block size can be rounded down to the nearest value in the candidate block size set to obtain the candidate block size, and the upper limit of candidate continuous execution can be determined based on the candidate block size.

[0068] Step S33: Based on the inference execution cost estimation data, determine the next consecutive execution limit of the target long pre-filled request from the candidate consecutive execution limits.

[0069] Specifically, when the estimated delay benefit corresponding to the candidate continuous execution limit meets the preset benefit condition and the estimated block cost does not exceed the preset limit, the candidate continuous execution limit can be determined as the next continuous execution limit for the target long pre-filled request.

[0070] In an optional embodiment of the present invention, step S32 may include: step S321, matching the corresponding execution cost data from a predetermined cost estimation source based on the candidate continuous execution limit and the runtime state data.

[0071] Specifically, when the cost estimation source is the bucket profile table, the online scheduling phase can query the bucket profile table based on the request length and candidate block size to obtain the execution cost data corresponding to the candidate continuous execution limit.

[0072] Step S322: Based on the execution cost data, generate inference execution cost estimation data corresponding to the candidate continuous execution upper limit. The cost estimation source includes at least one of the following: bucket profile table, online sliding statistics table, regression prediction model, or hardware counter estimation model.

[0073] Specifically, online sliding statistics tables, regression prediction models, or hardware counter estimation models can replace bucket profile tables to provide comparable execution cost estimates for candidate blocks and candidate execution windows.

[0074] In another optional embodiment of the present invention, step S3 may also include: step S31, determining the candidate block size from the candidate block size set based on the runtime load pressure signal.

[0075] Step S32: Determine the upper limit of candidate continuous execution based on the candidate block size and the remaining pre-filling workload of the target long pre-filling request.

[0076] Step S33: Determine the next consecutive execution limit based on the inference execution cost estimation data corresponding to the candidate consecutive execution limit.

[0077] Step S4: Write the next consecutive execution limit as a temporary control limit into the scheduler state, so that the target long pre-fill request returns to the scheduler after execution reaches the temporary control limit.

[0078] In an optional embodiment of the present invention, step S4 may include: step S41, associating the next consecutive execution limit with the request identifier of the target long pre-filled request to generate a temporary control state.

[0079] Step S42: Based on the temporary control state, limit the amount of pre-filling work that the target long pre-filling request can continuously advance within the execution window, so that the target long pre-filling request returns to the scheduler after being executed to the temporary control limit.

[0080] Specifically, the temporary control state is used to change the number of tags that can be continuously advanced in the next target long pre-fill request, without changing the underlying model execution operators, key-value cache management, adapter execution, and GPU kernel execution paths.

[0081] Step S5: After the scheduler returns control, construct a candidate benefit request set from the active inference request set based on the scheduling boundary formed by the most recent continuous execution limit.

[0082] Specifically, step S5 corresponds to Figure 5 The diagram illustrates the candidate set construction process during the selection of benefit requests and placement of the next execution window. The candidate benefit request set is used to determine which requests can proceed to benefit score calculation after the scheduler returns control.

[0083] In an optional embodiment of the present invention, step S5 may include: step S51, determining the boundary adaptation range based on the scheduling boundary formed by the most recent continuous execution upper limit.

[0084] Step S52: Exclude the target long pre-filled request and requests that do not meet the pre-filled state conditions from the active inference request set to obtain the request set to be filtered.

[0085] Specifically, requests that do not meet the pre-filling status conditions can include requests that have already entered the decoding stage and requests whose remaining pre-filling workload is less than a preset threshold.

[0086] Step S53: Filter the set of requests to be filtered according to the boundary adaptation range to obtain the set of candidate benefit requests.

[0087] Specifically, requests whose remaining pre-filled workload is within the boundary adaptation range can be retained as requests in the candidate beneficiary request set.

[0088] In an optional embodiment of the present invention, step S51 may include: step S511, determining the size of the most recent long pre-filled block based on the most recent continuous execution limit.

[0089] Step S512: Determine the boundary adaptation range based on the most recent long pre-filled block size and the preset boundary expansion conditions.

[0090] Specifically, the preset boundary expansion conditions may include a preset magnification factor or a preset margin. The boundary adaptation range can be configured such that the remaining pre-fill workload of the candidate request does not exceed the product of the most recent long pre-fill block size and the preset magnification factor, or does not exceed the sum of the most recent long pre-fill block size and the preset margin.

[0091] Step S6: Determine the benefit score of each candidate benefit request based on the runtime state data and the adaptation relationship between each candidate benefit request and the most recent continuous execution limit.

[0092] In an optional embodiment of the present invention, step S6 may include: step S61, determining the waiting priority of each candidate benefit request based on the waiting status data in the runtime status data.

[0093] Step S62: Determine the boundary fit of each candidate benefit request based on the fit relationship between the remaining pre-filled workload of each candidate benefit request and the upper limit of the most recent continuous execution.

[0094] Step S63: Determine the benefit score for each candidate benefit request based on the waiting priority and the boundary fit.

[0095] In one specific implementation, the waiting priority, boundary fit, and benefit score of a candidate benefit request can be expressed as: in, Indicates a candidate benefit request Waiting priority; Indicates a candidate benefit request Boundary fit; Indicates a candidate benefit request Benefit rating; Indicates a candidate benefit request The waiting time has expired; This represents the maximum waiting time in the set of candidate benefit requests; Indicates a candidate benefit request The remaining pre-filling workload; Indicates the size of the most recent long pre-filled block; This represents a constant to prevent the denominator from being zero; This indicates the priority weight for waiting. Boundary fit can be limited to a range of 0 to 1.

[0096] In an optional embodiment of the present invention, step S63 may include: step S631, determining the scoring weights corresponding to the waiting priority and the boundary fit degree based on the runtime load pressure signal.

[0097] Specifically, the scoring weight can be a preset value or adjusted based on the runtime load pressure signal. In one specific embodiment, 0.4 can be taken.

[0098] Step S632: Based on the scoring weight, the waiting priority and the boundary fit are fused to obtain the benefit score of each candidate benefit request.

[0099] Step S7: Select a benefit request from the candidate benefit request set according to the benefit score, place the benefit request in the next execution window of the deep learning inference runtime, and perform bounded delay or bounded supplementary placement for the unselected requests.

[0100] Specifically, the process of selecting benefit requests and placing them in the next execution window may include placing requests whose benefit scores reach a threshold into the benefit request set, treating requests whose benefit scores do not reach the threshold as low benefit requests, and performing bounded supplementary placement based on the available capacity of the next execution window.

[0101] In an optional embodiment of the present invention, step S7 may include: step S71, sorting the candidate benefit requests in the candidate benefit request set according to the benefit score, and selecting the candidate benefit request that meets the benefit selection condition as the benefit request.

[0102] Specifically, candidate benefit requests with benefit scores higher than a preset threshold can be sorted in descending order of benefit scores, and no more than a preset maximum number of requests can be selected as benefit requests.

[0103] Step S72: The benefit request is written first into the next execution window of the deep learning inference runtime.

[0104] Step S73: Based on the available capacity of the next execution window, perform bounded delay or bounded supplementary placement for the unselected requests.

[0105] Specifically, bounded deferral means that low-benefit requests are temporarily downgraded only within the current execution window and re-enter scheduling after the execution window is completed. Bounded supplementary placement means that when there is still capacity in the next execution window, only an additional number of requests, not exceeding a preset supplementary number, are allowed to enter the next execution window along with the benefiting requests.

[0106] In an optional embodiment of the present invention, step S71 may include: step S711, adjusting the benefit selection condition according to the runtime load pressure signal.

[0107] Specifically, when execution load pressure increases, the long request judgment threshold, the heterogeneity threshold, or the number of allowed supplementary requests can be increased. The heterogeneity threshold is used to determine whether the differences in input length, adapter rank, adapter type, or tenant identifier among requests within the execution window reach a preset range. When the urgency of short requests increases, the size of long pre-filled blocks can be reduced, the benefit score weight of short requests can be increased, or the benefit score threshold can be decreased.

[0108] Step S712: Select the benefit request from the candidate benefit request set according to the adjusted benefit selection criteria and the benefit score.

[0109] In an optional embodiment of the present invention, step S73 may include: step S731, when there is available capacity in the next execution window, determining a supplementary placement request from the unselected requests based on the supplementary placement conditions.

[0110] Specifically, supplementary placement requests can be determined based on benefit score, whether they are short requests, and expected block cost. The number of supplementary placement requests shall not exceed a preset supplementary quantity, which is a non-negative integer.

[0111] Step S732: Write the supplementary placement request into the next execution window, and determine the remaining unselected requests as temporarily postponed requests.

[0112] Step S733: After the next execution window is completed, clear the temporary preference state corresponding to the temporary postponement request.

[0113] Specifically, after the temporary preference state is cleared, the temporarily postponed request will be re-scheduled in a subsequent execution window.

[0114] This invention also provides a deep learning inference request scheduling system for executing the above-described methods, such as... Figure 6 As shown, it includes: The request status acquisition module is used to acquire runtime status data of the active inference request set at the scheduling boundary during deep learning inference runtime. The load pressure generation module is used to determine the target long pre-fill request from the active inference request set based on the runtime state data, and generate a runtime load pressure signal. The block upper limit determination module is used to determine the next continuous execution upper limit of the target long pre-filled request based on the runtime load pressure signal and the inference execution cost estimation data corresponding to the candidate continuous execution upper limit; The scheduling status writing module is used to write the next consecutive execution limit as a temporary control limit into the scheduler status, so that the target long pre-fill request returns to the scheduler after execution reaches the temporary control limit; The candidate set construction module is used to construct a candidate benefit request set from the active inference request set based on the scheduling boundary formed by the most recent continuous execution limit after the scheduler returns control. The benefit score determination module is used to determine the benefit score of each candidate benefit request based on the runtime state data and the adaptation relationship between each candidate benefit request and the most recent continuous execution limit. The execution window placement module is used to select a benefit request from the candidate benefit request set according to the benefit score, place the benefit request in the next execution window during the deep learning inference runtime, and perform bounded delay or bounded supplementary placement for the unselected requests.

[0115] Method performance testing: In this experiment, a deep learning inference request scheduling method was run on an NVIDIA A100 80GB GPU. The inference runtime used vLLM 0.10.1, PyTorch 2.7.1 and CUDA 12.6. The workload consisted of UltraChat200k short requests and LongBench long requests. The short requests and long requests were mixed in a ratio of (2-3):1 and replay was performed at four Poisson arrival densities: low, mid, high and peak.

[0116] Compared to the default vLLM shared runtime, this method reduces the p99 first marking time for all requests from 1418.6 ms to 628.9 ms, a reduction of 55.7%; reduces the p99 first marking time for short requests from 394.7 ms to 232.2 ms, a reduction of 41.2%; at the same time, the round completion time changes from 10348.3 ms to 10163.4 ms, and the throughput changes from 3.97 requests / second to 4.07 requests / second.

[0117] At peak density, the p99 first marking time for short requests is reduced by 73.7% compared to the default vLLM, indicating that this method can more significantly improve the visible response latency for short requests under high load and strong scheduling boundary contention.

[0118] In another set of runtime adaptive experiments with 8 models and 32 density models, the average first mark time of the combined path of boundary timing control and benefit request placement was improved by 2.309 times, the average round completion time was improved by 1.037 times, and the average deceleration ratio was improved by 1.117 times.

[0119] Although the present invention has been disclosed above with reference to embodiments, it is not intended to limit the present invention. Appropriate modifications or equivalent substitutions made by those skilled in the art to the technical solutions of the present invention should be covered within the protection scope of the present invention, which is defined by the claims.

Claims

1. A deep learning inference request scheduling method, characterized in that, Includes the following steps: At the scheduling boundary of the deep learning inference runtime, obtain runtime state data of the active inference request set; Based on the runtime state data, a target long pre-filled request is determined from the active inference request set, and a runtime load pressure signal is generated; Based on the runtime load pressure signal and the inference execution cost estimation data corresponding to the candidate continuous execution limit, determine the next continuous execution limit of the target long pre-filled request; Write the next consecutive execution limit as a temporary control limit into the scheduler state, so that the target long pre-fill request returns to the scheduler after execution reaches the temporary control limit; After the scheduler returns control, a set of candidate benefit requests is constructed from the set of active inference requests based on the scheduling boundary formed by the most recent continuous execution limit. Based on the runtime state data and the adaptation relationship between each candidate benefit request and the most recent continuous execution limit, the benefit score of each candidate benefit request is determined. Based on the benefit score, a benefit request is selected from the set of candidate benefit requests. The benefit request is then placed in the next execution window during the deep learning inference runtime. Unselected requests are then placed with bounded delay or bounded supplementation.

2. The method as described in claim 1, characterized in that, Based on the runtime state data, a target long pre-filled request is determined from the active inference request set, and a runtime load pressure signal is generated, including: Based on the remaining pre-filling workload in the runtime state data, determine the long pre-filling request set from the active inference request set; The target long pre-fill request is determined based on the remaining pre-fill workload and waiting status data of each long pre-fill request in the long pre-fill request set. The runtime load pressure signal is generated based on the runtime state data, the long pre-fill request set, and the target long pre-fill request.

3. The method as described in claim 2, characterized in that, Based on the runtime state data, the long pre-filled request set, and the target long pre-filled request, the runtime load pressure signal is generated, including: The urgency of short requests is determined based on the number of waiting short requests, the maximum waiting time, and the workload of short requests in the runtime status data. The execution load pressure is determined based on the queue length in the runtime status data and the remaining pre-filling workload of the target length pre-filling request; The runtime load pressure signal is generated based on the urgency of the short request and the execution load pressure.

4. The method as described in claim 1, characterized in that, Based on the runtime load pressure signal and the inference execution cost estimation data corresponding to the candidate consecutive execution upper limit, the next consecutive execution upper limit of the target long pre-filled request is determined, including: Based on the runtime load pressure signal, the target block size corresponding to the target length pre-fill request is generated; The candidate continuous execution limit is determined based on the target block size, and the inference execution cost estimation data corresponding to the candidate continuous execution limit is obtained; Based on the inference execution cost estimation data, the next consecutive execution limit for the target long pre-filled request is determined from the candidate consecutive execution limits.

5. The method as described in claim 1, characterized in that, Writing the next consecutive execution limit as a temporary control limit into the scheduler state, so that the target long pre-fill request returns to the scheduler after reaching the temporary control limit, includes: Associate the next consecutive execution limit with the request identifier of the target long pre-filled request to generate a temporary control state; The temporary control state limits the amount of pre-filling work that the target long pre-filling request can continuously advance within the execution window, so that the target long pre-filling request returns to the scheduler after being executed to the temporary control limit.

6. The method as described in claim 1, characterized in that, After the scheduler returns control, a candidate benefit request set is constructed from the active inference request set based on the scheduling boundary formed by the most recent continuous execution limit, including: The boundary adaptation range is determined based on the scheduling boundary formed by the most recent continuous execution limit. The target long pre-filled request and requests that do not meet the pre-filled state conditions are excluded from the active inference request set to obtain the request set to be filtered. The set of requests to be filtered is filtered according to the boundary adaptation range to obtain the set of candidate benefit requests.

7. The method as described in claim 1, characterized in that, Based on the runtime state data and the adaptation relationship between each candidate benefit request and the most recent continuous execution limit, the benefit score of each candidate benefit request is determined, including: Based on the waiting status data in the runtime status data, determine the waiting priority of each candidate benefit request; The boundary fit of each candidate benefit request is determined based on the fit relationship between the remaining pre-filled workload of each candidate benefit request and the upper limit of the most recent continuous execution. Based on the waiting priority and the boundary fit, the benefit score of each candidate benefit request is determined.

8. The method as described in claim 1, characterized in that, Based on the benefit score, a benefit request is selected from the candidate benefit request set. This benefit request is then preferentially placed in the next execution window during the deep learning inference runtime. Unselected requests are then subject to bounded deferral or bounded supplementary placement, including: Based on the benefit score, the candidate benefit requests in the candidate benefit request set are sorted, and the candidate benefit requests that meet the benefit selection criteria are selected as benefit requests. The benefit request will be written first into the next execution window during the deep learning inference runtime; Based on the available capacity of the next execution window, bounded deferral or bounded supplementary placement is performed on unselected requests.

9. The method as described in claim 8, characterized in that, Based on the available capacity of the next execution window, bounded deferral or bounded supplementary placement is performed on unselected requests, including: When the next execution window has available capacity, a supplementary placement request is determined from the requests that have never been selected, based on the supplementary placement conditions; Write the supplementary placement request into the next execution window, and determine the remaining unselected requests as temporarily postponed requests; After the next execution window is completed, clear the temporary preference state corresponding to the temporary postponement request.

10. A deep learning inference request scheduling system, characterized in that, include: The request status acquisition module is used to acquire runtime status data of the active inference request set at the scheduling boundary during deep learning inference runtime. The load pressure generation module is used to determine the target long pre-fill request from the active inference request set based on the runtime state data, and generate a runtime load pressure signal. The block upper limit determination module is used to determine the next continuous execution upper limit of the target long pre-filled request based on the runtime load pressure signal and the inference execution cost estimation data corresponding to the candidate continuous execution upper limit; The scheduling status writing module is used to write the next consecutive execution limit as a temporary control limit into the scheduler status, so that the target long pre-fill request returns to the scheduler after execution reaches the temporary control limit; The candidate set construction module is used to construct a candidate benefit request set from the active inference request set based on the scheduling boundary formed by the most recent continuous execution limit after the scheduler returns control. The benefit score determination module is used to determine the benefit score of each candidate benefit request based on the runtime state data and the adaptation relationship between each candidate benefit request and the most recent continuous execution limit. The execution window placement module is used to select a benefit request from the candidate benefit request set according to the benefit score, place the benefit request in the next execution window during the deep learning inference runtime, and perform bounded delay or bounded supplementary placement for the unselected requests.

Images