Inference service scheduling method and device for multi-token prediction scenario

Abstract

This invention discloses an inference service scheduling method and apparatus for multi-token prediction scenarios. The method includes: receiving an inference request; obtaining multi-token prediction parameters and service level target parameters corresponding to the inference request; the multi-token prediction parameters include at least the target token quantity for a single iteration; calculating an equivalent scheduling delay target based on the target token quantity for a single iteration and the service level target parameters; determining the partitioning ratio of GPU computing resources between the prefix computation stage and the decoding stage according to the equivalent scheduling delay target and the current system load state; dividing the inference request in the prefix computation stage into multiple prefix computation blocks; and concurrently executing the multiple prefix computation blocks and decoding batches under the partitioning ratio of GPU computing resources. Using this invention, the allocation of GPU computing resources can be made more precise, improving GPU utilization.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to a method and apparatus for scheduling inference services in multi-token prediction scenarios. Background Technology

[0002] The reasoning process of a Large Language Model (LLM) is typically divided into a Prefill phase (also known as the prefix computation or processing phase) and a decoding phase (also known as the token generation or autoregressive phase). In the traditional decoding model, only one token is generated per decoding iteration, resulting in high inference latency and making it difficult to meet the strict Service Level Objective (SLO).

[0003] To improve the decoding efficiency of the model itself, some advanced LLMs have introduced the Multi-Token Prediction (MTP) mechanism. MTP allows the model to generate multiple consecutive tokens simultaneously in a single decoding iteration, thereby significantly reducing the number of iterations and lowering generation latency at the model level. This mechanism has been proven to effectively improve inference throughput and response speed without significantly sacrificing generation quality.

[0004] However, existing service scheduling technologies for LLM inference are mainly designed for single-token generation scenarios and fail to fully consider the new features and challenges brought by the MTP mechanism, resulting in the following main technical problems: First, existing MTP optimization techniques at the model layer cannot address the resource allocation issues at the inference service scheduling layer. While MTP improves model decoding efficiency, how to achieve resource allocation and accurate SLO guarantees at the service scheduling level that match MTP remains an unresolved issue. Relying solely on traditional scheduling methods can easily lead to resource imbalances and reduced service quality.

[0005] Second, existing scheduling technologies lack a SLO conversion mechanism for MTP scenarios. Traditional scheduling frameworks use single-token generation latency as a constraint, but the characteristic of MTP generating multiple tokens in a single iteration cannot be directly mapped to an equivalent latency constraint that the framework can understand. The lack of a reasonable conversion method makes it difficult for scheduling strategies to effectively utilize the latency advantages brought by MTP.

[0006] Third, in actual inference, the number of tokens generated in each decoding iteration of the MTP mechanism is uncertain (e.g., affected by model confidence or stopping conditions). Existing scheduling methods cannot adapt to this dynamic change. When the actual number of tokens generated deviates from the expectation, it is impossible to avoid SLO defaults or to recover or replenish resources in a timely manner, resulting in resource allocation mismatch and waste.

[0007] Fourth, the MTP mechanism introduces additional inference startup overhead (such as multi-position input preprocessing, parallel output processing, etc.), and the existing scheduling strategy does not take this overhead into account. As a result, in task splitting, especially in the calculation of the Prefill block size, there will be idle bubbles in the Graphics Processing Unit (GPU), which will affect the overall execution efficiency.

[0008] Therefore, there is an urgent need for an inference service scheduling method for multi-token prediction scenarios, which can systematically solve problems such as resource allocation, SLO conversion, adaptation to uncertain token numbers, and elimination of startup overhead in MTP scenarios from the service scheduling level. Summary of the Invention

[0009] This invention provides a method and apparatus for scheduling inference services in multi-Token prediction scenarios, in order to overcome the problems existing in the inference service scheduling layer of the existing LLM with the introduction of MTP mechanism.

[0010] On one hand, embodiments of the present invention provide a method for scheduling inference services in multi-token prediction scenarios, the method comprising: Receive an inference request; obtain the multi-Token prediction parameters and service level target parameters corresponding to the inference request; the multi-Token prediction parameters include at least the target number of Tokens in a single iteration; Based on the target number of tokens for a single iteration and the service level target parameters, calculate the equivalent scheduling delay target; Based on the equivalent scheduling delay target and the current system load status, determine the partition ratio of GPU computing resources between the prefix calculation stage and the decoding stage; The inference request in the prefix calculation phase is divided into multiple prefix calculation blocks; Under the specified partition ratio of the GPU computing resources, the multiple prefix computing blocks and decoding batches are executed concurrently.

[0011] Optionally, the service level target parameters include: first token time target and inter-token delay target; The calculation of the equivalent scheduling delay target based on the single-iteration target token quantity and the service level target parameters includes: Calculate the ratio of the inter-Token delay target to the single-iteration target number of Tokens, and use the ratio as the equivalent scheduling delay target.

[0012] Optionally, determining the partitioning ratio of GPU computing resources between the prefix computation stage and the decoding stage based on the equivalent scheduling delay target and the current system load status includes: Using a pre-built delay prediction model, and with the equivalent scheduling delay target as a constraint, the minimum decoding stage computation resource allocation amount that satisfies the constraint is iteratively searched. The latency prediction model predicts the iteration latency of the decoding stage under different partition ratios; the inputs of the latency prediction model include: decoding batch size, number of streaming multiprocessors, and number of target tokens per iteration; the output of the latency prediction model is the predicted iteration latency.

[0013] Optionally, the multi-token prediction parameters further include a model identifier; the method further includes: The delay prediction model is selected based on the model identifier.

[0014] Optionally, dividing the inference request in the prefix calculation stage into multiple prefix calculation blocks includes: dividing the inference request in the prefix calculation stage into multiple prefix calculation blocks according to the expected execution time of the current decoding batch and the startup time of the prefix calculation kernel.

[0015] Optionally, the method further includes: During the decoding phase, monitor the actual number of tokens generated in a single iteration; If the deviation between the actual number of tokens generated in a single iteration and the target number of tokens in a single iteration is greater than a set deviation threshold, a dynamic adjustment process for the partition ratio is triggered to recalculate and update the partition ratio.

[0016] Optionally, updating the partition ratio includes: If the number of tokens actually generated in a single iteration is less than the target number of tokens in a single iteration, then the computing resource quota in the decoding stage is reduced, and the released computing resources are transferred to the prefix computing stage. If the number of tokens actually generated in a single iteration is greater than the target number of tokens in a single iteration, then the computing resource quota in the prefix stage is reduced, and the released computing resources are transferred to the decoding stage.

[0017] Optionally, updating the partition ratio further includes: making the recalculated partition ratio take effect gradually.

[0018] Optionally, the multi-Token prediction parameters further include a prediction strategy type; the prediction strategy type is used to characterize the statistical characteristics of the actual number of tokens output by the model in a single iteration; the method further includes: setting the deviation threshold according to the prediction strategy type.

[0019] Optionally, the method further includes: when a mixed inference request, including single-Token inference requests and multi-Token inference requests, is scheduled into the same decoding batch, determining the partition ratio based on the equivalent scheduling delay target corresponding to the multi-Token inference request.

[0020] On the other hand, embodiments of the present invention also provide an inference service scheduling device for multi-token prediction scenarios, the device comprising: The request parsing module is used to receive inference requests; obtain multi-Token prediction parameters and service level target parameters corresponding to the inference request; the multi-Token prediction parameters include at least the target number of Tokens in a single iteration; The equivalent scheduling target conversion module is used to calculate the equivalent scheduling delay target based on the number of single-iteration target tokens and the service level target parameters; The resource partitioning decision module is used to determine the partitioning ratio of GPU computing resources between the prefix calculation stage and the decoding stage based on the equivalent scheduling delay target and the current system load status. The task segmentation module is used to segment the inference request in the prefix calculation stage into multiple prefix calculation blocks; The concurrent execution control module is used to concurrently execute the multiple prefix calculation blocks and decoding batches under the partition ratio of the GPU computing resources.

[0021] Optionally, the device further includes: an execution deviation feedback module, used to monitor the actual number of tokens generated in a single iteration during the decoding phase; and to trigger a dynamic adjustment process for the partition ratio when the deviation between the actual number of tokens generated in a single iteration and the target number of tokens in a single iteration is greater than a set threshold, so as to recalculate and update the partition ratio.

[0022] On the other hand, embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when run by a processor, executes the steps of the inference service scheduling method for multi-token prediction scenarios.

[0023] The inference service scheduling method and apparatus for multi-Token prediction scenarios provided in this invention calculates an equivalent scheduling delay target based on the multi-Token prediction parameters and service level target parameters corresponding to the inference request. Then, based on the equivalent scheduling delay target and the current system load state, it determines the partition ratio of GPU computing resources between the prefix calculation stage and the decoding stage. Furthermore, it divides the inference request in the prefix calculation stage into multiple prefix calculation blocks, thereby enabling the concurrent execution of multiple prefix calculation blocks and decoding batches under this partition ratio. This allows existing scheduling infrastructure based on the single-Token assumption to seamlessly adapt to the MTP model, achieving precise GPU computing resource allocation without refactoring the underlying scheduling framework, thus improving GPU utilization.

[0024] Furthermore, the closed-loop control mechanism of deviation feedback effectively solves the resource allocation mismatch problem caused by the uncertainty of the actual number of tokens generated by MTP.

[0025] Furthermore, for application scenarios that combine single-token prediction mechanisms and MTP prediction mechanisms, the Streaming Multiprocessor (SM) quota is determined by adopting the minimum equivalent scheduling delay target, ensuring that the SLOs of all requests are met and achieving compatibility support for hybrid model deployment scenarios.

[0026] The solution of this invention does not require modification of the model decoding strategy, but only optimization at the inference service scheduling layer. Therefore, it does not interfere with the existing MTP optimization scheme of the model layer and is naturally decoupled. When the two are used together, their effects can be directly superimposed to produce synergistic gains. Attached Figure Description

[0027] Figure 1 This is a flowchart of an inference service scheduling method for multi-Token prediction scenarios provided in an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating the timing process of actual token count deviation feedback and dynamic resource rescheduling during the decoding stage in this embodiment of the invention. Figure 3 This is a schematic diagram of a reasoning service scheduling device for multi-token prediction scenarios provided in an embodiment of the present invention; Figure 4 This is another structural schematic diagram of the inference service scheduling device for multi-Token prediction scenarios provided in this embodiment of the invention. Detailed Implementation

[0028] To make the above-mentioned objectives, features and beneficial effects of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0029] The traditional LLM inference process is divided into two core stages: the prefix computation stage and the decoding stage.

[0030] During the prefix computation phase, the model breaks down the user-input text into several tokens and inputs these tokens into the neural network in parallel, calculating the attention relationships between each token. Simultaneously, the model generates the token for the first response and calculates a key-value cache (KV cache) for all tokens in this input. The prefix computation phase is characterized by parallel processing of the entire input prompt, requiring extensive matrix operations and placing high demands on GPU computing power.

[0031] During the decoding phase, the model uses the KV cache and the latest input token to predict the next token. Only one token is generated at a time, and this process is repeated multiple times until an end-of-line character is generated or the maximum length limit is reached. The characteristics of the decoding phase are: each token is generated serially; because only one token is processed at a time, the computational cost is low, but it requires frequent reading and updating of the KV cache, which is limited by GPU memory bandwidth.

[0032] The MTP mechanism is an optimization of the traditional decoding stage. Under the MTP mechanism, multiple tokens can be generated simultaneously in one iteration, thereby significantly reducing the number of iterations and improving decoding efficiency. However, the MTP mechanism brings problems such as longer and dynamically changing single iteration delays and uncertain number of tokens generated (which may stop early). Traditional scheduling strategies are difficult to adapt directly. For example, this uncertainty can lead to serious resource allocation mismatch.

[0033] To address the problems existing in the inference service scheduling layer of the LLM with the introduction of the MTP mechanism, this invention provides an inference service scheduling method and apparatus for multi-Token prediction scenarios. This enables the existing scheduling infrastructure based on the single-Token assumption to seamlessly adapt to the MTP model, achieving precise allocation of GPU computing resources without reconstructing the underlying scheduling framework, thereby improving GPU utilization.

[0034] like Figure 1 The diagram shown is a flowchart of an inference service scheduling method for multi-token prediction scenarios provided by an embodiment of the present invention, including the following steps: Step 101: Receive the inference request and obtain the multi-Token prediction parameters and service level target parameters corresponding to the inference request.

[0035] The multi-Token prediction parameters include at least the number of target tokens k in a single iteration, and may further include: prediction strategy type (such as greedy decoding, sampling decoding) and model identifier.

[0036] The prediction strategy type characterizes the statistical properties of the actual number of tokens output in a single iteration of the model: under the greedy decoding strategy, the actual number of tokens generated in a single iteration is relatively stable and close to the target number of tokens k; under the sampling decoding strategy, the randomness of sampling makes it easier to trigger early termination, and the deviation between the actual number of generated tokens and the target value is relatively larger. Therefore, different deviation thresholds and rescheduling sensitivities can be set based on the prediction strategy type. For example, a relatively larger deviation threshold can be set for inference requests using the sampling decoding strategy. The model identifier is used to index the pre-built latency prediction model or performance parameter table corresponding to the model of the inference request, so that the subsequent resource partitioning decision process calls the latency prediction model that matches the model identifier.

[0037] The multi-token prediction parameters are determined by the deployed model architecture. For example, when using the DeepSeek-V3 model, the target token number k=4 in a single iteration. Depending on the model architecture, the corresponding token prediction parameters may be the same or different, and this embodiment of the invention does not limit this.

[0038] The service level target parameters include: first token time target (TTFT) and inter-token delay target (ITL).

[0039] The first token time target (TTFT) refers to the maximum allowed time from the arrival of the request to the return of the first token; the inter-token delay target (ITL) refers to the maximum allowed interval between adjacent tokens.

[0040] Step 102: Calculate the equivalent scheduling delay target based on the single iteration target token quantity and the service level target parameters.

[0041] The MTP model generates k tokens in a single iteration. From the perspective of the scheduling system, this is equivalent to compressing the latency constraint of the original k single-token iterations into a single iteration. Therefore, in a non-limiting embodiment, the ratio of the inter-token latency target ITL to the target number of tokens k in a single iteration can be used as the equivalent scheduling latency target ITL', i.e. Instead of using the original ITL, use ITL' as a delay constraint for each iteration.

[0042] For example, when the inter-token latency target ITL is 50ms / token and k=4, the equivalent scheduling latency target ITL' = 50ms / 4 = 12.5ms. This means that the scheduling system needs to ensure that the execution time of each decoding iteration does not exceed 12.5ms in order to guarantee that the user's inter-token latency target ITL meets the SLO requirement.

[0043] Step 103: Determine the partition ratio of GPU computing resources between the prefix calculation stage and the decoding stage based on the equivalent scheduling delay target and the current system load status.

[0044] Specifically, a pre-built delay prediction model can be used to iteratively search for the minimum decoding stage computational resource allocation that satisfies the constraints, with the equivalent scheduling delay target ITL' as the constraint.

[0045] The latency prediction model can be built based on offline performance analysis to predict the iteration latency of the decoding stage under different partition ratios. The inputs of the latency prediction model include the decoding batch size, the number of SMs, and the number of target tokens k in a single iteration; the output is the predicted iteration latency.

[0046] The offline construction process of the latency prediction model specifically includes: on the target GPU hardware platform, iterating through various combinations of decoding batch size, number of SMs, and the number of target tokens k per iteration, running decoding benchmark tests for each combination, and collecting actual iteration latency samples under each combination to form an offline performance dataset; fitting the latency prediction model based on the offline performance dataset, which can be implemented using any of the following methods: lookup table combined with interpolation, piecewise linear fitting, or regression model. During online inference, using the current decoding batch size, number of SMs, and number of target tokens k per iteration as input, querying the latency prediction model to obtain the predicted iteration latency. When the deployed model version or hardware platform changes, the above offline sampling process is re-executed to update the latency prediction model.

[0047] For example, if the search results are 24 SMs that can satisfy the constraint ITL' ≤ 12.5ms, then the remaining SMs (e.g., 132-24=108) can be allocated to the prefix calculation stage.

[0048] The computational resources allocated to the decoding phase are constrained by satisfying the equivalent scheduling delay target, while the remaining computational resources are allocated to the prefix computation phase. This allocation strategy ensures that the SLOs (Solution Time Limits) of the decoding phase are satisfied first, while maximizing the throughput of the prefix computation phase.

[0049] Step 104: Divide the inference request in the prefix calculation stage into multiple prefix calculation blocks.

[0050] Specifically, the kernel startup time can be calculated based on the expected execution time of the current decoding batch and the prefix calculation kernel startup time. In other words, the size of the prefix calculation block is determined based on the expected execution time of the current decoding batch and the prefix calculation kernel startup time.

[0051] In a non-limiting embodiment, the specific segmentation process is as follows: (1) Obtain the estimated execution time T_decode for the current decoding batch; The estimated execution time T_decode of the current decoding batch can be determined in any of the following ways: Method 1: Online prediction is performed using the aforementioned delay prediction model. This involves taking the current decoding batch size, the number of SMs allocated to the decoding stage, and the target token number k for a single iteration as inputs, and then outputting the estimated execution time T_decode from the delay prediction model.

[0052] Method 2: Estimation is based on runtime measured data, that is, the actual execution time of the most recent decoding iterations is averaged (e.g., exponential moving average), and the average result is used as the expected execution time T_decode.

[0053] In one non-limiting embodiment, method one is used when there is no historical measured data at the initial stage of system startup, and method two is used after the system is running stably, or a weighted fusion result of the two methods is used.

[0054] (2) Determine the maximum allowed startup time T_max for a single prefix computation block, wherein the maximum allowed startup time T_max does not exceed T_decode; The principle for setting the maximum allowable startup time T_max is to ensure that the prefix calculation block can be processed within the execution window of the current decoding batch, thereby filling the GPU bubbles during the execution of the decoding batch. In one non-limiting embodiment, T_max can be directly set to T_decode to make full use of the execution window; in another non-limiting embodiment, to absorb the jitter of the decoding batch execution time, T_max can be set to α × T_decode, where α is a safety factor greater than 0 and less than 1, for example, α = 0.9, and the specific value of α can be configured according to the actual jitter of the system.

[0055] (3) Based on the maximum allowed startup time T_max of a single prefix computation block, the startup latency T_std of the standard prefix computation kernel, and the additional inference startup overhead time T_mtp introduced by the multi-Token prediction mechanism, determine the number of tokens N_chunk contained in each prefix computation block: N_chunk= (T_max - T_std - T_mtp) / T_per_token; Where T_per_token is the prefix calculation processing time for a single token.

[0056] In this embodiment of the invention, the additional inference startup overhead introduced by the MTP mechanism includes: multi-Token output cache pre-allocation time, parallel decoding path initialization time, and data transmission time between MTP modules.

[0057] For example, if the current decoding batch is expected to execute in 8ms, the standard prefix computation kernel startup latency T_std is 2ms, and the additional inference startup overhead T_mtp introduced by the MTP prediction mechanism is 1ms, then the available computation time is 8 - 2 - 1 = 5ms. Assuming 0.1ms per token, each prefix computation block contains approximately 50 tokens.

[0058] In step 105, the multiple prefix computation blocks and decoding batches are executed concurrently under the partition ratio of the GPU computing resources.

[0059] For the multiple prefix computation blocks obtained from the segmentation, GPU space sharing technology is used to execute multiple prefix computation blocks and decoding batches concurrently. The prefix computation blocks and decoding batches are executed overlappingly in the time dimension. During the execution of the decoding batch, the GPU synchronously starts the processing of the prefix computation blocks, minimizing the GPU bubble (the time period when the GPU computing core SM is idle and does not perform any useful computation).

[0060] For example, NVIDIA Green Context (an advanced tool specifically designed for resource isolation and deterministic concurrency) can be used to create multiple CUDA (Compute Unified Device Architecture) streams, each allocated independent GPU SM resources. For instance, a prefix computation stream could be created with 108 SMs, and a decoding computation stream with 24 SMs. The kernels on both streams execute concurrently without interference. By allocating independent CUDA streams and dedicated SM resources to prefix and decoding computations, hardware-level execution isolation is achieved, effectively avoiding performance fluctuations caused by resource contention.

[0061] The inference service scheduling method for multi-Token prediction scenarios provided in this invention achieves both deterministic performance assurance and GPU utilization improvement for MTP model inference without reconstructing the underlying scheduling framework, through a three-pronged mechanism of scheduling semantic adaptation, resource quota actuarial calculation, and task splitting compensation. Specifically, as follows: (1) Equivalent scheduling delay conversion mechanism To address the characteristic of MTP generating k tokens in a single iteration, this invention transforms this into an equivalent delay constraint recognizable by traditional scheduling frameworks. Through this conversion, existing scheduling infrastructures based on the single-token assumption can seamlessly perceive the iteration rhythm of the MTP model without any underlying reconstruction, thus bridging the semantic gap between upper-layer model capabilities and lower-layer scheduling logic.

[0062] (2) Resource partitioning decision based on offline latency prediction model

[0063] To further accurately match the aforementioned equivalent latency target, this invention introduces offline performance analysis to construct a latency prediction model. This model takes the decoding batch size, the number of SMs, and the MTP prediction step size (the number of target tokens k in a single iteration) as input, and outputs the iterative latency prediction value. Using the equivalent latency constraint as the boundary, it iteratively searches for the minimum SM quota that meets the latency requirements. This avoids resource redundancy or insufficiency caused by traditional experience-based configuration, achieving precise and optimized SM resource allocation, and reserving sufficient hardware margin for subsequent concurrent execution.

[0064] (3) Task segmentation strategy for MTP perception and prefix computation

[0065] To eliminate the additional GPU bubbles introduced by long MTP iterations, this invention explicitly incorporates the inference startup overhead unique to the MTP mechanism during the determination of the prefix computation block size. This includes pre-allocation of multi-Token output cache, initialization of parallel decoding paths, and data transfer between MTP modules. By absorbing these additional time overheads in advance during the front-end segmentation stage, the prefix computation block and decoding batches can seamlessly overlap and coordinate efficiently on the timeline, thereby significantly compressing the idle time window caused by the complex logic of MTP and maintaining a consistently high GPU utilization rate.

[0066] Considering that the actual number of tokens output in a single MTP iteration may deviate from the target number of tokens in a single iteration, in some embodiments, a deviation feedback mechanism may be further introduced to dynamically adjust the SM partition ratio accordingly, so that the resource quota adaptively tracks the actual computing needs, thereby avoiding SLO default or resource waste caused by the fixed partition strategy.

[0067] Specifically, during the decoding phase, the actual number of tokens generated in a single iteration can be monitored. If the deviation between the actual number of tokens generated in a single iteration and the target number of tokens in a single iteration (i.e., the k value) exceeds a set threshold, a dynamic adjustment process for the partition ratio is triggered to recalculate and update the partition ratio. If the actual number of tokens generated in a single iteration, k_actual, is less than the target number of tokens in a single iteration, k, the computational resource quota for the decoding phase is reduced, and the released computational resources are transferred to the prefix computation phase. Conversely, if the actual number of tokens generated in a single iteration, k_actual, is greater than the target number of tokens in a single iteration, the computational resource quota for the prefix phase is reduced, and the released computational resources are transferred to the decoding phase.

[0068] For example, in a non-limiting embodiment, when the actual number of tokens generated in a single iteration, k_actual, is less than the target number of tokens in a single iteration, k, the minimum computational resources required for the decoding phase are recalculated according to the new equivalent scheduling delay target ITL'' = ITL / k_actual. Since k_actual < k and ITL'' > ITL', the required SM resources are reduced. The computational resource quota for the decoding phase is reduced, and the released resources are transferred to the prefix computation phase; when the actual number of tokens generated in a single iteration is greater than the target number of tokens in a single iteration, k, the computational resource quota for the decoding phase is increased, and resources are reclaimed from the prefix computation phase to meet the updated equivalent scheduling delay target ITL''.

[0069] Figure 2 The timing process of actual token number deviation feedback and dynamic resource rescheduling during the decoding phase is shown.

[0070] In this example, the initial scheduling is: MTP prediction k=4, and the SM allocation ratio (prefix calculation SM: decoded SM) is 108:24. Figure 2 As shown in (a).

[0071] Monitoring showed that an average of approximately 2.5 tokens were actually generated across multiple iterations. The dynamically adjusted SM allocation ratio (prefix-calculated SM: decoded SM) was 117:15. Figure 2 As shown in (b).

[0072] In a preferred embodiment, the new partition ratio can be implemented gradually to avoid performance jitter caused by resource adjustments. For example, the transition can be completed within 5-10 iterations through phased migration (each adjustment not exceeding 10% of the total resources).

[0073] The inference service scheduling method for multi-token prediction scenarios provided in this embodiment of the invention can also be compatible with handling standard per-token inference requests (k=1).

[0074] For a standard per-token inference request, the equivalent scheduling delay target ITL' is equal to the inter-token delay target (i.e., ITL' = ITL); for a multi-token inference request, the equivalent scheduling delay target is equal to the ratio of the inter-token delay target ITL to the single-iteration target token number k, i.e., ITL' = ITL / k.

[0075] When mixed requests are scheduled into the same decoding batch, a conservative strategy can be adopted when determining the partition ratio. The SM quota is calculated based on the minimum equivalent scheduling delay target (i.e., the most stringent constraint) to ensure that the SLOs of all requests are met. The minimum equivalent scheduling delay target typically comes from multi-Token inference requests: the equivalent scheduling delay target for multi-Token inference requests is the ratio of the inter-Token latency target (ITL) to the target number of Tokens (k) per iteration. When k > 1, this ratio is necessarily less than the ITL of a single-Token inference request. Therefore, taking the minimum equivalent scheduling delay target in the mixed batch is equivalent to taking the equivalent scheduling delay target for the corresponding multi-Token inference request. Requests within the same decoding batch share the computational resources of the decoding stage and execute at a unified iteration pace. Configuring the SM quota according to the most stringent equivalent scheduling delay target within the batch ensures that the inter-Token latency of each request does not exceed the limit; for single-Token requests, this is equivalent to resource over-allocation, making their SLOs easier to meet.

[0076] Typical application scenarios where mixed requests are orchestrated into the same decoding batch include: simultaneously deploying multi-Token prediction models and standard single-Token prediction models in the same inference service cluster, or different tenants using the two types of models in a multi-tenant shared inference service; to improve GPU utilization, the scheduling system adopts a continuous batch processing approach, orchestrating the decoding iterations of inference requests from different models into the same decoding batch for execution. For example, the same decoding batch includes request A and request B. Request A corresponds to the multi-Token prediction model, with a single iteration target of Token number k=4 and an inter-Token latency target ITL=50ms, corresponding to an equivalent scheduling latency target ITL'=50 / 4=12.5ms; request B corresponds to the standard single-Token prediction model (k=1), with an inter-Token latency target ITL=20ms, corresponding to an equivalent scheduling latency target ITL'=20ms. In this case, taking the minimum value of 12.5ms as the SM quota for the constraint calculation decoding stage ensures that the SLOs of both request A and request B are satisfied simultaneously.

[0077] Accordingly, embodiments of the present invention also provide an inference service scheduling device for multi-token prediction scenarios, such as... Figure 3 The diagram shown is a structural schematic of the device.

[0078] In this embodiment, the inference service scheduling device 300 for multi-token prediction scenarios includes the following modules: The request parsing module 301 is used to receive inference requests; obtain multi-Token prediction parameters and service level target parameters corresponding to the inference requests; the multi-Token prediction parameters include at least the number of target Tokens in a single iteration; The equivalent scheduling target conversion module 302 is used to calculate the equivalent scheduling delay target based on the number of single-iteration target tokens and the service level target parameters; The resource partitioning decision module 303 is used to determine the partitioning ratio of GPU computing resources between the prefix calculation stage and the decoding stage based on the equivalent scheduling delay target and the current system load status. Task segmentation module 304 is used to segment the inference request in the prefix calculation stage into multiple prefix calculation blocks; The concurrent execution control module 305 is used to concurrently execute the multiple prefix calculation blocks and decoding batches under the partition ratio of the GPU computing resources.

[0079] In this embodiment, the multi-Token prediction parameters may further include: prediction strategy type and model identifier. The service level target parameters include: first-Token time target and inter-Token delay target. Accordingly, the equivalent scheduling target conversion module 302 can calculate the ratio of the inter-Token delay target to the single-iteration target number of Tokens, and use the ratio as the equivalent scheduling delay target.

[0080] The resource partitioning decision module 303 can use a pre-built delay prediction model to iteratively search for the minimum decoding stage that satisfies the constraint, using the equivalent scheduling delay target as a constraint, to calculate the resource allocation amount. For details, please refer to the description in the previous embodiments of the present invention, which will not be repeated here.

[0081] Specifically, the task segmentation module 304 can segment the inference request in the prefix calculation stage into multiple prefix calculation blocks based on the expected execution time of the current decoding batch and the prefix calculation kernel startup time.

[0082] The inference service scheduling device for multi-Token prediction scenarios provided in this embodiment of the invention can achieve both deterministic guarantee of MTP model inference performance and improvement of GPU utilization without reconstructing the underlying scheduling framework.

[0083] like Figure 4 The diagram shown is another structural schematic of the inference service scheduling device for multi-Token prediction scenarios provided in an embodiment of the present invention.

[0084] and Figure 3 Unlike the previous embodiment, in this example, the inference service scheduling device 300 further includes an execution deviation feedback module 306, used to monitor the actual number of tokens generated in a single iteration during the decoding phase; if the deviation between the actual number of tokens generated in a single iteration and the target number of tokens in a single iteration is greater than a set threshold, a dynamic adjustment process for the partition ratio is triggered to recalculate and update the partition ratio. Specific adjustments can be found in the descriptions in the preceding embodiments of the present invention, and will not be repeated here.

[0085] The inference service scheduling device for multi-token prediction scenarios in this embodiment introduces a deviation feedback mechanism to dynamically adjust the SM partition ratio, enabling resource quotas to adaptively track actual computing needs, thereby avoiding SLO defaults or resource waste caused by fixed partitioning strategies.

[0086] Compared with the prior art, one or more embodiments of the present invention have the following significant advantages: (1) It fills the technical gap in the service scheduling layer of MTP scenarios. The solution of this invention addresses the resource allocation problem in MTP scenarios by starting from the inference service scheduling layer, rather than relying on model-level modifications. It complements rather than overlaps with existing model-level decoding optimization solutions. Without modifying the model structure and decoding algorithm, it achieves efficient deployment and stable operation of the MTP model in the inference service system.

[0087] (2) Seamless adaptation of existing scheduling infrastructure to the MTP model. The present invention uses a conversion mechanism of equivalent scheduling delay target to convert the characteristic of MTP generating multiple tokens in a single iteration into an equivalent delay constraint that can be understood by traditional scheduling frameworks. This enables existing scheduling systems based on the single token assumption (such as PD-Multiplexing) to support the MTP model without reconstruction, greatly reducing system upgrade costs.

[0088] (3) Closed-loop adaptive correction of resource allocation mismatch. Closed-loop control is achieved through the execution deviation feedback module. When the actual number of tokens generated in a single MTP iteration deviates from the expected number, the SM partition ratio can be automatically adjusted to ensure that the resource quota is accurately matched with the actual computing load of each iteration, thus fundamentally avoiding SLO default or idle computing power caused by the fixed allocation strategy. Test data shows that in scenarios where some requests are terminated early, GPU resource utilization is improved by 15%–25%.

[0089] (4) The GPU bubble caused by long MTP iterations is compressed or eliminated. By using the task splitting module, the additional startup overhead introduced by the MTP mechanism is explicitly included in the calculation of the prefix calculation block size, so that the prefix calculation and decoding can be seamlessly overlapped on the time axis. The GPU idle time window is effectively compressed, and the actual GPU utilization rate is stably maintained at over 90%.

[0090] (5) Orthogonal compatibility and synergistic effect with model layer optimization schemes. The scheme of this invention and the existing model layer optimization scheme are independent of each other in terms of technical path and do not interfere with each other. The two can be deployed simultaneously, and the effects are linearly superimposed. Under the same 8×H100 hardware and DeepSeek-V3 model conditions, the existing model layer optimization alone can improve the end-to-end throughput by about 12%, the scheduling optimization of the scheme of this invention alone can improve the throughput by about 18%, and the throughput can be improved by about 35% when the two are deployed together, which is a significant gain.

[0091] (6) Significantly improved SLO guarantee capability in mixed workload scenarios. In mixed workload scenarios of OpenThoughts, ShareGPT and LooGLE, compared with the traditional PD-Multiplexing baseline, the solution of this invention can improve the SLO satisfaction rate of MTP requests from 72% to 98%, while the SLO satisfaction rate of standard requests remains above 99%, and the overall throughput is improved by 15%-22%, which fully verifies its robustness in complex mixed workload scenarios.

[0092] (7) Strong engineering practicality and low deployment cost. The solution of this invention is fully compatible with the existing NVIDIA Green Context space sharing capability and standard MTP model interface. No fine-tuning or modification of the model is required, nor is any modification to the GPU driver and underlying software stack required. It can achieve smooth access and rapid deployment, and has good engineering practicality and promotion value.

[0093] Regarding the modules / units included in the various devices and products described in the above embodiments, they can be software modules / units, hardware modules / units, or a combination of both. For example, for various devices and products applied to or integrated into a chip, all of their modules / units can be implemented using hardware methods such as circuits, or at least some modules / units can be implemented using software programs that run on a processor integrated within the chip, while the remaining (if any) modules / units can be implemented using hardware methods such as circuits; for various devices and products applied to or integrated into a chip module, all of their modules / units can be implemented using hardware methods such as circuits, and different modules / units can be located in the same component (e.g., chip, circuit module, etc.) or different components of the chip module, or at least some modules / units can be implemented using hardware methods such as circuits. The implementation is achieved through a software program that runs on the processor integrated within the chip module. The remaining modules / units (if any) can be implemented using hardware methods such as circuits. For various devices and products applied to or integrated into terminal equipment, each of their modules / units can be implemented using hardware methods such as circuits. Different modules / units can be located in the same component (e.g., chip, circuit module, etc.) or different components within the terminal equipment. Alternatively, at least some modules / units can be implemented through a software program that runs on the processor integrated within the terminal equipment, while the remaining modules / units (if any) can be implemented using hardware methods such as circuits.

[0094] This invention also discloses a storage medium, which is a computer-readable storage medium storing a computer program thereon, the computer program being executable during runtime. Figure 1 The steps of the method shown are illustrated. The storage medium may include read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, etc. The storage medium may also include non-volatile memory or non-transitory memory, etc.

[0095] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article indicates that the preceding and following related objects have an "or" relationship.

[0096] In the embodiments of this invention, "multiple" refers to two or more.

[0097] The descriptions of "first," "second," etc., appearing in the embodiments of this invention are for illustrative purposes and to distinguish the objects being described. They do not indicate any particular order and do not imply any special limitation on the number of devices in the embodiments of this invention. They do not constitute any limitation on the embodiments of this invention.

[0098] While the present invention has been disclosed above, it is not limited thereto. Any person skilled in the art can make various modifications and alterations without departing from the spirit and scope of the invention; therefore, the scope of protection of the present invention should be determined by the scope defined in the claims.

Claims

1. A method for scheduling inference services in multi-token prediction scenarios, characterized in that, The method includes: Receive an inference request; obtain the multi-Token prediction parameters and service level target parameters corresponding to the inference request; the multi-Token prediction parameters include at least the target number of Tokens in a single iteration; Based on the target number of tokens for a single iteration and the service level target parameters, calculate the equivalent scheduling delay target; Based on the equivalent scheduling delay target and the current system load status, determine the partition ratio of GPU computing resources between the prefix calculation stage and the decoding stage; The inference request in the prefix calculation phase is divided into multiple prefix calculation blocks; Under the specified partition ratio of the GPU computing resources, the multiple prefix computing blocks and decoding batches are executed concurrently.

2. The inference service scheduling method for multi-token prediction scenarios according to claim 1, characterized in that, The service level target parameters include: first token time target and inter-token delay target; The calculation of the equivalent scheduling delay target based on the single-iteration target token quantity and the service level target parameters includes: Calculate the ratio of the inter-Token delay target to the single-iteration target number of Tokens, and use the ratio as the equivalent scheduling delay target.

3. The inference service scheduling method for multi-token prediction scenarios according to claim 1, characterized in that, The step of determining the partitioning ratio of GPU computing resources between the prefix computation stage and the decoding stage based on the equivalent scheduling delay target and the current system load status includes: Using a pre-built delay prediction model, and with the equivalent scheduling delay target as a constraint, the minimum decoding stage computation resource allocation amount that satisfies the constraint is iteratively searched. The latency prediction model predicts the iteration latency of the decoding stage under different partition ratios; the inputs of the latency prediction model include: decoding batch size, number of streaming multiprocessors, and number of target tokens per iteration; the output of the latency prediction model is the predicted iteration latency.

4. The inference service scheduling method for multi-token prediction scenarios according to claim 1, characterized in that, The multi-Token prediction parameters also include a model identifier; the method further includes: The delay prediction model is selected based on the model identifier.

5. The inference service scheduling method for multi-token prediction scenarios according to claim 1, characterized in that, The step of dividing the inference request in the prefix calculation stage into multiple prefix calculation blocks includes: Based on the expected execution time of the current decoding batch and the startup time of the prefix calculation kernel, the inference requests in the prefix calculation stage are divided into multiple prefix calculation blocks.

6. The inference service scheduling method for multi-token prediction scenarios according to claim 1, characterized in that, The method further includes: During the decoding phase, monitor the actual number of tokens generated in a single iteration; If the deviation between the actual number of tokens generated in a single iteration and the target number of tokens in a single iteration is greater than a set deviation threshold, a dynamic adjustment process for the partition ratio is triggered to recalculate and update the partition ratio.

7. The inference service scheduling method for multi-token prediction scenarios according to claim 6, characterized in that, The update of the partition ratio includes: If the number of tokens actually generated in a single iteration is less than the target number of tokens in a single iteration, then the computing resource quota in the decoding stage is reduced, and the released computing resources are transferred to the prefix computing stage. If the number of tokens actually generated in a single iteration is greater than the target number of tokens in a single iteration, then the computing resource quota in the prefix stage is reduced, and the released computing resources are transferred to the decoding stage.

8. The inference service scheduling method for multi-token prediction scenarios according to claim 6, characterized in that, The update of the partition ratio also includes: The recalculated partition ratios will take effect gradually.

9. The inference service scheduling method for multi-token prediction scenarios according to claim 6, characterized in that, The multi-Token prediction parameters also include a prediction strategy type; the prediction strategy type is used to characterize the statistical characteristics of the actual number of tokens output in a single iteration of the model; the method further includes: The deviation threshold is set according to the prediction strategy type.

10. The inference service scheduling method for multi-token prediction scenarios according to any one of claims 1 to 9, characterized in that, The method further includes: When a mixed inference request, including single-Token inference requests and multi-Token inference requests, is scheduled into the same decoding batch, the partition ratio is determined based on the equivalent scheduling delay target corresponding to the multi-Token inference request.

11. A reasoning service scheduling device for multi-token prediction scenarios, characterized in that, The device includes: The request parsing module is used to receive inference requests; obtain multi-Token prediction parameters and service level target parameters corresponding to the inference requests; the multi-Token prediction parameters include at least the target number of Tokens in a single iteration; The equivalent scheduling target conversion module is used to calculate the equivalent scheduling delay target based on the number of single-iteration target tokens and the service level target parameters; The resource partitioning decision module is used to determine the partitioning ratio of GPU computing resources between the prefix calculation stage and the decoding stage based on the equivalent scheduling delay target and the current system load status. The task segmentation module is used to segment the inference request in the prefix calculation stage into multiple prefix calculation blocks; The concurrent execution control module is used to concurrently execute the multiple prefix calculation blocks and decoding batches under the partition ratio of the GPU computing resources.

12. The inference service scheduling device for multi-token prediction scenarios according to claim 11, characterized in that, The device further includes: The execution deviation feedback module is used to monitor the actual number of tokens generated in a single iteration during the decoding phase. If the deviation between the actual number of tokens generated in a single iteration and the target number of tokens in a single iteration is greater than a set threshold, the partition ratio dynamic adjustment process is triggered to recalculate and update the partition ratio.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, The computer program, when run by a processor, executes the steps of the inference service scheduling method for multi-token prediction scenarios as described in any one of claims 1 to 10.

Images