Large language model reasoning scheduling method and device, equipment, medium and program product

Abstract

This application relates to the field of artificial intelligence computing technology, and provides a method, apparatus, device, medium, and program product for scheduling large language model inference, including: responding to multiple inference requests and acquiring real-time running status data and offline hardware profile data of the inference server. Based on the service completion results within the previous preset sliding time window, the real-time running status data, and the offline hardware profile data, determining the task execution ratio of pre-filled request subtasks and decoding request subtasks within the current preset sliding time window. Based on the real-time hardware resource utilization rate in the real-time running status data, the number of inference requests in the current inference request queue, and the shape information of the pre-compiled computation graph in the offline hardware profile data, generating batch composition optimization suggestion data. Based on the real-time running status data, task execution ratio, and batch composition optimization suggestion data, determining the task execution mode and batch size for each scheduling iteration within the current preset sliding time window.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence computing technology, and in particular relates to a method, apparatus, device, medium and program product for large language model reasoning scheduling. Background Technology

[0002] Currently, Large Language Models (LLMs) have become the core foundation for various intelligent applications such as search, dialogue, code assistance, and document processing. Their online inference services handle massive concurrent requests. LLM inference is typically deployed on intelligent AI accelerator cards such as GPUs / NPUs on a request-by-request basis. The inference engine batches multiple requests and delivers them to the model for computation and execution. Computational cost, throughput, and latency are key metrics for measuring the efficiency of the inference service.

[0003] The mainstream LLM architecture employs the Transformer framework and an autoregressive generation method. The inference process for each request is naturally divided into two phases with distinct computational characteristics: the Prefill phase and the Decode phase. The computational and storage bandwidth requirements of these two phases are naturally complementary. Specifically, when all requests are in the Prefill phase, computational power is heavily utilized while GPU memory bandwidth remains relatively idle; conversely, when all requests are in the Decode phase, GPU memory bandwidth is heavily utilized while computational power resources remain relatively idle. Therefore, effectively allocating resources between the Prefill and Decode phases is crucial for optimizing the efficiency of single-GPU inference services.

[0004] However, existing large language model LLMs all have technical problems in maximizing the effective throughput of the inference service during the execution of inference requests. For example, the existing PD coupled architecture has mutual interference between the Prefill / Decode two stages, and the existing PD separated architecture has the overhead of KV Cache migration across physical resources. Both of these have unavoidable structural costs and cause significant bottlenecks in the effective throughput of the inference service under real dynamic loads. Furthermore, the existing PD dynamic mixed deployment scheme fails to take into account both hardware universality and scheduling flexibility under real dynamic loads, making it difficult to maximize the effective throughput of the inference service. Summary of the Invention

[0005] This application provides a method, apparatus, device, medium, and program product for scheduling inference in large language models, which can solve the technical problem that existing large language model LLMs cannot maximize the effective throughput of inference services during the execution of inference requests.

[0006] In a first aspect, embodiments of this application provide a large language model inference scheduling method, applied to a large language model inference server, the method comprising: In response to multiple inference requests initiated by the user, the real-time running status data and offline hardware profile data of the large language model inference server are obtained; wherein, each inference request includes a pre-filled request subtask and a decoding request subtask; Based on the service completion results of multiple inference requests within the previous preset sliding time window obtained after the previous preset sliding time window ends, the real-time running status data, and the offline hardware profile data, the task execution ratio of the pre-filled request subtask and the decoding request subtask within the current preset sliding time window is determined. Based on the real-time hardware resource utilization rate in the real-time running status data, the number of inference requests in the current inference request queue, and the shape information of the pre-compiled computation graph in the offline hardware profile data, batch composition optimization suggestion data is generated. Based on the real-time running status data, the task execution ratio, and the batch composition optimization suggestion data, the task execution mode and batch size for each scheduling iteration within the current preset sliding time window are determined; wherein, the task execution mode is used to indicate whether to execute the pre-filled request subtask or the decoding request subtask.

[0007] In one possible implementation of the first aspect, before determining the task execution ratio of the pre-filled request subtask and the decoding request subtask within the current preset sliding time window based on the service completion results of multiple inference requests within the previous preset sliding time window obtained after the previous preset sliding time window ends, the real-time running status data, and the offline hardware profile data, the method includes: Obtain the previous running status data of the large language model inference server within the previous preset sliding time window; Based on the baseline execution latency corresponding to different batch sizes, token counts, and pre-compiled computation graph shape information in the offline hardware profile data, as well as the remaining latency margin for the first token request, the latency margin for each token decoding request, and the hardware resource utilization rate in the previous running state data, the estimated first token latency and the estimated per-token generation latency for each inference request within the previous preset sliding time window are obtained respectively. The estimated first token delay and estimated token generation delay for each inference request are compared with the preset first token delay constraint threshold and the preset token generation delay constraint threshold, respectively, to determine the service completion result corresponding to each inference request within the previous preset sliding time window; wherein, the service completion result is either the service indicator is not completed or the service indicator is completed.

[0008] In one possible implementation of the first aspect, determining the task execution ratio of the pre-filled request subtask and the decoding request subtask within the current preset sliding time window, based on the service completion results of multiple inference requests within the previous preset sliding time window obtained after the previous preset sliding time window ends, the real-time running status data, and the offline hardware profile data, includes: If, within the previous preset sliding time window, there is at least one inference request whose service completion result is that the service indicator is not completed, then the proportion of the pre-filled request subtask in the task execution ratio is reduced to obtain the task execution ratio. If, within the previous preset sliding time window, the service completion results corresponding to the multiple inference requests are all the service completion results, and there are multiple pending inference requests, then the proportion of the pre-filled request subtask in the task execution ratio is increased to obtain the task execution ratio.

[0009] In one possible implementation of the first aspect, generating batch composition optimization suggestion data based on the real-time hardware resource utilization rate in the real-time operating status data, the number of inference requests in the current inference request queue, and the shape information of the pre-compiled computation graph in the offline hardware profile data includes: Based on the shape information of the pre-compiled computation graph in the offline hardware profile data and the real-time hardware resource utilization rate in the real-time running status data, the computational cost and data filling redundancy under the shape information of multiple candidate pre-compiled computation graphs are quantified. Based on the number of inference requests in the current inference request queue, the current inference request queue is matched with the shape information of the candidate pre-compiled computation graph to identify the first scheduling point when the shape mismatch risk is at its maximum and the second scheduling point when the shape alignment benefit is at its maximum. Based on the computational cost, the data filling redundancy, the first scheduling time point, and the second scheduling time point, the batch composition optimization suggestion data is generated.

[0010] In one possible implementation of the first aspect, determining the task execution mode and batch size for each scheduling iteration within the current preset sliding time window based on the real-time running status data, the task execution ratio, and the batch composition optimization suggestion data includes: The task execution mode for the current scheduling iteration is determined based on the preset anti-starvation hard constraint and the task execution ratio; Based on the task execution mode and the batch composition optimization suggestion data, the batch size of the current scheduling iteration is determined.

[0011] Secondly, embodiments of this application provide a large language model inference scheduling device, applied to a large language model inference server, the device comprising: The status acquisition module is used to respond to multiple inference requests initiated by the user and acquire real-time running status data and offline hardware profile data of the large language model inference server; wherein, each inference request includes a pre-filled request subtask and a decoding request subtask; The adaptive control module is used to determine the task execution ratio of the pre-filled request subtask and the decoding request subtask in the current preset sliding time window based on the service completion results of multiple inference requests within the previous preset sliding time window obtained after the previous preset sliding time window ends, the real-time running status data, and the offline hardware profile data. The graph recognition control module is used to generate batch composition optimization suggestion data based on the real-time hardware resource utilization rate in the real-time running status data, the number of inference requests in the current inference request queue, and the shape information of the pre-compiled computation graph in the offline hardware profile data. The adaptive control module is further configured to determine the task execution mode and batch size for each scheduling iteration within the current preset sliding time window based on the real-time running status data, the task execution ratio, and the batch composition optimization suggestion data; wherein, the task execution mode is used to indicate whether to execute the pre-filled request subtask or the decoding request subtask.

[0012] Thirdly, embodiments of this application provide a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the large language model inference scheduling method described above.

[0013] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the large language model inference scheduling method described in any of the preceding claims.

[0014] Fifthly, embodiments of this application provide a computer program product that, when run on a terminal device, causes the terminal device to execute the large language model inference scheduling method described in any one of the first aspects.

[0015] The beneficial effects of the embodiments in this application compared with the prior art are: This application provides a method for scheduling inference for a large language model. The method includes: responding to multiple inference requests initiated by a user, acquiring real-time runtime status data and offline hardware profile data of the large language model inference server. Each inference request includes a pre-filling request subtask and a decoding request subtask. Then, based on the service completion results of multiple inference requests within the previous preset sliding time window, the real-time runtime status data, and the offline hardware profile data, the task execution ratio of the pre-filling request subtask and the decoding request subtask within the current preset sliding time window is determined. Based on the real-time hardware resource utilization rate in the real-time runtime status data, the number of inference requests in the current inference request queue, and the shape information of the pre-compiled computation graph in the offline hardware profile data, batch composition optimization suggestion data is generated. Finally, based on the real-time runtime status data, the task execution ratio, and the batch composition optimization suggestion data, the task execution mode and batch size for each scheduling iteration within the current preset sliding time window are determined. The task execution mode indicates whether to execute the pre-filling request subtask or the decoding request subtask. This method fundamentally avoids the structural costs of existing coupled and separated architectures by alternately executing pre-filled request subtasks and decoding request subtasks on the same accelerator card along the time dimension. At the same time, this method maximizes the effective throughput of the inference service under the current load by adjusting the task execution ratio of pre-filled request subtasks and decoding request subtasks in real time according to the actual dynamic load at the iteration granularity, thus making up for the shortcomings of existing dynamic hybrid deployment schemes in balancing hardware universality and scheduling flexibility. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating a large language model inference scheduling method according to an embodiment of this application; Figure 2 This is a schematic diagram of the structure of a large language model inference scheduling device provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application. Detailed Implementation

[0018] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0019] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0020] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0021] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0022] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0023] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0024] Currently, Large Language Models (LLMs) have become the core foundation for various intelligent applications such as search, dialogue, code assistance, and document processing. Their online inference services handle massive concurrent requests. LLM inference is typically deployed on intelligent AI accelerator cards such as GPUs / NPUs, with the inference engine batching multiple requests and delivering them to the model for computation and execution. Computational cost, throughput, and latency are key metrics for measuring the efficiency of the inference service. On the cloud service side, the resource overhead of the accelerator cards constitutes the main operating cost of the LLM service. In many inference deployment scenarios, the number of available accelerator cards may be very limited. Therefore, the optimization potential of single-card resource utilization directly determines the scale of concurrency that the service can handle.

[0025] The mainstream LLM architecture employs the Transformer framework and an autoregressive generation method. The inference process for each request is naturally divided into two computationally distinct phases: Prefill and Decode. The Prefill phase processes the complete prompt sequence input by the user in one go, performing attention and forward computation in parallel to generate the first output token and write the intermediate results to the key-value cache (KV Cache). The parallelism of this Prefill phase depends primarily on the length of the prompt; its single computation involves a large operator size and demands high computing power from the accelerator card, but it has fewer memory accesses, making it a typical "computation-intensive" phase. The Decode phase generates output tokens one by one based on the existing KV Cache, outputting only a new token in each iteration, but it requires repeatedly reading the entire KV Cache for attention computation. This Decode phase has a smaller single computation and lower parallelism, but it consumes a large amount of GPU memory, making it a typical "memory-intensive" phase. The two phases naturally complement each other in their demands for computing power and storage bandwidth. Specifically, when all requests are in the Prefill phase, computing power is heavily utilized while memory bandwidth remains relatively idle; conversely, when all requests are in the Decode phase, memory bandwidth is heavily utilized while computing power remains relatively idle. Therefore, better resource allocation between the Prefill and Decode phases is crucial for optimizing the efficiency of single-card inference services.

[0026] Performance evaluation of LLM inference services typically revolves around the following metrics: Time-To-First-Token (TTFT), Time-Per-Output-Token (TPOT / TBT), end-to-end latency, and throughput. TTFT describes the latency from when a user submits a request to when they receive the first output token, primarily determined by request queuing time and the execution time of the Prefill phase, reflecting the system's responsiveness to newly received requests. TPOT / TBT describes the latency between any two adjacent output tokens, mainly determined by the execution time of each iteration in the Decode phase, reflecting the stability of the output frequency of the requests being processed. End-to-end latency and throughput characterize the overall response time and service capacity.

[0027] In interactive applications, TTFT and TPOT are key service level objectives (SLOs), while in batch processing applications, throughput is usually the primary objective. In real-world inference service scenarios, the scheduler typically needs to maximize the system's effective throughput (Goodput, i.e., request throughput while satisfying SLOs) under strict TTFT and TPOT SLO constraints. This constitutes the core problem of LLM inference service scheduling.

[0028] Given the complementary resource requirements and SLO constraints of the Prefill / Decode phases, coordinating the execution of the Prefill and Decode phases in an inference system has become one of the most active research directions in LLM inference service systems in recent years. Existing mainstream LLM inference engines generally adopt an iterative-level scheduling paradigm, where the scheduler organizes batch processing and calls model execution based on a single inference iteration, and supports request processing and KV cache management under high concurrency through techniques such as continuous batching and paged attention. Regarding the aforementioned Prefill / Decode collaborative scheduling problem, there are three different existing technical solutions.

[0029] The first type is the PD coupled architecture, where the Prefill and Decode phases share the same resources and are processed together in the same batch. Early PD coupled approaches did not distinguish between the Prefill and Decode phases, instead placing all requests into a single batch for execution. While simple to implement, a long Prefill phase could potentially affect the output latency of Decode requests within the same batch. To mitigate this issue, research has proposed the Chunked Prefill technique. This technique divides a long Prefill phase into smaller chunks, which are then mixed with Decode requests in the same batch for execution. By adjusting the chunk size, the interference between the Prefill and Decode phases is mitigated. Chunked Prefill is a very mature optimization technique for current coupled architectures and is built into mainstream LLM inference engines.

[0030] The second type is the PD separation architecture, which separates the Prefill and Decode stages onto different instances for independent execution. High-speed interconnects are used to transfer the KV cache between stages, physically eliminating inter-stage interference. This is currently the mainstream choice for large-scale deployments. However, this solution requires at least two accelerator cards and incurs the overhead of cross-card KV cache migration, limiting performance in resource-constrained deployments.

[0031] The third type is dynamic hybrid deployment (PD), which dynamically coordinates the execution of the Prefill / Decode stages within the same accelerator card based on load characteristics to further improve Goodput. Dynamic hybrid deployment mainly involves two technical paths: spatial multiplexing and time-division multiplexing. Spatial multiplexing relies on the fine-grained hardware computing unit partitioning capabilities (such as Multi-Process Service (MPS) and GreenContext on GPUs) to schedule the Prefill and Decode stages simultaneously to different computing units on the same accelerator card. Time-division multiplexing, on the other hand, allows the Prefill and Decode stages to alternately and exclusively execute on the same physical resources, leveraging the complementary computing power and bandwidth requirements of the Prefill and Decode stages to improve resource utilization efficiency.

[0032] In summary, the three existing technical solutions described above have explored feasible paths for PD collaborative scheduling, and their common goal is to maximize the effective throughput (Goodput) of inference services. However, they all still have the following problems: (1) Existing PD coupling / decoupling architectures have unavoidable structural costs, making it difficult to maximize Goodput under real dynamic loads. Real LLM online inference loads fluctuate continuously in terms of concurrency scale, prompt length, and request arrival patterns. The demands of the Prefill / Decode phases on computing power and memory access bandwidth change accordingly, making the coordination of the Prefill / Decode phases the core contradiction in maximizing Goodput. The two existing standard deployment schemes each have unavoidable structural costs: the PD coupling architecture places the Prefill / Decode phases in the same batch to share resources. Prefill calculations within the same batch will increase the TPOT of Decode requests, and the two phases interfere with each other, making it difficult to simultaneously meet the TTFT and TPOT requirements; although the PD coupling architecture has developed dynamic mechanisms such as Chunked Prefill, the basic form of the PD coupling architecture, "hybrid batch processing," determines that the interference between the Prefill / Decode phases cannot be fundamentally eliminated. The PD decoupling architecture physically eliminates interference by deploying the Prefill / Decode phases across instances, but Prefill and Decode are executed on different physical resources, and the KV Cache cross-instance migration overhead must be incurred when switching between the two phases. The cost of the two architectures stems from their underlying execution patterns, both of which cause significant bottlenecks in Goodput under real fluctuating loads.

[0033] (2) Existing dynamic PD mixed deployment schemes fail to balance hardware universality and scheduling flexibility under real dynamic loads, making it difficult to maximize Goodput. Spatial multiplexing and time-division multiplexing in dynamic PD deployment are two fundamentally different paths in terms of resource utilization models. The spatial multiplexing path relies on the hardware's fine-grained computing unit partitioning capability, and its scheduling mechanism can dynamically adjust the partition ratio according to the load at the iteration granularity, which has good scheduling flexibility; however, all the innovations of this spatial multiplexing path depend on the above hardware capabilities and cannot be applied to accelerators that do not have mature spatial multiplexing capabilities. Time-division multiplexing (TDM) allows the Prefill / Decode phases to alternate and execute exclusively on the same physical resources without any partitioning hardware abstraction, offering a unique advantage in hardware universality. However, the phase mutual exclusion of TDM means that each iteration-level phase decision instantaneously affects the TPOT of the Decode request and the TTFT of the newly arrived request, placing higher demands on the load awareness of the scheduling mechanism. Existing TDM schemes mostly use fixed P / D time sharding ratios or heuristic strategies to arrange P / D alternation, and lack fine-grained SLO feedback mechanisms, resulting in relatively coarse adjustments to the P / D execution ratio under real dynamic loads. Both paths have their strengths and weaknesses. Among existing dynamic PD deployment schemes, none can simultaneously achieve hardware universality and iteration-level load-aware scheduling, making it difficult to maximize Goodput on general-purpose accelerators.

[0034] In summary, this application proposes a large language model inference scheduling method. Addressing the problems of mutual interference between P / D stages in existing PD coupled architectures, the overhead of KV Cache migration across physical resources in PD separated architectures, the inherent structural costs of both, and significant bottlenecks in Goodput under real dynamic loads, as well as the failure of existing PD dynamic deployment schemes to balance hardware universality and scheduling flexibility under real dynamic loads, this application adopts a time-division multiplexing path. It alternates between the Prefill and Decode stages on the same accelerator card via time, and adaptively adjusts the execution ratio of the two stages at the iteration granularity based on the dynamic characteristics of the load. The overall solution does not rely on spatial partitioning hardware abstractions such as MPS or GreenContext, and is applicable to any accelerator that supports pre-compiled computation graphs.

[0035] This method uses an adaptive control module as its core, which adaptively adjusts the P / D execution ratio at the iteration granularity by feeding back the SLO margin within the sliding time window. A state acquisition module provides low-overhead hardware state input, together achieving load-aware scheduling at the iteration granularity. Meanwhile, to enable efficient implementation of this method on accelerators using pre-compiled computation graphs, a design graph recognition control module is added to incorporate the shape constraints of the pre-compiled computation graph into batch organization decisions, reducing padding waste. This allows for the simultaneous achievement of hardware universality and load-aware scheduling at the iteration granularity on general-purpose accelerators.

[0036] Please see Figure 1 , Figure 1 This is a flowchart illustrating a large language model inference scheduling method according to an embodiment of this application. As an example and not a limitation, this method is applied to or runs on a large language model inference server, and includes: S11. In response to multiple inference requests initiated by the user, obtain real-time running status data and offline hardware profile data of the large language model inference server. Each inference request includes a pre-filling request subtask and a decoding request subtask.

[0037] S12. Based on the service completion results, real-time running status data, and offline hardware profile data of multiple inference requests within the previous preset sliding time window obtained after the previous preset sliding time window ends, determine the task execution ratio of pre-filled request subtasks and decoding request subtasks within the current preset sliding time window.

[0038] S13. Generate batch composition optimization suggestion data based on the real-time hardware resource utilization rate in the real-time running status data, the number of inference requests in the current inference request queue, and the shape information of the pre-compiled computation graph in the offline hardware profile data.

[0039] S14. Based on real-time running status data, task execution ratio, and batch composition optimization suggestions, determine the task execution mode and batch size for each scheduling iteration within the current preset sliding time window. The task execution mode indicates whether to execute a pre-filled request subtask or a decoding request subtask.

[0040] This embodiment provides a large language model inference scheduling method, which aims to optimize the running efficiency of large language models on the inference server. Through an intelligent scheduling mechanism, it balances the resource requirements of different types of tasks to improve overall service performance and user experience.

[0041] It's important to note that a large language model inference server can be understood as the hardware and software environment that deploys large language models and provides inference services. This server typically includes high-performance computing units (such as GPUs or NPUs) and a corresponding inference engine. An inference request can be understood as a model inference call initiated by a user to the large language model inference server. This inference request usually includes a prompt input by the user and the output expected to be generated by the model.

[0042] The inference request includes a prefill request subtask and a decoding request subtask. The prefill request subtask corresponds to the prefill phase of the inference request, which primarily processes the complete sequence of prompt words input by the user, performs parallel computation to generate the first output word, and builds a key-value cache. This prefill phase is typically computationally intensive. The decoding request subtask corresponds to the decoding phase of the inference request, which generates subsequent outputs word by word based on the existing key-value cache. This decoding phase is typically memory-intensive.

[0043] Real-time runtime status data refers to various operational metrics of the large language model inference server at the current moment. Examples include the real-time utilization of hardware resources (such as GPU computing power and memory bandwidth), the number of requests in the current inference request queue, and the remaining processing time for each request. Offline hardware profiling data can be understood as hardware performance characteristic data obtained offline before the inference server is deployed, such as baseline execution latency and resource consumption corresponding to different batch sizes, lexical counts, and pre-compiled computation graph shapes.

[0044] A preset sliding time window refers to a pre-defined time period during which scheduling decisions are evaluated and adjusted. This preset sliding time window slides forward over time to continuously adapt to dynamically changing loads. Task execution ratio refers to the relative priority or time percentage of pre-filled request subtasks and decoding request subtasks in the allocation of scheduling resources within a specific time window (i.e., the preset sliding time window).

[0045] Real-time hardware resource utilization can be understood as the degree to which the hardware resources (such as GPUs and NPUs) of the large language model inference server are actually used at the current moment, such as GPU computing unit utilization and memory bandwidth utilization. The current inference request queue refers to the ordered set of inference requests waiting to be processed by the large language model inference server. The shape information of the pre-compiled computation graph refers to the structure and size information of the computation graph formed after compilation optimization under different input sequence lengths and batch sizes. This shape information of the pre-compiled computation graph can affect the efficiency and resource consumption of the large language model computation.

[0046] Batch composition optimization suggestions can be understood as optimization recommendations on how to organize batches (e.g., batch size, included request types, etc.) for the current scheduling iteration, based on real-time runtime status and offline hardware profile data. This batch composition optimization suggestion data may include a recommended batch size range; for example, for pre-filling tasks, a recommended batch size of [8, 16] is suggested; for decoding tasks, a recommended batch size of [32, 64] is suggested. This batch composition optimization suggestion data may also include a priority list, indicating which batch size combinations are more efficient under different hardware resource utilization levels.

[0047] Scheduling iteration can be understood as the basic unit of time in the large language model inference scheduling process, where the scheduler makes a decision and executes the corresponding task. Multiple scheduling iterations occur within a preset time window. Task execution mode can be understood as the scheduler's choice, within a single scheduling iteration, to execute either a batch of pre-filled request subtasks or a batch of decoding request subtasks. Batch size refers to the number of inference requests or lexical units processed simultaneously within a single scheduling iteration.

[0048] In this embodiment, firstly, in response to multiple inference requests initiated by the user, the real-time running status data and offline hardware profile data of the large language model inference server are obtained. Each inference request includes a pre-filled request subtask and a decoding request subtask.

[0049] In some examples, this real-time runtime status data can be obtained by periodically querying the operating system or hardware monitoring interface, such as reading GPU utilization, memory usage, and the length of the current inference request queue at fixed time intervals (e.g., 100 milliseconds). Offline hardware profiling data can be pre-stored offline in a configuration file or database, such as recording the average execution time of the pre-filling and decoding stages under different model input lengths and batch sizes, as well as throughput and latency data for different pre-compiled computation graph shapes.

[0050] After obtaining the real-time running status data and offline hardware profile data of the large language model inference server, the task execution ratio of the pre-filled request subtask and the decoding request subtask in the current preset sliding time window can be determined based on the service completion results of multiple inference requests within the previous preset sliding time window obtained after the previous preset sliding time window ends, the real-time running status data, and the offline hardware profile data.

[0051] In some examples, an initial task execution ratio can be calculated using a simple weighted average based on the number of pre-filled request subtasks and decoding request subtasks in the queue from real-time runtime status data, and the average processing time of the two types of tasks from offline hardware profiling data. For example, if the number of pre-filled request subtasks is much greater than that of decoding request subtasks, the ratio of pre-filled request subtasks can be appropriately increased. In other examples, the task execution ratio can be dynamically adjusted based on real-time hardware resource utilization (e.g., GPU computing power utilization and memory bandwidth utilization) and the resource preferences of the two types of tasks from offline hardware profiling data. For example, if current computing resources are relatively idle but memory access bandwidth is tight, the ratio of computationally intensive pre-filled request subtasks can be appropriately increased.

[0052] In some examples, the specific process for determining the task execution ratio of the pre-filled request subtask to the decoding request subtask within the current preset sliding time window is as follows: At the beginning of each scheduling cycle, a preliminary task execution ratio of the pre-filled request subtask to the decoding request subtask is determined based on real-time running status data and offline hardware profile data. In subsequent scheduling cycles, to ensure the effectiveness and adaptability of the scheduling strategy, after the previous preset sliding time window ends, the actual service completion results of the multiple inference requests processed within that window are statistically analyzed. That is, the actual performance feedback of the large language model inference server in processing inference requests within the previous scheduling cycle (i.e., the previous preset sliding time window) is obtained.

[0053] By analyzing service completion results, we can assess the effectiveness of scheduling strategies over a past period and whether they have met service quality requirements. This statistical process can be achieved by recording whether preset service metrics (e.g., first token latency, token generation latency, etc.) are met upon completion of each inference request, and then summarizing these records at the end of the window. Alternatively, we can monitor system logs or performance counters, process this data in batches at the end of the window, and extract the service completion status of each request. In other words, the service completion result is a comprehensive set of performance statistics after all inference requests have been completed within the previous sliding time window. For example, whether requests achieved SLOs within the window, whether there were any SLO timeouts; average request latency, queuing time; task backlog, hardware resource utilization saturation; new request acceptance rate, and latency performance of existing Decode tasks, etc. These service completion results, such as whether preset latency requirements are met, constitute direct feedback on the previous scheduling strategy. Based on this feedback information, we can further analyze and dynamically adjust the task execution ratio within the current preset sliding time window. For example, if a request with incomplete service metrics is found in the previous window, it indicates that the current scheduling strategy may be too aggressive or the resources may be unevenly allocated. The ratio of pre-filled request subtasks to decoded request subtasks can be adjusted accordingly to improve service quality in the current window.

[0054] Subsequently, based on this historical performance feedback, the task execution ratio for the current scheduling cycle (i.e., the current preset sliding time window) is dynamically adjusted to optimize future scheduling decisions, making them more adaptive and accurate. For example, if service completion results show poor historical performance, the ratio needs to be adjusted to improve it; if service completion results show good performance, further optimization can be attempted. This determination process can employ rule-based strategies. For instance, if a large number of requests failed to complete service metrics in the previous window, the proportion of pre-filled request subtasks is reduced, while the proportion of decoding request subtasks is increased to ensure that requests that have begun decoding can complete faster. Conversely, if all requests complete successfully, the proportion of pre-filled request subtasks can be appropriately increased to improve throughput. Alternatively, more complex adaptive algorithms, such as reinforcement learning or PID controllers, can be used to automatically learn and adjust the task execution ratio based on service completion results as feedback signals, maximizing a specific performance metric (such as throughput, average latency, etc.).

[0055] It should be understood that the execution ratio of pre-filled request subtasks to decoding request subtasks can be flexibly adjusted based on real-time changes in the request queue and hardware utilization, as well as offline preset hardware performance characteristics. Compared with traditional fixed ratios or simple heuristic strategies, this avoids interference from pre-filled tasks to decoding tasks and eliminates the overhead of cross-instance KV Cache migration. This achieves more refined inter-stage coordination in scenarios with limited single-card resources, significantly improving effective throughput. Furthermore, by introducing a statistical and feedback mechanism for the completion results of inference request services within the previous preset sliding time window, the determination of the task execution ratio is no longer static but adaptively responds to changes in system load and performance. This makes scheduling decisions more adaptive and accurate. It not only effectively addresses the dynamically changing load and hardware resource conditions of the large language model inference server, avoiding service quality degradation due to improper scheduling ratios, but also continuously optimizes resource allocation, ensuring that inference requests are completed efficiently and promptly, thereby significantly improving overall service performance and user satisfaction.

[0056] Then, based on the real-time hardware resource utilization, the number of inference requests in the current inference request queue, and the shape information of the pre-compiled computation graph in the offline hardware profile data, batch composition optimization suggestion data is generated. Specifically, the maximum batch capacity that the current system can handle can be initially estimated based on the real-time hardware resource utilization (e.g., GPU core utilization) and the number of requests in the current inference request queue. Then, combined with the performance characteristics corresponding to different pre-compiled computation graph shapes in the offline hardware profile data (e.g., a certain shape has higher throughput at a specific batch size), one or more candidate schemes for batch composition are generated. For example, if there are multiple short-hint requests and a few long-hint requests in the current queue, it can be suggested to form a large batch containing multiple short-hint requests, or to form a small batch for long-hint requests separately. In another implementation, the optimal batch size and request combination can be directly matched based on the real-time hardware resource utilization, the number of inference requests, and the shape information of the pre-compiled computation graph through a preset lookup table or heuristic rules. For example, if the real-time hardware resource utilization is low and there are a large number of requests with similar shapes in the queue, it is suggested to form a large batch of the same shape.

[0057] Finally, based on the real-time running status data, the task execution ratio, and the batch composition optimization suggestion data, the task execution mode and batch size for each scheduling iteration within the current preset sliding time window are determined. The task execution mode is used to indicate whether to execute the pre-filled request subtask or the decoding request subtask.

[0058] In some examples, the scheduler can prioritize executing the task type with the longest waiting time or closest to the Service Level Objective (SLO) deadline, based on the waiting time of each task in the real-time runtime status data, the set execution ratio of that task, and the batch composition optimization suggestion data. For example, if the task execution ratio favors decoding request subtasks, and there is an efficient decoding batch scheme in the batch composition optimization suggestion data, the scheduler will choose to execute the decoding request subtask batch and adopt the suggested batch size.

[0059] In other examples, the scheduler can select the task execution mode and batch size that maximizes the throughput of the current iteration based on the availability of hardware resources in real-time runtime status data, combined with task execution ratios and batch composition optimization suggestions. For example, if the current GPU computing power is sufficient and there is a large pre-filled batch scheme in the batch composition optimization suggestions, the scheduler may choose to execute the large pre-filled batch to make full use of computing power, even if the task execution ratio is slightly biased towards decoding.

[0060] It should be understood that by combining real-time hardware resource utilization, the number of requests in the inference request queue, and the shape information of the pre-compiled computation graph, batch composition optimization suggestion data is generated, and the task execution mode and batch size for each scheduling iteration are determined accordingly. This overcomes the shortcomings of existing dynamic PD (Power Deployment) schemes in terms of hardware universality and scheduling flexibility. This method not only considers real-time resource conditions and the number of requests but also makes in-depth use of the shape information of the pre-compiled computation graph to optimize batch composition, thereby reducing data filler redundancy and improving computational efficiency. This batch optimization strategy based on multi-dimensional data feedback allows the method to achieve iterative-granular load-aware scheduling on general-purpose accelerators without relying on specific hardware fine-grained computation unit partitioning capabilities. Compared with existing batch processing methods with fixed batch sizes or based solely on the number of requests, this method can more effectively utilize hardware resources and adapt to dynamically changing loads, thereby maximizing the system's effective throughput while meeting service level objectives.

[0061] For example, suppose a user continuously initiates inference requests on a large language model inference server. This server needs to process these requests and maximize effective throughput while meeting Service Level Objectives (SLOs). First, the server continuously acquires real-time runtime status data and offline hardware profiling data. For instance, every 50 milliseconds, the system collects GPU compute unit utilization, memory bandwidth utilization, the number of requests waiting to be processed in the current inference request queue, and the current processing progress of each request (e.g., the number of tokens processed in the pre-filling stage and the number of tokens generated in the decoding stage). Simultaneously, the server pre-stores offline hardware profiling data, which records the baseline execution time, resource consumption, and shape information of the pre-compiled computation graph for pre-filling and decoding tasks on the current hardware under different batch sizes and input sequence lengths.

[0062] Next, based on the inference request service completion results, real-time running status data, and offline hardware profile data acquired within the previous preset sliding time window, the task execution ratio of pre-filling request subtasks to decoding request subtasks is dynamically determined within the current preset sliding time window. For example, at the beginning of a certain time window, if real-time running status data indicates that the pre-filling request subtask queue is growing rapidly due to a large influx of new requests, while the average token generation time (TPOT) of decoding request subtasks begins to fluctuate, the task execution ratio can be initially adjusted to, for example, 60% for pre-filling and 40% for decoding, based on the characteristic in the offline hardware profile data that the pre-filling task requires more computing power than the decoding task, and the upward trend in computing power utilization in the real-time data. If subsequent real-time running status data further shows that, despite a high pre-filling ratio, the first token generation time (TTFT) or TPOT of some decoding request subtasks is approaching their SLO threshold, the task execution ratio can be adjusted again, for example, increasing the decoding ratio to 50% or even higher to avoid SLO violations.

[0063] Simultaneously, batch composition optimization suggestions can be generated based on real-time hardware resource utilization from real-time runtime status data, the number of inference requests in the current inference request queue, and the shape information of the pre-compiled computation graph from offline hardware profiling data. For example, the current real-time hardware resource utilization shows that the GPU computing power utilization is 85%, the memory bandwidth utilization is 70%, and there are 20 pre-filled request subtasks (average prompt word length of 200) and 30 decoding request subtasks (average number of generated tokens of 50) in the inference request queue. Offline hardware profiling data indicates that for pre-filled tasks with a prompt word length of 200, a batch size of 8 is more efficient; for decoding tasks, a batch size of 16 is more efficient. In this case, the batch composition optimization suggestion might be: for pre-filled tasks, two batches can be formed, each containing 8 requests; for decoding tasks, two batches can be formed, each containing 15 requests. These suggestions consider how to minimize data fill redundancy and maximize throughput under the current hardware resources.

[0064] Finally, based on real-time runtime data, the determined task execution ratio, and batch composition optimization suggestions, the task execution mode and batch size for each scheduling iteration within the current preset sliding time window are determined. For example, in a certain scheduling iteration, the task execution ratio indicates that pre-filling tasks should be prioritized, the batch composition optimization suggestions provide two pre-filling batch schemes, and the real-time runtime data indicates that there is still spare GPU computing power. In this case, the scheduler will choose to execute one pre-filling request subtask batch, using the batch size recommended in the batch composition optimization suggestions (e.g., 8 requests). In the next scheduling iteration, if the task execution ratio is adjusted to prioritize decoding, and there is an efficient decoding batch scheme in the batch composition optimization suggestions, then the scheduler will choose to execute one decoding request subtask batch, using its recommended batch size (e.g., 16 requests). Through this dynamic adjustment, it is possible to flexibly switch between pre-filling request subtasks and decoding request subtasks, and adjust the batch size according to real-time load and hardware characteristics, thereby maximizing overall effective throughput while meeting SLO requirements.

[0065] It is understood that this application provides a large language model inference scheduling method, including: responding to multiple inference requests initiated by a user, acquiring real-time runtime status data and offline hardware profile data of the large language model inference server. Each inference request includes a pre-filling request subtask and a decoding request subtask. Then, based on the service completion results of multiple inference requests within the previous preset sliding time window, the real-time runtime status data, and the offline hardware profile data obtained after the previous preset sliding time window ends, the task execution ratio of the pre-filling request subtask and the decoding request subtask within the current preset sliding time window is determined. Based on the real-time hardware resource utilization rate in the real-time runtime status data, the number of inference requests in the current inference request queue, and the shape information of the pre-compiled computation graph in the offline hardware profile data, batch composition optimization suggestion data is generated. Finally, based on the real-time runtime status data, the task execution ratio, and the batch composition optimization suggestion data, the task execution mode and batch size for each scheduling iteration within the current preset sliding time window are determined. The task execution mode is used to indicate whether to execute the pre-filling request subtask or the decoding request subtask. This method fundamentally avoids the structural costs of existing coupled and separated architectures by alternately executing pre-filled request subtasks and decoding request subtasks on the same accelerator card along the time dimension. At the same time, this method maximizes the effective throughput and resource utilization efficiency of the inference service under the current load by adjusting the task execution ratio of pre-filled request subtasks and decoding request subtasks in real time according to the actual dynamic load at the iteration granularity, thus making up for the shortcomings of existing dynamic hybrid deployment schemes in balancing hardware universality and scheduling flexibility.

[0066] In one possible implementation, before determining the task execution ratio of pre-filled request subtasks to decoding request subtasks within the current preset sliding time window based on the service completion results of multiple inference requests within the previous preset sliding time window, real-time running status data, and offline hardware profile data obtained after the previous preset sliding time window has ended, the method includes: Retrieve the previous running status data of the large language model inference server within the previous preset sliding time window.

[0067] Based on the baseline execution latency corresponding to different batch sizes, token counts, and pre-compiled computation graph shape information in the offline hardware profiling data, as well as the remaining latency margin for the first token request, the latency margin for each token decoding request, and the hardware resource utilization rate in the previous running status data, the estimated first token latency and the estimated per-token generation latency for each inference request within the previous preset sliding time window are obtained respectively.

[0068] The estimated first token latency and estimated token generation latency for each inference request are compared with the preset first token latency constraint threshold and the preset token generation latency constraint threshold, respectively, to determine the service completion result for each inference request within the previous preset sliding time window; wherein, the service completion result is either the service indicator is not completed or the service indicator is completed.

[0069] In actual operation, accurately and effectively evaluating the service completion status of each inference request within the previous preset sliding time window is crucial to ensuring the rationality of subsequent task execution ratio adjustments. Failure to accurately determine whether the service has met the standards may result in a lack of basis for adjusting the task execution ratio, thereby affecting the overall performance and user experience of the large language model inference server. Therefore, it is necessary to accurately obtain the service completion results corresponding to each inference request within the previous preset sliding time window.

[0070] First, the previous runtime status data of the large language model inference server within the previous preset sliding time window is obtained. This aims to collect runtime status information of the large language model inference server within a specific time period. This previous runtime status data may include, but is not limited to, real-time usage of hardware resources (such as GPU utilization, memory usage, and network bandwidth), the length of the inference request queue, the number of requests currently being processed, and system load. Obtaining this runtime status data is fundamental to evaluating service performance, providing necessary contextual information for subsequent latency estimation. For example, GPU utilization and memory usage can be obtained by periodically querying the operating system or hardware monitoring interface, or request queue status and system load can be obtained through the service's internal log system and metric collector.

[0071] Then, based on the baseline execution latency corresponding to different batch sizes, token counts, and pre-compiled computation graph shapes in the offline hardware profiling data, and the remaining latency margin for the first token request in the queue head request, the per-token latency margin for decoding requests, and hardware resource utilization in the previous running state data, the estimated first token latency and estimated per-token generation latency for each inference request within the previous preset sliding time window are obtained. The core of this step is to accurately predict the first token latency and per-token generation latency for each inference request. The offline hardware profiling data provides a performance reference under ideal or baseline conditions, such as the pre-measured execution latency for different batch sizes, input token counts, and pre-compiled computation graph shapes. Combining the real-time dynamic information in the previous running state data, such as the remaining latency margin for the first token request in the queue head request, the per-token latency margin for decoding requests, and the current hardware resource utilization, the baseline latency can be corrected to obtain an estimated latency that better reflects the actual operating environment. For example, a machine learning-based model can be built that takes offline hardware profile data and previous running state data as input features and outputs the estimated first token latency and token generation latency; or, a rule-based heuristic method can be used to adjust the baseline latency linearly or non-linearly based on hardware resource utilization.

[0072] Here, batch size refers to the number of prefill or decode request subtasks executed in parallel during a single scheduling iteration. Token count refers to the number of input tokens after input prompt word segmentation and the number of output tokens generated during inference; it is a core independent variable for latency measurement in large language models. The shape information of the pre-compiled computation graph refers to the structural features of the model after static compilation, such as input length range, operator arrangement, memory usage characteristics, performance curves under different input tokens / batches, and computational parallelism constraints. Baseline execution latency refers to the theoretical reference execution latency under standard hardware idle / standard load conditions in the offline hardware profile, given batch size, token count, and computation graph shape; it serves as the baseline for latency prediction.

[0073] The remaining latency margin for the first token request at the front of the queue refers to the available time remaining before the first token's SLO (Solution Timeout) threshold. The first token is the first output token returned by the model, corresponding to the TTFT (Time-To-Flight) metric. The per-token latency margin for decoding requests refers to the remaining time margin corresponding to the maximum allowed time for generating each token during the Decode phase, which can measure whether the continuous generation process will violate latency constraints. Hardware resource utilization refers to the real-time utilization rate of accelerator card computing power, video memory, memory access bandwidth, and computing units within the previous window, used to correct for load disturbances in the baseline latency.

[0074] The estimated first token latency refers to the total estimated time from the initial access to the return of the first token for a single inference request, calculated by combining offline baseline latency and historical window load status. The estimated per-token generation latency refers to the average estimated time for generating each token during the Decode phase. The preset first token latency constraint threshold refers to the pre-set maximum allowed latency limit for the first token (SLO hard threshold); exceeding this limit constitutes a breach. The preset per-token generation latency constraint threshold refers to the pre-set maximum generation time limit for a single token during Decode, serving as a standard for judging per-token service compliance.

[0075] Finally, the estimated first token latency and estimated per-token generation latency for each inference request are compared with the preset first token latency constraint threshold and the preset per-token generation latency constraint threshold, respectively, to determine the service completion result for each inference request within the previous preset sliding time window. This step aims to determine whether each inference request meets its Quality of Service (QoS) requirements. The preset first token latency constraint threshold and the preset per-token generation latency constraint threshold are pre-set performance targets that indicate the user's expectations for response speed.

[0076] By comparing the estimated first token latency and estimated per-token generation latency obtained in the previous steps with these thresholds, it can be determined whether the request completed the generation of the first token and the subsequent token generation within the specified time. If the estimated latency is less than or equal to the corresponding constraint threshold, the service metric for the request is considered complete; otherwise, the service metric is considered incomplete. For example, the comparison logic can be simply implemented using conditional statements: if the estimated first token latency is greater than the preset first token latency constraint threshold, the service metric is incomplete; if the estimated per-token generation latency is greater than the preset per-token generation latency constraint threshold, the service metric is incomplete.

[0077] Suppose that within the previous preset sliding time window, the large language model inference server processed multiple inference requests. To evaluate the service completion status of these requests, the previous runtime data within that time window is first retrieved. For example, it records an average GPU utilization of 85%, a memory usage of 70%, and an average inference request queue length of 5. Simultaneously, offline hardware profiling data stores baseline latency for generating the first token and token-by-token under different batch sizes (e.g., 1, 4, 8), token counts (e.g., 32, 64, 128), and pre-compiled computation graph shapes. For instance, for a specific inference request, its batch size is 4, and the number of input tokens is 64. Then, based on the offline hardware profiling data, the baseline first token latency for this configuration is found to be 200ms, and the baseline token-by-token generation latency is 50ms. Combining the waiting time of the request in the queue recorded in the previous runtime data, the current hardware resource utilization (e.g., 85%), and other factors, the baseline latency is corrected using a pre-trained prediction model (e.g., a regression model based on historical data). Assuming the adjusted estimated first token latency is 250ms and the estimated per-token generation latency is 60ms, these estimated latencies are then compared to preset first token latency constraint thresholds (e.g., 300ms) and preset per-token generation latency constraint thresholds (e.g., 70ms). Since 250ms is less than 300ms and 60ms is less than 70ms, the service completion result for this inference request is determined as service metric completion. If another request has an estimated first token latency of 350ms, even if its estimated per-token generation latency meets the requirements, the service completion result for that request will still be determined as service metric incomplete. By performing similar evaluations on all requests, the set of service completion results for all inference requests within the previous preset sliding time window can be obtained.

[0078] It should be understood that this embodiment can provide a refined evaluation of the actual performance of the large language model inference server within the previous preset sliding time window. By comprehensively considering offline hardware profile data and real-time operational status data, the first token latency and token generation latency of each inference request can be accurately estimated, and compared with preset service quality constraints to objectively determine whether the service meets the standards.

[0079] This precise service completion result evaluation mechanism provides a solid data foundation and decision-making basis for subsequently adaptively adjusting the task execution ratio of pre-filled request subtasks and decoding request subtasks. It avoids the drawbacks of adjustments based on rough estimates or empirical rules, significantly improves the accuracy and responsiveness of scheduling strategies, helps optimize hardware resource utilization, reduces request latency, and ultimately enhances user experience, ensuring that the large language model inference service can continuously and stably provide high-quality service.

[0080] In one possible implementation, based on the service completion results of multiple inference requests within the previous preset sliding time window, real-time running status data, and offline hardware profile data obtained after the previous preset sliding time window has ended, the task execution ratio of pre-filled request subtasks to decoding request subtasks within the current preset sliding time window is determined, including: If, within the previous preset sliding time window, at least one inference request results in a service indicator that is not completed, then the proportion of pre-filled request subtasks in the task execution ratio is reduced to obtain the task execution ratio.

[0081] If, within the previous preset sliding time window, the service completion results corresponding to multiple inference requests are all service completion results, and there are multiple pending inference requests, then the proportion of pre-filled request subtasks in the task execution ratio is increased to obtain the task execution ratio.

[0082] This embodiment proposes a specific method for determining the task execution ratio within the current preset sliding time window based on the service completion results of multiple inference requests within the previous preset sliding time window. Specifically, when at least one inference request in the previous preset sliding time window has a service completion result indicating that the service metric is not met, the proportion of the pre-filled request subtask in the task execution ratio is reduced to obtain the task execution ratio. This step aims to prioritize user experience, especially for requests sensitive to the latency of first token generation, by reducing the priority of the pre-filled task when system performance fails to meet the service metric, thereby allocating more computing resources to the decoding task. For example, the proportion of the pre-filled request subtask can be decreased according to a preset fixed step size (such as 5% or 10%), or the decreasing step size can be dynamically adjusted through an adaptive algorithm based on factors such as the number of incomplete requests and the degree of incompleteness (such as the extent to which the latency threshold is exceeded).

[0083] Conversely, if within the previous preset sliding time window, multiple inference requests all result in service completion, and there are multiple pending inference requests, then the proportion of pre-filled request subtasks in the task execution ratio is increased to obtain the task execution ratio. This step aims to accelerate the preprocessing of new requests by increasing the proportion of pre-filled tasks when the system performance is good and there are sufficient pending requests, thereby improving overall throughput, shortening the average waiting time of requests, and making fuller use of hardware resources. For example, the proportion of pre-filled request subtasks can be increased in a preset fixed step, or the increment step can be dynamically adjusted by an optimization algorithm based on factors such as the number of pending requests and real-time hardware resource utilization (e.g., GPU utilization) to maximize the proportion of pre-filled tasks while meeting service metrics.

[0084] Suppose that the large language model inference server has a preset sliding time window of 10 seconds. At the end of a certain sliding time window, the service completion results of all inference requests processed within that window are statistically analyzed. If it is found that the first token generation latency of one of the 10 inference requests exceeds a preset threshold, i.e., the service metric is not completed, then it is determined that the latency-sensitive decoding task needs to be prioritized. In this case, the proportion of the current pre-filling request subtask is reduced from, for example, 70% to 60%, to provide more scheduling opportunities for the decoding task in the next 10-second sliding time window. Conversely, if all of the 10 inference requests successfully meet their latency requirements, i.e., the service metric is completed, and there are still a large number (e.g., 50) of pending inference requests in the current inference request queue, this indicates that the system has the capacity to handle more pre-filling tasks to improve throughput. In this case, the proportion of the pre-filling request subtask is increased from 70% to 75% to accelerate the preprocessing of new requests. This adjustment ensures that the task execution ratio can be optimized according to actual operating conditions, thereby better balancing service quality and system throughput.

[0085] It should be understood that in this embodiment, a feedback mechanism based on historical service completion results is introduced to achieve dynamic adaptive adjustment of the task execution ratio of pre-filling request subtasks and decoding request subtasks. After each preset sliding time window ends, the task execution ratio of the current window is intelligently adjusted based on the actual performance of the previous window, i.e., the service completion result. When a decline in service quality is detected, the proportion of pre-filling tasks is proactively reduced to ensure the timely completion of decoding tasks and avoid request "starvation" or excessive latency. Conversely, when service quality is guaranteed and the system still has processing capacity, the proportion of pre-filling tasks is increased to improve overall request processing efficiency and throughput. This bidirectional adjustment mechanism allows the scheduling strategy to flexibly adapt to constantly changing inference load and hardware resource conditions, avoiding resource waste or service quality degradation caused by a fixed ratio. Thus, while ensuring service quality (such as first token latency and token generation latency), it maximizes hardware resource utilization and system throughput.

[0086] In one possible implementation, batch composition optimization suggestion data is generated based on the real-time hardware resource utilization from the real-time runtime status data, the number of inference requests in the current inference request queue, and the shape information of the pre-compiled computation graph from the offline hardware profiling data. This includes: Based on the shape information of pre-compiled computation graphs in offline hardware profiling data and the real-time hardware resource utilization in real-time running status data, the computational cost and data filling redundancy under the shape information of multiple candidate pre-compiled computation graphs are quantified.

[0087] Based on the number of inference requests in the current inference request queue, the current inference request queue is matched with the shape information of the candidate pre-compiled computation graph to identify the first scheduling point when the shape mismatch risk is at its maximum and the second scheduling point when the shape alignment benefit is at its maximum.

[0088] Based on computational cost, data redundancy, the first scheduling point, and the second scheduling point, batch composition optimization suggestion data is generated.

[0089] In this embodiment, firstly, based on the shape information of the pre-compiled computation graph in the offline hardware profiling data and the real-time hardware resource utilization in the real-time runtime status data, a quantitative evaluation is performed on multiple potential batch composition schemes (i.e., the resource consumption and efficiency loss when executing pre-compiled computation graphs with different batch sizes and sequence lengths on specific hardware). This quantification includes calculating the computational cost and data padding redundancy of each scheme, thereby providing basic data for subsequent optimization decisions. In this way, the resource consumption and efficiency loss of different batch configurations under the current hardware conditions can be clearly understood.

[0090] In this context, computational cost refers to the computational resource consumption required to execute a pre-compiled computation graph shape (i.e., a combination of batch size and sequence length) on specific hardware. Examples include floating-point operations (FLOPs), memory bandwidth requirements, and core utilization. Computational cost can be quantified by pre-recording benchmark execution times and power consumption of different computation graph shapes on different hardware in offline hardware profiling data, or by online simulation and performance counter sampling. For instance, benchmark tests can be performed on computation graphs with different batch sizes and sequence length combinations, recording their execution times on specific hardware as quantified computational cost values. Alternatively, based on the structure of the computation graph and hardware characteristics, the required computational and memory access volumes can be estimated using theoretical models, thereby quantifying the computational cost.

[0091] Data padding redundancy refers to the additional computational or storage overhead incurred when combining multiple inference requests into a batch. This is because the sequence lengths or batch sizes of different requests vary, necessitating padding of shorter sequences to meet the fixed shape requirements of the pre-compiled computation graph. Data padding redundancy can be quantified by calculating the proportion of padding tokens to the total number of tokens in the batch, or by converting the additional computation time or memory consumption from padding operations into redundancy. For example, if a batch contains tokens of length... , ,..., Given a sequence and a precompiled computation graph with a maximum sequence length of L_max, the padding redundancy can be quantified as... Where N is the number of requests in the batch, i = 1, 2, ..., N. Alternatively, based on the actual hardware resource usage (such as memory bandwidth and computing units) of the filling operation, it can be converted into a cost value.

[0092] Then, combining the number and characteristics of requests in the current inference request queue, the requests in the queue are matched with the shape information of these candidate precompiled computation graphs. Through this matching, two key scheduling points are identified: the first scheduling point corresponds to the maximum shape mismatch risk, indicating that the current queue requests have a very low matching degree with the existing computation graph shape, which may lead to serious resource waste or processing delays; the second scheduling point corresponds to the maximum shape alignment benefit, indicating that resource utilization or processing efficiency can be significantly improved through some batch composition adjustment.

[0093] Shape mismatch risk refers to the risk that requests in the current inference request queue do not match the shape of the available precompiled computation graph, resulting in inefficient batching and potentially causing resource waste or scheduling delays. For example, shape mismatch occurs when the queue consists of short-sequence requests, but the available precompiled computation graphs are designed for long sequences. The first scheduling point can be identified by analyzing the sequence length distribution and batch size requirements of requests in the current queue, and their matching degree with the shape of the existing precompiled computation graph. When this mismatch reaches a preset threshold or shows a worsening trend, it is considered the point of maximum shape mismatch risk. For example, an indicator can be set, such as "the percentage of requests in the queue that cannot be completely filled into any existing computation graph shape." When this percentage exceeds a certain threshold, the shape mismatch risk is considered to be maximum.

[0094] Shape alignment benefits refer to the performance improvements achieved by adjusting batch composition or selecting a specific pre-compiled computation graph shape, which maximizes the reduction of data fill redundancy, improves hardware utilization, or shortens overall processing time. Identifying the second scheduling point involves simulating the performance gains obtained by processing requests in the queue under different batch composition schemes. The point at which a particular batch composition scheme delivers a significant performance improvement is the point where shape alignment benefits are maximized. For example, one can evaluate metrics such as the average fill rate and hardware utilization of requests in the queue under different combinations of batch size and sequence length. When these metrics reach or are close to optimal, the corresponding scheduling point is the point where shape alignment benefits are maximized.

[0095] Finally, by comprehensively considering the quantified computational cost, data filling redundancy, and the identified first and second scheduling points, batch composition optimization suggestion data is generated. This suggestion data not only reflects the efficiency of the current batch composition but also incorporates predictions of future scheduling risks and potential benefits, enabling batch scheduling decisions to be more forward-looking and optimized. This effectively solves the problem of how to accurately generate batch composition optimization suggestion data in a dynamic inference environment.

[0096] It should be understood that in this embodiment, by quantifying the computational cost and data filling redundancy of different batch composition schemes, the efficiency of each scheme can be accurately evaluated. Simultaneously, by combining the analysis of the current inference request queue, the scheduling points with the greatest shape mismatch risk and the greatest shape alignment benefit are identified. This ensures that the batch composition optimization suggestion data not only considers the current state but also incorporates predictions of future scheduling risks and potential benefits. This enables the large language model inference scheduling method to make more forward-looking and optimized decisions when determining the task execution mode and batch size for each scheduling iteration, thereby effectively reducing resource waste caused by data filling, improving hardware resource utilization, and ultimately enhancing the overall throughput and response efficiency of the large language model inference service.

[0097] In one possible implementation, based on real-time runtime status data, task execution ratios, and batch composition optimization suggestions, the task execution mode and batch size for each scheduling iteration within the current preset sliding time window are determined, including: Based on the preset starvation prevention hard constraints and the task execution ratio, the task execution mode of the current scheduling iteration is determined.

[0098] Based on the task execution mode and batch composition optimization suggestions, determine the batch size for the current scheduling iteration.

[0099] Preset anti-starvation hard constraints refer to a set of pre-defined rules or conditions designed to prevent specific types of tasks (such as pre-filled request subtasks) from being delayed for extended periods or completely ignored during scheduling. These preset anti-starvation hard constraints ensure that all task types receive a basic chance of execution, avoiding the "starvation" phenomenon caused by over-optimization.

[0100] For example, a maximum waiting time threshold can be set. If the waiting time of a pre-filled request subtask exceeds this threshold, the scheduler is forced to execute the pre-filled request subtask in the next scheduling cycle. Alternatively, a minimum execution frequency can be set. For example, within every N scheduling cycles, regardless of the current optimization suggestions, the pre-filled request subtask must be executed in at least M cycles.

[0101] The task execution ratio refers to the relative proportion of execution opportunities or resources allocated between pre-fill request subtasks and decoding request subtasks within the current preset sliding time window. This ratio can be a floating-point number, such as 0.3, indicating that 30% of scheduling resources or time should be allocated to pre-fill request subtasks within a certain time window. It can also be an integer ratio, such as 3:7, indicating that in a certain number of scheduling decisions, 3 parts are used for pre-filling and 7 parts for decoding.

[0102] At each scheduling decision, a decision tree or rule engine can be used, combined with anti-starvation constraints and task execution ratios, to determine whether the current scheduling iteration will execute pre-filled request subtasks or decoding request subtasks. After determining the task execution mode for the current scheduling iteration, the number of requests to be processed in this scheduling can be determined based on batch composition optimization suggestion data. That is, a recommended batch size can be selected from the batch composition optimization suggestion data and fine-tuned according to the current task execution mode; alternatively, an optimal batch size can be dynamically calculated based on the number of requests waiting to be processed in the current queue, combined with the optimization suggestion data.

[0103] In this embodiment, by introducing a preset anti-starvation hard constraint and combining task execution ratio and batch composition optimization suggestion data, a hierarchical decision-making approach is used to determine the task execution mode and batch size for each scheduling iteration. First, it prioritizes assessing whether any task faces the risk of "starvation," i.e., checking whether the preset anti-starvation hard constraint has been triggered. If the waiting time of a pre-filled request subtask or a decoding request subtask has reached a preset threshold, or its execution frequency is lower than a preset minimum requirement, the scheduler will force the current scheduling iteration's task execution mode to be set to execute the "starvation-risk" task to ensure it is processed in a timely manner. If there is no "starvation" risk, the scheduler will balance the execution opportunities of pre-filled request subtasks and decoding request subtasks based on the current task execution ratio, thereby determining the task execution mode for the current scheduling iteration.

[0104] After determining the task execution mode, batch composition optimization suggestion data is further referenced. This data provides the optimal batch size range or specific value for different task modes under the current hardware resource utilization and request queue status. The scheduler will select or calculate the most suitable batch size from this data based on the determined task execution mode. This decision-making process ensures that while pursuing overall scheduling efficiency and resource utilization, it effectively avoids long waiting times for specific tasks, thereby maintaining system fairness and responsiveness.

[0105] It should be understood that this embodiment effectively addresses the task "starvation" problem that may occur when pursuing scheduling efficiency for large language model inference. By introducing a preset anti-starvation hard constraint and making it the primary consideration in determining the task execution mode, it ensures that all types of requests (especially pre-filled request subtasks) can be processed promptly, even in complex scheduling optimization scenarios, avoiding a decline in user experience due to long waiting times. Furthermore, by balancing the execution of different task types based on task execution ratios and refining batch size using batch composition optimization suggestions, it maximizes hardware resource utilization and overall throughput while ensuring fairness. This comprehensive scheduling strategy not only improves the response speed and stability of the large language model inference service but also optimizes the balance of resource allocation, thereby significantly improving the overall service performance and user satisfaction.

[0106] It should be noted that the embodiments of this application implement load-aware scheduling with hardware universality and iterative granularity on a general-purpose accelerator. Compared with the prior art, this large language model inference scheduling method does not rely on the fine-grained computing unit partitioning capability of specific hardware. It can dynamically coordinate the execution of the pre-filling and decoding stages through time-division multiplexing in scenarios with limited single-card resources. Under real dynamic load, the system can accurately adjust the P / D execution ratio and batch composition based on real-time feedback, which avoids inter-stage interference in the PD coupled architecture and eliminates the KV Cache migration overhead in the PD separated architecture, thereby significantly improving the effective throughput of the large language model inference service.

[0107] Understandably, this large language model inference scheduling method also solves the problem of efficient implementation on accelerators using pre-compiled computation graphs, making it widely applicable to accelerators and feasible for engineering implementation.

[0108] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0109] Corresponding to the large language model inference scheduling method in the above embodiment, Figure 2 The diagram shows a structural schematic of a large language model inference scheduling device according to an embodiment of this application. For ease of explanation, only the parts related to the embodiment of this application are shown.

[0110] Reference Figure 2 The large language model inference scheduling device of this embodiment is applied to or runs on the large language model inference server. The large language model inference scheduling device 1 includes: The status acquisition module 10 is used to respond to multiple inference requests initiated by the user and obtain real-time running status data and offline hardware profile data of the large language model inference server; wherein, each inference request includes a pre-filled request subtask and a decoding request subtask.

[0111] The adaptive control module 20 is used to determine the task execution ratio of pre-filled request subtasks and decoding request subtasks in the current preset sliding time window based on the service completion results, real-time running status data and offline hardware profile data of multiple inference requests within the previous preset sliding time window, which are statistically obtained after the previous preset sliding time window ends.

[0112] The graph recognition control module 30 is used to generate batch composition optimization suggestion data based on the real-time hardware resource utilization rate in the real-time running status data, the number of inference requests in the current inference request queue, and the shape information of the pre-compiled computation graph in the offline hardware profile data.

[0113] The adaptive control module 20 is also used to determine the task execution mode and batch size for each scheduling iteration within the current preset sliding time window based on real-time running status data, task execution ratio, and batch composition optimization suggestion data; wherein, the task execution mode is used to indicate whether to execute a pre-filled request subtask or a decoding request subtask.

[0114] like Figure 2 As shown, the large language model inference scheduling device 1 can be applied to or run on a large meta-model inference server. The large language model inference scheduling device 1 mainly consists of three cooperating modules: a state acquisition module 10, an adaptive control module 20, and a graph recognition control module 30. Figure 2 In this context, P represents the pre-filling request subtask, and D represents the decoding request subtask. xP:yD represents the execution ratio of the pre-filling request subtask to the decoding request subtask within the current preset sliding time window. By combining the adaptive control module 20 and the graph recognition control module 30 in a collaborative manner, the execution ratio of the pre-filling request subtask and the decoding request subtask is dynamically adjusted at the iteration granularity based on SLO margin feedback. Simultaneously, the batch composition is optimized by considering the shape information of the pre-compiled computation graph, achieving a balance between hardware universality and scheduling flexibility on a general-purpose accelerator, maximizing the effective throughput of the large language model inference service.

[0115] Specifically, the large language model inference scheduling device 1 acquires real-time running status data and offline hardware profile data through the status acquisition module 10. The adaptive control module 20 dynamically adjusts the task execution ratio within the sliding time window based on the service completion results of multiple inference requests within the previous preset sliding time window, the real-time running status data, and the offline hardware profile data obtained after the previous preset sliding time window ends. The graph recognition control module 30 optimizes the batch composition by using the shape information of the pre-compiled computation graph. The three work together to achieve the unity of hardware universality and iterative granular load-aware scheduling.

[0116] In practical implementation, the status acquisition module 10 can periodically query the operating system or hardware monitoring interface, for example, reading the GPU utilization, video memory usage, and the length of the current inference request queue every fixed time interval (e.g., 100 milliseconds). Offline hardware profiling data can be pre-stored in a configuration file or database, for example, recording the average execution time of the pre-filling and decoding stages under different model input lengths and batch sizes.

[0117] The adaptive control module 20 calculates the optimal task execution ratio of pre-filled request subtasks to decoding request subtasks within the current preset sliding time window based on the service completion results of multiple inference requests in the previous preset sliding time window, real-time running status data, and offline hardware profile data. This balances the resource requirements of computationally intensive and memory-intensive tasks. For example, when real-time running status data shows low GPU computing power utilization but high memory bandwidth utilization, the adaptive control module 20 will appropriately increase the proportion of pre-filled request subtasks to fully utilize idle computing resources.

[0118] The graph recognition control module 30 analyzes the real-time hardware resource utilization and the number of requests in the inference request queue, and combines the shape information of the pre-compiled computation graph to evaluate the computational cost and data filling redundancy of different batch composition schemes.

[0119] Specifically, the graph recognition control module 30 can quantify the computational cost and data padding redundancy under the shape information of multiple candidate pre-compiled computation graphs based on the shape information of the pre-compiled computation graph in the offline hardware profiling data and the real-time hardware resource utilization in the real-time running status data. Then, based on the number of inference requests in the current inference request queue, it matches the current inference request queue with the shape information of the candidate pre-compiled computation graphs to identify the first scheduling point corresponding to the maximum shape mismatch risk and the second scheduling point corresponding to the maximum shape alignment benefit. Finally, based on the computational cost, data padding redundancy, the first scheduling point, and the second scheduling point, it generates batch composition optimization suggestion data. For example, when there are multiple short prompt word requests and a small number of long prompt word requests in the queue, the graph recognition control module 30 may suggest forming a large batch containing multiple short prompt word requests, or forming long prompt word requests into a separate small batch to reduce the performance loss caused by data padding.

[0120] The adaptive control module 20 can also integrate task execution ratio and batch composition optimization suggestion data to determine the task execution mode and batch size for each scheduling iteration.

[0121] Specifically, the adaptive control module 20 can determine the task execution mode of the current scheduling iteration based on preset anti-starvation hard constraints and task execution ratios; and determine the batch size of the current scheduling iteration based on the task execution mode and batch composition optimization suggestion data. For example, if the task execution ratio favors decoding request subtasks, and there is an efficient decoding batch scheme in the batch composition optimization suggestion data, the scheduler will choose to execute the decoding request subtask batch and adopt the suggested batch size. In the next scheduling iteration, if the task execution ratio is adjusted to prioritize pre-filling, and there is an efficient pre-filling batch scheme in the batch composition optimization suggestion data, the scheduler will choose to execute a pre-filling request subtask batch and adopt its recommended batch size. Through this dynamic adjustment, the large language model inference scheduling device 1 can flexibly coordinate the execution of the pre-filling and decoding stages under real dynamic loads, avoiding inter-stage interference in the PD coupled architecture and the overhead of KV Cache cross-instance migration in the PD separated architecture, thereby maximizing effective throughput in scenarios with limited single-card resources.

[0122] In some examples, the large language model inference scheduling device 1 can be deployed on AI accelerator cards such as GPUs or NPUs, and integrated with existing inference engines (such as vLLM, SGLang, etc.). During operation, the status acquisition module 10 can continuously monitor the system status, and the adaptive control module 20 and the graph recognition control module 30 work together to ensure that the effective throughput of the system is maximized while meeting the service level objectives (SLO). For example, when the token generation time (TPOT) of a decoding request is detected to be close to the SLO threshold, the adaptive control module 20 can automatically increase the proportion of decoding request subtasks to ensure service quality; at the same time, the graph recognition control module 30 can optimize the composition of decoding batches, reduce data filler redundancy, and improve resource utilization efficiency.

[0123] It should be understood that, through the above technical solution, this embodiment provides a device for realizing hardware universality and iterative granular load-aware scheduling on a general-purpose accelerator, which effectively solves the problem that the existing PD dynamic mixed deployment scheme fails to take into account both hardware universality and scheduling flexibility, and significantly improves the resource utilization efficiency and effective throughput of large language model inference services.

[0124] It should be noted that the information interaction and execution process between the modules in the above-mentioned large language model inference scheduling device 1 are based on the same concept as the method embodiment of this application. For details on their specific functions and technical effects, please refer to the method embodiment section, which will not be repeated here.

[0125] This application also provides a terminal device, such as... Figure 3 As shown, Figure 3 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application. (Refer to...) Figure 3 The terminal device 4 in this embodiment includes a memory 41, a processor 42, and a computer program stored in the memory 41 and executable on the processor 42. When the processor 42 executes the computer program, it implements the steps in the embodiment of the large language model inference scheduling method described above.

[0126] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement the steps in the above-described method embodiments.

[0127] This application provides a computer program product that, when run on a mobile terminal, enables the mobile terminal to implement the steps described in the various method embodiments.

[0128] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying the computer program code to a photographic device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks.

[0129] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0130] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0131] In the embodiments provided in this application, it should be understood that the disclosed apparatus / network devices and methods can be implemented in other ways. For example, the apparatus / network device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0132] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0133] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A large language model inference scheduling method, characterized in that, The method, applied to a large language model inference server, includes: In response to multiple inference requests initiated by the user, the real-time running status data and offline hardware profile data of the large language model inference server are obtained; wherein, each inference request includes a pre-filled request subtask and a decoding request subtask; Based on the service completion results of multiple inference requests within the previous preset sliding time window obtained after the previous preset sliding time window ends, the real-time running status data, and the offline hardware profile data, the task execution ratio of the pre-filled request subtask and the decoding request subtask within the current preset sliding time window is determined. Based on the real-time hardware resource utilization rate in the real-time running status data, the number of inference requests in the current inference request queue, and the shape information of the pre-compiled computation graph in the offline hardware profile data, batch composition optimization suggestion data is generated. Based on the real-time running status data, the task execution ratio, and the batch composition optimization suggestion data, the task execution mode and batch size for each scheduling iteration within the current preset sliding time window are determined; wherein, the task execution mode is used to indicate whether to execute the pre-filled request subtask or the decoding request subtask.

2. The large language model inference scheduling method of claim 1, wherein, Before determining the task execution ratio of the pre-filled request subtask and the decoding request subtask within the current preset sliding time window based on the service completion results of multiple inference requests within the previous preset sliding time window (obtained statistically after the end of the previous preset sliding time window), the real-time running status data, and the offline hardware profile data, the method includes: Obtain the previous running status data of the large language model inference server within the previous preset sliding time window; Based on the baseline execution latency corresponding to different batch sizes, token counts, and pre-compiled computation graph shape information in the offline hardware profile data, as well as the remaining latency margin for the first token request, the latency margin for each token decoding request, and the hardware resource utilization rate in the previous running state data, the estimated first token latency and the estimated per-token generation latency for each inference request within the previous preset sliding time window are obtained respectively. The estimated first token delay and estimated token generation delay for each inference request are compared with the preset first token delay constraint threshold and the preset token generation delay constraint threshold, respectively, to determine the service completion result corresponding to each inference request within the previous preset sliding time window; wherein, the service completion result is either the service indicator is not completed or the service indicator is completed.

3. The large language model inference scheduling method of claim 2, wherein, The step of determining the task execution ratio of the pre-filled request subtask and the decoding request subtask within the current preset sliding time window based on the service completion results of multiple inference requests within the previous preset sliding time window (obtained after the previous preset sliding time window ends), the real-time running status data, and the offline hardware profile data includes: If, within the previous preset sliding time window, there is at least one inference request whose service completion result is that the service indicator is not completed, then the proportion of the pre-filled request subtask in the task execution ratio is reduced to obtain the task execution ratio. If, within the previous preset sliding time window, the service completion results corresponding to the multiple inference requests are all the service completion results, and there are multiple pending inference requests, then the proportion of the pre-filled request subtask in the task execution ratio is increased to obtain the task execution ratio.

4. The large language model inference scheduling method of claim 1, wherein, The step of generating batch composition optimization suggestion data based on the real-time hardware resource utilization rate in the real-time operating status data, the number of inference requests in the current inference request queue, and the shape information of the pre-compiled computation graph in the offline hardware profile data includes: Based on the shape information of the pre-compiled computation graph in the offline hardware profile data and the real-time hardware resource utilization rate in the real-time running status data, the computational cost and data filling redundancy under the shape information of multiple candidate pre-compiled computation graphs are quantified. Based on the number of inference requests in the current inference request queue, the current inference request queue is matched with the shape information of the candidate pre-compiled computation graph to identify the first scheduling point when the shape mismatch risk is at its maximum and the second scheduling point when the shape alignment benefit is at its maximum. Based on the computational cost, the data filling redundancy, the first scheduling time point, and the second scheduling time point, the batch composition optimization suggestion data is generated.

5. The large language model inference scheduling method of claim 4, wherein, The step of determining the task execution mode and batch size for each scheduling iteration within the current preset sliding time window based on the real-time running status data, the task execution ratio, and the batch composition optimization suggestion data includes: The task execution mode for the current scheduling iteration is determined based on the preset anti-starvation hard constraint and the task execution ratio; Based on the task execution mode and the batch composition optimization suggestion data, the batch size of the current scheduling iteration is determined.

6. A large language model inference scheduling apparatus, characterized by comprising: The device, used in a large language model inference server, includes: The status acquisition module is used to respond to multiple inference requests initiated by the user and acquire real-time running status data and offline hardware profile data of the large language model inference server; wherein, each inference request includes a pre-filled request subtask and a decoding request subtask; The adaptive control module is used to determine the task execution ratio of the pre-filled request subtask and the decoding request subtask in the current preset sliding time window based on the service completion results of multiple inference requests within the previous preset sliding time window obtained after the previous preset sliding time window ends, the real-time running status data, and the offline hardware profile data. The graph recognition control module is used to generate batch composition optimization suggestion data based on the real-time hardware resource utilization rate in the real-time running status data, the number of inference requests in the current inference request queue, and the shape information of the pre-compiled computation graph in the offline hardware profile data. The adaptive control module is further configured to determine the task execution mode and batch size for each scheduling iteration within the current preset sliding time window based on the real-time running status data, the task execution ratio, and the batch composition optimization suggestion data; wherein, the task execution mode is used to indicate whether to execute the pre-filled request subtask or the decoding request subtask.

7. A terminal device, characterized by comprising: The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method as claimed in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 5.

9. A computer program product, characterized in that, Includes a computer program, which, when run, causes the method as described in any one of claims 1 to 5 to be performed.

Images