Methods and Systems for Scheduling Computational Power for Large Language Model Inference

By optimizing the distributed inference scheduling of large language models through global cache indexing and differentiated resource veto thresholds, the highest cache hit rate is ensured, achieving global optimization of load balancing and green energy utilization, and solving the shortcomings of cache reuse rate and green energy utilization rate in existing technologies.

CN123114729APending Publication Date: 2026-11-17HANGZHOU ZHIYUAN KAIWU TECHNOLOGY CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202611150009.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-07-31
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing technologies lack a unified multi-level scheduling architecture in distributed reasoning environments for large language models. The cache hit rate has not been given the highest priority, and there is a lack of differentiated resource veto mechanisms, which makes it difficult to guarantee cache reuse rate and optimize green energy utilization.

Method used

The prefix matching degree is calculated by using a global cache index to generate a candidate node list. Differentiated resource rejection thresholds are set according to session type, and green cost information is combined to select the best node, ensuring the highest priority of cache hit rate and achieving global optimization of load balancing and green energy utilization.

Benefits of technology

While ensuring cache hit rate, the system achieves global optimization of load balancing and green energy utilization in the inference cluster, solving the problems of equalizing scheduling targets and lacking differentiated resource veto mechanisms in existing technologies.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123114729A_ABST
    Figure CN123114729A_ABST
Patent Text Reader

Abstract

This invention discloses a method and system for scheduling computational power for large language model inference. The method includes: acquiring large model inference requests carrying session identifiers and prompt words; calculating and sorting the prefix matching degree between the prompt words and each node based on a global cache index to generate a first candidate list; determining a resource rejection threshold based on the session type, and, in conjunction with node resource status indicators, sequentially removing nodes exceeding the threshold to generate a second candidate list; if nodes with similar prefix matching degrees exist in the second list, selecting a target node based on green cost information; otherwise, selecting the node ranked highest; routing the request to the target node, and establishing or updating the mapping relationship between the session identifier and the target node. By implementing the method of this invention, the maximum hit rate of the KV Cache is ensured, while achieving global optimization of inference cluster load balancing and green energy utilization.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of distributed artificial intelligence inference technology, and more specifically to a method and system for scheduling computing power for large language model inference. Background Technology

[0002] With the widespread application of large language models in natural language processing, computational efficiency and resource utilization during inference have become key technical challenges. In autoregressive generation based on the Transformer architecture, the model needs to compute the key-value matrix corresponding to each input token. To avoid redundant computation of the same prefix, modern inference engines (such as vLLM and SGLang) generally introduce a KV cache mechanism. This mechanism resides the computed token's KV matrix in GPU memory. When subsequent requests contain the same prefix, the cached result can be reused directly, thus skipping the corresponding pre-filling stage. The KV cache hit rate directly determines the inference latency and the effective throughput of the hardware. In a distributed multi-computing center environment, user requests can be routed to any computing node with the required model. However, due to different historical request distributions, the prefix sets cached in the GPU memory of different nodes also differ. Therefore, the routing decision itself significantly affects the KV cache hit rate.

[0003] Existing technologies primarily explore scheduling based on session affinity, cache awareness, and resource thresholds. One approach is session affinity-based scheduling, such as the adaptive load balancing method disclosed in patent CN121501463B, which improves reuse by routing consecutive requests from the same session to the same inference instance. However, this scheme can only reuse the cache within consecutive rounds of the same conversation, failing to detect shared prefixes between different sessions and not prioritizing cache hit rate as the highest priority scheduling consideration. Another approach is key-value (KV) cache awareness-based scheduling, such as patent application CN121501479A, which establishes a global KV cache index and combines it with performance data for weighted comprehensive scoring. This type of scheme typically uses cache hit rate as a factor in the weighted scoring, alongside load and cost. Its drawback is that when nodes with high cache hit rates are under high load, the weighted scoring may sacrifice cache hit principles, failing to guarantee service level objectives. Furthermore, existing technologies include node rejection mechanisms based on resource status thresholds, typically using uniform CPU or memory utilization thresholds to filter nodes. This uniform threshold cannot distinguish the difference in cache dependency between historical and new sessions: for historical sessions with multiple rounds of dialogue, migration will lead to KV cache invalidation and significant performance loss; while for new sessions, the migration cost is smaller. A uniform threshold either leads to unnecessary migration of historical sessions or to new sessions being scheduled to overloaded nodes, failing to achieve a balance between ensuring cache hit rate and maintaining load balancing. Thirdly, green energy-driven computing power scheduling, such as patent CN121008931B, prioritizes minimizing energy costs or carbon emissions, failing to incorporate cache hit rate into core scheduling factors, and thus failing to optimize green energy utilization while ensuring inference performance.

[0004] In summary, existing technologies share the following common shortcomings: First, they lack a unified multi-level scheduling architecture, with each optimization objective operating independently and failing to form an independent decision-making hierarchy that can be prioritized. Second, they do not establish the highest priority for cache hits, often employing a weighted summation approach, which can lead to cache hits being sacrificed due to load or cost factors. Third, they lack differentiated resource veto mechanisms, and using a uniform threshold cannot distinguish the different degrees of dependence on cache reuse between historical and new sessions, making it difficult to effectively guarantee the cache hit rate of historical sessions. Fourth, green energy and cache awareness are not coordinated, making it impossible to optimize green energy utilization while ensuring cache hit rates.

[0005] Therefore, it is necessary to design a new method to achieve global optimization of inference cluster load balancing and green energy utilization while ensuring the highest hit rate of KV Cache. This would address the shortcomings of existing technologies, such as the equalization of cache-aware scheduling targets and the lack of differentiated resource veto mechanisms, which makes it difficult to guarantee the KV Cache reuse rate of high-value sessions and the inability to collaboratively optimize green energy utilization. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method and system for scheduling computing power for large language model inference.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: a method for scheduling computational power for large language model inference, characterized in that it includes:

[0008] Obtain a large model inference request, wherein the large model inference request carries a session identifier and a prompt word;

[0009] Based on the global cache index, the prefix matching degree between the prompt word and the cached prefixes on each computing power node is calculated, and each computing power node is sorted according to the prefix matching degree to generate a first candidate node list; wherein, the global cache index is used to record the information of the cached prefixes on each computing power node;

[0010] The session type to which the large model inference request belongs is determined based on the session identifier. The resource rejection threshold corresponding to the session type is obtained, wherein different session types correspond to different resource rejection thresholds. The resource status indicators of each computing power node are obtained. According to the sorting of the first candidate node list, the resource status indicators of each computing power node are compared with the resource rejection threshold in turn. Computing power nodes whose resource status indicators exceed the resource rejection threshold are eliminated, and a second candidate node list is generated.

[0011] When there are at least two computing power nodes with similar prefix matching in the second candidate node list, the target node is selected from the at least two computing power nodes according to the green cost information corresponding to each computing power node; otherwise, the computing power node ranked first in the second candidate node list is determined as the target node.

[0012] The large model inference request is routed to the target node, and a mapping relationship between the session identifier and the target node is established or updated, wherein the mapping relationship is recorded in the session mapping table.

[0013] This invention also provides a large-model inference computing power scheduling system, comprising:

[0014] A request receiving unit is used to obtain a large model inference request, wherein the large model inference request carries a session identifier and a prompt word;

[0015] A cache affinity scoring unit is used to calculate the prefix matching degree between the prompt word and the cached prefixes on each computing power node based on the global cache index, sort each computing power node according to the prefix matching degree, and generate a first candidate node list; wherein, the global cache index is used to record the information of the cached prefixes on each computing power node;

[0016] The resource status monitoring unit is used to determine the session type to which the large model inference request belongs based on the session identifier, obtain the resource rejection threshold corresponding to the session type, wherein different session types correspond to different resource rejection thresholds; obtain the resource status indicators of each computing power node, and compare the resource status indicators of each computing power node with the resource rejection threshold in turn according to the sorting of the first candidate node list, remove computing power nodes whose resource status indicators exceed the resource rejection threshold, and generate a second candidate node list;

[0017] The green cost assessment unit is used to select a target node from at least two computing power nodes with similar prefix matching degrees when there are at least two computing power nodes in the second candidate node list; otherwise, the computing power node ranked first in the second candidate node list is determined as the target node.

[0018] The routing and scheduling unit is used to route the large model inference request to the target node and establish or update the mapping relationship between the session identifier and the target node, and the mapping relationship is recorded in the session mapping table.

[0019] The advantages of this invention compared to existing technologies are as follows: This invention establishes the highest priority for cache hits by calculating prefix matching degree based on global cache index and generating a first candidate list; subsequently, it generates a second candidate list by matching differentiated resource rejection thresholds according to session type and removing nodes with excessive resources, thus achieving fine-grained load balancing while ensuring cache reuse for high-value sessions; finally, it introduces green cost information for optimization among nodes with similar prefix matching degrees, thereby achieving global optimization of inference cluster load balancing and green energy utilization while ensuring the highest KV Cache hit rate, effectively solving the defects of equalizing scheduling targets and lacking differentiated rejection mechanisms in existing technologies.

[0020] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 A flowchart illustrating the large language model inference computing power scheduling method provided in an embodiment of the present invention;

[0023] Figure 2 A schematic block diagram of a large language model inference computing power scheduling system provided in an embodiment of the present invention;

[0024] Figure 3 A schematic block diagram of a computer device provided for an embodiment of the present invention. Detailed Implementation

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0027] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0028] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0029] Please see Figure 1 , Figure 1This is a flowchart illustrating the large language model inference computing power scheduling method provided in this embodiment of the invention. This method is applied in a distributed multi-computing-power-center environment. Each computing center deploys several computing nodes running inference engines. Each computing node runs a large language model and maintains a key-value cache (KV Cache, i.e., historical attention key-value pairs cached during the large language model inference process; requests hitting this cache can skip the pre-filling stage and be directly generated, thus significantly reducing the first token latency). The scheduling platform deploys a scheduling system responsible for receiving users' large model inference requests and routing them to appropriate computing nodes.

[0030] This method achieves an organic unity of cache hit, load balancing, and green energy optimization through a three-layer progressive scheduling architecture: First, it calculates the prefix matching degree between the prompt word and the cached prefixes of each computing node based on the global cache index, and generates a first-candidate node list accordingly, establishing cache hit as the highest priority that cannot be overridden by subsequent factors, maximizing KVCache reuse. Second, it configures differentiated resource rejection thresholds based on the session type to which the session identifier belongs—setting higher thresholds for historical sessions with established cache affinity to allow more cache-hitting sessions and prevent them from being frequently migrated due to load fluctuations, thus preventing cache loss. If a cache hit fails, a lower threshold is set for new sessions to guide them to nodes with lower loads for global load balancing and to reserve resource redundancy for future historical sessions. A veto check is performed according to the cache sorting order of the first candidate node list, ensuring cache hits while preventing overloaded nodes from being selected. Only when multiple nodes that pass the veto check have similar prefix matching degrees are the target nodes selected based on the proportion of green energy and real-time electricity price. This ensures that green energy factors are strictly limited to playing an equal role among nodes with similar cache hit rates, fundamentally preventing green costs from overriding the cache hit priority principle. Therefore, this scheme, with its progressive decision-making mechanism of "cache affinity sorting → differentiated resource veto → green cost decision-making" without each subsequent layer overriding the previous one, solves the technical problems of existing technologies, such as difficulty in guaranteeing KV cache reuse rates, frequent cache jitter in historical sessions, and the inability to coordinate and optimize green energy utilization, which are caused by the lack of differentiated resource veto mechanisms and the equalization of cache-aware scheduling targets. It achieves global optimization of inference cluster load balancing and green energy utilization while maximizing cache hit rates.

[0031] Figure 1 This is a flowchart illustrating the large language model inference computing power scheduling method provided in this embodiment of the invention. Figure 1 As shown, the method includes the following steps S110 to S150.

[0032] S110. Obtain a large model inference request, wherein the large model inference request carries a session identifier and a prompt word.

[0033] In this embodiment, a model inference request refers to a service request initiated by the user to the scheduling platform, requesting the large language model to perform inference on the input text and return the generated result. A prompt is the text content input to the large language model carried in the large model inference request, which, after word segmentation, forms a token sequence. A token is the smallest processing unit obtained after the large language model's word segmenter segments the text.

[0034] A session identifier is a string used to uniquely identify a consecutive session. Specifically, when a user makes its first request, the scheduling platform generates a globally unique identifier (e.g., using UUID v4, which is the fourth version of a universally unique identifier generated based on random numbers). The user carries this session identifier in subsequent requests so that multiple requests within the same consecutive session can be identified as related by the scheduling platform. In one embodiment, the character set of the session identifier is limited to [a-zA-Z0-9-_], with a maximum length of 512 characters.

[0035] After receiving the large model inference request, the request receiving and parsing module of the scheduling platform parses the fields such as session identifier, model identifier, and prompt words. The model identifier is used to indicate the specific large language model called by the request. At the same time, it determines whether the session to which the request belongs already has a historical mapping, providing a basis for session type determination in subsequent steps.

[0036] S120. Based on the global cache index, calculate the prefix matching degree between the prompt word and the cached prefixes on each computing power node, sort each computing power node according to the prefix matching degree, and generate a first candidate node list; wherein, the global cache index is used to record the information of the cached prefixes on each computing power node.

[0037] In this embodiment, the prefix matching degree is the number of tokens with the longest common prefix between the token sequence of the prompt word and the token sequence of the cached prefix on the computing power node;

[0038] The global cache index is updated by listening to cache events reported by each computing node through a message queue. The cache events include creation events, update events, and eviction events of cached prefixes.

[0039] The cached prefix refers to the historical hint prefix corresponding to the currently cached KV Cache on the computing power node. That is, the token sequence prefix that has been pre-filled and calculated in previous requests and whose KV Cache is still retained in the node's memory. The global cache index is an index service deployed on the scheduling platform side to centrally record the cached prefix information on each computing power node, reflecting the real-time status of each computing power node's KV Cache.

[0040] In this embodiment, the global cache index adopts a two-level mapping data structure: the first level is the mapping from the token sequence hash to the physical block index set, where each item in the physical block index set includes the node identifier, block identifier, last access time and reference count, and the block identifier is used to locate the physical block in the node's video memory that stores the KV Cache; the second level is the mapping from the node identifier to the cache metadata, which includes information such as the model the node is running, video memory usage and the most recent heartbeat time.

[0041] In this embodiment, the global cache index is updated by listening to cache events reported by each computing node through a message queue. Specifically, the inference engine of each computing node reports KV Cache events to the scheduling platform through a message queue (e.g., ZeroMQ or Kafka). In a preferred embodiment, the large language model is deployed in a vLLM high-performance inference and service architecture. This architecture has a built-in KV Cache event extrapolation mechanism, with the core configuration being KVEventsConfig, which can publish structured events to the outside world through ZeroMQ's PUB socket. The cache events include creation events, update events, and eviction events with cached prefixes, corresponding to three scenarios: adding a cache block, updating cache block content or changing reference count, and eviction of a cache block, respectively. Each event includes a node identifier, model identifier, token sequence hash, block identifier, event type, and timestamp, which the scheduling platform uses to update the global cache index in real time.

[0042] In this embodiment, prefix matching degree refers to a quantitative indicator used to measure the degree of reusability between the prompt word and the cached prefix on a certain computing power node. Specifically, the prefix matching degree is the number of tokens in the longest common prefix between the token sequence of the prompt word and the token sequence of the cached prefix on the computing power node. Let the token sequence of the prompt word P be [t1, t2, ..., t n The set of cached prefixes on computing node N is S_N={s1, s2, ..., s...} m}, then the prefix matching degree M(P, N) between the prompt word P and the computing power node N is:

[0043] M(P, N)=max{LCP(P, s)|s∈S_N};

[0044] Where LCP(P, s) represents the number of tokens with the longest common prefix between the prompt word P and the cached prefix s. In one embodiment, to facilitate comparison between prompt words of different lengths, the prefix matching degree can also be normalized as follows:

[0045] M_norm(P, N)=M(P, N) / n×100%;

[0046] Where n is the total number of tokens for prompt word P. The computing nodes are arranged in descending order of M(P, N) or M_norm(P, N).

[0047] In this embodiment, the first candidate node list refers to the set of candidate computing power nodes output after sorting in this step and arranged in descending order of prefix matching degree, namely L1. It carries the output results of the cache affinity scoring layer. Subsequent steps only exercise the power of rejection or preference on this list without changing its essential attribute of cache sorting.

[0048] Step S120 further includes:

[0049] When a mapping relationship for the session identifier already exists in the session mapping table, the computing node corresponding to the mapping relationship is placed at the beginning of the first candidate node list. The session affinity strategy refers to prioritizing scheduling sessions with established historical mappings to their historical target nodes. Since the historical target node already caches the session's KVCache (or at least has a cache expectation), placing it at the beginning maximizes the continuous reuse of the session's cache.

[0050] S130. Determine the session type to which the large model inference request belongs based on the session identifier, obtain the resource rejection threshold corresponding to the session type, wherein different session types correspond to different resource rejection thresholds, obtain the resource status indicators of each computing power node, and compare the resource status indicators of each computing power node with the resource rejection thresholds in turn according to the sorting of the first candidate node list, remove computing power nodes whose resource status indicators exceed the resource rejection thresholds, and generate a second candidate node list.

[0051] In this embodiment, the session type refers to the classification of large model inference requests based on whether the session to which the request belongs has established a cache affinity (i.e., whether there is a historical cache), including at least historical sessions and new sessions. Specifically, determining the session type to which the large model inference request belongs based on the session identifier includes: querying the session mapping table; if a valid mapping record for the session identifier exists in the session mapping table, the large model inference request is determined to belong to a historical session; if no valid mapping record for the session identifier exists in the session mapping table, the large model inference request is determined to belong to a new session. The valid mapping record is a mapping record that has not exceeded a preset expiration time. In one embodiment, the session mapping table sets an expiration time (TTL, Time To Live, i.e., the record's lifespan, for example, 30 minutes) for each mapping record. When a session becomes inactive after the expiration time, the mapping is automatically cleared, thereby avoiding invalid affinity caused by node cache eviction. Furthermore, the mapping records in the session mapping table can also record the request rounds. Historical sessions require the mapping record to be in a valid state and the request round to be no less than 1 (i.e., not the first request). The mapping states include three types: ACTIVE, STALE, and MIGRATING. Records in the STALE state are considered invalid mapping records, and the corresponding requests are determined to be new sessions.

[0052] The resource veto threshold is a critical indicator value used to determine whether a computing node is unsuitable for receiving new requests due to resource constraints. When the resource status indicator of a computing node exceeds this threshold, the node is vetoed and removed from the candidate set. The resource veto threshold is the core design of this invention: different session types correspond to different resource veto thresholds, wherein the resource veto threshold corresponding to historical sessions is higher than that corresponding to new sessions. A higher veto threshold is used for historical sessions to reserve more resource redundancy space. Even if the node load is at a medium to high level, as long as the higher tolerance line set for historical sessions is not reached, veto and migration will not be triggered, thereby ensuring the continuous reuse of the KV cache of historical sessions. A relatively lower veto threshold is used for new sessions, prioritizing their scheduling to nodes with lower loads, achieving global load balancing while ensuring basic performance. For example, if we don't differentiate between session types, let's assume the rejection threshold is uniformly set to 90% GPU load. If we differentiate between session types, the rejection threshold for historical sessions can be set to 95%, thus allowing more cache-capable historical sessions to be allowed at the current moment. The rejection threshold for new sessions can be set to 85%, thus reserving some resource space for future historical sessions. The purpose is also to allow more cache-capable historical sessions to be allowed in the future.

[0053] Resource status indicators refer to monitoring metrics used to characterize the current resource utilization and service capacity status of computing nodes, including at least one of the following: GPU utilization (the computing utilization of the node's GPU cores), request queue depth (the number of requests currently queued and waiting to be processed by the node), concurrent request count (the number of requests currently being processed by the node), average first token time (TTFT, the average latency from arrival to return of the first token for the most recent N requests), average adjacent token generation time (TPOT, the average time interval between the generation of tokens for the most recent N requests), and available GPU memory capacity (the currently available GPU memory capacity of the node). These indicators can be used individually or in combination as a basis for rejection. In a preferred embodiment, a combination of GPU utilization and request queue depth is used as the primary basis for rejection, while average first token time and average adjacent token generation time are used as auxiliary reference indicators.

[0054] The second candidate node list refers to the set of candidate computing power nodes output after the differential resource status rejection check, i.e., L2, which maintains the prefix matching degree sorting of the first candidate node list.

[0055] The resource status indicators include at least one of the following: GPU utilization, request queue depth, number of concurrent requests, average first token time, average adjacent token generation time, and available video memory capacity.

[0056] Specifically, determining the session type of the large model inference request based on the session identifier includes: querying the session mapping table; if a valid mapping record for the session identifier exists in the session mapping table, the large model inference request is determined to belong to a historical session; if a valid mapping record for the session identifier does not exist in the session mapping table, the large model inference request is determined to belong to a new session.

[0057] Wherein, the resource veto threshold corresponding to the historical session is higher than the resource veto threshold corresponding to the new session; the valid mapping record is a mapping record that has not exceeded the preset expiration time.

[0058] The resource veto threshold is set based on the physical overload limit of the computing node, whereby the physical overload limit is a preset overload threshold value based on the hardware capabilities of the computing node; including:

[0059] Based on the physical overload limit, a first target anchor point is set for the historical session, and a second target anchor point is set for the new session; wherein, the first target anchor point is the physical overload limit plus a first offset, and the second target anchor point is the physical overload limit minus a second offset, and both the first offset and the second offset are positive values.

[0060] The resource veto threshold is obtained by superimposing a trend-aware fine-tuning term on the corresponding first target anchor point or second target anchor point;

[0061] The trend-aware fine-tuning term is determined based on the positive deviation between the instantaneous value of the resource status indicator and the sliding average value of the instantaneous value within a preset time window, and the trend-aware fine-tuning term is not greater than zero, so as to lower the resource rejection threshold when the instantaneous value is higher than the sliding average value.

[0062] Specifically, the resource veto threshold is set using a method of "physical limit anchoring + differentiated safety margin + trend-aware fine-tuning", as follows:

[0063] Anchoring of physical overload limits. The resource veto threshold is set based on the physical overload limits of the computing power nodes. These physical overload limits are preset overload thresholds based on the hardware capabilities of the computing power nodes, denoted as... Specifically, an absolute physical overload threshold is set based on the node's hardware capabilities (GPU computing power, video memory capacity) and Service Level Agreement (SLA, i.e., the agreement on service quality indicators, such as latency limits), such as 90% GPU utilization. This value does not change with historical load fluctuations and serves as the absolute anchor point for setting all resource rejection thresholds.

[0064] Setting differentiated target anchor points. Based on the physical overload limit, a first target anchor point is set for the historical session, and a second target anchor point is set for the new session; wherein, the first target anchor point is the physical overload limit plus a first offset, i.e. The second target anchor point is the physical overload limit minus the second offset, i.e. The first offset With the second offset All values ​​are positive, with specific values ​​preset based on the node's hardware configuration and the session's cache dependency. Because migrating historical sessions is extremely costly (it will invalidate their accumulated KV cache), it is allowed to approach or even briefly tolerate slight overload to maximize cache hit rate. New sessions have no cache dependency, so a safety margin is reserved in advance, guiding them to other idle nodes before reaching physical limits to ensure service level targets are met. Since differentiation is entirely achieved through a fixed offset, the first target anchor point is always greater than the second target anchor point, the logic is clear, it does not rely on fluctuating statistical parameters, and the threshold difference is stable and reliable. Differentiation is entirely achieved through a fixed offset ( , )accomplish, It always holds true, has clear logic, and does not depend on fluctuation parameters.

[0065] The trend-aware fine-tuning item is superimposed. The resource rejection threshold is obtained by superimposing the trend-aware fine-tuning item on the corresponding first target anchor point or second target anchor point. The trend-aware fine-tuning item is determined based on the positive deviation between the instantaneous value of the resource status indicator and the sliding average value of the instantaneous value within a preset time window, and the trend-aware fine-tuning item is not greater than zero, so as to lower the resource rejection threshold when the instantaneous value is higher than the sliding average value. Specifically, the formula for calculating the trend-aware fine-tuning item Δ is: in, This is an instantaneous sample value of the resource status indicator at the current moment (e.g., real-time GPU utilization). This is the moving average of the indicator over a preset time window T, reflecting the recent steady-state load level of the node. The default value of T is 60 seconds. is the damping coefficient, used to control the fine-tuning sensitivity, with a value range of 0.1 to 0.8 and a default value of 0.5; max(0, This ensures that fine-tuning is triggered only when the instantaneous value is significantly higher than the recent average, meaning it only detects rapid increases in load. The final resource veto threshold is... ,in It is either the first target anchor point or the second target anchor point, and The value is limited to a preset limit, for example... .because Allowing only downward adjustments and never upward adjustments effectively tightens access control before the load rises rapidly but before reaching the target anchor point. This provides a buffer for upcoming sustained loads, preventing a performance cliff caused by a surge of requests at the anchor point. Within the range... The request within the fixed anchor point should have been approved, but the fine-tuning item rejected it prematurely.

[0066] The effect of the trend-aware fine-tuning item is equivalent to raising the effective rejection threshold from... Adjusted to: ;in, This is an equivalent effective veto threshold. For resource status indicators whose instantaneous values ​​are within the range... <X≤ The request within the fixed anchor point should have been allowed, but the trend-aware fine-tuning item rejected it in advance, thus achieving early access tightening before the load reaches the anchor point.

[0067] It should be noted that the trend-aware fine-tuning items are completely consistent for both conversation types; therefore, the difference in the differentiation threshold between the two conversation types remains constant.

[0068] This fine-tuning item is only an auxiliary correction and does not change the differentiated master structure determined by the physical anchor point.

[0069] In one example, the second target anchor point for a new session is 85%, the recent moving average μ is 81%, and the damping coefficient β is 0.5. When a sudden surge in traffic causes the instantaneous load X to reach 84.5%, although this value does not reach the fixed anchor point of 85%, the trend-aware fine-tuning term Δ = -0.5 × (84.5% - 81%) = -1.75%, effectively lowering the rejection threshold to 85% - 1.75% = 83.25%. Since 84.5% is greater than 83.25%, the request is rejected in advance. Without the fine-tuning term, 84.5% being less than 85% would have been allowed. This demonstrates that the mechanism can intervene in advance based on a rapid upward trend before the load reaches its physical limit.

[0070] To further illustrate the technical effectiveness of the threshold setting method in this embodiment, it is compared with the traditional dynamic threshold scheme (μ+kσ, i.e., using the moving average plus k times the standard deviation as the threshold): In low-load scenarios (e.g., the moving average μ is 20%), the threshold of the traditional scheme is pulled down to about 70% by the average, leading to false rejections when the load is 75%; while the anchor point of the method in this embodiment is fixed at 90%, allowing normal access when the load is 75%, without false rejections; in scenarios with rapidly increasing load (instantaneous load 84.5%, moving average 81%), the threshold of the traditional scheme is approximately 85%. The request was allowed; however, the effective threshold of the method in this embodiment is approximately 83.25%, and the request was rejected prematurely. In high-load scenarios (e.g., a moving average μ of 85%), the threshold of traditional solutions is raised to approximately 95% by the average, leading to missed detections at 92% load. In contrast, the anchor point of the method in this embodiment is fixed, and new sessions are accurately rejected at 92% load. Regarding differentiated protection, the differentiated capability of traditional solutions depends on the setting of the k value and is affected by fluctuations in the standard deviation σ, resulting in poor stability. In contrast, the method in this embodiment ensures that the first target anchor point is always greater than the second target anchor point through a fixed offset, making it stable and reliable. It can be seen that the threshold setting method of "physical limit anchoring + differentiated safety margin + trend-aware fine-tuning" in this embodiment not only avoids the false rejections and missed detections caused by the drift of traditional dynamic thresholds with the load average, but also achieves stable and reliable session type differentiated protection with a fixed offset.

[0071] In this embodiment, the rejection check in step S130 strictly follows the prefix matching degree sorting of the first candidate node list. Furthermore, an early stopping strategy can be adopted: when the first computing power node that passes the rejection check is encountered, the check is immediately stopped, and this node is designated as the optimal candidate in the second candidate node list; subsequent nodes are no longer checked because their prefix matching degrees are all no higher than the current node. This early stopping strategy reduces scheduling computation overhead while ensuring cache hit priority.

[0072] In this embodiment, when all computing power nodes in the first candidate node list are removed, a global emergency scheduling is triggered. The computing power node with the lowest load is selected from all available computing power nodes as the target node. The load is determined based on the resource status indicators, such as selecting the computing power node with the smallest current request queue depth. This global emergency scheduling mechanism ensures that the system can still complete request routing under extreme load, avoiding scheduling deadlock.

[0073] S140. When there are at least two computing power nodes with similar prefix matching degrees in the second candidate node list, select the target node from the at least two computing power nodes according to the green cost information corresponding to each computing power node; otherwise, determine the computing power node ranked first in the second candidate node list as the target node.

[0074] The term "similar prefix matching degree" means that the difference in prefix matching degree between computing power nodes does not exceed a preset tolerance threshold.

[0075] The green cost information includes the proportion of green energy and real-time electricity price of the computing center to which the computing node belongs;

[0076] The step of selecting a target node from at least two computing power nodes based on the green cost information corresponding to each computing power node includes: scoring each computing power node for green cost based on the green energy ratio and the real-time electricity price, and selecting the computing power node with the best score as the target node.

[0077] Specifically, the term "similar prefix matching degree" means that the difference in prefix matching degree between computing power nodes does not exceed a preset tolerance threshold. The preset tolerance threshold ranges from, for example, 1% to 10%, and in one example, is 5%. If the matching degree difference between preceding nodes in the second candidate node list does not exceed this preset tolerance threshold, then these nodes enter the green cost assessment. If there are no multiple nodes with similar matching degrees, this step does not interfere with the result of step S130. Through this design, the green cost factor is strictly limited to a tie-break mechanism and will not override the cache hit priority principle.

[0078] In this embodiment, the green cost information includes the green energy ratio and real-time electricity price of the computing center to which the computing node belongs. The green energy ratio refers to the proportion of electricity consumed by the computing center from renewable energy sources such as wind power and solar power; the real-time electricity price refers to the real-time settlement price of the electricity market where the computing center is located. In one embodiment, the green energy ratio and real-time electricity price information of each computing center can be obtained based on an existing energy storage business platform or an external data platform.

[0079] In this embodiment, selecting a target node from at least two computing power nodes based on the green cost information corresponding to each computing power node includes: scoring each computing power node for green costs based on the green energy ratio and the real-time electricity price, and selecting the computing power node with the best score as the target node. The green cost score is calculated according to a preset scoring function; the higher the green energy ratio and the lower the real-time electricity price, the better the score of the computing power node.

[0080] S150. The large model inference request is routed to the target node, and a mapping relationship between the session identifier and the target node is established or updated, wherein the mapping relationship is recorded in the session mapping table.

[0081] In this embodiment, the mapping relationship refers to the routing association record established in the session mapping table with the session identifier as the key. The record content includes at least the target node, request round, first request time, and last access time, and may also include the mapping status (ACTIVE / STALE / MIGRATING). After each request completes routing, the scheduling platform writes or updates the target node of this route to the mapping record corresponding to the session identifier and refreshes the last access time, so that subsequent requests of this session can obtain cache protection and differentiated threshold treatment based on the session affinity policy in step S120 and the session type determination in step S130.

[0082] It should be noted that the above steps S110 to S150 constitute a three-layer progressive scheduling in terms of execution logic: step S120 corresponds to the highest priority cache affinity scoring layer, step S130 corresponds to the medium priority differentiated resource status veto layer, and step S140 corresponds to the lowest priority green cost decision layer; the subsequent layer only exercises veto or decision power on the candidate set output by the previous layer, that is, resource status veto does not change the cache order, and green cost decision does not cover cache hit, thereby establishing the highest priority of cache hit in the scheduling decision.

[0083] Examples illustrating the method of this embodiment:

[0084] Example 1: Complete scheduling process for a single inference request (new session). Assume a distributed computing power scheduling platform manages three computing power centers A, B, and C. Each computing power center deploys several inference instances, all running the vLLM inference engine and enabling KVCache event reporting. The client initiates an inference request, carrying the session identifier sess_new_001 (new session, no record in the session mapping table) and the prompt word P; the cache affinity scoring module calculates the matching degree of each node: node A-1 is 85%, node B-2 is 60%, node C-1 is 0%, generating the first candidate node list [A-1, B-2, C-1]; the resource status monitoring and rejection module determines that the session type is a new session, adopts the second target anchor point of 85% (the physical overload limit of 90% is lowered by 5%), and checks in the list order: the GPU utilization of node A-1 is 92%, and the effective threshold after superimposed trend perception fine-tuning is about 83%, 92% is greater than 83%, so it is rejected; the GPU utilization of node B-2 is 65%, 65% is less than 85%, so it passes the check and generates the second candidate node list [B-2]; since there is only one node in the second candidate node list, step S140 is not triggered, B-2 is selected as the target node, and (sess_new_001, B-2) is written into the session mapping table.

[0085] Example 2: Historical Sessions Protected by Differentiated Thresholds. Continuing from Example 1, after the first round of requests by the same session sess_new_001, a record is established in the session mapping table, and subsequent requests become historical sessions. Several minutes later, the user initiates a second round of requests, at which point the GPU utilization of node B-2 has risen to 93%. Upon arrival of the request, the session mapping table matches B-2, and the cache affinity scoring module places B-2 at the top of the first candidate node list; the resource status monitoring and rejection module determines that the session type is a historical session, using the first target anchor point of 95% (90% physical overload limit shifted up by 5%). 93% is less than 95%, so B-2 passes the rejection check, the request is successfully routed to B-2, the KV Cache is hit, and the pre-filling stage is skipped. In contrast, if a uniform threshold of 90% is used, B-2 will be rejected when the GPU utilization is 93%, causing the session to be forcibly migrated and the cached KV Cache to become invalid; with the differentiated threshold, this historical session is effectively protected.

[0086] Example 3: Early Intervention of Trend-Aware Fine-Tuning Items. Assume a node's second target anchor point for new sessions is 85%, the recent moving average μ is 81%, and the damping coefficient β is 0.5. At a certain moment, a sudden surge in traffic causes the instantaneous load X to reach 84.5%. Although this value does not reach the fixed anchor point of 85%, it deviates from the historical average by 3.5 percentage points, indicating that the node is rapidly approaching overload. The fine-tuning item Δ = -0.5 × (84.5% - 81%) = -1.75%, effectively lowering the rejection threshold to 83.25%. Since 84.5% is greater than 83.25%, the request is rejected early; whereas under the fixed anchor point scheme without fine-tuning, this request would be allowed. This mechanism ensures early intervention based on a rapid upward trend before the load reaches its physical limit, avoiding a performance cliff caused by a large influx of requests at the moment the anchor point is reached.

[0087] Example 4: Green Cost Decision When Multiple Nodes Have Similar Hit Rates. Assume the prefix matching results for a certain request are: Node A-1 is 82%, Node B-2 is 80%, and Node C-1 is 78%. A preset tolerance threshold of 5% is set; if the difference between any two nodes does not exceed 5%, the prefix matching rates are considered similar. All three nodes pass the resource status check. The green cost assessment module obtains real-time data from each computing center: Center A has an electricity price of 0.45 yuan / kWh and a green electricity ratio of 70%; Center B has an electricity price of 0.52 yuan / kWh and a green electricity ratio of 45%; Center C has an electricity price of 0.38 yuan / kWh and a green electricity ratio of 85%. According to the preset green cost scoring function, Center C has the best score; therefore, C-1 is selected as the final target node.

[0088] Example 5: Global Emergency Scheduling. Assume that during a peak period, the first candidate node list is [A-1 (85%), B-2 (80%), C-1 (78%)]. The GPU utilization rates of these three nodes are 96%, 95%, and 94%, respectively, all exceeding the corresponding resource rejection thresholds and thus being eliminated. At this point, global emergency scheduling is triggered. Among all available nodes (including D-1, E-1, etc.), the node with the smallest current request queue depth is selected as the target node, ensuring that the system can still complete request routing under high load and avoiding scheduling deadlock.

[0089] Example 6: Comparison of the effects of differentiated thresholds and unified thresholds. Using a physical overload limit of 90% as a benchmark, the unified threshold scheme and the differentiated threshold scheme (historical session first target anchor point 95%, new session second target anchor point 85%) are compared: When the node load is 85%, both schemes allow both historical and new sessions; when the node load is 92%, the unified threshold scheme rejects historical sessions, causing cache invalidation, while the differentiated scheme allows historical sessions to achieve cache protection and rejects new sessions; when the node load is 96%, the unified threshold scheme rejects both types of sessions, while the differentiated scheme still allows historical sessions; when the node load is 98%, the unified threshold scheme rejects both types of sessions, and in the differentiated scheme, whether new sessions are rejected and whether historical sessions are allowed depends on its resource rejection threshold (including the comprehensive effect of trend-aware fine-tuning). It should be noted that the threshold values ​​of the differentiated scheme in the above scenarios already include the comprehensive effect of trend-aware fine-tuning. From a statistical perspective, the historical session migration rate significantly decreased from approximately 8% / hour under the unified threshold scheme to approximately 2% / hour under the differentiated scheme, and the stability of the cache hit rate was significantly improved.

[0090] To verify the technical effectiveness of the method in this embodiment, a systematic comparative experiment was conducted on four scheduling schemes based on a discrete event-driven simulation framework. The simulation environment was: a discrete event-driven simulation platform based on Python; the model size was set according to Qwen-32B or a model of the same size; the computing cluster included 3 data centers and 8 inference nodes, with 22 concurrent GPU slots per node and a hard concurrency limit (hard_limit) of 44 per node; 300,000 inference requests were processed within an 8-hour continuous period (6:00 to 14:00), and the test load was a mixed load model, including multi-turn dialogues (50% of historical sessions), system prefix shared sessions (15%), topic prefix shared sessions (5%), and independent new sessions (30%), with request arrival following a diurnal variation non-homogeneous Poisson process. The comparison schemes are: Scheme A (the method in this embodiment, three-layer progressive + differentiated veto + green energy wins), Scheme B (three-layer progressive but with a uniform veto threshold, not distinguishing session type), Scheme C (three-layer progressive but with the second-layer veto mechanism disabled), and Scheme D (cache hit rate, load, and green energy weighted scoring routing, where cache weight is 0.20, load weight is 0.50, green energy weight is 0.15, and price weight is 0.15). The simulation results are shown in Table 1.

[0091] Table 1. Simulation Results

[0092] index Solution A (the method in this embodiment) Option B (Unanimously rejected) Option C (Affinity Priority) Option D (Rating Route) Historical session cache hit rate 93.10% 68.50% 99.10% 67.50% Historical session migration rate 2.80% 19.60% 1.00% 11.20% TTFT (P50) 81.2ms 84.6ms 84.7ms 88.6ms TTFT (P95) 250.1ms 271.3ms 320.1ms 285.6ms SLO achievement rate 98.70% 96.50% 93.90% 96.00% Extremely poor node load 2.10% 3.20% 8.60% 26.40% Green electricity consumption ratio 49.20% 49.30% 48.30% 54.90% GPU average utilization 58.00% 59.10% 55.20% 60.40% Hit rate stability (fluctuation range) 6.90% 31.00% 0.90% 12.10%

[0093] Experimental results show that Scheme A achieves the best performance in core metrics such as SLO achievement rate (98.7%), TTFT (81.2ms for P50 and 250.1ms for P95), node load balancing (range 2.1%), and hit rate stability (fluctuation 6.9%). Simultaneously, it achieves a historical session cache hit rate of 93.1% and a migration rate of only 2.8%, effectively balancing cache protection and load balancing. Scheme B's uniform threshold design leads to a historical session migration rate of 19.6%, with a peak hit rate dropping sharply to 29.6%. While Scheme C achieves the highest hit rate (99.1%), its abandonment of load balancing results in an SLO achievement rate of 93.9%. Scheme D, lacking a veto mechanism, has a historical migration rate of 11.2% and a load range of 26.4%. These results verify the effectiveness of the differentiated veto mechanism in this embodiment. The SLO achievement rate refers to the proportion of requests with a first token delay (TTFT) of no more than 300 milliseconds out of the total number of requests.

[0094] Specifically, in order to systematically verify the technical effect of the scheduling scheme of this embodiment, a high-fidelity large model inference computing power scheduling simulation environment is constructed. The performance differences of four scheduling schemes—Scheme A (layered veto, i.e. the method scheme of this embodiment), Scheme B (unified veto), Scheme C (affinity priority, no veto) and Scheme D (scoring routing, no veto)—are fully simulated and statistically analyzed in terms of cache hit rate, session migration rate, first token latency and SLO achievement rate, GPU utilization balance, and green electricity consumption.

[0095] This experiment uses a discrete event-driven simulation framework to simulate a computing cluster containing 3 data centers and a total of 8 inference nodes. The simulation parameters are configured as follows: simulation duration is 8 hours (6:00 to 14:00); total number of requests is 300,000; session composition is 50% historical sessions and 50% new sessions, of which historical session requests number 214,485 and new session requests number 85,515; request arrival model is a diurnal non-homogeneous Poisson process; KV cache capacity is limited and supports eviction; SLO threshold is dynamically determined based on node load. The request arrival rate in the experiment followed a typical daily variation pattern: 6:00 to 8:00 was a low-load warm-up period, with the system generally under light load; 8:00 to 10:00 was the ramp-up period, with the request volume gradually increasing; 10:00 to 12:00 was a medium-to-high load period, with a significant increase in cluster pressure; and 12:00 to 14:00 was the peak period, with the request volume reaching the highest level of the day, which is the key period for testing the adaptability of the scheduling scheme to extreme operating conditions. This daily variation model ensures that the experiment covers the full spectrum of operating conditions from light to heavy load, allowing the performance differences of each scheme under different system pressures to be fully exposed. It should be noted that because random variables (including request arrival time, session allocation, and other random processes) are introduced into the simulation program, the results of each run will fluctuate to some extent, but this does not affect the overall comparative analysis conclusions; all data in this embodiment are the results of a single run, and the relative ranking and difference trends between the schemes remain consistent in multiple repeated experiments.

[0096] Scheme A (layered veto, the method in this embodiment) implements differentiated veto thresholds for historical and new sessions: a higher veto threshold (load score threshold of 35) is set for historical sessions, and the scheduling selection of affinity nodes is rejected only when the load score of candidate nodes deviates significantly, thereby maximizing the protection of the reuse of existing KV cache; a lower veto threshold (load score threshold of 19) is set for new sessions, allowing the scheduler to select nodes with lighter loads from a wider range, making full use of the flexibility of new sessions without cache dependencies to achieve load balancing. Scheme B (uniform veto) uses a uniform veto threshold (load score threshold of 27), imposing the same scheduling constraints on all session types. Its advantage is its simplicity of implementation, but for historical sessions, the threshold of 27 is more sensitive than Scheme A's 35, making it easier to trigger veto and cause cache migration, while for new sessions it is more lenient than Scheme A's 19, resulting in insufficient scheduling flexibility. Option C (Affinity Priority) never rejects affinity nodes: For historical sessions, the scheduler always routes them to the affinity node where the cache resides, regardless of the node's current load status, with maximizing KV cache reuse as the sole objective. However, it completely abandons the ability to achieve load balancing through session migration. Option D (Rating Routing) uses a pure rating mechanism for routing decisions, combining four rating factors: cache affinity (weight 0.20), node load status (weight 0.50), green electricity availability (weight 0.15), and electricity price (weight 0.15). There is no rejection mechanism; all sessions select target nodes entirely based on the comprehensive rating. Historical sessions may obtain the highest cache rating (match=1.0) on affinity nodes, but nodes with higher scores in load, green electricity, or price dimensions may still win. Its cache protection relies entirely on rating competition.

[0097] The evaluation metrics used in this experiment are defined as follows: Cache hit rate, which refers to the proportion of requests routed to nodes that have stored the corresponding session KV Cache. It is divided into three levels: historical session hit rate, new session hit rate, and overall hit rate. The higher the hit rate, the more fully the KV Cache is reused, and the greater the probability of skipping the pre-filling stage. Session migration rate, which refers to the proportion of historical sessions scheduled to non-affinity nodes (i.e., nodes that are not the original cache locations). The lower the migration rate, the more effective the scheduling scheme is in protecting the cache of historical sessions. First Token Delay, which refers to the time interval from request submission to the user receiving the first generated token. It is calculated using two quantile values: P50 (median) and P95 (95th percentile). P50 reflects the typical user experience, and P95 reflects the severity of tail latency. SLO Achievement Rate, which refers to the proportion of requests that meet the preset service quality target out of the total number of requests. The latency judgment standard is a TTFT of no more than 300 milliseconds. GPU utilization includes the average and peak utilization of GPUs on each node, reflecting the overall efficiency of resource utilization and the pressure level during the busiest periods, respectively; green electricity consumption rate refers to the proportion of requests scheduled to nodes with sufficient green electricity to the total number of requests; node load balancing refers to the range of utilization between nodes (the difference between the maximum and minimum values) and the percentage of relative difference, while also calculating the range of full load time for each node to reflect the duration of extreme overload.

[0098] The performance of the four scheduling schemes across all core metrics is shown in Table 2 below.

[0099] Table 2. Performance of the four scheduling schemes across all core metrics

[0100] index Option A (Stratified Rejection) Option B (Unanimously rejected) Option C (Affinity Priority) Option D (Rating Route) Historical hit rate 93.10% 68.50% 99.10% 67.50% New session hit rate 19.90% 19.90% 19.90% 19.90% Overall hit percentage 72.20% 54.70% 76.50% 53.90% Historical migration rate 2.80% 19.60% 1.00% 11.20% P50 TTFT 81.2ms 84.6ms 84.7ms 88.6ms P95 TTFT 250.1ms 271.3ms 320.1ms 285.6ms TTFT > 300ms percentage 1.30% 3.50% 6.10% 4.00% SLO achievement rate 98.70% 96.50% 93.90% 96.00% GPU average utilization 58.00% 59.10% 55.20% 60.40% GPU peak utilization 68.60% 70.40% 63.80% 69.30% Green electricity consumption rate 49.20% 49.30% 48.30% 54.90%

[0101] The overall metrics show that each of the four schemes has its own focus: Scheme A has a significant advantage in SLO achievement rate and TTFT; Scheme C performs best in cache hit rate; and Scheme D leads in green energy consumption. The new session hit rate remains at 19.9% ​​across all schemes, indicating that new sessions do not depend on historical caches, and their hit behavior is determined solely by the probability of cache collisions between nodes, regardless of the scheduling strategies of each scheme. This result also verifies the correctness of the session classification logic in the simulation experiment.

[0102] The overall hit rates of the four schemes were ranked as follows: Scheme C (76.5%) > Scheme A (72.2%) > Scheme B (54.7%) > Scheme D (53.9%), with a difference of 22.6 percentage points between the highest and lowest values. Scheme C ranked first with a historical hit rate of 99.1%, fully meeting its "never veto" design philosophy. Scheme A ranked second with a historical hit rate of 93.1%. Although its tiered veto mechanism allowed a small number of historical sessions to migrate under extreme conditions, the cache affinity of the vast majority of historical sessions was effectively protected. Scheme B's historical hit rate of 68.5% was 24.6 percentage points lower than that of Scheme A, indicating that the unified threshold of 27 was significantly insufficient in protecting historical sessions. When node load fluctuated, too many historical sessions were misjudged as needing migration, leading to cache invalidation. Scheme D had a historical hit rate of 67.5%. Its cache score weight of 0.20 with match=1.0 effectively protected historical sessions to a certain extent, making the cache protection effect better than expected.

[0103] The SLO achievement rates are ranked as follows: Scheme A (98.7%) > Scheme B (96.5%) > Scheme D (96.0%) > Scheme C (93.9%). Scheme A leads significantly with 98.7%, exceeding Scheme C by nearly 5 percentage points. The fundamental reason for Scheme A's superior performance lies in the hierarchical veto mechanism, which achieves an organic unity of cache protection and load balancing: on the one hand, it protects the cache affinity of historical sessions through a high threshold, reducing the increase in pre-fill latency caused by cache misses; on the other hand, it gives new sessions greater scheduling flexibility through a low threshold, effectively avoiding node overload, resulting in a P50 TTFT of only 81.2ms and a P95 TTFT of only 250.1ms. Scheme C had the lowest SLO achievement rate (93.9%), and its highest hit rate but worst SLO revealed the non-linear relationship between cache affinity and system performance: the "never veto" strategy caused a large number of historical sessions to flood the same affinity node during peak hours, drastically increasing the queuing latency of that node. Even if the cache hit occurred, inference could not be completed within the SLO time limit, with a P95 TTFT as high as 320.1ms, which is a direct manifestation of the deterioration of queuing latency caused by overload. Scheme B increased prefilling latency due to excessive historical session migration and decreased cache hit rate caused by the uniform threshold. Scheme D had an SLO achievement rate of 96.0% and a historical migration rate of only 11.2%, showing relatively good cache protection, with a P95 TTFT (285.6ms) better than Scheme C.

[0104] The historical migration rates for the four schemes are as follows: Scheme A (2.8%), Scheme B (19.6%), Scheme C (1.0%), and Scheme D (11.2%). Scheme C has the lowest migration rate, with only a very small number of sessions migrating due to extreme conditions (such as simulated node failures). Scheme A has a migration rate of 2.8%, which is extremely low in a production system, indicating that the high threshold (35) in the hierarchical veto mechanism is very effective in protecting historical sessions, triggering historical session migration only under extreme conditions of severe imbalance in node load. In terms of absolute numbers, Scheme A has a total of 5,924 historical migrations, far lower than the 42,139 migrations of Scheme B and the 23,983 migrations of Scheme D. The migration rate of Scheme B is about 7 times that of Scheme A, indicating that the unified threshold 27 is too sensitive, frequently triggering historical session migration decisions, resulting in a large number of KV Caches being discarded and rebuilt on new nodes. Although the migration rate of scheme D is higher than that of schemes A and C, it is significantly lower than that of scheme B, indicating that the cached score weight of 0.20 with match=1.0 in pure score routing effectively protects historical sessions to some extent, but it is still insufficient compared to the direct protection of the veto mechanism.

[0105] The load ranges for the four schemes are as follows: Scheme A (2.1%, relative difference 4%), Scheme B (3.2%, relative difference 5%), Scheme C (8.6%, relative difference 16%), and Scheme D (26.4%, relative difference 44%). Scheme A has the best load balancing, with the GPU utilization of the eight nodes distributed in a very narrow range of 56.9% to 59.0%, and the number of requests processed by each node ranging from 36,633 to 38,067, which is extremely evenly distributed. This is due to the low threshold (19) design of Scheme A for new sessions, which allows the scheduler to allocate new sessions to nodes with lighter loads, achieving fine-tuning of the global load. Scheme C has historical sessions that are always routed to affinity nodes. When some affinity nodes carry a large number of high-frequency historical sessions, their load continues to rise, while other nodes are relatively idle because they cannot share the historical load. Scheme D exhibits the worst load balancing, with node utilization ranging widely from 49.0% to 75.4%. This is because the pure scoring mechanism lacks veto protection, and the clustering effect of the comprehensive scores leads to scheduling decisions becoming overly concentrated on specific "highest-scoring" nodes during certain periods, creating a positive feedback loop—nodes with higher scores are more likely to be selected, ultimately resulting in load skew. Details of the load distribution for each node are shown in Table 3 below.

[0106] Table 3. Load distribution details for each node

[0107] node Data Center Green electricity rate A load B load C load D load A Full Load Duration B Full load duration C Full load duration D Full load duration 0 0 0.45 57.80% 59.00% 54.20% 62.10% 104.7min 135.4min 96.8min 101.3min 1 1 0.25 59.00% 58.30% 59.60% 49.40% 98.8min 130.3min 131.1min 37.0min 2 2 0.92 58.00% 58.10% 53.10% 75.40% 103.2min 130.7min 93.9min 218.3min 3 0 0.45 57.70% 58.70% 53.20% 61.70% 97.8min 128.4min 90.8min 99.1min 4 1 0.25 56.90% 58.00% 53.20% 49.00% 97.6min 130.4min 92.8min 34.5min 5 2 0.92 57.60% 61.20% 53.20% 75.10% 100.1min 136.2min 90.8min 215.3min 6 0 0.45 58.80% 58.80% 53.90% 61.30% 95.6min 130.3min 95.0min 95.7min

[0108] In terms of full load duration: Scheme A's full load duration ranges from 95.6 to 104.7 minutes, with minimal differences between nodes, further confirming its excellent load balancing capabilities; Scheme B's full load duration ranges from 128.4 to 136.2 minutes, generally higher than Scheme A, reflecting the additional pre-filling overhead brought by cache migration under a unified threshold; Scheme C's full load duration ranges from 90.8 to 131.1 minutes, where differences between nodes begin to appear; Scheme D's full load duration ranges from 34.5 to 218.3 minutes, with significant differences between nodes. Among them, nodes 2 and 5 (data center 2, green electricity rate 0.92) are selected by a large number of requests due to their dual advantages of high green electricity score and load score, with full load durations as high as 218.3 minutes and 215.3 minutes respectively, while the full load durations of nodes 1, 4, and 7 are only 34.5 to 37.0 minutes, a difference of about 6 times, further confirming its load imbalance problem.

[0109] The historical migration rate time history of each scheduling scheme under different load intensities is shown in Table 4 below:

[0110] Table 4. Historical mobility variation under different load intensities over time

[0111] Time period Option A Option B Option C Option D 6:00–8:00 0.02% 0.31% 0.00% 16.17% 8:00–10:00 0.24% 2.70% 0.07% 11.64% 10:00–12:00 1.30% 14.65% 0.78% 9.55% 12:00–14:00 8.54% 52.90% 2.89% 9.94%

[0112] During low-load periods (6:00–8:00), the migration rates of all schemes remained low because the load differences between nodes were small, and the rejection conditions were rarely triggered. As the request volume increased, the differences in migration rates among the schemes widened rapidly. During peak periods (12:00–14:00), the migration rate of Scheme A rose from 0.02% during the off-peak period to 8.54%, but the absolute level remained manageable, meaning that even during the busiest period, approximately 91% of historical sessions remained running on affinity nodes. Scheme B's peak migration rate reached as high as 52.90%, indicating that the uniform threshold was too sensitive under peak pressure. The migration rate of Scheme D showed a trend of first decreasing and then stabilizing (16.17%→9.94%). During low-load periods, the load factor weight made it easier for nodes with green electricity and higher price scores to win. As the load increased, the load balancing score constraints began to take effect.

[0113] The historical hit rate changes of each scheduling scheme over time are shown in Table 5 below:

[0114] Table 5. Historical Hit Rate Variation of Each Scheduling Scheme

[0115] Time period Option A Option B Option C Option D 6:00–8:00 99.90% 99.00% 100.00% 55.60% 8:00–10:00 99.30% 91.90% 99.90% 66.00% 10:00–12:00 96.10% 68.20% 99.30% 71.30% 12:00–14:00 79.50% 29.60% 97.50% 70.90%

[0116] Scheme C maintained a high historical hit rate of 97.5% to 100.0% throughout the entire period; Scheme A's hit rate gradually decreased from 99.9% during the trough to 79.5% during the peak, a decrease of about 20.4 percentage points overall. Considering that the migration rate rose to 8.54% during the same period, it can be inferred that the decrease in hit rate was mainly due to forced migration under extreme load, rather than a defect in the strategy itself; Scheme B's hit rate plummeted to 29.6% during the peak period, indicating that the cache protection capability of the uniform threshold was severely degraded under high-pressure conditions; Scheme D's historical hit rate was the lowest (55.6%) during the low-load period, and then gradually rose to 71.3% and stabilized at 70.9%, showing a time-series characteristic opposite to that of the migration rate. The time-history analysis also showed that the historical hit rates of the four schemes were relatively similar during low-load periods, but during peak periods, the gap between scheme A (79.5%) and scheme B (29.6%) widened to 49.9 percentage points, indicating that the real value of the scheduling scheme lies in its robustness under extreme conditions, rather than its average performance.

[0117] This experiment uses the standard deviation and volatility (difference between the maximum and minimum values) of the hit rate over four time periods to quantify the stability of each scheme. The results are shown in Table 6 below:

[0118] Table 6. Stability of each scheme

[0119] Stability Indicators Option A Option B Option C Option D Standard deviation 2.30% 10.70% 0.30% 4.50% minimum value 93.10% 68.50% 99.10% 55.40% maximum value 100.00% 99.60% 100.00% 67.50% Fluctuation range 6.90% 31.10% 0.90% 12.10%

[0120] Option C exhibits the best stability, with a standard deviation of only 0.3% and a fluctuation range of only 0.9 percentage points. Option A follows suit, with a standard deviation of 2.3% and a fluctuation range of 6.9%. Although the hit rate decreases during peak periods, the overall trend is gradual and controllable, without any sharp drops. Option B has a fluctuation range of 31.1 percentage points and a standard deviation of 10.7%, indicating that its hit rate is greatly affected by load intensity, with a significant performance gap between low and peak loads. Option D has a fluctuation range of 12.1% and a standard deviation of 4.5%, falling between Option A and Option B. From an operational perspective, request load naturally fluctuates in production environments. Option A, with a hit rate fluctuation range of only 6.9%, provides more predictable and reliable performance assurance for upper-layer services compared to options with fluctuation ranges of 31.1% or 12.1%, reducing alarm events caused by performance fluctuations.

[0121] Based on the above experimental results, the following conclusions can be drawn: First, the hierarchical veto mechanism significantly outperforms other schemes in overall performance. Scheme A, by setting differentiated veto thresholds for historical and new sessions, achieves optimal performance in four key indicators: SLO achievement rate, TTFT, node load balancing, and hit rate stability. It also maintains a high level of historical hit rate. Its core innovation lies in acknowledging the fundamental differences in cache value and scheduling flexibility among different session types and transforming these differences into an operable scheduling strategy through parameterized threshold design. Second, cache protection and load balancing must be optimized synergistically and neither should be neglected. The experimental results of Scheme C demonstrate that excessively pursuing a high cache hit rate (99.1%) while completely abandoning load balancing capabilities will lead to a deterioration of the SLO achievement rate to 93.9% (P95). The TTFT deteriorated to 320.1ms. The success of Scheme A lies in finding the optimal balance between the two. Third, the differentiated threshold design is superior to the unified threshold design. Compared with Scheme B (unified threshold 27), Scheme A (layered threshold 35 / 19) improved the historical hit rate by 24.6 percentage points, the SLO achievement rate by 2.2 percentage points, and the migration rate by 16.8 percentage points, verifying the design assumption that implementing differentiated scheduling strategies for historical sessions and new sessions can systematically improve scheduling quality. Fourth, the peak performance of each scheme differs much more than the average performance. Scheme A combines high SLO guarantee, good cache hit rate, and excellent load balancing capability, making it suitable for engineering implementation as a production-grade scheduling scheme for large-scale model inference computing power clusters.

[0122] In summary, compared with the prior art, the method of this embodiment has at least the following beneficial technical effects:

[0123] By combining the session affinity strategy with the precise prefix cache awareness strategy, and setting the cache hit as the highest priority that cannot be overwritten, the KV cache reuse rate is maximized, the first token latency and the computational overhead of the pre-filling stage are significantly reduced, and the effective token throughput per unit of hardware resources is improved.

[0124] By setting a higher resource rejection threshold for historical sessions, resource redundancy is reserved for sessions that have established cache affinity, effectively avoiding frequent migration of historical sessions due to load fluctuations, reducing cache jitter risk, and ensuring the stability of cache hit rate in long dialogue scenarios; (3) By setting a relatively low rejection threshold for new sessions, new requests are ensured to be scheduled to nodes with sufficient load, maintaining global load balance while ensuring cache affinity, and ensuring service level objectives.

[0125] By using trend-aware fine-tuning based on instantaneous deviation, the threshold can be tightened in advance when the load rises rapidly but has not yet reached the physical limit, thus avoiding a performance cliff caused by a sudden surge of requests at the anchor point.

[0126] Green cost assessment only intervenes between nodes with similar prefix matching degrees, optimizes green energy utilization without sacrificing cache hit rate, and avoids scheduling deadlock under extreme load through global emergency scheduling.

[0127] The aforementioned large language model inference computing power scheduling method establishes the highest priority for cache hits by calculating prefix matching degree based on global cache index and generating a first candidate list. Subsequently, it generates a second candidate list by matching differentiated resource rejection thresholds according to session type and removing nodes with excessive resources, thus achieving fine-grained load balancing while ensuring cache reuse for high-value sessions. Finally, it introduces green cost information for optimization among nodes with similar prefix matching degrees, thereby achieving global optimization of inference cluster load balancing and green energy utilization while ensuring the highest KV Cache hit rate. This effectively solves the shortcomings of existing technologies in terms of equalizing scheduling targets and lacking differentiated rejection mechanisms.

[0128] Figure 2 This is a schematic block diagram of a large-model inference computing power scheduling system 300 provided in an embodiment of the present invention. Figure 2 As shown, corresponding to the above-described large-model inference computing power scheduling method, the present invention also provides a large-model inference computing power scheduling system 300. This large-model inference computing power scheduling system 300 includes a unit for executing the above-described large-model inference computing power scheduling method, and the system can be configured in a server. Specifically, please refer to... Figure 2 The large model inference computing power scheduling system 300 includes a request receiving unit 301, a cache affinity scoring unit 302, a resource status monitoring unit 303, a green cost assessment unit 304, and a routing scheduling unit 305.

[0129] Request receiving unit 301 is used to obtain a large model inference request, wherein the large model inference request carries a session identifier and a prompt word;

[0130] The cache affinity scoring unit 302 is used to calculate the prefix matching degree between the prompt word and the cached prefixes on each computing power node based on the global cache index, and sort each computing power node according to the prefix matching degree to generate a first candidate node list; wherein, the global cache index is used to record the information of the cached prefixes on each computing power node; the resource status monitoring unit 303 is used to determine the session type to which the large model inference request belongs according to the session identifier, obtain the resource rejection threshold corresponding to the session type, wherein different session types correspond to different resource rejection thresholds; obtain the resource status indicators of each computing power node, and sort the resource status indicators of each computing power node according to the order of the first candidate node list. The resource rejection threshold is compared with the target node, and computing power nodes whose resource status indicators exceed the resource rejection threshold are removed to generate a second candidate node list; the green cost assessment unit 304 is used to select a target node from the at least two computing power nodes with similar prefix matching degree when there are at least two computing power nodes in the second candidate node list; otherwise, the computing power node ranked first in the second candidate node list is determined as the target node; the routing scheduling unit 305 is used to route the large model inference request to the target node and establish or update the mapping relationship between the session identifier and the target node, and the mapping relationship is recorded in the session mapping table.

[0131] It should be noted that those skilled in the art can clearly understand that the specific implementation process of the above-mentioned large language model inference computing power scheduling system and each unit can be referred to the corresponding description in the foregoing method embodiments. For the sake of convenience and brevity, it will not be repeated here.

[0132] The aforementioned large language model inference computing power scheduling system can be implemented as a computer program, which can, for example, Figure 3 It runs on the computer device shown.

[0133] Please see Figure 3 , Figure 3 This is a schematic block diagram of a computer device provided in an embodiment of this application. The computer device 500 can be a server, wherein the server can be a standalone server or a server cluster composed of multiple servers.

[0134] See Figure 3 The computer device 500 includes a processor 502, a memory, and a network interface 505 connected via a system bus 501. The memory may include a non-volatile storage medium 503 and internal memory 504.

[0135] The non-volatile storage medium 503 may store an operating system 5031 and a computer program 5032. The computer program 5032 includes program instructions that, when executed, cause the processor 502 to perform a large language model inference computing power scheduling method.

[0136] The processor 502 provides computing and control capabilities to support the operation of the entire computer device 500.

[0137] The internal memory 504 provides an environment for the execution of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute a large language model inference computing power scheduling method.

[0138] This network interface 505 is used for network communication with other devices. Those skilled in the art will understand that... Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device 500 to which the present application is applied. The specific computer device 500 may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0139] The processor 502 is used to run the computer program 5032 stored in the memory to implement all the steps of the large language model inference computing power scheduling method.

[0140] It should be understood that in the embodiments of this application, the processor 502 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0141] It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program includes program instructions and can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the embodiments of the above methods.

[0142] Therefore, the present invention also provides a storage medium. This storage medium can be a computer-readable storage medium. The storage medium stores a computer program, wherein when executed by a processor, the computer program causes the processor to perform all steps of the large language model inference computing power scheduling method.

[0143] The storage medium can be any computer-readable storage medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0144] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0145] In the embodiments provided by this invention, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For example, the division of each unit is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0146] The steps in the method of this invention can be adjusted, merged, or reduced in order according to actual needs. The units in the system of this invention can be merged, divided, or reduced according to actual needs. Furthermore, the functional units in the various embodiments of this invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0147] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0148] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A large language model inference computing power scheduling method, characterized in that, include: Obtain a large model inference request, wherein the large model inference request carries a session identifier and a prompt word; Based on the global cache index, the prefix matching degree between the prompt word and the cached prefixes on each computing power node is calculated, and each computing power node is sorted according to the prefix matching degree to generate a first candidate node list; wherein, the global cache index is used to record the information of the cached prefixes on each computing power node; The session type to which the large model inference request belongs is determined based on the session identifier. The resource rejection threshold corresponding to the session type is obtained, wherein different session types correspond to different resource rejection thresholds. The resource status indicators of each computing power node are obtained. According to the sorting of the first candidate node list, the resource status indicators of each computing power node are compared with the resource rejection threshold in turn. Computing power nodes whose resource status indicators exceed the resource rejection threshold are eliminated, and a second candidate node list is generated. When there are at least two computing power nodes with similar prefix matching in the second candidate node list, the target node is selected from the at least two computing power nodes according to the green cost information corresponding to each computing power node; otherwise, the computing power node ranked first in the second candidate node list is determined as the target node. The large model inference request is routed to the target node, and a mapping relationship between the session identifier and the target node is established or updated, wherein the mapping relationship is recorded in the session mapping table.

2. The large language model inference computing power scheduling method of claim 1, wherein, Determining the session type of the large model inference request based on the session identifier includes: The session mapping table is queried. If a valid mapping record for the session identifier exists in the session mapping table, the large model inference request is determined to belong to a historical session; if no valid mapping record for the session identifier exists in the session mapping table, the large model inference request is determined to belong to a new session. Wherein, the resource veto threshold corresponding to the historical session is higher than the resource veto threshold corresponding to the new session; the valid mapping record is a mapping record that has not exceeded the preset expiration time.

3. The large language model inference computing power scheduling method of claim 2, characterized in that, The resource veto threshold is set based on the physical overload limit of the computing node, whereby the physical overload limit is a preset overload threshold value based on the hardware capabilities of the computing node; including: Based on the physical overload limit, a first target anchor point is set for the historical session, and a second target anchor point is set for the new session; wherein, the first target anchor point is the physical overload limit plus a first offset, and the second target anchor point is the physical overload limit minus a second offset, and both the first offset and the second offset are positive values.

4. The large language model inference computing power scheduling method of claim 3, characterized in that, The resource veto threshold is obtained by superimposing a trend-aware fine-tuning term on the corresponding first target anchor point or second target anchor point; The trend-aware fine-tuning term is determined based on the positive deviation between the instantaneous value of the resource status indicator and the sliding average value of the instantaneous value within a preset time window, and the trend-aware fine-tuning term is not greater than zero, so as to lower the resource rejection threshold when the instantaneous value is higher than the sliding average value.

5. The large language model inference computing power scheduling method of claim 1, wherein, The prefix matching degree is the number of tokens with the longest common prefix between the token sequence of the prompt word and the token sequence of the cached prefix on the computing power node. The step of calculating the prefix matching degree between the prompt word and the cached prefixes on each computing power node based on the global cache index, and sorting each computing power node according to the prefix matching degree to generate a first candidate node list, further includes: When the session identifier mapping relationship already exists in the session mapping table, the computing power node corresponding to the mapping relationship is placed at the beginning of the first candidate node list.

6. The large language model inference computing power scheduling method of claim 1, wherein, The global cache index is updated by listening to cache events reported by each computing node through a message queue. The cache events include creation events, update events, and eviction events of cached prefixes.

7. The large language model inference compute scheduling method of claim 1, wherein, The resource status indicators include at least one of the following: GPU utilization, request queue depth, number of concurrent requests, average first token time, average adjacent token generation time, and available video memory capacity.

8. The large language model inference compute scheduling method of claim 1, wherein, When all computing power nodes in the first candidate node list are removed, a global emergency scheduling is triggered to select the computing power node with the lowest load from all available computing power nodes as the target node, wherein the load is determined according to the resource status index.

9. The large language model inference computing power scheduling method of claim 1, wherein, The term "similar prefix matching degree" means that the difference in prefix matching degree between computing power nodes does not exceed a preset tolerance threshold. The green cost information includes the proportion of green energy and real-time electricity price of the computing center to which the computing node belongs; The step of selecting a target node from at least two computing power nodes based on the green cost information corresponding to each computing power node includes: scoring each computing power node for green cost based on the green energy ratio and the real-time electricity price, and selecting the computing power node with the best score as the target node.

10. A large model inference computing power scheduling system, characterized in that, include: A request receiving unit is used to obtain a large model inference request, wherein the large model inference request carries a session identifier and a prompt word; A cache affinity scoring unit is used to calculate the prefix matching degree between the prompt word and the cached prefixes on each computing power node based on the global cache index, sort each computing power node according to the prefix matching degree, and generate a first candidate node list; wherein, the global cache index is used to record the information of the cached prefixes on each computing power node; The resource status monitoring unit is used to determine the session type to which the large model inference request belongs based on the session identifier, obtain the resource rejection threshold corresponding to the session type, wherein different session types correspond to different resource rejection thresholds; obtain the resource status indicators of each computing power node, and compare the resource status indicators of each computing power node with the resource rejection threshold in turn according to the sorting of the first candidate node list, remove computing power nodes whose resource status indicators exceed the resource rejection threshold, and generate a second candidate node list; The green cost assessment unit is used to select a target node from at least two computing power nodes with similar prefix matching degrees when there are at least two computing power nodes in the second candidate node list; otherwise, the computing power node ranked first in the second candidate node list is determined as the target node. The routing and scheduling unit is used to route the large model inference request to the target node and establish or update the mapping relationship between the session identifier and the target node, and the mapping relationship is recorded in the session mapping table.

Citation Information

Patent Citations

  • Green computing power scheduling system and method based on multi-objective optimization

    CN121501463B

  • Large model cluster shunting system and method based on performance perception

    CN121501479A