Strategy factory construction system and implementation method for large model inference cluster

Abstract

The application provides a strategy factory construction system and implementation method for a large model inference cluster, comprising: a request classification and strategy selection module for receiving an external request, identifying the external request, and determining a target strategy template corresponding to the external request according to the identification result; a state mirror layer for receiving incremental state events returned by each inference instance, and maintaining an instance state snapshot with a unified version number; a prefix signature index for estimating the reusable context length of the external request on different inference instances respectively; a measure generator for generating the residual preprocessing amount, concurrent occupancy degree and resource risk value corresponding to each inference instance; an execution graph compiler for parallel evaluation of multiple candidate inference instances to determine a target inference instance; a routing submission and pre-occupancy write-back module for routing the external request to the target inference instance, and writing a transient placeholder state corresponding to the target inference instance in the state mirror layer.

Description

Technical Field

[0001] This invention relates to the field of large model technology, and more specifically, to a policy factory construction system and implementation method for large model inference clusters. Background Technology

[0002] As large-model inference services evolve from single-machine deployment to multi-instance cluster deployment, how requests are routed among multiple inference instances has become a critical factor affecting initial response latency, average output latency, and resource utilization. Unlike traditional stateless services, large-model inference has significant context state dependencies. If an instance already retains a prefix context that is the same as or similar to the current request, that instance can directly reuse the existing cache, reducing redundant computations and shortening the preprocessing stage time.

[0003] However, simply pursuing context reuse will cause requests to continuously favor a few cached hot instances, leading to load imbalance among instances; while simply pursuing balanced distribution will forgo the speedup benefits brought by existing caching. More complicatedly, large model inference workloads are characterized by strong burstiness, large fluctuations in input length, and unstable output length, making scheduling strategies unable to rely solely on fixed thresholds or static parameters.

[0004] At the engineering implementation level, current practices often hardcode state acquisition, prefix hit estimation, scoring formulas, and fallback logic directly within the scheduler. This type of structure requires repeated modifications to state maintenance and execution chains for each new strategy, resulting in high development costs and difficulty in stably supporting the differentiated strategy requirements of various businesses, models, and tenants within the same scheduler.

[0005] Therefore, the industry urgently needs a unified scheduling platform for large-scale model inference clusters. This platform should not only be able to maintain instance states related to requests, but also be able to transform states into reusable scheduling metrics, and support the rapid assembly, execution, and switching of different strategies without modifying the underlying data acquisition chain, thereby forming a strategy factory suitable for actual production environments.

[0006] One existing approach employs a fixed load balancing routing method. This method uses a global scheduler to collect the number of queued requests, the number of running requests, or the queue length for each instance. It then allocates new requests to the seemingly least busy instance using a combination of shortest queue priority, smallest batch priority, or round-robin with load weighting. The core idea is to distribute requests as evenly as possible, avoiding backlogs on individual instances. However, this approach ignores the differences in context reuse across different instances for the same request. For requests that have already formed a high-hit prefix on a particular instance, if distributed using a pure load balancing method, preprocessing calculations will be repeated on other instances, resulting in wasted computing power and increased first-response latency. These drawbacks are even more pronounced in scenarios with long contexts, high-repetition prefixes, or multi-turn dialogues.

[0007] Another existing approach attempts to combine contextual hit factors with instance load factors for decision-making. Common implementations include: linearly weighted summation of multiple metrics; filtering out high-risk instances based on thresholds and then selecting those with higher hit rates from the remaining instances; or predicting and comparing the completion costs of requests landing on each instance based on an inference latency simulator. While this approach introduces more information, it still has significant shortcomings. Linear combination relies on empirical weights, which often become invalid when the workload changes; threshold filtering is sensitive to models, hardware, and request structures, resulting in poor portability; simulator methods, while more refined in form, have higher modeling, calibration, and maintenance costs. More importantly, these methods typically propose only a single specific strategy without providing a unified mechanism for strategy generation, execution, rollback, and state reconciliation, leading to insufficient strategy scalability and engineering reusability. Summary of the Invention

[0008] To address the shortcomings of existing technologies, the purpose of this invention is to provide a policy factory construction system and implementation method for large model inference clusters.

[0009] The present invention provides a policy factory construction system for large model inference clusters, comprising: a request classification and policy selection module, a state mirroring layer, a prefix signature index, a metric generator, a policy template library, an execution graph compiler, and a route submission and pre-placement write-back module; The request classification and strategy selection module is used to receive external requests, identify the business tag, model identifier, tenant identifier or session identifier of the external request, and determine the target strategy template corresponding to the external request from the strategy template library based on the identification result. The state mirror layer is used to receive incremental state events returned by each inference instance and maintain instance state snapshots with a unified version number. The instance state snapshots are used to provide the current state information of each inference instance to the prefix signature index and the metric generator. The prefix signature index is used to estimate the reusable context length of the external request on different inference instances based on the request prefix fingerprint of the external request and the cache index in the instance state snapshot. The metric generator is used to generate the remaining preprocessing volume, concurrency consumption, and resource risk value for each inference instance based on the instance state snapshot and the reusable context length. The execution graph compiler is used to call the measure generated by the measure generator according to the target policy template, and to evaluate multiple candidate inference instances in parallel according to the scoring expression, candidate filtering rules and backoff rules in the target policy template, so as to determine the target inference instance; The route submission and pre-placement write-back module is used to route the external request to the target inference instance and write a transient placeholder state corresponding to the target inference instance into the state mirror layer, so that the scheduling of subsequent requests can be aware of the resource occupation of the target inference instance by the external request.

[0010] Preferably, the state mirror layer includes: Each inference instance sends incremental state back to the global scheduler via long connections, response header piggybacking, event streams, or shared message channels; wherein, the incremental state includes at least: request entering queue event, request completion preprocessing event, request completion generation event, cache block addition event, cache block eviction event, and resource exception event; the state mirror layer merges the above events and assigns a monotonically increasing version number to each update; When a new user request is received, the snapshot with the same version number is read all at once through the global scheduler.

[0011] Preferably, the prefix signature index includes: The external request input sequence is divided into blocks of fixed token length B. A hash signature is generated for each prefix block to form the signature sequence RequestFingerprint=(sig1,sig2,...,sigN) formed by the current request prefix after block division and hashing. Each inference instance i maintains CacheIndex(i), which records the current cached prefix signature sequence and the corresponding cache block position of the instance. During scheduling, the RequestFingerprint of the current request is matched with the CacheIndex(i) for the longest prefix to obtain the longest prefix block k that can be directly reused on instance i. Then, the prefix length ReuseLen(i) that the current request can reuse on inference instance i is k×B.

[0012] If the last block is less than B or requires precise verification, it can be adjusted according to the actual number of tokens.

[0013] Preferably, the metric generator includes: The remaining preprocessing amount corresponding to each inference instance includes: ResidualPrompt(i) = PromptLen - ReuseLen(i) Where PromptLen is the total length of the input for the current request; ReuseLen(i) is the length of the prefix that can be reused in the current request on inference instance i; The concurrent occupancy includes: Occupancy(i) = WaitingReqs(i) + RunningReqs(i) Where WaitingReqs(i) is the number of requests waiting to be processed on instance i; RunningReqs(i) is the number of requests currently being processed on instance i; The resource risk values ​​include: QueueCost(i) = PendingPrompt(i) + ResidualPrompt(i) Where PendingPrompt(i) is the cumulative input workload of instance i that has not yet completed the preprocessing stage; ResidualPrompt(i) is the input length that still needs to be recalculated after the request is routed to instance i.

[0014] Preferably, the scoring expression in the execution graph compiler is: Score(i) = (PendingPrompt(i) + ResidualPrompt(i)) × (1 + Occupancy(i)) Where PendingPrompt(i) is the amount of preprocessing work that instance i still needs to complete at the current time; ResidualPrompt(i) is the amount of preprocessing work that will be added if the current request is routed to instance i; Occupancy(i) is the degree of concurrency congestion of instance i; the scheduler selects the candidate instance with the smallest Score from the candidate instances that meet the resource constraints as the target inference instance. The resource constraints include: candidate filtering rules and fallback rules; the candidate filtering rules include: pre-excluding unavailable instances based on available video memory, maximum context length, tenant isolation rules, circuit breaker flags or device anomaly flags; If the remaining candidate set is less than the preset threshold, or if a certain type of request is detected to continuously hit the same instance set and the queuing cost rises rapidly, a rollback will be triggered. Once a rollback is triggered, the strategy will switch to load balancing priority, tiered round-robin, or abnormal instance isolation.

[0015] The present invention provides a method for implementing a policy factory construction system for large model inference clusters, comprising: Step S1: Receive external requests through the request classification and strategy selection module, identify the business tag, model identifier, tenant identifier or session identifier of the external request, and determine the target strategy template corresponding to the external request from the strategy template library based on the identification result; Step S2: Receive incremental state events returned by each inference instance through the state mirror layer, and maintain instance state snapshots with a unified version number. The instance state snapshots are used to provide the current state information of each inference instance to the prefix signature index and the metric generator. Step S3: Estimate the reusable context length of the external request on different inference instances based on the request prefix fingerprint of the external request and the cache index in the instance state snapshot using the prefix signature index; Step S4: Based on the instance state snapshot and the reusable context length, the metric generator generates the remaining preprocessing amount, concurrency consumption, and resource risk value for each inference instance. Step S5: The graph compiler calls the measure generated by the measure generator according to the target policy template, and evaluates multiple candidate inference instances in parallel according to the scoring expression, candidate filtering rules and backoff rules in the target policy template to determine the target inference instance; Step S6: The external request is routed to the target inference instance through the route submission and pre-placement write-back module, and a transient placeholder state corresponding to the target inference instance is written into the state mirror layer, so that the scheduling of subsequent requests can be aware of the resource occupation of the target inference instance by the external request.

[0016] Preferably, the state mirror layer includes: Each inference instance sends incremental state back to the global scheduler via long connections, response header piggybacking, event streams, or shared message channels; wherein, the incremental state includes at least: request entering queue event, request completion preprocessing event, request completion generation event, cache block addition event, cache block eviction event, and resource exception event; the state mirror layer merges the above events and assigns a monotonically increasing version number to each update; When a new user request is received, the snapshot with the same version number is read all at once through the global scheduler.

[0017] Preferably, the prefix signature index includes: The external request input sequence is divided into blocks of fixed token length B. A hash signature is generated for each prefix block to form the signature sequence RequestFingerprint=(sig1,sig2,...,sigN) formed by the current request prefix after block division and hashing. Each inference instance i maintains CacheIndex(i), which records the current cached prefix signature sequence and the corresponding cache block position of the instance. During scheduling, the RequestFingerprint of the current request is matched with the CacheIndex(i) for the longest prefix to obtain the longest prefix block k that can be directly reused on instance i. Then, the prefix length ReuseLen(i) that the current request can reuse on inference instance i is k×B.

[0018] If the last block is less than B or requires precise verification, it can be adjusted according to the actual number of tokens.

[0019] Preferably, the metric generator includes: The remaining preprocessing amount corresponding to each inference instance includes: ResidualPrompt(i) = PromptLen - ReuseLen(i) Where PromptLen is the total length of the input for the current request; ReuseLen(i) is the length of the prefix that can be reused in the current request on inference instance i; The concurrent occupancy includes: Occupancy(i) = WaitingReqs(i) + RunningReqs(i) Where WaitingReqs(i) is the number of requests waiting to be processed on instance i; RunningReqs(i) is the number of requests currently being processed on instance i; The resource risk values ​​include: QueueCost(i) = PendingPrompt(i) + ResidualPrompt(i) Where PendingPrompt(i) is the cumulative input workload of instance i that has not yet completed the preprocessing stage; ResidualPrompt(i) is the input length that still needs to be recalculated after the request is routed to instance i.

[0020] Preferably, the scoring expression in the execution graph compiler is: Score(i) = (PendingPrompt(i) + ResidualPrompt(i)) × (1 + Occupancy(i)) Where PendingPrompt(i) is the amount of preprocessing work that instance i still needs to complete at the current time; ResidualPrompt(i) is the amount of preprocessing work that will be added if the current request is routed to instance i; Occupancy(i) is the degree of concurrency congestion of instance i; the scheduler selects the candidate instance with the smallest Score from the candidate instances that meet the resource constraints as the target inference instance. The resource constraints include: candidate filtering rules and fallback rules; the candidate filtering rules include: pre-excluding unavailable instances based on available video memory, maximum context length, tenant isolation rules, circuit breaker flags or device anomaly flags; If the remaining candidate set is less than the preset threshold, or if a certain type of request is detected to continuously hit the same instance set and the queuing cost rises rapidly, a rollback will be triggered. Once a rollback is triggered, the strategy will switch to load balancing priority, tiered round-robin, or abnormal instance isolation.

[0021] Compared with the prior art, the present invention has the following beneficial effects: 1. By constructing a unified policy factory, this invention can decouple state maintenance, metric generation, policy evaluation, and rollback mechanism from specific policies, significantly reducing the cost of integrating new policies. 2. This invention reduces misjudgments caused by inconsistent state reading timing and improves the stability of scheduling decisions through a consistent snapshot mechanism; 3. This invention generates a prefix signature index and derived measure, which can simultaneously perceive cache reuse opportunities and instance load status under low overhead conditions; 4. This invention, through a pre-occupied write-back mechanism, can alleviate the problem of hotspot amplification under sudden traffic surges and reduce the instantaneous imbalance caused by concentrated traffic. 5. This invention maintains service continuity in scenarios such as cache hotspots, resource shortages, and instance failures through templated rollback and exception isolation mechanisms.

[0022] 6. This invention is applicable to different models, different inference engines and different cluster sizes, and has good engineering versatility and scalability. Attached Figure Description

[0023] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a schematic diagram of the overall structure of the strategy factory.

[0024] Figure 2 To measure the relationship between generation and strategy assembly.

[0025] Figure 3 This is a flowchart of the strategy factory scheduling process. Detailed Implementation

[0026] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0027] Example 1 A policy factory construction system for large model inference clusters provided by the present invention includes: The module includes a request classification and strategy selection module, a state mirroring layer, a prefix signature index, a metric generator, a strategy template library, an execution graph compiler, and a route submission and pre-placement write-back module. The request classification and strategy selection module is used to receive external requests, identify the business tag, model identifier, tenant identifier or session identifier of the external request, and determine the target strategy template corresponding to the external request from the strategy template library based on the identification result. The state mirror layer is used to receive incremental state events returned by each inference instance and maintain instance state snapshots with a unified version number. The instance state snapshots are used to provide the current state information of each inference instance to the prefix signature index and the metric generator. Specifically, the state mirror layer includes: Each inference instance sends incremental state back to the global scheduler via long connections, response header piggybacking, event streams, or shared message channels; wherein, the incremental state includes at least: request entering queue event, request completion preprocessing event, request completion generation event, cache block addition event, cache block eviction event, and resource exception event; the state mirror layer merges the above events and assigns a monotonically increasing version number to each update; When a new user request is received, the snapshot with the same version number is read all at once through the global scheduler.

[0028] The prefix signature index is used to estimate the reusable context length of the external request on different inference instances based on the request prefix fingerprint of the external request and the cache index in the instance state snapshot. Specifically, the prefix signature index includes: The external request input sequence is divided into blocks of fixed token length B. A hash signature is generated for each prefix block to form the signature sequence RequestFingerprint=(sig1,sig2,...,sigN) formed by the current request prefix after block division and hashing. Each inference instance i maintains CacheIndex(i), which records the current cached prefix signature sequence and the corresponding cache block position of the instance. During scheduling, the RequestFingerprint of the current request is matched with the CacheIndex(i) for the longest prefix to obtain the longest prefix block k that can be directly reused on instance i. Then, the prefix length ReuseLen(i) that the current request can reuse on inference instance i is k×B.

[0029] If the last block is less than B or requires precise verification, it can be adjusted according to the actual number of tokens.

[0030] The reusable prefix length ReuseLen(i) refers to the length of the request prefix that can be directly reused in the cache of the current request if it is routed to inference instance i. This length can be obtained by longest prefix matching between the prefix signature of the current request and the cache index of instance i. The larger ReuseLen(i) is, the more context can be reused after routing to this instance, and the fewer input parts need to be recalculated.

[0031] The metric generator is used to generate the remaining preprocessing volume, concurrency consumption, and resource risk value for each inference instance based on the instance state snapshot and the reusable context length. Specifically, the metric generator includes: The remaining preprocessing amount corresponding to each inference instance includes: ResidualPrompt(i) = PromptLen - ReuseLen(i) The ResidualPrompt(i) refers to the input length or workload that still needs to be newly computed after routing the current request to inference instance i, excluding the reusable prefix portion. This metric measures how much additional preprocessing computation burden is incurred after assigning the current request to an instance. PromptLen is the total input length of the current request; ReuseLen(i) is the reusable prefix length of the current request on inference instance i. The concurrent occupancy includes: Occupancy(i) = WaitingReqs(i) + RunningReqs(i) Here, Occupancy(i) refers to the degree to which inference instance i is currently occupied by requests, which can be obtained by combining the number of waiting requests, the number of running requests, batch processing slot occupancy, KV cache occupancy, or other resource occupancy indicators. This indicator is used to avoid continuing to allocate requests to already congested instances. WaitingReqs(i) is the number of requests waiting to be processed on instance i; RunningReqs(i) is the number of requests currently being processed on instance i; The resource risk values ​​include: QueueCost(i) = PendingPrompt(i) + ResidualPrompt(i) Here, PendingPrompt(i) refers to the cumulative input workload that has entered the queue but has not yet completed the preprocessing stage on inference instance i. Compared with the simple number of queued requests, PendingPrompt(i) can reflect the computational pressure caused by the difference in input length of different requests. For example, although a long prompt request and a short prompt request are both counted as only one waiting request, they cause different preprocessing burdens. Therefore, this invention uses PendingPrompt(i) to more precisely describe the workload to be processed on the instance. ResidualPrompt(i) is the input length that still needs to be recalculated after the request is routed to instance i.

[0032] The execution graph compiler is used to call the measure generated by the measure generator according to the target policy template, and to evaluate multiple candidate inference instances in parallel according to the scoring expression, candidate filtering rules and backoff rules in the target policy template, so as to determine the target inference instance; Specifically, the scoring expression in the execution graph compiler is: Score(i) = (PendingPrompt(i) + ResidualPrompt(i)) × (1 + Occupancy(i)) Where PendingPrompt(i) is the amount of preprocessing work that instance i still needs to complete at the current time; ResidualPrompt(i) is the amount of preprocessing work that will be added if the current request is routed to instance i; Occupancy(i) is the degree of concurrency congestion of instance i; the scheduler selects the candidate instance with the smallest Score from the candidate instances that meet the resource constraints as the target inference instance. The resource constraints include: candidate filtering rules and fallback rules; the candidate filtering rules include: pre-excluding unavailable instances based on available video memory, maximum context length, tenant isolation rules, circuit breaker flags or device anomaly flags; If the remaining candidate set is less than the preset threshold, in this embodiment, the preferred example of the current threshold is 1 instance, or if a certain type of request is detected to continuously hit the same instance set and the queuing cost rises rapidly, then a rollback is triggered. Once a rollback is triggered, the strategy will switch to load balancing priority, tiered round-robin, or abnormal instance isolation.

[0033] The route submission and pre-placement write-back module is used to route the external request to the target inference instance and write a transient placeholder state corresponding to the target inference instance into the state mirror layer, so that the scheduling of subsequent requests can be aware of the resource occupation of the target inference instance by the external request.

[0034] The key to this invention lies not in simply proposing a new scoring value, but in decomposing the large-scale model scheduling process into reusable, factory-like links: first, maintaining instance states through incremental events; then, generating metrics by combining current requests with instance snapshots; next, executing pre-compiled policy graphs on consistent snapshots; and finally, completing stable routing with pre-placement and reconciliation mechanisms. This structure can significantly reduce policy implementation costs and improve the consistency, scalability, and engineering maintainability of scheduling results.

[0035] Example 2 This invention provides a policy factory construction system and implementation method for large-scale model inference clusters. This method is not limited to a fixed scoring formula, but rather establishes a unified state mirror layer, prefix signature index, metric generator, policy template library, execution graph compiler, and route submission and write-back modules within the global scheduler. Under this architecture, any scheduling policy is abstracted as a combination of "metric dependency + candidate filtering rule + scoring expression + fallback rule," thereby achieving rapid assembly and reuse.

[0036] The strategy factory of this invention is deployed inside the cluster-level global scheduler, and its overall structure is as follows: Figure 1 As shown, after an external request enters, the target policy template is first determined by the request classification and policy selection module; then, the state mirror layer provides a unified version of the instance snapshot, and the prefix signature index and metric generator obtain the reusability, remaining workload, and risk value of the current request on each instance; then, the execution graph compiler completes parallel evaluation based on the policy template, and finally, the route submission and pre-placement write-back module selects the target instance and updates the transient state.

[0037] The state mirror layer receives incremental events from instances and maintains a global state snapshot with version numbers, ensuring that a single scheduling decision uses the same view. State mirroring means that the global scheduler does not directly access all inference instances in real time during each scheduling iteration. Instead, based on the incremental events returned by each inference instance, the scheduler maintains a logical state copy with a version number on the scheduler side. This state copy may include the number of queued requests, the number of running requests, the amount of preprocessing work not yet completed, cache block mapping relationships, resource usage, and abnormal states for each instance. The scheduler reads the state mirror with the same version number during a single routing decision, thus ensuring that the comparison between candidate instances is based on a consistent view.

[0038] The prefix signature index estimates the reusable context length on different instances based on the request prefix fingerprint and cache block index. Specifically, the prefix signature involves dividing the input prefix of the user request according to a fixed token block length or other preset rules, and generating a hash value, digest value, or other comparable identifier for each prefix block, forming a set of prefix identifier sequences. This prefix signature is used to match with the existing cache block indexes on different inference instances, thereby quickly determining how much context cache the current request can reuse on each instance without comparing the complete text token by token.

[0039] The metric generator is used to convert the basic state into derived metrics such as remaining preprocessing volume, concurrency occupancy, and resource risk value.

[0040] The strategy template library is used to store scoring expressions, candidate filtering rules, fallback rules, and different business binding relationships.

[0041] The execution graph compiler is used to precompile policy templates into execution graphs, which are then evaluated directly in parallel when a request arrives.

[0042] The route submission and pre-placement write-back module is used to write a temporary placeholder after the target instance is selected, so as to avoid sudden requests being concentrated on the same instance.

[0043] The state mirroring layer includes the following: each inference instance sends incremental state back to the global scheduler via long connections, response header piggybacking, event streams, or shared message channels. The incremental state includes at least: request entering queue events, request completion preprocessing events, request completion generation events, cache block addition events, cache block eviction events, and resource exception events. The state mirroring layer merges these events and assigns a monotonically increasing version number to each update.

[0044] When a new user request arrives, the scheduler does not read the latest point-in-time state of each instance separately, but instead reads a snapshot with the same version number all at once. This avoids comparison bias caused by some instances using updated data while others still use old data, improving the consistency and interpretability of routing decisions.

[0045] The prefix signature index includes: dividing the request input sequence into segments of a fixed token block length B, generating a hash signature for each prefix block, forming RequestFingerprint = (sig1, sig2, ..., sigN). Each inference instance i maintains CacheIndex(i), which records the currently cached prefix signature sequence and the corresponding cache block position. During scheduling, the RequestFingerprint of the current request is matched with CacheIndex(i) for the longest prefix, obtaining the longest prefix block k that can be directly reused on instance i, then ReuseLen(i) = k × B; if the last block is less than B or requires precise verification, it can be corrected according to the actual number of tokens. Subsequently, ResidualPrompt(i) = PromptLen - ReuseLen(i) is calculated, representing the input length that still needs to be recalculated after the request is routed to instance i.

[0046] The metric generator includes: This invention divides the information on which scheduling depends into basic measures and derived measures. Basic measures can be obtained directly from the state image, such as the number of queued requests, the number of running requests, the amount of preprocessing in the queue, the cache block set, and the device available budget. Derived measures are calculated jointly from the current requests and instance snapshots, such as the remaining preprocessing amount, cache affinity, concurrency utilization, and resource risk value.

[0047] ReuseLen(i) = PrefixMatch(RequestFingerprint, CacheIndex(i)) ResidualPrompt(i) = PromptLen - ReuseLen(i) Occupancy(i) = WaitingReqs(i) + RunningReqs(i) QueueCost(i) = PendingPrompt(i) + ResidualPrompt(i) Wherein, ReuseLen(i) represents the reusable prefix length of the current request on inference instance i; PrefixMatch represents the longest prefix matching function; RequestFingerprint represents the signature sequence formed by the current request prefix after block splitting and hashing; CacheIndex(i) represents the cache block signature index currently maintained by inference instance i; ResidualPrompt(i) represents the input length that still needs to be recalculated after the request is routed to instance i; PromptLen represents the total input length of the current request; Occupancy(i) represents the concurrency of instance i; WaitingReqs(i) represents the number of requests waiting to be processed on instance i; RunningReqs(i) represents the number of requests currently being processed on instance i; QueueCost(i) represents the total queuing cost of instance i; PendingPrompt(i) represents the cumulative input workload that instance i has not yet completed the preprocessing stage.

[0048] like Figure 2 As shown, the policy template declaratively describes the required measures, dependencies, filtering conditions, scoring expressions, and fallback rules. The execution graph compiler completes dependency unrolling and execution graph generation during the template registration phase, allowing the target instance to be computed in parallel without reinterpreting the entire set of expressions when a request arrives; it only needs to populate the current measure value. This approach reduces online scheduling overhead and enables different policies to share the underlying measure generation chain.

[0049] In this invention, policy execution does not end with "forwarding immediately after calculating a score," but includes a transient write-back phase. After selecting a target instance, the scheduler first writes the instance's concurrency usage, queuing cost, and necessary resource budget into a pre-placement state before forwarding the request. When the instance subsequently sends back real enqueue, completion, or exception events, the state mirror layer performs reconciliation and correction. Specifically, pre-placement write-back means that after the scheduler selects the target inference instance according to the policy, it does not wait for the instance to actually send back a queuing event before updating the state mirror, but instead writes a transient placeholder into the state mirror maintained by the scheduler. For example, it first increases the instance's concurrency usage, queuing cost, or resource budget usage. In this way, when multiple requests arrive consecutively in a short period of time, subsequent requests can be aware of the transient resources already occupied by the preceding requests, avoiding multiple requests simultaneously selecting the same instance based on the same old snapshot. After the target instance subsequently sends back real state events, the state mirror layer confirms, corrects, or rolls back the transient placeholder.

[0050] The advantage of using a pre-placement write-back mechanism is that, under sudden traffic surges, even if multiple requests share the same snapshot for decision-making within a very short period of time, subsequent requests can still perceive the transient resources already occupied by previous requests, avoiding a large number of requests simultaneously selecting the same hot instance and reducing instantaneous fluctuations.

[0051] This invention incorporates a candidate filter and a fallback controller within the strategy factory. The candidate filter can pre-exclude unavailable instances based on available memory, context length limits, tenant isolation rules, circuit breaker flags, or device anomaly flags. If the remaining candidate set is less than a preset threshold, or if a certain type of request is detected to continuously hit the same instance set with a rapidly increasing queuing cost, a fallback is triggered.

[0052] Once a rollback is triggered, this invention can switch to a load balancing priority strategy, a hierarchical round-robin strategy, or an abnormal instance isolation strategy. Since the rollback rules themselves are part of the strategy template, protective switching can be completed without rewriting the scheduler's core logic, thus enhancing system stability.

[0053] In a preferred embodiment, the present invention employs the following scoring expression: Score(i) = (PendingPrompt(i) + ResidualPrompt(i)) × (1 + Occupancy(i)) Here, PendingPrompt(i) reflects the amount of preprocessing work that instance i still needs to complete at the current moment, ResidualPrompt(i) reflects the additional preprocessing work if the current request is routed to instance i, and Occupancy(i) reflects the degree of concurrency congestion of instance i. The scheduler selects the instance with the lowest score from the candidate instances that meet the resource constraints as the target instance.

[0054] The present invention is not limited to the preferred scoring expression described above.

[0055] This preferred embodiment has the following characteristics: First, the more fully the cache is reused, the smaller ResidualPrompt(i) is, and the lower the score; second, the more congested the instances are, the higher the Occupancy(i) is, and the higher the score; third, compared to a linear combination structure that requires manually setting multiple sets of weights, the above expression is more suitable for direct access to the policy factory for stable reuse. In other embodiments, piecewise functions, lexicographical comparisons, multi-objective constraints, or risk correction terms can also be used to adapt to different deployment objectives.

[0056] For different deployment objectives, the strategy template can adopt addition, multiplication, piecewise functions, sorting rules, risk penalty items or multi-stage decision structures; the prefix signature index can adopt block-level index, tree index or hash mapping; the status return method can adopt event stream, response header piggyback, shared memory messages or other equivalent mechanisms.

[0057] Example 3 According to the present invention, a method for constructing a policy factory system for large-scale model inference clusters is provided, such as... Figure 3 As shown, it includes: Step 1: Receive the request and identify its business tag, model identifier, tenant identifier, or session identifier.

[0058] Step 2: Select the corresponding strategy template and fallback rule based on the preset mapping relationship.

[0059] Step 3: Read the instance status snapshot under the same version number to form a consistent view.

[0060] Step 4: Estimate the reusable prefix length based on the request prefix signature and the cache index of each instance.

[0061] Step 5: Generate basic measures, derived measures, and risk measures.

[0062] Step 6: Call the pre-compiled execution graph according to the strategy template and evaluate all candidate instances in parallel.

[0063] Step 7: Perform candidate filtering and anomaly exclusion, and trigger hotspot detection and rollback control if necessary.

[0064] Step 8: Select the target instance and write it into the pre-placement state.

[0065] Step 9: Forward the request to the target instance to perform inference.

[0066] Step 10: Receive the real state increment returned by the instance, and reconcile and correct the image state.

[0067] Those skilled in the art will understand that, in addition to implementing the system, apparatus, and their modules provided by this invention in purely computer-readable program code, the same program can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system, apparatus, and their modules provided by this invention can be considered a hardware component, and the modules included therein for implementing various programs can also be considered structures within the hardware component; alternatively, modules for implementing various functions can be considered both software programs implementing the method and structures within the hardware component.

[0068] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A policy factory construction system for large model inference clusters, characterized in that, include: The module includes a request classification and strategy selection module, a state mirroring layer, a prefix signature index, a metric generator, a strategy template library, an execution graph compiler, and a route submission and pre-placement write-back module. The request classification and strategy selection module is used to receive external requests, identify the business tag, model identifier, tenant identifier or session identifier of the external request, and determine the target strategy template corresponding to the external request from the strategy template library based on the identification result. The state mirror layer is used to receive incremental state events returned by each inference instance and maintain instance state snapshots with a unified version number. The instance state snapshots are used to provide the current state information of each inference instance to the prefix signature index and the metric generator. The prefix signature index is used to estimate the reusable context length of the external request on different inference instances based on the request prefix fingerprint of the external request and the cache index in the instance state snapshot. The metric generator is used to generate the remaining preprocessing volume, concurrency consumption, and resource risk value for each inference instance based on the instance state snapshot and the reusable context length. The execution graph compiler is used to call the measure generated by the measure generator according to the target policy template, and to evaluate multiple candidate inference instances in parallel according to the scoring expression, candidate filtering rules and backoff rules in the target policy template, so as to determine the target inference instance; The route submission and pre-placement write-back module is used to route the external request to the target inference instance and write a transient placeholder state corresponding to the target inference instance into the state mirror layer, so that the scheduling of subsequent requests can be aware of the resource occupation of the target inference instance by the external request.

2. The policy factory construction system for large model inference clusters according to claim 1, characterized in that, The state mirroring layer includes: Each inference instance sends incremental state back to the global scheduler via long connections, response header piggybacking, event streams, or shared message channels; wherein, the incremental state includes at least: request entering queue event, request completion preprocessing event, request completion generation event, cache block addition event, cache block eviction event, and resource exception event; the state mirror layer merges the above events and assigns a monotonically increasing version number to each update; When a new user request is received, the snapshot with the same version number is read all at once through the global scheduler.

3. The policy factory construction system for large model inference clusters according to claim 1, characterized in that, The prefix signature index includes: The external request input sequence is divided into blocks of fixed token length B. A hash signature is generated for each prefix block to form the signature sequence RequestFingerprint=(sig1,sig2,...,sigN) formed by the current request prefix after block division and hashing. Each inference instance i maintains CacheIndex(i), which records the current cached prefix signature sequence and the corresponding cache block position of the instance. During scheduling, the RequestFingerprint of the current request is matched with the CacheIndex(i) for the longest prefix to obtain the longest prefix block k that can be directly reused on instance i. Then, the prefix length ReuseLen(i) that the current request can reuse on inference instance i is k×B. If the last block is less than B or requires precise verification, it can be adjusted according to the actual number of tokens.

4. The policy factory construction system for large model inference clusters according to claim 1, characterized in that, The metric generator includes: The remaining preprocessing amount corresponding to each inference instance includes: ResidualPrompt(i) = PromptLen - ReuseLen(i) Where PromptLen is the total length of the input for the current request; ReuseLen(i) is the length of the prefix that can be reused in the current request on inference instance i; The concurrent occupancy includes: Occupancy(i) = WaitingReqs(i) + RunningReqs(i) Where WaitingReqs(i) is the number of requests waiting to be processed on instance i; RunningReqs(i) is the number of requests currently being processed on instance i; The resource risk values ​​include: QueueCost(i) = PendingPrompt(i) + ResidualPrompt(i) Where PendingPrompt(i) is the cumulative input workload of instance i that has not yet completed the preprocessing stage; ResidualPrompt(i) is the input length that still needs to be recalculated after the request is routed to instance i.

5. The policy factory construction system for large model inference clusters according to claim 1, characterized in that, The scoring expression in the execution graph compiler is: Score(i) = (PendingPrompt(i) + ResidualPrompt(i)) × (1 + Occupancy(i)) Where PendingPrompt(i) is the amount of preprocessing work that instance i still needs to complete at the current time; ResidualPrompt(i) is the amount of preprocessing work that will be added if the current request is routed to instance i; Occupancy(i) is the degree of concurrency congestion of instance i; the scheduler selects the candidate instance with the smallest Score from the candidate instances that meet the resource constraints as the target inference instance. The resource constraints include: candidate filtering rules and fallback rules; the candidate filtering rules include: pre-excluding unavailable instances based on available video memory, maximum context length, tenant isolation rules, circuit breaker flags or device anomaly flags; If the remaining candidate set is less than the preset threshold, or if a certain type of request is detected to continuously hit the same instance set and the queuing cost rises rapidly, a rollback will be triggered. Once a rollback is triggered, the strategy will switch to load balancing priority, tiered round-robin, or abnormal instance isolation.

6. A method for implementing a policy factory construction system for large model inference clusters, characterized in that, include: Step S1: Receive external requests through the request classification and strategy selection module, identify the business tag, model identifier, tenant identifier or session identifier of the external request, and determine the target strategy template corresponding to the external request from the strategy template library based on the identification result; Step S2: Receive incremental state events returned by each inference instance through the state mirror layer, and maintain instance state snapshots with a unified version number. The instance state snapshots are used to provide the current state information of each inference instance to the prefix signature index and the metric generator. Step S3: Estimate the reusable context length of the external request on different inference instances based on the request prefix fingerprint of the external request and the cache index in the instance state snapshot using the prefix signature index; Step S4: Based on the instance state snapshot and the reusable context length, the metric generator generates the remaining preprocessing amount, concurrency consumption, and resource risk value for each inference instance. Step S5: The graph compiler calls the measure generated by the measure generator according to the target policy template, and evaluates multiple candidate inference instances in parallel according to the scoring expression, candidate filtering rules and backoff rules in the target policy template to determine the target inference instance; Step S6: The external request is routed to the target inference instance through the route submission and pre-placement write-back module, and a transient placeholder state corresponding to the target inference instance is written into the state mirror layer, so that the scheduling of subsequent requests can be aware of the resource occupation of the target inference instance by the external request.

7. The method for implementing a policy factory construction system for large model inference clusters according to claim 6, characterized in that, The state mirroring layer includes: Each inference instance sends incremental state back to the global scheduler via long connections, response header piggybacking, event streams, or shared message channels; wherein, the incremental state includes at least: request entering queue event, request completion preprocessing event, request completion generation event, cache block addition event, cache block eviction event, and resource exception event; the state mirror layer merges the above events and assigns a monotonically increasing version number to each update; When a new user request is received, the snapshot with the same version number is read all at once through the global scheduler.

8. The method for implementing a policy factory construction system for large model inference clusters according to claim 6, characterized in that, The prefix signature index includes: The external request input sequence is divided into blocks of fixed token length B. A hash signature is generated for each prefix block to form the signature sequence RequestFingerprint=(sig1,sig2,...,sigN) formed by the current request prefix after block division and hashing. Each inference instance i maintains CacheIndex(i), which records the current cached prefix signature sequence and the corresponding cache block position of the instance. During scheduling, the RequestFingerprint of the current request is matched with the CacheIndex(i) for the longest prefix to obtain the longest prefix block k that can be directly reused on instance i. Then, the prefix length ReuseLen(i) that the current request can reuse on inference instance i is k×B. If the last block is less than B or requires precise verification, it can be adjusted according to the actual number of tokens.

9. The method for implementing a policy factory construction system for large model inference clusters according to claim 6, characterized in that, The metric generator includes: The remaining preprocessing amount corresponding to each inference instance includes: ResidualPrompt(i) = PromptLen - ReuseLen(i) Where PromptLen is the total length of the input for the current request; ReuseLen(i) is the length of the prefix that can be reused in the current request on inference instance i; The concurrent occupancy includes: Occupancy(i) = WaitingReqs(i) + RunningReqs(i) Where WaitingReqs(i) is the number of requests waiting to be processed on instance i; RunningReqs(i) is the number of requests currently being processed on instance i; The resource risk values ​​include: QueueCost(i) = PendingPrompt(i) + ResidualPrompt(i) Where PendingPrompt(i) is the cumulative input workload of instance i that has not yet completed the preprocessing stage; ResidualPrompt(i) is the input length that still needs to be recalculated after the request is routed to instance i.

10. The method for implementing a policy factory construction system for large model inference clusters according to claim 6, characterized in that, The scoring expression in the execution graph compiler is: Score(i) = (PendingPrompt(i) + ResidualPrompt(i)) × (1 + Occupancy(i)) Where PendingPrompt(i) is the amount of preprocessing work that instance i still needs to complete at the current time; ResidualPrompt(i) is the amount of preprocessing work that will be added if the current request is routed to instance i; Occupancy(i) is the degree of concurrency congestion of instance i; the scheduler selects the candidate instance with the smallest Score from the candidate instances that meet the resource constraints as the target inference instance. The resource constraints include: candidate filtering rules and fallback rules; the candidate filtering rules include: pre-excluding unavailable instances based on available video memory, maximum context length, tenant isolation rules, circuit breaker flags or device anomaly flags; If the remaining candidate set is less than the preset threshold, or if a certain type of request is detected to continuously hit the same instance set and the queuing cost rises rapidly, a rollback will be triggered. Once a rollback is triggered, the strategy will switch to load balancing priority, tiered round-robin, or abnormal instance isolation.

Images