An API gateway kv cache awareness scheduling method and system

Abstract

The application discloses an API gateway KV Cache sensing scheduling method and system, receives a large model inference request, performs KV Cache sensing matching on the large model inference request and historical prefix records, determines a candidate model service gateway node, acquires the load of an inference node subordinate to the candidate node, performs load balancing when distribution conditions are met, and determines a target node; wherein the target node contains a primary backup copy, and a sidecar-competition lease is used to dynamically determine a ready state; a request is forwarded to the primary backup copy through a gateway external plug-in, polling and retrying between the primary backup copy are performed when an exception is captured, a standby node chain is used for reselecting an available node when all the failures occur, request content is converted into a hash value sequence, and the hash value sequence is stored in the prefix records. The application integrates KV Cache sensing, load balancing sensing and primary backup high-availability scheduling at a cloud native API gateway layer, solves the problems of low cache hit rate, uneven resource utilization and insufficient high availability in the prior art, and significantly improves the overall performance and stability of large model inference services.

Description

Technical Field

[0001] This invention relates to the fields of cloud computing, artificial intelligence and API gateway technology, and in particular to an API gateway KV Cache-aware scheduling method and system for large model inference services. Background Technology

[0002] With the widespread application of large language models, the performance and resource utilization of their inference services have become key challenges. KVCache technology avoids redundant computation during the Decode stage of large model inference by storing the attention calculation results of historical tokens, and is a core means to improve inference performance.

[0003] like Figure 1 As shown, a typical inference service chain includes three layers: a cloud-native API gateway, a model service gateway, and a model inference service. In existing technologies, the model inference service itself has KV cache reuse capabilities, and the model service gateway also has a cache-aware mechanism that can route requests to nodes with high cache hit rates based on request prefix matching. However, the cloud-native API gateway, as the entry point, lacks KV cache awareness. When multiple model service gateway instances provide the same model service, the cloud-native API gateway cannot intelligently select the backend gateway that maximizes KV cache reuse based on the request content. This results in requests being randomly or round-robin distributed, easily leading to low KV cache reuse rates, repeated prefill calculations, increased inference latency, node load imbalance, concentrated requests for hot topics while other nodes are idle, and overall low resource utilization and throughput.

[0004] Existing consistent hashing schemes can ensure that requests from the same session land on the same node, but they are prone to causing concentrated requests from hot users, resulting in uneven load distribution.

[0005] Therefore, the existing technology has problems and needs further improvement and development. Summary of the Invention

[0006] (I) Purpose of the invention: In order to solve the problems existing in the prior art, the purpose of the present invention is to provide an API gateway KV Cache-aware scheduling method and system, which solves the problem that the existing cloud-native API gateway lacks KV Cache awareness capability, resulting in low performance and resource utilization of large model inference services.

[0007] (II) Technical Solution: To address the aforementioned technical problems, this technical solution provides an API gateway KV Cache-Aware Scheduling Method, applied to cloud-native API gateways. The method includes:

[0008] The system receives a large model inference request sent by the client, performs KV Cache-aware matching between the large model inference request and the prefix records of historical inference information to obtain prefix hit information, and determines at least one candidate model service gateway node with prefix matching based on the prefix hit information; the at least one candidate model service gateway node is set in the same service mesh.

[0009] The load information of the inference nodes under the candidate model service gateway node is obtained. When the preset allocation conditions are not met, the corresponding candidate model service gateway node is determined as the final target model service gateway node according to the prefix hit information. When the preset allocation conditions are met, load balancing is performed to determine the candidate model service gateway node with the lowest load as the final target model service gateway node. The target model service gateway node includes one primary replica and at least one backup replica. The primary replica in the ready state and one or more backup replicas in the non-ready state are dynamically determined through sidecar-competitive lease.

[0010] The large model inference request is forwarded to the primary replica currently in a ready state in the target model service gateway node through the external gateway plugin; if an abnormal state is captured after forwarding, the external gateway plugin is used to poll and retry between the primary replica and the backup replica; if both the primary replica and the backup replica fail to retry, the available node with the least load is selected again according to the backup node chain to continue forwarding the large model inference request.

[0011] The content of the large model inference request is converted into a hash value sequence and stored in the prefix record.

[0012] Preferably, the prefix hit information is obtained by performing KV Cache-aware matching between the large model inference request and the prefix record of historical inference information, including:

[0013] The message content of the large model inference request is divided into a message array by sequence number;

[0014] Calculate the digest value of each message in the message array, and convert the message array into a corresponding hash value array based on the digest value;

[0015] The hash value array is fed into the prefix record for matching, and the matching length and prefix hit rate of a successful match are obtained. The matching length and the prefix hit rate are then used as the prefix hit information.

[0016] Preferably, determining at least one candidate model service gateway node with prefix matching based on the prefix hit information includes:

[0017] Based on the matching length, a search is conducted among all model service gateway nodes managed within the service mesh to select one or more model service gateway nodes that can achieve the longest prefix hash match with the hash value array, thus forming the candidate model service gateway nodes.

[0018] Preferably, obtaining the load information of the inference nodes under the candidate model service gateway node includes:

[0019] Continuously collect dynamic load data for each pre-filled node managed by the candidate model service gateway node;

[0020] The load information includes the pre-filled maximum load value and the pre-filled minimum load value for each candidate model service gateway node.

[0021] Preferably, the preset allocation conditions include a prefix hit rate less than a preset cache hit threshold; and,

[0022] There is at least one candidate load balancer node within the service mesh. The candidate load balancer node satisfies the following sub-conditions: the difference between the pre-filled maximum load value of the candidate model service gateway node and the pre-filled minimum load value of the candidate load balancer node is greater than or equal to a preset request gap threshold; and the ratio of the pre-filled maximum load value of the candidate model service gateway node to the pre-filled minimum load value of the candidate load balancer node is greater than or equal to a preset request ratio threshold; the difference between the pre-filled maximum load value of the candidate load balancer node itself and the pre-filled minimum load value of its own subordinates is greater than or equal to the request gap threshold; and the ratio of the pre-filled maximum load value of the candidate load balancer node itself to the pre-filled minimum load value of its own subordinates is greater than or equal to the request ratio threshold.

[0023] Preferably, load balancing is performed when preset allocation conditions are met, and determining the candidate model service gateway node with the lowest load as the final target model service gateway node includes:

[0024] The candidate equilibrium nodes are used to form an equilibrium candidate set;

[0025] By comparing the pre-filled minimum load values ​​of each candidate balanced node in the balanced candidate set, the candidate balanced node with the smallest pre-filled minimum load value is selected as the final target model service gateway node.

[0026] Preferably, the step of dynamically determining one primary replica in a ready state and one or more backup replicas in a non-ready state through sidecar-competitive leases includes:

[0027] Inject a sidecar-competitive lease container into each initial replica of the target model service gateway node. The sidecar-competitive lease container competes for distributed locked leases after startup.

[0028] The copy of the sidecar container that wins the competition and holds a fixed lease is marked as ready and becomes the primary copy.

[0029] The copy of the sidecar container that has not obtained a lease is marked as not ready and is used as a backup copy.

[0030] When the primary replica in the ready state fails, it actively releases the lease and switches to the not ready state. The sidecars in the other backup replicas in the not ready state then compete for the lease and switch to the ready state.

[0031] Preferably, the abnormal state includes at least one of the following: returning a preset error status code, request timeout, transmission control protocol connection failure, node circuit breaker failure, and network health check failure;

[0032] When the external gateway plugin detects that the currently sent replica returns the abnormal state, it resends the large model inference request to the standby replica in the target model service gateway node that is not ready, according to the polling rule, for fault tolerance processing.

[0033] Preferably, if both the primary and backup replicas fail to retry, the available node with the lowest load is reselected from the backup node chain to continue forwarding the large model inference request, including:

[0034] When both the primary replica and the backup replica within the target model service gateway node fail to retry, the gateway external plugin reports a group failure status to the cloud-native API gateway.

[0035] The cloud-native API gateway activates the backup node chain that was pre-established for the service gateway node of the target model, and after excluding the node set that has failed in this request, filters and removes nodes in abnormal state to obtain available substitute nodes.

[0036] From the selected available backup nodes, the node with the lowest load is chosen to continue forwarding the large model inference request.

[0037] Preferably, the content of the large model inference request is converted into a hash value sequence and stored in the prefix record, including:

[0038] Read the message set field in the large model inference request and obtain them one by one to form an input message queue;

[0039] Each message in the input message queue is independently calculated sequentially using a preset one-way hash algorithm to obtain a continuous chain of hash digests as the hash value sequence.

[0040] The hash value sequence is inserted as a new path into the prefix record according to the trie data structure.

[0041] An API gateway KV Cache-Aware Scheduling System is provided for executing the API gateway KV Cache-Aware Scheduling Method. The system includes a request receiving and awareness module, a load balancing module, a forwarding and scheduling module, and a prefix state update module.

[0042] The request receiving and sensing module is used to receive large model inference requests sent by the client, perform KV Cache sensing matching between the large model inference request and the prefix records of historical inference information to obtain prefix hit information, and determine at least one candidate model service gateway node with prefix matching based on the prefix hit information; the at least one candidate model service gateway node is set in the same service mesh;

[0043] The load balancing module is used to obtain the load information of the inference nodes under the candidate model service gateway node. When the preset allocation conditions are not met, the corresponding candidate model service gateway node is determined as the final target model service gateway node according to the prefix hit information. When the preset allocation conditions are met, load balancing is performed to determine the candidate model service gateway node with the lowest load as the final target model service gateway node. The target model service gateway node includes one primary replica and at least one backup replica. Each replica dynamically determines one primary replica in the ready state and one or more backup replicas in the non-ready state through sidecar-competition lease.

[0044] The forwarding and scheduling module is used to forward the large model inference request to the primary replica currently in a ready state in the target model service gateway node through the gateway external plugin; when an abnormal state is captured after forwarding, the module performs a round-robin retry between the primary replica and the backup replica through the gateway external plugin; when both the primary replica and the backup replica fail to retry, the module reselects the available node with the least load according to the backup node chain to continue forwarding the large model inference request.

[0045] The prefix state update module is used to convert the content of the large model inference request into a hash value sequence and store it in the prefix record.

[0046] (III) Beneficial Effects: This invention provides an API gateway KV Cache-Aware Scheduling Method and System. First, by implementing request content prefix-based awareness matching at the cloud-native API gateway layer, it ensures that requests are preferentially routed to the model service gateway with the richest cache, significantly improving KV Cache reuse and reducing inference latency. Second, by introducing load balancing triggering conditions, based on cache awareness, it can intervene when node load is severely uneven, scheduling requests to the node with the lowest load, avoiding resource hotspot issues caused by traditional consistent hashing schemes. Furthermore, through the sidecar-competitive lease mechanism, a primary-backup high-availability mode for the model service gateway is achieved without compromising the service mesh's native circuit breaking and health check capabilities. Combined with the gateway's external plugin's fast retry and backup node chain, service failures are effectively addressed, ensuring the continuity of inference services. Finally, compared to methods that synchronize KV Cache states through complex message channels, this invention avoids performance bottlenecks and consistency issues caused by state synchronization between multiple replicas through prefix recording and awareness scheduling at the cloud-native gateway layer, resulting in a simpler and more stable architecture. Attached Figure Description

[0047] Figure 1 This is an architecture diagram of the inference service link provided by the present invention;

[0048] Figure 2 This is a flowchart of the steps of an API gateway KV Cache-aware scheduling method according to the present invention;

[0049] Figure 3 This is a flowchart of the KV Cache-Aware Scheduling Process of the present invention;

[0050] Figure 4 This is a diagram of the high-availability architecture of the AI ​​gateway, including the sidecar-competition lease mechanism, as described in this invention.

[0051] Figure 5 This is a schematic diagram of the structure of an API gateway KV Cache-aware scheduling system according to the present invention. Detailed Implementation

[0052] The present invention will be further described in detail below with reference to preferred embodiments. Further details are set forth in the following description to provide a thorough understanding of the invention. However, the invention can obviously be implemented in many other ways different from those described herein, and those skilled in the art can make similar extensions and derivations based on actual applications without departing from the spirit of the invention. Therefore, the scope of protection of the present invention should not be limited by the content of this specific embodiment.

[0053] The accompanying drawings are schematic diagrams of embodiments of the present invention. It should be noted that these drawings are for illustrative purposes only and are not drawn to scale, and should not be construed as limiting the actual scope of protection of the present invention.

[0054] This invention provides an API gateway KV Cache-Aware Scheduling Method, applicable to cloud-native API gateways, such as... Figure 2 , Figure 3 As shown, the method includes the following steps:

[0055] Step 1: Receive a large model inference request sent by the client, perform KV Cache-aware matching between the large model inference request and the prefix records of historical inference information to obtain prefix hit information, and determine at least one candidate model service gateway node with prefix matching based on the prefix hit information; the at least one candidate model service gateway node is set in the same service mesh.

[0056] Step 2: Obtain the load information of the inference nodes under the candidate model service gateway node. When the preset allocation conditions are not met, determine the corresponding candidate model service gateway node as the final target model service gateway node based on the prefix hit information. When the preset allocation conditions are met, perform load balancing processing and determine the candidate model service gateway node with the smallest load as the final target model service gateway node. The target model service gateway node includes one primary replica and at least one backup replica. The primary replica in the ready state and one or more backup replicas in the non-ready state are dynamically determined through sidecar-competitive lease.

[0057] Step 3: Forward the large model inference request to the primary replica currently in a ready state in the target model service gateway node through the gateway external plugin; if an abnormal state is captured after forwarding, poll and retry between the primary replica and the backup replica through the gateway external plugin; if both the primary replica and the backup replica fail to retry, select the available node with the least load according to the backup node chain to continue forwarding the large model inference request.

[0058] Step 4: Convert the content of the large model inference request into a hash value sequence and store it in the prefix record.

[0059] In this implementation, fixed routing can also be performed based on request headers such as Session-ID, thereby enabling the reuse of KV Cache within the dialog context. However, in a real production environment, this can easily lead to severe load imbalance, causing some nodes to be overloaded while other nodes are idle, thus limiting overall throughput.

[0060] In addition, the ai-load-balancer capability provided by the existing cloud-native API gateway Higress only considers prefix matching and ignores the actual load of the backend nodes; at the same time, it relies on exposing fixed IPs, and in a cloud-native environment, once a node fails or migrates, it lacks automated fault recovery and fault tolerance capabilities.

[0061] Step 1 specifically includes:

[0062] Step 101: Receive and parse the large model inference request sent by the client.

[0063] The cloud-native API gateway listens for large model inference requests from clients via standard HTTP / 2 or gRPC protocols. Upon receiving a large model inference request, the cloud-native API gateway does not directly distribute the request. Instead, it first performs basic checks such as protocol parsing, authentication, and parameter validity, and then retrieves the messages array from the large model inference request.

[0064] The large model inference request follows a preset API format, including a message array and core parameters such as model name, temperature, and maximum text length for a single output (max_tokens).

[0065] The messages array is an array that stores the history of messages between the user and the assistant, as well as the current input, in the order of the conversation. Each message is an object, typically containing a role and text content.

[0066] Step 102: Convert the messages array into a fixed-length hash value array.

[0067] To perform prefix matching efficiently and avoid direct string comparison of long texts, the messages array is split into individual message entries in their original order, resulting in a message list. ,in This represents the total number of messages carried in the large model inference request. For each message... Based on the digest value of each message, its hash value is calculated using a preset one-way hash algorithm. Arrange all hash values ​​into a hash value array in order. .

[0068] The one-way hash algorithm is an algorithm that transforms input data of arbitrary length into a fixed-length output value through mathematical operations. It includes unencrypted one-way hash algorithms, such as MurmurHash3, SHA-256, or xxHash, and encrypted one-way hash algorithms, such as SHA-256. The unencrypted one-way hash algorithms are extremely fast, suitable for high-throughput API gateway scenarios; although they do not provide cryptographic-level collision resistance, they are secure and reliable enough for engineering requirements such as prefix matching. In contrast, the encrypted one-way hash algorithms significantly increase computational overhead, impacting gateway performance. To balance computational speed and collision probability, unencrypted one-way hash algorithms are typically used in industrial practice, outputting 64-bit or 128-bit integers.

[0069] Step 103: Send the hash value array into the prefix record for matching, obtain the matching length and prefix hit rate of a successful match, and use the matching length and prefix hit rate as the prefix hit information.

[0070] When a large model inference request arrives, the cloud-native API gateway will return its hash array. As the query key, it is sent to the prefix record module for matching.

[0071] Specifically, the prefix record module stores an array of hash values ​​for all historical inference requests using a trie data structure. The trie data structure includes a node structure and a root node. The node structure includes multiple nodes, each containing a hash value, a mapping to its child nodes, and a list of associated model service gateway nodes. The root node is empty and does not store a hash value.

[0072] Starting from the root node of the trie, try matching sequentially. When a certain The search stops when a corresponding child node is not found in the trie. The length of the successfully matched prefix is ​​recorded as L. When L=0, it means that the first hash value was not matched, and no historical inference request shares a prefix with this large model inference request; in this case, the prefix match information is empty. When L≥1, the longest common prefix of the current large model inference request is obtained. .

[0073] Once a large model inference request is successfully processed, the cloud-native API gateway will process its hash array. Insert into the trie. During insertion, for each hash value in the array, create a child node if it does not exist; at the last node of the path, record the identifier of the model service gateway that handles the large model inference request. Through the trie, any prefix path... All are available It can quickly locate and retrieve the set of all SMG nodes that have ever matched the prefix within a time complexity.

[0074] Locate the trie node corresponding to the path of length L, search among all model service gateway nodes managed within the service mesh, and select one or more model service gateway nodes that can achieve the longest prefix hash match with the hash value array. Collect all model service gateway node identifiers as candidate model service gateway nodes. .

[0075] The model service gateway node is the inference instance that has previously processed the same message prefix as this large model inference request.

[0076] Calculate prefix hit information, which includes at least the matching length L and the prefix hit rate. .

[0077] The matching length L refers to the longest consecutive matching message count.

[0078] The prefix hit rate ,in This represents the total number of messages in the current large model inference request. The prefix hit rate reflects the similarity between the current large model inference request and historical inference requests.

[0079] To ensure that the model service gateway instances in the candidate model service gateway nodes are accessible and healthy, this invention requires that all model service gateways be registered in the same service mesh. The cloud-native API gateway, as the traffic entry point of the service mesh, can obtain the metadata of each model service gateway instance through the service discovery mechanism of each service mesh.

[0080] The metadata of the model service gateway instance includes IP address and port, health status, circuit breaker status, and current load metric.

[0081] Therefore, after obtaining the candidate model service gateway nodes, the cloud-native API gateway will further filter out instances that are marked as unhealthy, circuit-broken, or unreachable by the service mesh, ensuring that the remaining candidate model service gateway nodes are all available services.

[0082] Step 2 specifically includes:

[0083] Step 201: Continuously collect dynamic load data of each pre-filled node managed by the candidate model service gateway node.

[0084] The cloud-native API gateway includes a load information collection module, which continuously acquires the real-time load information of the Prefill inference nodes under each model service gateway.

[0085] Specifically, the load information collection module proactively queries the status of the Prefill nodes managed by each model service gateway at fixed intervals or based on event-driven mechanisms, or the model service gateways report the status via heartbeats. The load of each Prefill node is primarily measured by the number of large model inference requests currently being processed, and the maximum prefill load value for each model service gateway is calculated. and pre-filled minimum load value .

[0086] The load information is cached in the memory of the cloud-native API gateway with a timestamp to ensure fast retrieval during routing decisions. If no load update is received from a certain SMG within a specified time, the load information of that model service gateway is marked as outdated, and the most recent valid data is used first or an active query is triggered during decision-making.

[0087] Step 202: Determine the final target model service gateway node according to the preset allocation conditions.

[0088] To balance KV Cache hit rate and node load balancing, preset allocation conditions are defined. The cloud-native API gateway will only perform load balancing when these conditions are met simultaneously; otherwise, the candidate model service gateway node matched by KV Cache awareness will be used as the final target model service gateway node.

[0089] The preset allocation conditions include variables: candidate model service gateway nodes output based on KV Cache awareness matching. Candidate model service gateway node Management pre-filled maximum load value Candidate model service gateway node Management pre-filled minimum load value Prefix hit rate Where L is the longest matching prefix length, n is the total number of messages in the large model inference request; and the preset cache hit threshold. ; Preset request gap threshold D; Preset request ratio threshold R.

[0090] The preset allocation conditions include prefix hit rate. Less than the preset cache hit threshold If the cache hit rate for the current request is below the set lower limit, it indicates that cache reuse is not very valuable, and load balancing should be considered in this case.

[0091] And there is at least one candidate load balancer node in the service mesh. The minimum pre-filled load value under the candidate load balancer node .

[0092] The candidate load balancer node satisfies the following sub-condition: the difference between the pre-filled maximum load value of the candidate model service gateway node and the pre-filled minimum load value of the candidate load balancer node is greater than or equal to a preset request gap threshold, i.e. Furthermore, the ratio of the pre-filled maximum load value under the candidate model service gateway node to the pre-filled minimum load value under the candidate load balancer node is greater than or equal to a preset request ratio threshold, i.e. Ensure the candidate model service gateway node The load is significantly higher than that of the candidate load balancer nodes. Light load.

[0093] The difference between the maximum pre-filled load value and the minimum pre-filled load value of the candidate load balancer node is greater than or equal to the request gap threshold. Furthermore, the ratio of the maximum pre-filled load value of the candidate load balancer node to the minimum pre-filled load value of its own subordinate nodes is greater than or equal to the request ratio threshold, i.e. Ensure the selected equilibrium node There is also a certain degree of load imbalance within the system itself, and all of the above together prevent unnecessary sacrifice of cache hit rate when the load difference is small.

[0094] When the preset allocation conditions are met, the cloud-native API gateway scans all candidate model service gateway nodes to perform load balancing processing, forming a load balancing filter set. In the balanced screening set In the process, the pre-filled minimum load values ​​of each candidate equilibrium node in the equilibrium candidate set are compared. Select the candidate load balancer node with the smallest pre-filled minimum load value as the final target model service gateway node.

[0095] If any of the above conditions are not met, either the prefix hit rate is high, or there are no other model service gateways that meet the conditions, then the candidate model service gateway node is directly selected. As the final target model service gateway node.

[0096] Through the load balancing process, the cloud-native API gateway can direct requests to the least busy model service gateway instance in scenarios with low cache hit rates, thereby alleviating hotspots and improving system throughput.

[0097] Regardless of whether the target model service gateway node is selected through KV cache awareness or load balancing, such as Figure 4As shown, the deployment architecture includes one primary replica and at least one backup replica, and the readiness status of each replica is dynamically determined through a sidecar-contention lease mechanism. Specifically, the target model service gateway node is an abstraction of a set of service replicas, including multiple replicas. Each replica internally runs a primary container and a sidecar-contention lease container.

[0098] The main container is the model service gateway business container, which is responsible for handling the routing, prefix tree management, and request forwarding of large model inference requests.

[0099] The sidecar-competitive lease container is a competing lease client responsible for competing for distributed lock leases. Upon startup, it attempts to create a lease with a fixed expiration time or update an existing lease. The distributed lock ensures that only one replica of the sidecar-competitive lease container can successfully hold the lease. The sidecar-competitive lease container holding the lease sets its corresponding sidecar-competitive lease container's ready probe to success, and its replica is marked as the primary replica. Other sidecar-competitive lease containers that fail to acquire a lease set their respective sidecar-competitive lease container's ready probe to failure, and their replicas are marked as backup replicas, not receiving service traffic.

[0100] It's important to note that when the primary / standby access mode is enabled, the primary and standby SMG services are typically not located in the same service mesh, resulting in the loss of features such as circuit breaking and failure forwarding, which impacts high availability. When supporting failure and circuit breaking forwarding features and employing multi-replica load balancing, all SMG replicas will handle business traffic, distributing the prefix tree state across replicas. Requests cannot be concentrated on the node with the richest cache, reducing KV cache reuse. Since Dynamo and SMG are both AI gateways, the shared prefix tree structure across multiple instances in high-concurrency scenarios leads to severe lock contention.

[0101] While it is possible to listen for KV Cache events through message channels, this increases system complexity, and its own high availability and latency issues can easily lead to instability. The asynchronous propagation of KV Cache events causes eventual consistency, which may cause routing decisions to fail.

[0102] While related technologies also employ a sidecar model to deploy multiple replicas across different nodes, these address request retry and forwarding issues arising from replica failures in general containers. They do not address core requirements of large-model inference scenarios such as request content awareness, load balancing, and KV cache reuse. Furthermore, these technologies are not part of the primary and backup SMG services within the same service mesh as described in this embodiment, as this would cause the aforementioned problems. This implementation integrates mechanisms such as KV cache-aware matching, condition-triggered load balancing, high availability of primary / backup replicas via sidecar-competitive leases, and failover of the backup node chain at the cloud-native API gateway layer. Specifically optimized for the performance and stability of large-model inference services, it offers higher KV cache hit rates, superior load balancing capabilities, a clearer primary / backup replica high-availability architecture, and stronger fault recovery capabilities in large-model inference scenarios.

[0103] However, this implementation supports a primary / backup access mode within the same service mesh, enabling a primary SMG to carry traffic and a backup SMG to provide hot standby access within the same service mesh on the cloud-native API gateway. It also features native high availability capabilities for reuse, while ensuring circuit breaking, health checks, and failure forwarding capabilities for multiple AI gateway services. When the primary SMG encounters a problem, it can quickly switch to the backup SMG to provide service, eliminating situations where inference service is unavailable for a short period, thus ensuring uninterrupted inference service.

[0104] In actual deployment, all model service gateways are registered within the same service mesh. This service mesh automatically injects high-availability features such as circuit breakers, active / passive health checks, connection pool management, and failure forwarding based on exception status codes into each SMG instance. When the gateway's external plugin detects a failure of all replicas within a certain SMG group and reports a group-wide failure to the upper layer, the cloud-native API gateway does not immediately return an error to the client. Instead, it utilizes the service mesh's cross-group failure forwarding capability to reroute the current large model inference request to other SMG groups within the same service mesh.

[0105] Specifically, the gateway external plugin records the failed SMG group identifier and failure reason in the context metadata of the large model inference request and returns it to the main process of the cloud-native API gateway. The cloud-native API gateway switches the target service of the request to another SMG group within the service mesh that also provides the model service. Since all SMGs share the same service port and API protocol, the gateway does not need to modify the request body, only the hostname or service discovery key of the target service. Requests rerouted to the new SMG group will be re-executed according to the KV Cache-aware scheduling process. They are forwarded again to the primary replica of the new SMG group through the gateway external plugin, and primary-backup retry and backup node chain fault tolerance are performed. Even if a certain SMG group is completely unavailable, the cloud-native API gateway can automatically switch traffic to other healthy SMG groups within the same service mesh, and the switching process is transparent to the client. At the same time, since the KV Cache prefix record is globally shared, some cache may still be hit after switching groups, thus maintaining inference performance to a certain extent.

[0106] Step 3 specifically includes:

[0107] The cloud-native API gateway has a built-in extensible plugin framework. This invention deploys an external gateway plugin within the cloud-native API gateway. This external gateway plugin receives forwarding requests from the main process of the cloud-native API gateway and access logic for the primary and backup replicas within the management model service gateway.

[0108] The interaction between the main process of the cloud-native API gateway and the external plugin of the gateway adopts a non-blocking asynchronous callback mode to avoid blocking the gateway worker thread.

[0109] The external gateway plugin can only find replicas that are in a successful ready state through service discovery, so traffic will only be sent to the current primary replica.

[0110] When the primary replica cannot renew its lease due to process crash, node failure, or network isolation, its lease will expire. When the gateway external plugin detects that the current replica returns an abnormal state, it resends the large model inference request to the standby replica in the target model service gateway node that is not ready, according to the polling rules, for fault tolerance processing. The standby replica's sidecar-competitive lease container immediately initiates competition and successfully acquires the lease, and then sets its sidecar-competitive lease container's ready probe to success, thereby automatically upgrading it to a new primary replica.

[0111] The abnormal states include at least one of the following: returning a preset error status code, request timeout, transmission control protocol connection failure, node circuit breaker failure, network health check failure, etc. When the gateway external plugin detects any of these abnormal states, it immediately interrupts the current request, records the abnormal state as a failure of the replica, and initiates retry logic. Following the pre-established backup node chain, it sequentially selects candidate model service gateway nodes. The gateway external plugin iterates through each candidate model service gateway node in the backup node chain, prioritizing the selection of candidate load balancing nodes that satisfy the load balancing strategy, and these nodes are not part of the set of nodes that have failed in this request.

[0112] When there are multiple candidate model service gateway nodes that simultaneously meet the above conditions, the node with the lowest current load is selected to perform request forwarding.

[0113] When both the primary and backup replicas within the target model service gateway node fail to retry, the gateway external plugin reports a group-wide failure status to the cloud-native API gateway. After excluding the target model service gateway nodes whose requests have failed, nodes in abnormal states are filtered and removed to obtain available backup nodes. From the filtered available backup nodes, the node with the lowest load is selected to continue forwarding the large model inference request.

[0114] The backup node chain is an ordered list containing other alternative model service gateway groups. The backup node chain can be statically defined by the administrator in the gateway configuration or dynamically generated through the service mesh's priority tags.

[0115] The entire switching process is transparent to the cloud-native API gateway. The external plugin of the gateway automatically sends the request to the new ready replica during the next forwarding, realizing high availability of the primary and backup model service gateway. There is no need to maintain a complex primary and backup status table in the API gateway, while fully reusing the service mesh's native health check and load balancing capabilities.

[0116] Step 4 specifically includes:

[0117] Read the message array from the large model inference request and obtain them one by one to form an input message queue;

[0118] Each message in the input message queue is independently calculated sequentially using a preset one-way hash algorithm to obtain a continuous chain of hash digests as the hash value sequence.

[0119] The hash value sequence is inserted as a new path into the prefix record according to the trie data structure.

[0120] After the cloud-native API gateway completes the processing of a large model inference request, it determines the target model service gateway instance that actually handled the request and obtains the model service gateway identifier. The model service gateway identifier can be the service name, namespace, or unique ID of the model service gateway. The prefix record module's insertion interface is then called, passing the hash array H and the model service gateway identifier as parameters to the prefix record module.

[0121] A read-write lock is applied to the trie to prevent data races caused by concurrent insertions or queries. H is traversed to create or find candidate model service gateway nodes. After completing the traversal of the hash value sequence, the endpoint candidate model service gateway node is located, and its unique identifier is added to the association set of the endpoint candidate model service gateway node.

[0122] To control the capacity limit of a single node's associated set, a capacity threshold can be determined proportionally based on the total number of model service gateway instances in the deployment environment. When the current number of elements in the associated set exceeds the capacity threshold, one or more records that have not been accessed for the longest time are selected and removed from the associated set; alternatively, a least recently used strategy is adopted, based on the historical access frequency and timeliness of each gateway identifier on this node, to eliminate the entry with the longest access interval.

[0123] By using the above methods, we can ensure that the size of the associated set of any node is always kept within the capacity threshold, preventing unlimited memory growth, while retaining the associated information of cloud-native API gateways that have been used recently or frequently.

[0124] An API gateway KV Cache-aware scheduling system is provided for executing the API gateway KV Cache-aware scheduling method, such as... Figure 5 As shown, it includes a request receiving and sensing module, a load balancing module, a forwarding and scheduling module, and a prefix state update module.

[0125] The request receiving and sensing module is used to receive large model inference requests sent by the client, perform KV Cache sensing matching between the large model inference request and the prefix records of historical inference information to obtain prefix hit information, and determine at least one candidate model service gateway node with prefix matching based on the prefix hit information; the at least one candidate model service gateway node is set in the same service mesh;

[0126] The load balancing module is used to obtain the load information of the inference nodes under the candidate model service gateway node. When the preset allocation conditions are not met, the corresponding candidate model service gateway node is determined as the final target model service gateway node according to the prefix hit information. When the preset allocation conditions are met, load balancing is performed to determine the candidate model service gateway node with the lowest load as the final target model service gateway node. The target model service gateway node includes one primary replica and at least one backup replica. Each replica dynamically determines one primary replica in the ready state and one or more backup replicas in the non-ready state through sidecar-competition lease.

[0127] The forwarding and scheduling module is used to forward the large model inference request to the primary replica currently in a ready state in the target model service gateway node through the gateway external plugin; when an abnormal state is captured after forwarding, the module performs a round-robin retry between the primary replica and the backup replica through the gateway external plugin; when both the primary replica and the backup replica fail to retry, the module reselects the available node with the least load according to the backup node chain to continue forwarding the large model inference request.

[0128] The prefix state update module is used to convert the content of the large model inference request into a hash value sequence and store it in the prefix record.

[0129] This invention introduces a KV Cache-Aware Matching Mechanism at the cloud-native API gateway layer. First, it filters candidate model service gateway nodes with cache reuse potential based on the prefix hit rate of the request content. Then, when the prefix hit rate falls below a set threshold and there are significant load differences between nodes, the system automatically schedules large model inference requests to the model service gateway node with the lowest load. This ensures a high cache hit rate while effectively avoiding node load imbalance, significantly improving the overall resource utilization and request processing capacity of the service cluster. Secondly, through sidecar-competitive leases, it achieves substantial improvements in high availability compatibility, synergy between cache hit rate and load balancing, system architecture simplicity, concurrency performance, and routing consistency, better meeting the comprehensive requirements of cloud-native large model inference services for high performance, high resource utilization, and high stability.

[0130] The following is an explanation with reference to specific embodiments:

[0131] Taking a cloud-native environment based on Kubernetes and Istio service mesh as an example, the cloud-native API gateway uses Higress, the model service gateway uses an SMG instance based on the SGLang framework, and the model inference service consists of independent Prefill nodes and Decode nodes. This embodiment aims to illustrate the complete implementation process of the API gateway KV Cache-Aware Scheduling Method described in this invention.

[0132] The cloud-native API gateway is deployed on a Kubernetes cluster, with the Wasm plugin framework enabled, loading the external gateway plugin of this invention. The model service gateway deploys two SMG groups, respectively... and Each group contains one primary replica and one backup replica. Each replica's readiness state is dynamically maintained through a sidecar-competitive lease container. The prefix record module runs a trie in the gateway's memory, initially empty. The load information collection module obtains the request queue count of each SMG's Prefill nodes every 5 seconds via Istio's Prometheus adapter as a load metric.

[0133] The cloud-native API gateway receives a dialogue request from the client and splits the message array into three messages, namely: , , Using the xxHash64 algorithm, calculate the hash value of each message. Concatenate the role and content with \x00 and input them to get h1=xxhash64("user\x00What is KV Cache?")=0xA1B2C3D4E5F67890; h2=xxhash64("assistant\x00KV Cache is a technology used to accelerate large model inference.")=0x1234567890ABCDEF; h3=xxhash64("user\x00What are its advantages?")=0xFEDCBA0987654321. The corresponding hash value array H=[0xA1B2C3D4E5F67890,0x1234567890ABCDEF,0xFEDCBA0987654321] is obtained.

[0134] The hash array H is fed into the trie of the prefix record module for longest prefix matching. Assume that the trie already contains the path 0xA1B2C3D4E5F67890->0x1234567890ABCDEF, and the SMG set associated with this path node is { The third hash value, 0xFEDCBA0987654321, does not exist in the child nodes of the current longest path node. Therefore, the matching length L=2, and the prefix hit rate is... .

[0135] Since only smg-a is associated with this prefix, the set of candidate model service gateway nodes is { }

[0136] Load information collection module records: There are two prefill nodes under management, with loads of [12, 8] respectively, therefore max( )=12,min( =8. The managed Prefill nodes have loads of [15, 14], therefore max( )=15,min( =14.

[0137] Preset cache hit threshold =0.3, request gap threshold D=5, request ratio threshold R=2.0.

[0138] The current prefix hit rate of 0.667 is greater than 0.3, which does not meet the first condition for triggering load balancing. Therefore, load balancing will not be triggered, and the KV cache will be used directly for matching. As the gateway node for the final target model service.

[0139] Gateway external plugin query This corresponds to the Endpoints of the Kubernetes Service smg-a-svc. Currently, the IP returned by this Service is 10.0.1.10, which is the primary replica holding the lease. The gateway external plugin constructs a forwarding request with the target address http: / / 10.0.1.10 / v1 / chat / completions and sets a timeout of 30 seconds. After sending the request, the gateway external plugin receives a normal HTTP 200 response and inference results. The gateway external plugin then returns the response to the client via the original path. This forwarding proceeded without error and no retry is required.

[0140] If we assume the primary replica fails to return a TCP connection due to a process crash, the gateway external plugin will catch the exception and record the primary replica failure. After waiting 2 seconds, it will re-query the Endpoints and find a new primary replica IP, 10.0.1.11. The large model inference request will be resent to 10.0.1.11, and a successful response will be received.

[0141] Assumption All replicas failed. The gateway external plugin reported the entire group failure to the cloud-native API gateway master. The cloud-native API gateway read the pre-configured standby node chain: [ , ].

[0142] Traverse the alternative node chain, first check Obtain the load information of smg-b and determine whether the load condition is met: current prefix hit rate. The value remains 0.667, but the conditions can be relaxed in the Fallback phase, directly requiring... The internal load difference satisfies max( )-min( ) = 15 - 14 = 1 < 5, which does not satisfy the condition; That is not enough. Therefore It is not considered an optimal load candidate.

[0143] Since there are only two nodes in the backup node chain Failure has been ruled out; according to the rollback rules, select directly. And forward large model inference requests. The gateway external plugin sends... The primary replica sends a request, successfully receives a response, and returns it to the client. The original messages array for this large model inference request is read, and a hash array H=[h1,h2,h3] is calculated using the same rules. The actual SMG identifier that processed this large model inference request is obtained. .

[0144] Lock the trie, starting from the root node, process h1, h2, and h3. Upon reaching the end node, lock that node. Add Assuming the node's If the original size is 10, then iterate through the last_access_time of each item in the set, remove the oldest item, and then add it. Release the lock.

[0145] After insertion, the trie contains three complete message paths, which are then associated with... Any subsequent requests that begin with the same three messages will be matched directly with this path and become candidates. .

[0146] In this embodiment, the cloud-native API gateway successfully implemented KV Cache-aware matching, prioritizing the routing of requests to those with high cache hit rates. Maintain caching advantage when load balancing conditions are not triggered; utilize sidecar-contention leases and polling retries via external gateway plugins for rapid failover in case of transient primary replica failure; in case of overall group failure, automatically switch to backup node chain and minimum load selection mechanism. This ensures service continuity; asynchronously updating the prefix record allows subsequent requests to continuously benefit from historical cached information.

[0147] This invention provides an API gateway KV Cache-Aware Scheduling Method and System. First, by implementing request content prefix-based awareness matching at the cloud-native API gateway layer, it ensures that requests are preferentially routed to the model service gateway with the richest cache, significantly improving KV Cache reuse and reducing inference latency. Second, it introduces load balancing triggering conditions, enabling intervention when node loads are severely uneven, scheduling requests to the least loaded node, avoiding resource hotspots caused by traditional consistent hashing schemes. Third, through a sidecar-competitive lease mechanism, it achieves a master-slave high-availability mode for the model service gateway without compromising the service mesh's native circuit breaking and health check capabilities. Combined with the gateway's external plugin's fast retry and backup node chain, it effectively handles service failures and ensures the continuity of inference services. Finally, compared to methods that synchronize KV Cache states through complex message channels, this invention avoids performance bottlenecks and consistency issues caused by state synchronization between multiple replicas through prefix recording and awareness scheduling at the cloud-native gateway layer, resulting in a simpler and more stable architecture.

[0148] The above description illustrates preferred embodiments of the present invention and helps those skilled in the art to more fully understand the technical solution of the present invention. However, these embodiments are merely illustrative and should not be construed as limiting the specific implementation of the present invention to these embodiments. For those skilled in the art, several simple deductions and modifications can be made without departing from the inventive concept, and all such modifications should be considered within the protection scope of the present invention.

Claims

1. A method for scheduling API gateway KV Cache awareness, characterized in that, Applied to cloud-native API gateways, the method includes: The system receives a large model inference request sent by the client, performs KV Cache-aware matching between the large model inference request and the prefix records of historical inference information to obtain prefix hit information, and determines at least one candidate model service gateway node with prefix matching based on the prefix hit information; the at least one candidate model service gateway node is set in the same service mesh. The load information of the inference nodes under the candidate model service gateway node is obtained. When the preset allocation conditions are not met, the corresponding candidate model service gateway node is determined as the final target model service gateway node according to the prefix hit information. When the preset allocation conditions are met, load balancing is performed to determine the candidate model service gateway node with the lowest load as the final target model service gateway node. The target model service gateway node includes one primary replica and at least one backup replica. The primary replica in the ready state and one or more backup replicas in the non-ready state are dynamically determined through sidecar-competitive lease. The large model inference request is forwarded to the primary replica currently in a ready state in the target model service gateway node through the external gateway plugin; if an abnormal state is captured after forwarding, the external gateway plugin is used to poll and retry between the primary replica and the backup replica; if both the primary replica and the backup replica fail to retry, the available node with the least load is selected again according to the backup node chain to continue forwarding the large model inference request. The content of the large model inference request is converted into a hash value sequence and stored in the prefix record.

2. The API gateway KV Cache-Aware Scheduling Method according to claim 1, characterized in that, The prefix hit information is obtained by performing KV Cache-aware matching between the large model inference request and the prefix record of historical inference information, including: The message content of the large model inference request is divided into a message array by sequence number; Calculate the digest value of each message in the message array, and convert the message array into a corresponding hash value array based on the digest value; The hash value array is fed into the prefix record for matching, and the matching length and prefix hit rate of a successful match are obtained. The matching length and the prefix hit rate are then used as the prefix hit information.

3. The API gateway KV Cache-Aware Scheduling Method according to claim 2, characterized in that, Based on the prefix hit information, at least one candidate model service gateway node with prefix matching is determined, including: Based on the matching length, a search is conducted among all model service gateway nodes managed within the service mesh to select one or more model service gateway nodes that can achieve the longest prefix hash match with the hash value array, thus forming the candidate model service gateway nodes.

4. The API gateway KV Cache-Aware Scheduling Method according to claim 1, characterized in that, Obtain the load information of the inference nodes under the candidate model service gateway node, including: Continuously collect dynamic load data for each pre-filled node managed by the candidate model service gateway node; The load information includes the pre-filled maximum load value and the pre-filled minimum load value for each candidate model service gateway node.

5. The API gateway KV Cache-Aware Scheduling Method according to claim 1, characterized in that, The preset allocation conditions include a prefix hit rate less than a preset cache hit threshold; and, There is at least one candidate load balancer node in the service mesh. The candidate load balancer node satisfies the following sub-conditions: the difference between the pre-filled maximum load value under the candidate model service gateway node and the pre-filled minimum load value under the candidate load balancer node is greater than or equal to a preset request gap threshold; and the ratio of the pre-filled maximum load value under the candidate model service gateway node to the pre-filled minimum load value under the candidate load balancer node is greater than or equal to a preset request ratio threshold. The difference between the maximum pre-filled load value of the candidate load balancer node and the minimum pre-filled load value of its own subordinates is greater than or equal to the request gap threshold. Furthermore, the ratio of the maximum pre-filled load value of the candidate load balancer to the minimum pre-filled load value of its own subordinates is greater than or equal to the request ratio threshold.

6. The API gateway KV Cache-Aware Scheduling Method according to claim 5, characterized in that, When preset allocation conditions are met, load balancing is performed. The step of determining the candidate model service gateway node with the lowest load as the final target model service gateway node includes: The candidate equilibrium nodes are used to form an equilibrium candidate set; By comparing the pre-filled minimum load values ​​of each candidate balanced node in the balanced candidate set, the candidate balanced node with the smallest pre-filled minimum load value is selected as the final target model service gateway node.

7. The API gateway KV Cache-Aware Scheduling Method according to claim 1, characterized in that, The method of dynamically determining one primary replica in a ready state and one or more backup replicas in a non-ready state through sidecar-competitive leases includes: Inject a sidecar-competitive lease container into each initial replica of the target model service gateway node. The sidecar-competitive lease container competes for distributed locked leases after startup. The copy of the sidecar container that wins the competition and holds a fixed lease is marked as ready and becomes the primary copy. The copy of the sidecar container that has not obtained a lease is marked as not ready and is used as a backup copy. When the primary replica in the ready state fails, it actively releases the lease and switches to the not ready state. The sidecars in the other backup replicas in the not ready state then compete for the lease and switch to the ready state.

8. The API gateway KV Cache-Aware Scheduling Method according to claim 1, characterized in that, The abnormal status includes at least one of the following: returning a preset error status code, request timeout, transmission control protocol connection failure, node circuit breaker failure, and network health check failure. When the external gateway plugin detects that the currently sent replica returns the abnormal state, it resends the large model inference request to the standby replica in the target model service gateway node that is not ready, according to the polling rule, for fault tolerance processing.

9. The API gateway KV Cache-Aware Scheduling Method according to claim 8, characterized in that, If both the primary and backup replicas fail to retry, the large model inference request will be forwarded again by selecting the available node with the least load according to the backup node chain, including: When both the primary replica and the backup replica within the target model service gateway node fail to retry, the gateway external plugin reports a group failure status to the cloud-native API gateway. The cloud-native API gateway activates the backup node chain that was pre-established for the service gateway node of the target model, and after excluding the node set that has failed in this request, filters and removes nodes in abnormal state to obtain available substitute nodes. From the selected available backup nodes, the node with the lowest load is chosen to continue forwarding the large model inference request.

10. The API gateway KV Cache-Aware Scheduling Method according to claim 1, characterized in that, The content of the large model inference request is converted into a hash value sequence and stored in the prefix record, including: Read the message set field in the large model inference request and obtain them one by one to form an input message queue; Each message in the input message queue is independently calculated sequentially using a preset one-way hash algorithm to obtain a continuous chain of hash digests as the hash value sequence. The hash value sequence is inserted as a new path into the prefix record according to the trie data structure.

11. An API gateway KV Cache-aware scheduling system, used to execute the API gateway KV Cache-aware scheduling method, characterized in that, It includes a request receiving and sensing module, a load balancing module, a forwarding and scheduling module, and a prefix state update module. The request receiving and sensing module is used to receive large model inference requests sent by the client, perform KV Cache sensing matching between the large model inference request and the prefix records of historical inference information to obtain prefix hit information, and determine at least one candidate model service gateway node with prefix matching based on the prefix hit information; the at least one candidate model service gateway node is set in the same service mesh; The load balancing module is used to obtain the load information of the inference nodes under the candidate model service gateway node. When the preset allocation conditions are not met, the corresponding candidate model service gateway node is determined as the final target model service gateway node according to the prefix hit information. When the preset allocation conditions are met, load balancing is performed to determine the candidate model service gateway node with the lowest load as the final target model service gateway node. The target model service gateway node includes one primary replica and at least one backup replica. Each replica dynamically determines one primary replica in the ready state and one or more backup replicas in the non-ready state through sidecar-competition lease. The forwarding and scheduling module is used to forward the large model inference request to the primary replica currently in a ready state in the target model service gateway node through the gateway external plugin; when an abnormal state is captured after forwarding, the module performs a round-robin retry between the primary replica and the backup replica through the gateway external plugin; when both the primary replica and the backup replica fail to retry, the module reselects the available node with the least load according to the backup node chain to continue forwarding the large model inference request. The prefix state update module is used to convert the content of the large model inference request into a hash value sequence and store it in the prefix record.

Images