A reasoning service request scheduling system, method, and electronic device

Abstract

This application discloses an inference service request scheduling system, method, and electronic device, relating to the field of artificial intelligence technology. Because the routing module can score and preferentially schedule the decoding service instance cluster, replacing the traditional round-robin mechanism, it considers the load and inference resources of each instance during request distribution. Simultaneously, due to the addition of a prefix coverage condition judgment mechanism, if it is determined that the current key-value cache of the target decoding service instance with the highest first scheduling score is sufficient to complete the original inference request, the original inference request is directly sent to the target decoding service instance. This not only improves the cache reuse rate of the decoding service instance but also reduces unnecessary redundant calculations and data transmission, thereby improving the response efficiency of inference requests and reducing the response latency of inference requests.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an inference service request scheduling system, method and electronic device. Background Technology

[0002] Current inference models typically divide the inference process into two independent stages: pre-filling and decoding, using a separate deployment architecture. This architecture includes multiple pre-filling service instances and multiple decoding service instances. When a user's inference request is received, a round-robin scheduling method is used to select a target pre-filling service instance from among the multiple pre-filling service instances. After the target pre-filling service instance completes the initial inference and generates a key-value cache, a target decoding service instance is selected from among the multiple decoding service instances, and the key-value cache data is transferred to the target decoding service instance. Finally, the target decoding service instance completes all inference. However, the round-robin scheduling mechanism can lead to uneven instance load, reducing the utilization of inference resources and increasing the response latency of inference requests. Summary of the Invention

[0003] This application provides an inference service request scheduling system, method, and electronic device to at least solve the problem of increased response latency for inference requests in related technologies.

[0004] This application provides an inference service request scheduling system, including: a routing module, a gateway, and an inference pool, wherein the inference pool includes a pre-populated service instance cluster and a decoding service instance cluster; The gateway is used to receive the original inference requests sent by the user and forward them to the routing module. The routing module is used to determine the first scheduling score of each decoding service instance based on the first scoring information of the decoding service instance cluster when the original inference request is received. The decoding service instance with the highest first scheduling score is selected as the target decoding service instance. If the original inference request meets the preset prefix coverage condition, the first scheduling address of the target decoding service instance is fed back to the gateway. The gateway is used to send the original inference request to the target decoding service instance based on the first scheduling address; The target decoding service instance is used to perform inference calculations in response to the original inference request and obtain the request response result if it is determined that the original inference request does not require processing by the pre-populated service instance.

[0005] This application also provides a method for scheduling inference service requests, including: Obtain the original inference request issued by the user; Based on the first scoring information of the decoding service instance cluster, determine the first scheduling score of each decoding service instance; The decoding service instance with the highest first scheduling score is selected as the target decoding service instance. If the original inference request meets the preset prefix coverage condition, the first scheduling address of the target decoding service instance is fed back to the gateway, so that the gateway sends the original inference request to the target decoding service instance according to the first scheduling address; Based on the target decoding service instance, if it is determined that the original inference request does not require pre-filled service instance processing, inference calculation is performed in response to the original inference request to obtain the request response result.

[0006] This application also provides a reasoning service request scheduling apparatus, including: The acquisition module is used to acquire the original inference request issued by the user; The determination module is used to determine the first scheduling score of each decoding service instance based on the first scoring information of the decoding service instance cluster. The selection module is used to select the decoding service instance with the highest first scheduling score as the target decoding service instance; The feedback module is used to feed back the first scheduling address of the target decoding service instance to the gateway when the original inference request meets the preset prefix coverage condition, so that the gateway sends the original inference request to the target decoding service instance according to the first scheduling address. The response module is used to perform inference calculations in response to the original inference request, based on the target decoding service instance, if it is determined that the original inference request does not require pre-filled service instance processing, and obtain the request response result.

[0007] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for implementing the steps of any of the above-described inference service request scheduling methods when executing the computer program.

[0008] This application also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the above-described inference service request scheduling methods.

[0009] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described inference service request scheduling methods.

[0010] Through this application, the routing module can score and preferentially schedule the decoding service instance cluster, replacing the traditional round-robin mechanism, thus considering the load and inference resources of each instance when distributing requests. At the same time, due to the addition of a prefix coverage condition judgment mechanism, if it is determined that the current key-value cache of the target decoding service instance with the highest first scheduling score is sufficient to complete the original inference request, the original inference request is directly sent to the target decoding service instance. This not only improves the cache reuse rate of the decoding service instance, but also reduces unnecessary duplicate calculations and data transmission, improves the response efficiency of inference requests, and reduces the response latency of inference requests. Attached Figure Description

[0011] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 A schematic diagram of the interaction flow of the inference service request scheduling system provided in the embodiments of this application; Figure 2 A schematic diagram of the structure of the inference service request scheduling system provided in the embodiments of this application; Figure 3 This is a schematic diagram of the request-response process provided in an embodiment of this application; Figure 4 A flowchart illustrating the preparation stage provided for embodiments of this application; Figure 5 This application provides a schematic diagram of the execution flow of routing services in an embodiment. Figure 6 This is a schematic diagram of the target pre-filled service instance invocation process provided in the embodiments of this application; Figure 7 A flowchart illustrating the inference service request scheduling method provided in this application embodiment; Figure 8 A schematic diagram of the structure of the inference service request scheduling device provided in the embodiments of this application; Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0013] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0014] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0015] Large Language Models (LLMs) have made significant breakthroughs in the field of natural language processing in recent years, demonstrating strong generalization capabilities and practical effects in many tasks such as text generation, language understanding, code completion, question answering systems, and translation.

[0016] In the LLM inference process, a key-value (KV) caching mechanism is used to improve efficiency. This mechanism helps the model avoid recalculating previous lexical information during generation, thereby accelerating the inference process. The implementation of this mechanism is briefly described below: In the self-attention mechanism of LLM, each input term generates three vectors: query, key, and value. In standard inference, these three vectors need to be recalculated for each input term. With KV caching, the model caches the already computed keys and values. This allows the model to directly use these cached keys and values ​​when generating the next term, without needing to recalculate all the inputs. This caching mechanism is particularly efficient for generating long texts, significantly reducing computational cost.

[0017] In the inference process of large models, the inference process can generally be divided into two stages: the Prefill stage processes all prompts, generates the first word, and generates a KVCache. The Decode stage uses the KVCache generated in the Prefill stage for multiple rounds of iteration, generating one word in each round. Prefix cache matching refers to finding the longest prefix in the existing prefix cache that is completely identical to the beginning of the prompt of the request at the word level when a new request comes in, and reusing the KVCache corresponding to this prefix. Typically, the KVCache is stored in blocks, with each KV block containing several words of KVCache.

[0018] Currently, large-scale service deployments are starting to develop a separate deployment approach for the P (Prefill) and D (Decode) phases. That is, Prefill and Decode run on different Pods or different nodes, which can optimize resource usage in each phase independently, but places higher demands on cache transmission and scheduling strategies.

[0019] In the cloud-native Kubernetes architecture, a Deployment is a Kubernetes object used for declarative management of application replicas, responsible for lifecycle management such as service creation, upgrades, and rolling updates. A Deployment can manage multiple Pod instances, each of which can run an inference service. Each inference service is started by an inference engine, which typically provides metrics and monitoring items for the inference service, reflecting the service's status and resource utilization in real time.

[0020] For services with separate P and D components, the current inference engine provides a KV cache transfer method. By configuring KV transfer parameters between the P and D services, the KV cache generated in P can be transferred to D, thereby completing the coordination process of inference.

[0021] The current inference engine has designed a message publishing mechanism for the KV block caching mechanism. By publishing KVEvents, messages such as block storage and removal are published. It relies on ZMQ PUB to publish messages, and the receiver subscribes in the ZMQSUB way. Currently, service instances can be managed using Deployments within a cluster. For P / D separation type services, since these services have instances with two roles, two Deployments can be used to manage multiple instances of P and D respectively. These multiple instances form a service. When a user sends a request, the gateway layer selects a P type instance according to a round-robin routing strategy and sends the request to that instance. This instance performs inference and generates the first word. After the P type instance finishes execution, the routing layer selects a D type instance and then passes the P type instance's IP, port, request ID, and other KV cache transmission parameters to the D type instance. After receiving the parameters, the D type instance retrieves the information needed to transmit the KV cache from the P instance, transmits the KV cache to the D instance, begins decoding, and returns the decoding result to the user.

[0022] Currently, P / D separation type services generally adopt a round-robin strategy at the routing layer when scheduling user requests, that is, selecting instances to execute requests sequentially from multiple P instances and D instance replicas in a round-robin manner.

[0023] Currently, P / D separation services use a round-robin scheduling approach, which doesn't consider the individual instance conditions. This can easily lead to high request latency, decreased throughput, and load imbalance. When there are multiple P and D instances, without considering instance load, high load often results in some Pods being idle while others are overloaded, causing some requests to wait. Furthermore, without considering the distribution of key-value caches, requests are randomly scattered, disrupting prefix cache reuse. Predictably hit prefix caches become invalid, leading to repeated prefill executions. Alternatively, if the prefix is ​​too short, the transmission latency caused by the P / D separation strategy is often less beneficial than performing prefill directly in the decode phase. These are typical request scheduling problems that need to be considered when using P / D separation.

[0024] To address the aforementioned technical problems, this application provides an inference service request scheduling system, method, and electronic device. Because the routing module can score and prioritize the scheduling of the decoding service instance cluster, replacing the traditional round-robin mechanism, it considers the load and inference resources of each instance during request distribution. Simultaneously, by adding a prefix coverage condition judgment mechanism, if it is determined that the current key-value cache of the target decoding service instance with the highest first scheduling score is sufficient to complete the original inference request, the original inference request is directly sent to the target decoding service instance. This not only improves the cache reuse rate of the decoding service instance but also reduces unnecessary redundant calculations and data transmission, thereby improving the response efficiency of inference requests and reducing the response latency of inference requests.

[0025] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0026] This application provides an inference service request scheduling system for scheduling pre-filled service instances and decoding service instances to respond to user inference requests.

[0027] like Figure 1 The diagram shown is an interactive flow diagram of the inference service request scheduling system provided in this application embodiment. The system includes: a routing module, a gateway, and an inference pool. The inference pool includes a pre-populated service instance cluster and a decoding service instance cluster.

[0028] The gateway receives the original inference request from the user and forwards it to the routing module. Upon receiving the original inference request, the routing module determines the first scheduling score of each decoding service instance based on the first scoring information of the decoding service instance cluster. It then selects the decoding service instance with the highest first scheduling score as the target decoding service instance. If the original inference request meets a preset prefix coverage condition, the routing module sends the first scheduling address of the target decoding service instance back to the gateway. The gateway then sends the original inference request to the target decoding service instance based on the first scheduling address. If it is determined that the original inference request does not require pre-filled service instance processing, the target decoding service instance responds to the original inference request by performing inference calculations and obtaining the request response result.

[0029] Specifically, the gateway is the network access component of this system. It receives inference requests initiated by users and forwards the requests to the routing module according to preset routing rules. When the gateway receives the first scheduling address of the target decoding service instance from the routing module, it forwards the original inference request to that target decoding service instance through an address mapping mechanism.

[0030] Specifically, the routing module obtains the first scoring information of the decoding service instance cluster. This information includes at least data such as KV cache distribution (current inference key-value cache), instance load metrics, and cache utilization. Using various preset scoring strategies, it calculates the score for each decoding service instance, obtaining the first scheduling score for each decoding service instance in the cluster. The decoding service instance with the highest first scheduling score is selected as the target decoding service instance. Simultaneously, it calculates the prefix cache hit rate of the target decoding service instance. The prefix cache hit rate refers to the proportion of blocks in the request prompt block that continuously match the instance's existing KV cache to the total number of blocks. If it meets the preset prefix coverage condition, the first scheduling address of the target decoding service instance is fed back to the gateway.

[0031] Furthermore, after the target decoding service instance receives the original inference request sent by the gateway, it confirms that there is no need for a prefill service instance to process it, meaning that its own KV cache (prefix cache) can already cover the inference dependency data required by the request. It then directly calls its own prefix cache to perform inference calculations and generate the request response result.

[0032] There is a one-to-one correspondence between the routing module and the inference pool. Each inference pool is configured with one instance of the routing module, and each routing module instance serves one inference pool. Each routing module instance can be configured with different routing policies. The routing module implements Envoy's ext-proc. Before Envoy forwards traffic, it first finds the corresponding routing instance by the inference pool name, and then uses gRPC to call the routing instance. The routing instance instructs Envoy to route the request to which specific Pod (inference service instance).

[0033] The inference service request scheduling system provided in this application has the advantage that the routing module can score and prioritize the decoding service instance cluster, replacing the traditional round-robin mechanism. This allows the system to consider the load and inference resources of each instance when distributing requests. Furthermore, by adding a prefix coverage condition judgment mechanism, if the current key-value cache of the target decoding service instance with the highest first scheduling score is sufficient to complete the original inference request, the original inference request is directly sent to the target decoding service instance. This not only improves the cache reuse rate of the decoding service instance but also reduces unnecessary redundant calculations and data transmission, thereby improving the response efficiency and reducing the response latency of the inference request.

[0034] Based on the above embodiments, as an implementable approach, in one embodiment, the routing module is further configured to: If the original inference request does not meet the preset prefix coverage condition, the second scheduling score of each pre-filled service instance is determined according to the second scoring information of the pre-filled service instance cluster, and the pre-filled service instance with the highest second scheduling score is selected as the target pre-filled service instance; the request header information of the original inference request is modified according to the second scheduling address of the target pre-filled service instance to obtain the target inference request; the first scheduling address of the target decoding service instance and the target inference request are fed back to the gateway so that the gateway sends the target inference request to the target decoding service instance according to the first scheduling address.

[0035] It should be noted that the pre-filling service instance refers to the inference service instance responsible for processing the generation of the first word unit of the prompt word and the KV cache; the decoding service instance refers to the inference service instance that uses the KV cache to perform multiple rounds of iterative decoding to generate the results. The first scheduling score and the second scheduling score can use the same scoring mechanism.

[0036] Furthermore, in one embodiment, as Figure 2 The diagram shown is a structural schematic of the inference service request scheduling system provided in this application embodiment. The decoding service instance in the decoding service instance cluster includes a request forwarding module and a decoding inference module, which is also called an inference service.

[0037] The request forwarding module intercepts the original inference request or the target inference request sent by the gateway. If it is determined that the original inference request does not require processing by the pre-populated service instance, it forwards the original inference request to the decoding inference module. If it is determined that the target inference request requires processing by the target pre-populated service instance, it forwards the target inference request to the target pre-populated service instance according to the second scheduling address represented by the request header information. The decoding inference module is used to respond to the original inference request, perform inference calculations, and obtain the request response result.

[0038] Specifically, the request forwarding module, also known as the sidecar module, captures the original inference request or target inference request sent by the gateway through the registered interception interface. It performs processing requirement judgment on the original inference request. When it is confirmed that no pre-filling service instance is required, the original inference request is directly forwarded to the decoding inference module within the same decoding service instance. This avoids invalid pre-filling process calls, reduces request flow time, and improves inference response efficiency.

[0039] Accordingly, when the request forwarding module confirms that the target inference request needs to be processed by the target pre-filled service instance, it extracts the second scheduling address from the request header information and forwards the target inference request to the target pre-filled service instance based on the second scheduling address. This enables flexible scheduling of the target pre-filled service instance and the decoding service instance. By initiating the pre-filled inference process when the target inference request is received, it provides a data foundation for subsequent KV cache migration and decoding inference, ensuring the continuity and accuracy of the inference process under the pre-filled and decoding separation architecture, and also ensuring that the target pre-filled service instance can accurately receive the target inference request.

[0040] Specifically, in one embodiment, the request forwarding module can check whether there is a preset convention key value in the request header information of the original inference request or the target inference request to determine the request type; wherein, if there is no preset convention key value, the request type is determined to be the original inference request; if there is a preset convention key value, the request type is determined to be the target inference request.

[0041] Specifically, the request forwarding module analyzes whether the request header information of the inference request has a preset convention key value (second scheduling address, such as prefill-url) to determine whether it is the original inference request or the target inference request. The original inference request is forwarded to the decoding inference module, and the target inference request is forwarded to the corresponding target prefill service instance.

[0042] Specifically, in one embodiment, the request forwarding module can search for the target network connection corresponding to the second scheduling address from the preset instance connection pool; based on the target network connection, the target inference request is forwarded to the target pre-populated service instance.

[0043] The preset instance connection pool is used to cache network connections between the decoding service instance and multiple pre-filled service instances.

[0044] Specifically, after obtaining the second scheduling address, the request forwarding module retrieves the preset instance connection pool and selects the network connection corresponding to the second scheduling address as the target network connection through address matching. This reduces the resource consumption of connection establishment operations and shortens the preparation time for request forwarding. After confirming that the target network connection is valid, the request forwarding module sends the target inference request to the target pre-populated service instance through the target network connection, ensuring stable transmission of request data within the established network connection.

[0045] Specifically, by using a pre-defined instance connection pool for network connection caching, network connection reuse and efficient utilization are achieved, reducing the time and resource overhead of repeatedly establishing network connections and improving request forwarding efficiency and transmission stability.

[0046] Accordingly, if there is no target network connection corresponding to the second scheduling address in the preset instance connection pool, a new connection is created and added to the connection pool.

[0047] Specifically, in one embodiment, such as Figure 3 The diagram shown is a request-response flow diagram provided in an embodiment of this application. The request forwarding module is also used to determine whether the second scheduling address represented by the request header information belongs to the preset security address set; if it is determined that the second scheduling address does not belong to the preset security address set, the module sends a request-response error message to the gateway.

[0048] Specifically, after extracting the second scheduling address from the request header information, the request forwarding module retrieves a preset set of secure addresses. This set consists of the addresses of legitimate pre-filled service instances within the inference pool, and is dynamically refreshed and maintained by periodically scanning instances with pre-filled role tags in the inference pool. Based on an address whitelist verification mechanism, access permissions are determined by comparing whether the second scheduling address is within the legitimate set. Illegal pre-filled service instance addresses are filtered to ensure the security of the inference service.

[0049] Accordingly, in one embodiment, the request forwarding module may, upon determining that the second scheduling address belongs to a preset set of secure addresses, extract original parameter information from the target inference request; construct pre-filled request parameters based on the original parameter information; wherein the pre-filled request parameters include at least a unique request identifier, a maximum number of generated lexical units, and key-value cache transmission parameters for the pre-filling stage; generate a target pre-filled request corresponding to the target inference request based on the pre-filled request parameters; and forward the target pre-filled request to the target pre-filled service instance.

[0050] The original parameter information includes the basic configuration information required for the inference task. It is the unmodified configuration data carried by the user when initiating the inference request, such as prompt words, generation length control parameters, and output format information.

[0051] Specifically, after confirming that the second scheduling address belongs to the preset security address set, the request forwarding module parses the dictionary-formatted original parameter information from the target inference request, generates pre-filled request parameters based on the extracted original parameter information, and adds a unique request identifier (UUID, Universally Unique Identifier) ​​to it. This identifier is generated by the universally unique identifier generation function uuid() and is used to uniquely mark the KV cache corresponding to the current pre-filled request. The maximum number of generated tokens (max_tokens) is set to 1 to ensure that the pre-filled service instance only generates the first token. The KV (key-value) cache transmission parameters in the pre-filling stage are configured, and then the corresponding target pre-filled request is generated. This clarifies that the pre-filled service instance only performs pre-filled inference and allows KV cache to be transmitted across instances, so as to ensure that the pre-filled service instance completes the first token generation and KV cache storage as expected.

[0052] Specifically, by maintaining a set of secure addresses for Prefill instances and combining a connection pool with an LRU eviction mechanism, this invention reduces the connection establishment overhead of P and D while preventing SSRF risks and ensuring connection security.

[0053] Specifically, the request forwarding module (sidecar module) is responsible for intercepting user requests from the routing module, analyzing whether the request needs to perform the Prefill step, and if so, forwarding the request to the Prefill instance. After the Prefill instance has completed the inference of the first token, it coordinates the Decode instance to migrate the KV cache from the corresponding Prefill instance to the Decode instance. Finally, the subsequent inference steps are executed in the Decode instance.

[0054] Specifically, this application embodiment achieves instance-level autonomy of P / D coordination by decentralizing the request split between Prefill and Decode, KV Cache migration, and parameter coordination logic to the sidecar module of the Decode instance. This avoids the complexity and coupling problems caused by centralized management of KV transmission by the routing layer, thereby improving system maintainability and scalability.

[0055] like Figure 4 The diagram shown illustrates the preparation phase of this application's embodiment. During the preparation phase, the gateway is deployed sequentially, the inference pool is deployed, and the reverse proxy service of the sidecar module and the routing service of the routing module are started. The sidecar module contains a reverse proxy service. When creating a Decode instance, the parameters of the reverse proxy service can be configured, including the reverse proxy service port, inference service port, and inference pool name. After the instance is created, the reverse proxy service is started. The start-up steps are as follows: 1. Obtain the reverse proxy service port and start an HTTP service on that port; 2. Register the health check interface with the reverse proxy service for health checks of the reverse proxy service; Kubernetes can set up health checks on Pods, and through this interface, Kubernetes can take over the maintenance of the reverse proxy service. 3. Register the interception interface: Register the interfaces that need to be intercepted in the inference service to the reverse proxy service, such as the commonly used chat interfaces / v1 / chat / completions and / v1 / completions; 4. Initialize the P instance connection pool: Creating a new TCP connection every time a Decode or Prefill instance connects is very time-consuming. Therefore, a connection pool is maintained to cache already established connections. When a specific Prefill connection is needed, the corresponding connection can be retrieved from the pool and used directly. Furthermore, because P and D type instances are usually configured for automatic scaling in production environments, the number of Prefill instances is dynamic and uncontrollable. To prevent the connection pool from growing indefinitely, a capped, automatically eviction caching mechanism is required. Here, a cap can be set on the size of the connection pool; connections exceeding the cap are evicted according to the LRU (Least Recently Used) mechanism. Initially, the connection pool is an empty set.

[0056] 5. Obtaining a Set of Secure Addresses: When processing a request, the reverse proxy service retrieves an address from the request header according to a predefined key-value pair (e.g., a key-value pair in the request header might be `prefill-url:100.10.10.1`, indicating that the request wants to use the P instance at address 100.10.10.1 for the Prefill inference phase). To prevent SSRF (Server-Side Request Forgery), this address cannot be unconditionally trusted; instead, it must be checked whether it is a valid Prefill instance address in the inference pool. Therefore, a set of secure addresses can be maintained. Specifically, the following steps are executed every S seconds: First, the scenario of the Decode instance where this sidecar module is located is read. Then, all Pods in that scenario are read, and Pods tagged with the inference pool name and those tagged with the role of Prefill are retrieved. Finally, the addresses of these Pods and the inference service port passed to the reverse proxy service form the address set. This set is refreshed every preset second.

[0057] After the reverse proxy service starts, it will listen for user requests from the routing module. Whenever a user request arrives, the reverse proxy service will perform the following steps: 1. When the reverse proxy service detects a request entering the interface that needs to be intercepted (e.g., / v1 / chat / completions), it checks whether there is a pre-defined key-value pair (e.g., prefill-url) in the request header. If so, it retrieves the address of the P instance from the request header according to the pre-defined key-value pair. If not, Set to empty; 2. If If the value is empty, then the request is directly forwarded to the inference service of the D instance where the current reverse proxy service resides, without proceeding with the following steps; if... If it is not empty, then perform the following checks first. Check if the address is in the safe address set. If not, return the response to the request and set the HTTP status code to 403. If it is, continue to the next step. 3. Read the raw parameters carried in the request. The original parameters are in dictionary format; 4. Define the parameter passed to the P instance as follows It is copied from the original parameters. Modify. To add a request ID, a new unique identifier (UUID) is generated using the `uuid()` function and used as the request ID. The reason for needing a request ID is that the user request will be split into two parts, P and D, resulting in two separate HTTP requests. After the P instance completes its request, the request ID is used to identify which KV cache segment belongs to which request when storing the data in the KV cache. The D instance needs to migrate its KV cache, and the request ID is required to index the KV cache during the migration process.

[0058] Revise The max_tokens of the P instance only need to infer the first word, and the streaming output stream is turned off. The streaming protocol is not required in the Prefill stage. In the KV cache transfer parameter kv_transfer_params, the parameters that need to be remotely decoded are turned on. That is, the P instance only needs to do Prefill. There are other parameters in kv_transfer_params. The method in this article does not involve modifying these parameters, so they will not be mentioned here.

[0059] 5. Retrieve from the Prefill instance connection pool If a corresponding connection is not available in the connection pool, a new connection is created and added to the connection pool. After obtaining a connection, ... The request is sent to the P instance as a request parameter. After processing the request, the P instance returns a response to the reverse proxy service.

[0060] 6. After receiving the response, the reverse proxy service parses the kv_transfer_params from the response and adds it to the original parameters. In, and add the request ID generated in step 4 to In the middle, finally As a request parameter, the request is sent to the local inference service of the D instance where the reverse proxy service is located.

[0061] 7. After receiving the request, the local inference service migrates the KV cache from the corresponding P instance to the local machine according to the parameters in kv_transfer_params and the request ID. Then, it starts the Decode process and returns the decoded tokens to the user in the response.

[0062] Accordingly, in one embodiment, the target pre-filling service instance is used to receive a target pre-filling request; in response to the target pre-filling request, perform pre-filling inference to generate target lexicals and corresponding key-value caches according to the maximum number of generated lexicals; associate the key-value caches with the unique identifier of the request and store them; and return a target pre-filling response to the request forwarding module, the target pre-filling response containing at least the key-value cache transmission parameters of the key-value cache.

[0063] Among them, the key-value cache transmission parameters include at least the key-value cache transmission method and address configuration information.

[0064] Specifically, the target pre-filling service instance responds to the target pre-filling request, invokes the self-attention mechanism of the large language model, processes the prompt in the request, and generates the first token (lexical) according to the preset maximum number of generated tokens (max_tokens=1). Simultaneously, during inference, a corresponding KV cache is generated. This cache contains the key and value vector data calculated during inference to generate the first token and KV cache required for the decoding stage. The target pre-filling service instance binds and stores the generated KV cache with a unique request identifier, ensuring that subsequent decoding service instances can accurately locate the KV cache corresponding to the current task using this unique identifier. This avoids KV cache confusion between different requests and ensures the accuracy and uniqueness of the KV cache migration process.

[0065] Accordingly, in one embodiment, the request forwarding module is further configured to receive the target pre-filled response returned by the target pre-filled service instance; parse the key-value cache transmission parameters from the target pre-filled response; add the unique request identifier and the parsed key-value cache transmission parameters to the original parameter information to obtain the updated target inference request parameters; and send the request containing the target inference request parameters to the decoding inference module, so that the decoding inference module can obtain the key-value cache from the target pre-filled service instance according to the target inference request parameters, perform inference calculations according to the key-value cache, and obtain the request response result.

[0066] Specifically, the request forwarding module parses the target pre-filled response, extracts the key-value cache transmission parameters that characterize the KV cache transmission rules, and adds the previously generated unique request identifier and the parsed key-value cache transmission parameters to the original parameter information extracted from the target inference request, integrating them to form the updated target inference request parameters. The request containing the updated target inference request parameters is then sent to the decoding inference module within the same decoding service instance, triggering the decoding inference process. This process involves retrieving the key-value cache from the target pre-filled service instance, performing inference calculations based on the key-value cache, and obtaining the request response result.

[0067] Based on the above embodiments, as one implementable approach, in one embodiment, the routing module is specifically used for: Based on the current inference key-value cache of the target decoding service instance and the hint word information of the original inference request, calculate the prefix cache hit rate of the target decoding service instance for the original inference request; based on the prefix cache hit rate and the number of hint word blocks in the original inference request, determine whether the original inference request meets the preset prefix coverage condition.

[0068] Specifically, the routing module uses a word segmentation tool to convert the prompt words into a list of lexical identifiers (lexical IDs). This list is then divided into multiple consecutive lexical blocks according to a preset block size. The request hash value of each lexical block is calculated to form a hash chain. Based on the hash chain, the module sequentially matches the cached lexical blocks in the current inference key-value cache of the target decoding service instance. The number of consecutively matched lexical blocks is counted, and the ratio of this number to the total number of prompt word blocks is used as the prefix cache hit rate. Then, the product of (1 - prefix cache hit rate) and the number of prompt word blocks is calculated. This product is compared with a preset threshold. If the product is less than the preset threshold, the original inference request is determined to meet the preset prefix coverage condition, i.e., whether the KV cache of the target decoding service instance is sufficient to cover the current request prompt word inference requirements. Otherwise, the preset prefix coverage condition is not met.

[0069] Based on the above embodiments, as one implementable approach, in one embodiment, the routing module is specifically used for: Based on multiple preset scoring strategies and the first scoring information of the decoding service instance cluster, the corresponding multiple scoring results are determined; based on the multiple scoring results, the first scheduling score of each decoding service instance is determined.

[0070] It should be noted that multiple preset scoring strategies are used to score each decoding service instance from different perspectives. Finally, by weighting, fusing and normalizing the multiple scoring results, the first scheduling score of each decoding service instance is obtained.

[0071] Specifically, in one embodiment, when the preset scoring strategy includes a prefix cache matching strategy, the routing module can generate a lexical sequence based on the prompt words in the original inference request; divide the lexical sequence into multiple consecutive lexical blocks; calculate the request hash value of each lexical block to obtain a request hash chain; for any decoding service instance, determine the continuous matching length between the cached lexical blocks in the decoding service instance and the request hash chain based on the current inference key-value cache of the decoding service instance; wherein, the current inference key-value cache includes multiple cached lexical blocks, and the first scoring information includes the current inference key-value cache of the decoding service instance; and determine the prefix cache matching scoring result of the decoding service instance based on the continuous matching length.

[0072] Among them, there are multiple scoring results, including prefix cache matching scoring results.

[0073] Specifically, the routing module uses a word segmentation tool to convert the prompt words into a list of lexical identifiers (lexical IDs). This list is then divided into multiple consecutive lexical blocks according to a preset block size. The request hash value of each lexical block is calculated, forming a hash chain. Similar to the prefix cache hit rate determination process, the prefix cache matching score of the decoding service instance is determined by determining the consecutive matching length between the cached lexical blocks in the decoding service instance and the request hash chain. This prefix cache matching score can also be directly taken as the prefix cache hit rate.

[0074] Specifically, in one embodiment, the routing module can specifically target the i-th word block in the word block sequence; where i is an integer greater than or equal to 0; if i equals 0 and a parent hash value exists, then the request hash value of the i-th word block is determined based on the parent hash value, the word identifier sequence of the i-th word block, and the model name; if i equals 0 and no parent hash value exists, then the request hash value of the i-th word block is determined based on the word identifier sequence of the i-th word block and the model name; if i is greater than 0, then the request hash value of the i-th word block is determined based on the request hash value of the (i-1)-th word block, the word identifier sequence of the i-th word block, and the model name.

[0075] The parent hash value is the cache identifier parsed from the current inference key-value cache. The parent hash value is used to represent the cache dependency relationship of the token block in the current inference key-value cache.

[0076] Specifically, the request hash value of the i-th word block can be determined based on the following formula. :

[0077] in, This represents the request hash value of the i-th block (term block), where i takes the value [0, G-1]. The routing module divides the term ID list into G blocks according to the preset block size Z. `hash()` represents the function to calculate the hash value. Indicates the ID of the i-th word element. This represents the value of the (i-1)th request hash (the request hash value of the token block). The reason for using the previous value here is to ensure continuous prefix matching. Furthermore, considering that multiple models can be deployed within the same Pod, the model name is also taken into account. Here, the calculation... There are two cases: the first is the parent hash value. If it exists (is not empty), it can be calculated by hash( (Model name, Pod address), find the hash of the previous request based on the second mapping table. This allows us to calculate the 0th request hash for this message, and subsequent request hashes with i>0 can be calculated based on the 0th request hash; the second method is... If it does not exist, it means that this message does not depend on the previous block, and the hash of the 0th request is directly calculated using the word list.

[0078] Specifically, the prefix matching scoring strategy is a scoring strategy for KV cache reuse. It is used to evaluate how long the continuous match is between the KV block already cached by a Pod and the prompt prefix of the current request. The longer the match, the less prefilling is needed for this Pod and the more existing KV cache can be reused, so the higher the score is.

[0079] Specifically, the routing module converts prompt words into a list of lexical identifiers (lexical IDs), then divides them into multiple consecutive lexical blocks, calculates the request hash value of each lexical block, and forms a hash chain called PromptHash. Extract in sequence Based on Table 1, index which Pods have this block. If they do, the Pod's score is increased by 1. Once the match is broken, subsequent blocks will not be scored even if they can be matched (because prefix matching must be continuous).

[0080] Accordingly, in one embodiment, the routing module is further configured to obtain cache change messages for the decoding service instance; wherein, the cache change message includes a cache block storage message and a cache block removal message; when the cache change message is a cache block storage message, the request hash value and service instance address represented by the cache block storage message are mapped and stored in the first mapping table, and the corresponding combined hash value is calculated based on the actual hash value of the cache block, the model name, and the service instance address represented by the cache block storage message, and a mapping relationship between the combined hash value and the request hash value is established in the second mapping table based on the request hash value; when the cache change message is a cache block removal message, the corresponding combined hash value is determined based on the actual hash value of the cache block to be removed, the model name, and the service instance address represented by the cache block removal message, the request hash value mapped by the combined hash value is found in the second mapping table, and the request hash value and its mapped service instance address are deleted from the first mapping table.

[0081] The first mapping table records the mapping relationship between request hash values ​​and service instance addresses, while the second mapping table records the mapping relationship between combined hash values ​​and request hash values.

[0082] Specifically, the routing module starts a ZMQ SUB socket (Zero Message Queue SUB socket) and binds it to the ZMQ PUB socket (Zero Message Queue PUBsocket) of the inference engine to establish a message subscription channel. It periodically pulls messages from the channel and parses them to obtain cache change messages, which include cache block storage messages and cache block removal messages.

[0083] Furthermore, when the cache block storage message is obtained, the cache block storage message is parsed to extract the request hash value, service instance address, actual cache block hash value, and model name, etc., and a mapping storage between the request hash value and the service instance address is established in the first mapping table; the combined hash value of the actual cache block hash value, model name, and service instance address is calculated through a hash function, and a mapping relationship between the combined hash value and the request hash value is established in the second mapping table.

[0084] For example, an exemplary first mapping table is shown in Table 1 below, and an exemplary second mapping table is shown in Table 2 below: Table 1. Exemplary First Mapping Table

[0085] Table 2. Exemplary Second Mapping Table

[0086] Since the messages are obtained from multiple Pods, and sometimes even multiple models (multiple models can be deployed in the same Pod), there may be the same list of lexical IDs in multiple Pods and multiple models. Therefore, it is necessary to maintain a table (Table 2) that maps the actual hash value H to the request hash value. Each time a block is processed, a hash calculation is performed using the actual hash value of this block parsed from the message, the model name, and the Pod address. This hash is used as the key of the table, pointing to the request hash value generated by the block being processed.

[0087] Specifically, when a cache block removal message is received, the message is parsed to extract the actual hash value (H0~H5), model name (e.g., model1), and service instance address (e.g., 10.10.10.3) of the cache block to be removed. The corresponding combined hash value (internal hash value) is calculated. The request hash value corresponding to the combined hash value is found in the second mapping table. The corresponding mapping relationship is deleted in the first mapping table based on the request hash value. If the address column corresponding to the request hash value is empty in the first mapping table, the row corresponding to the request hash value is deleted, and the row corresponding to the combined hash value in the second mapping table is also deleted.

[0088] Specifically, based on the reverse index lookup mechanism, the request hash value is traced by combining hash values, which enables accurate cleaning of related data, timely removal of invalid cache mapping relationships, reduction of invalid traversal during queries, and further improvement of scheduling efficiency.

[0089] Specifically, in one embodiment, when the preset scoring strategy includes a cache utilization strategy, the routing module can determine the cache utilization score of the decoding service instance based on the current key-value cache utilization of the decoding service instance.

[0090] Among them, multiple scoring results include cache utilization scoring results, which are negatively correlated with the current key-value cache utilization. The first scoring information includes the current key-value cache utilization of the decoding service instance.

[0091] It should be noted that the cache utilization score is negatively correlated with the current key-value cache utilization. That is, the lower the current key-value cache utilization, the more sufficient the remaining cache space of the instance is, and the more it can meet the caching needs of new requests, resulting in a higher score. Conversely, the higher the current key-value cache utilization, the lower the score.

[0092] Specifically, in one embodiment, when the preset scoring strategy includes an instance load metric strategy, the routing module can determine the instance load metric scoring result of the decoding service instance based on the current instance load metric of the decoding service instance.

[0093] Among them, multiple scoring results include instance load metric scoring results. The instance load metric scoring results are negatively correlated with the current instance load metric. The first scoring information includes the current instance load metric of the decoding service instance.

[0094] Specifically, the routing module can obtain the current instance load metric of the decoding service instance. This metric is used to characterize the current request processing pressure of the decoding service instance, including real-time monitoring data such as waiting queue length and resource utilization. The instance load metric score is negatively correlated with the current instance load metric; that is, the lower the current instance load metric, the less processing pressure the instance has, the more remaining processing capacity it can handle, and the better it can handle new requests, resulting in a higher score, and vice versa.

[0095] Specifically, this application embodiment incorporates instance load metrics (such as waiting queue and GPU utilization) and KV Cache distribution status into a configurable scoring system, and normalizes and weights different strategies to dynamically select the optimal Prefill and Decode instances. This avoids the coexistence of hot instance overload and resource idleness caused by round-robin scheduling, thereby improving overall throughput and service stability.

[0096] Specifically, inference engines typically provide real-time monitoring metrics for inference services. Based on these metrics, corresponding scoring strategies can be built, thus requiring the maintenance of a monitoring metric table. Monitoring metrics are periodically pulled from the Pods in the inference pool, retaining only the data from the most recent Q seconds.

[0097] Table 3 Monitoring Indicators

[0098] For example, such as Figure 5 The diagram shown illustrates the execution flow of the routing service provided in this embodiment of the application. After the routing service is started, it listens for user requests from the gateway. Whenever a user request arrives, the routing service executes the following steps: 1. After receiving a request, the routing service, based on the inference pool name configured when creating the routing instance, first reads the scenario where the current routing instance resides, then reads all Pods in that scenario, and retrieves the Pods tagged with the inference pool name and those tagged with the role "Decode". This collection of Pods can be referred to as... .

[0099] From i=1 to N, in order from Take a Pod from the set. For the i-th Pod, calculate the score for each Pod according to the scoring strategy j. :

[0100] To avoid inconsistent impacts on the final score due to varying value ranges for each scoring strategy, normalization is required here:

[0101] That is, the direction of the scoring strategy When the score is positive (the higher the score, the better), For each item, the minimum value is subtracted and the result is divided by the maximum value minus the minimum value for normalization. When the scoring strategy is negative (the smaller the score, the better), Each item is normalized by subtracting the current value from the maximum value and dividing by the minimum value of the maximum value. When each item is the same and not 0, the final score of each item is 1. When each item is the same and is 0, the final score of each item is 0.

[0102] Calculate the final weighted score for each scoring strategy. Read the scoring items and their weights from the configuration file in sequence:

[0103] That is, from 1 to M, obtain strategy j and its weights sequentially from the scoring strategy configuration. According to the specific algorithm of the strategy The score of the i-th Pod is evaluated, and then its weight is calculated. Multiply. Calculate the scores for j from 1 to M sequentially, then add them together to obtain the first scheduling score. .

[0104] In this application, an embodiment proposes a configurable scheduling strategy. By configuring which scoring strategies and their weights are used, different scheduling strategies can be controlled, and new strategy types can be extended in the future. This allows the scheduling logic to flexibly adapt to different scenarios, avoids the performance bottleneck caused by fixed strategies, and makes the scheduling strategy more flexible.

[0105] according to Size from Select the Pod with the highest score and obtain the prefix cache hit rate (Hit). If (1 - hit rate) × number of prompt chunks < threshold, it means that most prompts are in the cache or the prompts are very short. In this case, no separation is performed, and the user request is processed directly in the Decode instance, returning the selected Decode instance address to the gateway. Otherwise, the Prefill instance is selected for separated inference.

[0106] Specifically, based on the prefix cache hit rate and the number of prompt blocks, it dynamically determines whether to perform P / D separation inference; when the number of reusable prefixes in the Prefill is small or the prompt is short, it automatically degenerates to completing the inference directly in the Decode instance, avoiding the negative benefits caused by KV transmission and additional scheduling, thereby achieving better request response performance.

[0107] For example, such as Figure 6 The diagram shown illustrates the target prefill service instance invocation process provided in this application embodiment. If a Prefill instance needs to be selected for separate inference, then the Pods tagged with the inference pool name and those tagged with the role of Prefill are obtained. These Pod sets are referred to as... Calculate the score according to the above scoring method. The second scheduling score for all Pods (pre-populated service instances) The system selects the Pod with the highest score and sets the Prefill instance to the address of that Pod in the user request header (request header information) (for example, setting the request header prefill-url:10.10.10.3).

[0108] Finally, the selected Decode instance address (the first scheduling address) and the modified request headers are returned to the gateway, which then forwards the request to the reverse proxy service.

[0109] The inference service request scheduling system provided in this application has the advantage that the routing module can score and prioritize the decoding service instance cluster, replacing the traditional round-robin mechanism. This allows the system to consider the load and inference resources of each instance when distributing requests. Furthermore, by adding a prefix coverage condition judgment mechanism, if the current key-value cache of the target decoding service instance with the highest first scheduling score is sufficient to complete the original inference request, the original inference request is directly sent to the target decoding service instance. This not only improves the cache reuse rate of the decoding service instance but also reduces unnecessary redundant calculations and data transmission, thereby improving the response efficiency and reducing the response latency of the inference request.

[0110] Through the above description of the embodiments, those skilled in the art can clearly understand that the system according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0111] This application provides a method for scheduling inference service requests, which is applied to the inference service request scheduling system provided in the above embodiments.

[0112] like Figure 7 The diagram shown is a flowchart illustrating the inference service request scheduling method provided in this application embodiment. The method includes: Step 701: Obtain the original inference request issued by the user; Step 702: Determine the first scheduling score of each decoding service instance based on the first scoring information of the decoding service instance cluster. Step 703: Select the decoding service instance with the highest first scheduling score as the target decoding service instance; Step 704: If the original inference request meets the preset prefix coverage condition, the first scheduling address of the target decoding service instance is fed back to the gateway so that the gateway sends the original inference request to the target decoding service instance according to the first scheduling address. Step 705: Based on the target decoding service instance, if it is determined that the original inference request does not require pre-filled service instance processing, inference calculation is performed in response to the original inference request to obtain the request response result.

[0113] For a description of the features in the embodiment corresponding to the inference service request scheduling method, please refer to the relevant description of the embodiment corresponding to the inference service request scheduling system, which will not be repeated here.

[0114] The embodiments of this application also provide an inference service request scheduling apparatus for executing the inference service request scheduling method provided in the above embodiments.

[0115] like Figure 8The diagram shown is a structural schematic of the inference service request scheduling device provided in an embodiment of this application. The inference service request scheduling device 80 includes: an acquisition module 801, a determination module 802, a selection module 803, a feedback module 804, and a response module 805.

[0116] The system comprises the following modules: an acquisition module for acquiring the original inference request issued by the user; a determination module for determining the first scheduling score of each decoding service instance based on the first scoring information of the decoding service instance cluster; a selection module for selecting the decoding service instance with the highest first scheduling score as the target decoding service instance; a feedback module for feeding back the first scheduling address of the target decoding service instance to the gateway when the original inference request meets the preset prefix coverage condition, so that the gateway sends the original inference request to the target decoding service instance based on the first scheduling address; and a response module for performing inference calculations in response to the original inference request based on the target decoding service instance, if it is determined that the original inference request does not require pre-filling of service instance processing, to obtain the request response result.

[0117] For a description of the features in the embodiment corresponding to the inference service request scheduling device, please refer to the relevant description of the embodiment corresponding to the inference service request scheduling method, which will not be repeated here.

[0118] Embodiments of this application also provide an electronic device, such as... Figure 9 The diagram shown is a schematic diagram of the structure of an electronic device provided in an embodiment of this application, including a processor 10 and a memory 20. The memory 20 stores a computer program, and the processor 10 is configured to run the computer program to execute the steps in any of the above-described inference service request scheduling method embodiments.

[0119] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described inference service request scheduling method embodiments at runtime.

[0120] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0121] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described inference service request scheduling method embodiments.

[0122] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described inference service request scheduling method embodiments.

[0123] Any of the components, modules, units, parts, methods, and operations described herein can be implemented using software, firmware, hardware (e.g., fixed logic circuitry), manual processing, or any combination thereof. Alternatively or additionally, any functionality described herein can be executed at least in part by one or more hardware logic components, such as, but not limited to, a central processing unit (CPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), an application-specific standard product (ASSP), a system-on-a-chip (SoC), a complex programmable logic device (CPLD), a microprocessor (MCU), etc. The terms "system," "computing device," or "apparatus" as used herein encompass various means, devices, and machines for processing data, including, for example, one or more programmable processors, computers, SoCs, or combinations thereof. The apparatus may also include code that creates an execution environment for the computer program in question, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or one or more combinations thereof. The aforementioned computer program (also known as a program, software, software application, app, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as a standalone program or as a module, component, subroutine, object, or other unit suitable for a computing environment.

[0124] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0125] The foregoing has provided a detailed description of the inference service request scheduling system, method, and electronic device provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only intended to help understand the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A reasoning service request scheduling system, characterized in that, include: The system includes a routing module, a gateway, and an inference pool, wherein the inference pool comprises a pre-populated service instance cluster and a decoding service instance cluster. The gateway is used to receive the original inference request sent by the user and forward the original inference request to the routing module; The routing module is used to determine the first scheduling score of each decoding service instance based on the first scoring information of the decoding service instance cluster when the original inference request is received, and to take the decoding service instance with the highest first scheduling score as the target decoding service instance. If the original inference request meets the preset prefix coverage condition, the first scheduling address of the target decoding service instance is fed back to the gateway. The gateway is used to send the original inference request to the target decoding service instance according to the first scheduling address; The target decoding service instance is used to perform inference calculations in response to the original inference request and obtain a request response result when it is determined that the original inference request does not require processing by the pre-filled service instance.

2. The inference service request scheduling system according to claim 1, characterized in that, The routing module is also used for: If the original inference request does not meet the preset prefix coverage condition, the second scheduling score of each pre-filled service instance is determined according to the second scoring information of the pre-filled service instance cluster, and the pre-filled service instance with the highest second scheduling score is taken as the target pre-filled service instance. Based on the second scheduling address of the target pre-filled service instance, modify the request header information of the original inference request to obtain the target inference request; The first scheduling address of the target decoding service instance and the target inference request are fed back to the gateway, so that the gateway sends the target inference request to the target decoding service instance according to the first scheduling address.

3. The inference service request scheduling system according to claim 2, characterized in that, The decoding service instances in the decoding service instance cluster include a request forwarding module and a decoding inference module; The request forwarding module is used to intercept the original inference request or the target inference request sent by the gateway, and if it is determined that the original inference request does not need to be processed by the pre-filled service instance, the original inference request is forwarded to the decoding inference module. If it is determined that the target inference request needs to be processed by the target pre-filled service instance, the target inference request is forwarded to the target pre-filled service instance according to the second scheduling address represented by the request header information; The decoding and reasoning module is used to perform reasoning calculations in response to the original reasoning request and obtain the request response result.

4. The inference service request scheduling system according to claim 3, characterized in that, The request forwarding module is specifically used for: Check whether there is a preset key value in the request header information of the original inference request or the target inference request to determine the request type; If the preset convention key value does not exist, the request type is determined to be the original inference request; if the preset convention key value exists, the request type is determined to be the target inference request.

5. The inference service request scheduling system according to claim 3, characterized in that, The request forwarding module is specifically used for: Search the preset instance connection pool for the target network connection corresponding to the second scheduling address; Based on the target network connection, the target inference request is forwarded to the target pre-filled service instance; The preset instance connection pool is used to cache network connections between the decoding service instance and multiple pre-filled service instances.

6. The inference service request scheduling system according to claim 3, characterized in that, The request forwarding module is also used for: Determine whether the second scheduling address represented by the request header information belongs to a preset set of secure addresses; If it is determined that the second scheduling address does not belong to the preset security address set, a request response error is reported to the gateway.

7. The inference service request scheduling system according to claim 6, characterized in that, The request forwarding module is specifically used for: If it is determined that the second scheduling address belongs to the preset security address set, the original parameter information is extracted from the target inference request; Based on the original parameter information, pre-filled request parameters are constructed; wherein, the pre-filled request parameters include at least the unique request identifier, the maximum number of generated lexical units, and the key-value cache transmission parameters for the pre-filling stage; Based on the pre-filled request parameters, generate the target pre-filled request corresponding to the target inference request; The target pre-fill request is forwarded to the target pre-fill service instance.

8. The inference service request scheduling system according to claim 7, characterized in that, The target pre-fill service instance is used for: Receive the target pre-fill request; In response to the target pre-filling request, pre-filling inference is performed to generate the target lexical and its corresponding key-value cache according to the maximum number of generated lexicals; The key-value cache is associated with the unique identifier of the request and stored accordingly; The request forwarding module returns a target pre-filled response, which includes at least the key-value cache transmission parameters of the key-value cache.

9. The inference service request scheduling system according to claim 8, characterized in that, The request forwarding module is also used for: Receive the target prefill response returned by the target prefill service instance; The key-value cache transfer parameters are parsed from the target pre-filled response; The unique identifier of the request and the parsed key-value cache transmission parameters are added to the original parameter information to obtain the updated target inference request parameters; A request containing the target inference request parameters is sent to the decoding inference module, so that the decoding inference module can obtain the key-value cache from the target pre-filled service instance according to the target inference request parameters, perform inference calculation based on the key-value cache, and obtain the request response result.

10. The inference service request scheduling system according to claim 1, characterized in that, The routing module is specifically used for: Based on the current inference key-value cache of the target decoding service instance and the prompt word information of the original inference request, calculate the prefix cache hit rate of the target decoding service instance for the original inference request; Based on the prefix cache hit rate and the number of prompt word blocks in the original inference request, determine whether the original inference request meets the preset prefix coverage condition.

11. The inference service request scheduling system according to claim 1, characterized in that, The routing module is specifically used for: According to multiple preset scoring strategies, based on the first scoring information of the decoding service instance cluster, the corresponding multiple scoring results are determined; Based on the various scoring results, a first scheduling score is determined for each of the decoding service instances.

12. The inference service request scheduling system according to claim 11, characterized in that, When the preset scoring strategy includes a prefix cache matching strategy, the routing module is specifically used for: Generate a word sequence based on the prompt words in the original reasoning request; The word sequence is divided into multiple consecutive word blocks; Calculate the request hash value for each word block to obtain the request hash chain; For any of the decoding service instances, the consecutive matching length between the cached word blocks in the decoding service instance and the request hash chain is determined based on the current inference key-value cache of the decoding service instance; wherein, the current inference key-value cache includes multiple cached word blocks, and the first scoring information includes the current inference key-value cache of the decoding service instance; Based on the continuous matching length, the prefix cache matching score of the decoding service instance is determined; Among them, the multiple scoring results include the prefix cache matching scoring results.

13. The inference service request scheduling system according to claim 12, characterized in that, The routing module is specifically used for: For the i-th word block in the word sequence; where i is an integer greater than or equal to 0; If i equals 0 and a parent hash value exists, then the request hash value of the i-th word block is determined based on the parent hash value, the word identifier sequence of the i-th word block, and the model name. If i equals 0 and there is no parent hash value, then the request hash value of the i-th word block is determined based on the word identifier sequence of the i-th word block and the model name; If i is greater than 0, the request hash value of the i-th word block is determined based on the request hash value of the (i-1)th word block, the word identifier sequence of the i-th word block, and the model name. The parent hash value is a cache identifier parsed from the current inference key-value cache, and the parent hash value is used to characterize the cache dependency relationship of the term block in the current inference key-value cache.

14. The inference service request scheduling system according to claim 12, characterized in that, The routing module is also used for: Obtain the cache change message of the decoding service instance; wherein, the cache change message includes a cache block storage message and a cache block removal message; When the cache change message is a cache block storage message, the request hash value and service instance address represented by the cache block storage message are mapped and stored in the first mapping table. Based on the actual hash value of the cache block, the model name and the service instance address represented by the cache block storage message, the corresponding combined hash value is calculated. Based on the request hash value, a mapping relationship between the combined hash value and the request hash value is established in the second mapping table. When the cache change message is a cache block removal message, the corresponding combined hash value is determined based on the actual hash value, model name and service instance address of the cache block to be removed represented by the cache block removal message. The request hash value mapped by the combined hash value is found in the second mapping table, and the request hash value and its mapped service instance address are deleted from the first mapping table. The first mapping table is used to record the mapping relationship between request hash values ​​and service instance addresses, and the second mapping table is used to record the mapping relationship between combined hash values ​​and request hash values.

15. The inference service request scheduling system according to claim 11, characterized in that, When the preset scoring strategy includes a cache utilization strategy, the routing module is specifically used for: Based on the current key-value cache utilization of the decoding service instance, determine the cache utilization score of the decoding service instance; The various scoring results include the cache utilization scoring result, which is negatively correlated with the current key-value cache utilization. The first scoring information includes the current key-value cache utilization of the decoding service instance.

16. The inference service request scheduling system according to claim 11, characterized in that, When the preset scoring strategy includes an instance load metric strategy, the routing module is specifically used for: Based on the current instance load metric of the decoding service instance, determine the instance load metric score of the decoding service instance; The multiple scoring results include the instance load index scoring results, which are negatively correlated with the current instance load index. The first scoring information includes the current instance load index of the decoding service instance.

17. A reasoning service request scheduling method, applied to the reasoning service request scheduling system as described in any one of claims 1 to 16, characterized in that, The method includes: Obtain the original inference request issued by the user; Based on the first scoring information of the decoding service instance cluster, determine the first scheduling score of each decoding service instance; The decoding service instance with the highest first scheduling score is selected as the target decoding service instance. If the original inference request meets the preset prefix coverage condition, the first scheduling address of the target decoding service instance is fed back to the gateway, so that the gateway sends the original inference request to the target decoding service instance according to the first scheduling address; Based on the target decoding service instance, if it is determined that the original inference request does not require processing by the pre-filled service instance, inference calculation is performed in response to the original inference request to obtain the request response result.

18. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the inference service request scheduling method as described in claim 17 when executing the computer program.

19. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the inference service request scheduling method as described in claim 17.

20. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the inference service request scheduling method as described in claim 17.

Images