Large model inference system and inference method

Abstract

The application provides a large model inference system and an inference method. The system comprises a front-end layer configured to receive an inference request sent by a client; a core layer comprising a request router, a pre-filling calculation node and a decoding calculation node; the request router receives the inference request, generates and broadcasts a scoring request; the decoding calculation node calculates a corresponding scoring result based on the scoring request and returns the scoring result to the request router; the request router determines a target decoding calculation node based on the scoring result; a decoding scheduler in an inference layer acquires the inference request from the target decoding calculation node and determines a target pre-filling calculation node based on a preset strategy, and sends a KV cache calculation task of the inference request to the target pre-filling calculation node; a pre-filling scheduler and a pre-filling engine are used for KV cache calculation and transmission; and a decoding engine performs decoding calculation based on the KV cache to generate a token. The application improves the overall utilization rate of GPU computing power and video memory bandwidth.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and more specifically, to a large-scale model reasoning system and reasoning method. Background Technology

[0002] In recent years, with the widespread application of Large Language Models (LLMs) based on the Transformer architecture in fields such as natural language processing and multimodal understanding, how to efficiently deploy and infer these models has become a research hotspot in both industry and academia. A typical LLM inference process can be divided into two stages: the prefill stage and the decoding stage.

[0003] In large-scale cluster deployments, the industry generally adopts two deployment modes: Aggregated Deployment and Prefill-Decode Disaggregated Deployment.

[0004] Among them, the PD split deployment attempts to split the Prefill and Decode tasks to run on dedicated nodes. After receiving an inference request, a centralized router is used to allocate and schedule the inference request. At this time, the router needs to obtain the performance parameters of all computing nodes in advance and make routing decisions accordingly. This routing method cannot effectively perceive the working status of each computing node, resulting in low resource utilization. Summary of the Invention

[0005] The purpose of this application is to provide a large model inference system and inference method to improve resource utilization in the large model inference process.

[0006] In a first aspect, embodiments of this application provide a large-model inference system, including: The front-end layer is used to receive inference requests sent by the client; The core layer includes request routers and compute nodes; the compute nodes include pre-filled compute nodes and decoding compute nodes. The router is requested to receive the inference request, generate a scoring request based on the inference request, and broadcast the scoring request to the decoding computing node. The decoding computing node calculates the corresponding score based on the score request and returns the score result to the requesting router. The router is requested to determine the target decoding computing node based on the scoring results and send an inference request to the target decoding computing node; The inference layer includes the decoding engine, decoding scheduler, pre-filling engine, and pre-filling scheduler; The decoding scheduler obtains the inference request from the target decoding computing node, determines the target pre-filled computing node based on the preset strategy, and sends the KV cache computing task of the inference request to the target pre-filled computing node; The pre-fill scheduler sends the KV cache computation task to the pre-fill engine, which performs pre-fill computation based on the KV cache computation task to obtain the KV cache. The decoding engine performs decoding calculations based on the KV cache to generate a token.

[0007] This application embodiment achieves efficient inference scheduling under a PD separation architecture by constructing a collaborative workflow of front-end-decoding computing node-prefilling computing node-decoding computing node and combining it with a two-level routing strategy. The first-level routing, using a decentralized scoring mechanism, precisely allocates requests to the optimal decoding node through a global request router, reducing the performance bottleneck and state lag risk of traditional single-point routing. The second-level routing is asynchronously completed within the decoding node by the scheduler, reducing first-word latency (TTFT) by selecting appropriate prefilling computing nodes. This two-level routing collaboration achieves fine-grained awareness and elastic scheduling of computing resources, decouples resource contention between the Prefill and Decode stages, and improves the overall utilization of GPU computing power and memory bandwidth.

[0008] In one possible implementation of the first aspect, the core layer also includes a KV cache transfer engine; The KV cache transmission engine sends the KV cache calculated by the target pre-filled computing node to the target decoding computing node through a separate subprocess.

[0009] This application embodiment achieves decoupling between transmission and computation by setting up an independent KV cache transmission engine in the core layer and using an independent subprocess to perform cross-node transmission of the KV cache. This design overcomes the computation blocking problem caused by the transmission process occupying the resources of the inference main process in traditional schemes, allowing pre-filled computing nodes to asynchronously initiate transmission while computing the KV cache, thereby improving the overall system throughput and response efficiency.

[0010] In one possible implementation of the first aspect, the target pre-filled computing node calculates the KV cache layer by layer. After completing the KV cache calculation for one layer, it sends the KV cache of that layer to the target decoding computing node through the KV cache transmission engine.

[0011] This embodiment of the application achieves pipelined parallelism of computation and transmission by simultaneously computing the KV cache layer by layer at the target prefill computing node and sending the cache of each layer to the target decoding computing node through an independent KV cache transmission engine after each layer is completed. This layer-by-layer transmission mechanism effectively overlaps the computation time of the Prefill stage and the network transmission time of the KV cache, significantly reducing the first-word latency caused by waiting for all caches to be generated, and improving the overall inference response speed.

[0012] In one possible implementation of the first aspect, the front-end layer, core layer, and inference layer communicate via Redis. This application embodiment achieves efficient, flexible, and loosely coupled data interaction between the layers. Using Redis as a unified middleware, the system does not rely on a fixed network topology or centralized scheduling node. Each component can perceive cluster status changes in real time through a publish / subscribe (Pub / Sub) mechanism, supporting dynamic scaling and automatic node discovery.

[0013] In one possible implementation of the first aspect, the requesting router broadcasts the scoring request via Redis; wherein the broadcast address includes the node type and model identifier; The rating request is received by the decoding computing node that is compatible with the broadcast address.

[0014] In this application embodiment, the broadcast address includes node type and model identifier, so that only decoding computing nodes that meet specific task types and model configurations can receive the scoring request, thus achieving accurate message delivery and node filtering.

[0015] In one possible implementation of the first aspect, the front-end layer includes a task submitter and at least one client interface; the client sends inference requests through the client interface; and the task submitter stores the inference requests in Redis.

[0016] The large model inference system provided in this application supports at least one client interface method. After the client submits an inference request through a standardized interface, the task submitter writes it to Redis in a unified manner, which realizes the buffering and decoupling of requests and reduces the risk of request loss or connection timeout caused by instantaneous high load on the backend.

[0017] In one possible implementation of the first aspect, the Redis database stores node information of the registered and online compute nodes; Compute nodes obtain all available compute nodes in real time through Redis subscription.

[0018] This application embodiment centrally stores the node information of registered and online compute nodes in a Redis database, and utilizes Redis's publish / subscribe mechanism to enable each compute node to subscribe to and dynamically obtain the latest status of all available nodes in the cluster in real time, thereby achieving automated discovery and status synchronization between nodes.

[0019] Secondly, embodiments of this application provide a large-model inference method, including: The router is requested to generate a scoring request based on the received inference request; The requesting router broadcasts a scoring request to the decoding computing node, so that the decoding computing node that receives the scoring request can perform scoring calculation and return the scoring result to the requesting router; The router is requested to determine the target decoding computing node based on the scoring results and send an inference request to the target decoding computing node. The target decoding computing node determines the target pre-filling computing node based on a preset strategy and sends the KV cache computing task corresponding to the inference request to the target pre-filling computing node; The target pre-filling computation node performs pre-filling computation based on the KV cache computation task to obtain the KV cache; The target decoding computing node receives the KV cache and performs decoding computing based on the KV cache to generate a token.

[0020] This application embodiment achieves efficient inference scheduling under a PD separation architecture by constructing a collaborative workflow of front-end-decoding computing node-prefilling computing node-decoding computing node and combining it with a two-level routing strategy. The first-level routing, using a decentralized scoring mechanism, precisely allocates requests to the optimal decoding node through a global request router, reducing the performance bottleneck and state lag risk of traditional single-point routing. The second-level routing is asynchronously completed within the decoding node by the scheduler, reducing first-word latency by selecting appropriate prefilling computing nodes. This two-level routing collaboration achieves fine-grained awareness and elastic scheduling of computing resources, decouples resource contention between the Prefill and Decode stages, and improves the overall utilization of GPU computing power and memory bandwidth.

[0021] In one possible implementation of the second aspect, the scoring request includes a scoring strategy identifier; the decoding computing node calculates the corresponding scoring result based on the scoring request, including: The decoding computing node calculates the score based on the scoring strategy identifier and the corresponding scoring strategy to obtain the score result.

[0022] This application embodiment achieves flexible configuration and dynamic selection of routing strategies by carrying a scoring strategy identifier in the scoring request. Different scoring algorithms can be adopted according to actual needs (such as load balancing, cache affinity, minimum latency, etc.) in different scenarios, which improves the adaptability and intelligence of the system. At the same time, each decoding computing node independently performs local scoring according to a unified strategy, which not only ensures the consistency of scoring standards, but also avoids the performance bottlenecks and single point of failure risks caused by centralized strategy scheduling, thus enhancing the overall scalability and reliability of the system.

[0023] In one possible implementation of the second aspect, the target decoding computing node receives a KV buffer, including: The KV cache transmission engine of the target pre-filled computing node sends the KV cache to the target decoding computing node through a separate subprocess; The target decoding computing node receives the KV buffer.

[0024] This application embodiment decouples the KV cache transmission process into an independent sub-process, achieving complete separation of transmission and computation. This overcomes the performance degradation caused by data transmission blocking inference computation in traditional architectures. The independent KV cache transmission engine can initiate transmission immediately after the pre-filled computation node completes its computation, without waiting for the entire inference process to finish, thereby significantly improving the system's concurrent processing capabilities and resource utilization.

[0025] In one possible implementation of the second aspect, the KV cache transport engine of the target pre-filled computing node sends the KV cache to the target decoding computing node through a separate subprocess, including: Each time the target pre-filled computing node obtains a KV cache layer, it sends that KV cache layer to the target decoding computing node through an independent subprocess and executes the next layer of KV cache calculation in parallel.

[0026] This embodiment of the application achieves pipelined parallelism of computation and transmission by simultaneously computing the KV cache layer by layer at the target prefill computing node and sending the cache of each layer to the target decoding computing node through an independent KV cache transmission engine after each layer is completed. This layer-by-layer transmission mechanism effectively overlaps the computation time of the Prefill stage and the network transmission time of the KV cache, significantly reducing the first-word latency caused by waiting for all caches to be generated, and improving the overall inference response speed.

[0027] Other features and advantages of this application will be set forth in the following description and will be apparent in part from the description or may be learned by practicing embodiments of this application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description and the accompanying drawings. Attached Figure Description

[0028] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 An architecture diagram of a large model inference system provided in this application embodiment; Figure 2 This application provides a schematic diagram of a decentralized distributed routing strategy process. Figure 3 This application provides a schematic diagram of a KV cache generation and transmission process. Figure 4 This is a schematic diagram of a large model inference method provided in an embodiment of this application. Detailed Implementation

[0030] The embodiments of the technical solution of this application will now be described in detail with reference to the accompanying drawings. These embodiments are only used to more clearly illustrate the technical solution of this application and are therefore merely examples, and should not be used to limit the scope of protection of this application.

[0031] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this application; the terms “comprising” and “having”, and any variations thereof, in the specification and the foregoing description of the drawings are intended to cover non-exclusive inclusion.

[0032] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly defined.

[0033] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0034] In the description of the embodiments in this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0035] In the description of the embodiments of this application, the term "multiple" refers to two or more (including two), similarly, "multiple sets" refers to two or more (including two sets), and "multiple pieces" refers to two or more (including two pieces).

[0036] In the description of the embodiments of this application, unless otherwise expressly specified and limited, technical terms such as "installation," "connection," "joining," and "fixing" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. For those skilled in the art, the specific meaning of the above terms in the embodiments of this application can be understood according to the specific circumstances.

[0037] The continuous expansion of the Transformer architecture has endowed Large Language Models (LLMs) with powerful cognitive capabilities, but it has also led to a sharp increase in inference latency and resource consumption. Faced with massive requests, traditional homogeneous computing models are unsustainable. Therefore, researchers have deconstructed the inference process into two heterogeneous stages: prefilling and decoding, giving rise to a new generation of efficient inference architectures such as PD separation, aiming to achieve optimal allocation of computing and storage resources.

[0038] During the pre-filling phase, the system performs a complete forward propagation computation in parallel on all tokens of the input prompt, generates a key-value cache, and outputs the first generated token. The computational cost of this phase is proportional to the length of the input prompt, making it a typical computationally intensive operation that requires significant GPU computing power.

[0039] During the decoding phase, the system generates subsequent tokens autoregressively based on the generated KV Cache. Each step only requires forward computation on a single token and appending its corresponding KV value to the cache. Although the computational cost per step is extremely small, this phase is primarily limited by memory bandwidth and is a memory-intensive operation because the entire KV Cache needs to be read from GPU memory each time.

[0040] In a clustered deployment, each inference node independently handles the complete Prefill and Decode processes. While the architecture is simple, it has significant drawbacks in practical applications: First, Prefill and Decode tasks share resources on the same node, causing computationally intensive Prefill tasks to block bandwidth-sensitive Decode tasks, resulting in latency jitter in streaming output. Second, long-staying Decode requests continuously consume a large amount of GPU memory for storing the KV cache, severely compressing the GPU memory available for new Prefill requests, leading to frequent preemptions and impairing service quality. Third, because hardware configuration must balance computing power and GPU memory bandwidth, it is difficult to achieve optimal resource allocation for different stages, resulting in low resource utilization.

[0041] The PD split deployment attempts to separate the Prefill and Decode tasks and run them on dedicated nodes, equipped with high-performance computing cards and high-bandwidth cards with large memory, respectively. Theoretically, this can eliminate inter-stage interference and optimize resource allocation. While centralized routers can provide a globally unified scheduling view, their decisions heavily rely on pre-collected static performance profiles of nodes. This design implicitly assumes a key fact: the processing power of a node is constant. In real-world scenarios, the available resources of a node change dynamically over time, and static routing cannot capture this time-varying nature, leading to a disconnect between scheduling decisions and the actual service capacity of nodes. This ultimately manifests as high-priority nodes being overloaded while low-load nodes remain idle, significantly reducing the system's resource utilization efficiency.

[0042] To address the aforementioned technical problems, this application provides a large-model inference system. The system includes a front-end layer, a core layer, and an inference layer. The front-end layer interfaces with the client and receives inference requests sent by the client. The core layer includes a request router and compute nodes, with the compute nodes including decoding compute nodes and pre-filled compute nodes. The request router broadcasts scoring requests to the decoding compute nodes. Upon receiving a scoring request, the decoding compute nodes calculate their respective scores and write the calculated scores back to the request router. The request router selects a target decoding compute node based on the scoring results and then sends an inference request to the target decoding compute node. The target decoding compute node selects a target pre-filled compute node based on a preset routing strategy. The target pre-filled compute node performs key-value (KV) cache calculations and transmits the calculated KV cache to the target decoding compute node. The target decoding compute node decodes the KV cache and generates a token corresponding to the inference request.

[0043] The architecture of the large-scale model inference system is described below: Figure 1 This is an architecture diagram of a large-model inference system provided in an embodiment of this application. The system includes a front-end layer 101, a core layer 102, and an inference layer 103. Wherein: Frontend layer 101 serves as the user access portal for the large-scale model inference system, receiving inference requests initiated by clients. It includes at least one client interface and a task submitter. The client interface offers diverse access methods, such as access via the OpenAI-compatible API or the interactive terminal mode. The OpenAI-compatible API supports standard RESTful API calls, allowing users to seamlessly integrate with the existing ecosystem via HTTP requests. This interface fully supports both streaming and non-streaming response modes. Streaming responses meet the high latency requirements of scenarios like real-time chat and code completion, allowing users to see the generated results token by token. Furthermore, this interface supports multimodal content processing, capable of receiving and parsing input prompts in various formats, including images, videos, and audio. The interactive terminal mode provides a command-line interface (CLI) or a web terminal, facilitating interactive debugging, model testing, and rapid prototyping for developers and researchers.

[0044] The Task Submitter acts as a bridge between the front-end layer and the core layer. Its main function is to encapsulate and persist raw inference requests from the client interface. When an inference request is received, the Task Submitter serializes its key information (such as request ID, model identifier, input prompt, generation parameters, etc.) and submits it to the core layer 102 via a message queue.

[0045] The core layer (Core) 102's main functions are to implement global request distribution, intelligent routing decisions, and collaborative management of computation. Core layer 102 includes request routers and worker nodes.

[0046] Figure 2 This is a schematic diagram of a decentralized distributed routing strategy process provided in an embodiment of this application.

[0047] A computer cluster includes multiple computing nodes, and a large model may only be deployed on some computing nodes. Furthermore, for a large model with separate PD (Programmable Detection) stages, some computing nodes are responsible for the calculations related to the pre-filling stage of the large model, which are referred to as pre-filling computing nodes in this application embodiment, while some computing nodes are responsible for the calculations related to the decoding stage of the large model, which are referred to as decoding computing nodes in this application embodiment.

[0048] The request router is used to schedule decoding compute nodes in a decentralized manner. It generates a scoring request based on the inference request and then broadcasts the generated scoring request to the decoding compute nodes. Understandably, the scoring request can be in JSON format, specifically including a request ID, a hint, a cache block hash, and a routing policy name. In one implementation, the request router can use Redis to broadcast the scoring request to the decoding compute nodes. In another implementation, the request router can use NATS to broadcast the scoring request to the decoding compute nodes.

[0049] The router can add a model identifier and a decoding identifier to the broadcast address of the scoring request. The scoring request can then be broadcast to the decoding computing node corresponding to the large model using the model identifier and decoding identifier.

[0050] The task processor of the decoding compute node receives scoring requests by subscribing to Redis. Upon receiving a scoring request, it performs scoring calculations. During the scoring calculation process, if the decoding compute node has a calculation strategy stored internally, it will perform the calculation according to that strategy. Additionally, the generated scoring request may include a strategy identifier. After receiving the scoring request, the decoding compute node can extract the strategy identifier from it, then use the strategy corresponding to that identifier to calculate the score, obtain the scoring result, and write its scoring result back to the request router. The scoring result can be represented in the form of a score or other forms, such as representing it according to performance levels, i.e., high performance, medium-high performance, medium performance, medium-low performance, and low performance. Each decoding compute node writes its own scoring result back to the request router.

[0051] It should be noted that if the requesting router does not receive all the scoring results within a preset time (e.g., 500ms), the optimal node will be selected based on the existing responses.

[0052] The requesting router selects a target decoding computing node from multiple decoding computing nodes based on the received score results. Understandably, if the score results are represented numerically, a higher value indicates a higher prefix matching degree, resulting in less computation; and fewer computational tasks for the node, allowing it to enter the scheduling queue more quickly; furthermore, it indicates that the decoding computing node can complete the task with lower latency, and the requesting router can choose the decoding computing node with the highest value as the target decoding computing node. If the score results are represented by performance levels, the requesting router can choose a high-performance decoding computing node as the target decoding computing node. If the highest value of the score corresponds to multiple decoding computing nodes, or if multiple high-performance decoding computing nodes exist, the requesting router can randomly select one as the target decoding computing node, or choose the first high-performance decoding computing node received as the target decoding computing node.

[0053] After identifying the target decoding computing node, the requesting router sends an inference request to the target decoding computing node.

[0054] Inference layer 103 includes the vLLM backend and other backends. Other backends refer to other inference models; in a large-scale model inference system, multiple inference models can be deployed. Taking the vLLM inference model as an example, the vLLM backend includes a decoding engine, a decoding scheduler, a pre-filling engine, and a pre-filling scheduler. The decoding engine and decoding scheduler run on the decoding compute nodes, while the pre-filling engine and pre-filling scheduler run on the pre-filling compute nodes. For ease of description, Figure 1 The decoding engine and the pre-filling engine are referred to as the vLLM engine (AsyncLLMEngine), and the decoding scheduler and the pre-filling scheduler are collectively referred to as the scheduler (CustomScheduler).

[0055] During large-scale model inference, pre-filling computation is required before decoding computation. Therefore, upon receiving an inference request, the target decoding computation node asynchronously initiates routing for the pre-filling task within its internal decoding scheduler. Specifically, the optimal target pre-filling computation node can be selected for the request based on a preset strategy. It should be noted that this preset strategy refers to routing decision rules pre-configured in the system, which may include, but are not limited to: prefix cache hit strategy, load balancing strategy, latency prediction strategy, cost optimization strategy, etc. Each strategy can be dynamically registered and enabled through configuration files or APIs.

[0056] The core objective of the prefix cache hit strategy is to maximize the reuse rate of KV cache prefixes and reduce first-word latency and pre-fill computation load. When a new request shares the same prompt prefix as an existing request, the target decoding compute node prioritizes routing the request to a node that has cached the KV cache for that prefix, avoiding duplicate computation. The implementation is as follows: After the front-end receives a user request, it first segments the input Prompt and calculates the hash value of its token sequence. Then, it sequentially extracts the prefix subsequences of this hash sequence (e.g., the hashes of the first 10, first 50 tokens), forming a prefix hash chain. The decoding scheduler sends this calculated prefix hash chain to Redis to query whether any pre-fill compute node has cached the matching KV block. If Redis returns a matching result (e.g., P-Node-3 has the prefix cache), then when performing routing scoring for each pre-fill compute node, P-Node-3, which has the cache, will receive a very high priority score. The routing algorithm will preferentially select P-Node-3 to perform the Prefill task, meaning P-Node-3 is the target prefill computation node. After receiving the task, P-Node-3 can directly skip the computation of the cached prefix part, only need to compute the remaining, new token part, and then send the complete KV cache back to the target decoding computation node.

[0057] The core idea of ​​the prefill compute node load balancing strategy is to avoid concentrating all prefill tasks on a few nodes, preventing node overload and ensuring the stability and high throughput of the entire prefill cluster. This is implemented as follows: each prefill compute node (P-Node) periodically (e.g., every second) proactively reports its load status to Redis. This status information, as part of its score, is stored in a globally visible Redis key. Key load metrics include: the number of currently processed prefill requests, reflecting instantaneous computational pressure; GPU utilization, used to monitor the saturation of computing resources; KV cache utilization, used to measure the scarcity of GPU memory resources; high utilization may lead to preemption of new requests; and the length of the pending request queue, reflecting task backlog. When the decoding scheduler needs to route a prefill task, it pulls the latest load data of all online P-Nodes from Redis. The routing algorithm calculates a load score for each P-Node. This score is typically inversely proportional to the load metric. For example, a simple implementation is score = 1 / (1 + normalized load value). This means that nodes with lighter loads will have higher scores.

[0058] It should be noted that load balancing strategies can also be combined with other strategies (such as prefix cache hits). For example, the system can be set with a priority, first attempting to find nodes with prefix cache hits; if none are found, then selecting the lightest-loaded node. This approach ensures that tasks can be assigned to the healthiest node for execution even when caching advantages cannot be utilized.

[0059] The method by which the target decoding computation node selects the target pre-filled computation node is also decentralized. That is, the target decoding computation node broadcasts a request to multiple corresponding pre-filled computation nodes to perform score calculations. This request may or may not include a routing strategy. If a routing strategy is included, the pre-filled computation nodes, upon receiving the request, perform their own score calculations based on the specified routing strategy and feed the calculated score back to the target decoding computation node. If no routing strategy is included, the pre-filled computation nodes, upon receiving the request, perform score calculations based on the default routing strategy. After receiving the scores from multiple pre-filled computation nodes, the target decoding computation node selects the target pre-filled computation node from among them.

[0060] After identifying the target pre-filling computation node, the target decoding computation node sends the KV cache computation task to the target pre-filling computation node. The pre-filling scheduler of the target pre-filling computation node then sends the KV cache computation task to the pre-filling engine. The pre-filling engine performs the KV cache computation and obtains the KV cache. Then, it sends the generated KV cache to the target decoding computation node via the KV cache transmission interface, calling the KV cache transmission engine in the computation node. Upon receiving the KV cache, the target decoding computation node performs decoding computation and generates the token corresponding to the inference request.

[0061] Understandably, during model inference, the KV cache transfer engine runs as an independent subprocess, exchanging data handles with the main inference process via POSIX shared memory and communicating control signaling via Unix Domain Sockets. The main process monitors the health of the subprocess; if an abnormal exit is detected, a retry mechanism is triggered and the transfer process is restarted to ensure data integrity. The inference process is managed by the inference backend management process.

[0062] This application embodiment achieves efficient inference scheduling under a PD separation architecture by constructing a collaborative workflow of front-end-decoding computing node-prefilling computing node-decoding computing node and combining it with a two-level routing strategy. The first-level routing, using a decentralized scoring mechanism, precisely allocates requests to the optimal decoding node through a global request router, reducing the performance bottleneck and state lag risk of traditional single-point routing. The second-level routing is asynchronously completed within the decoding node by the scheduler, reducing first-word latency (TTFT) by selecting appropriate prefilling computing nodes. This two-level routing collaboration achieves fine-grained awareness and elastic scheduling of computing resources, decouples resource contention between the Prefill and Decode stages, and improves the overall utilization of GPU computing power and memory bandwidth.

[0063] The two levels of routing described above correspond to different scheduling targets. The first-level routing determines the target decoding compute node from multiple decoding compute nodes to accept the inference request. The second-level routing, based on the selected target decoding compute node, further determines the target pre-fill compute node from multiple pre-fill compute nodes to perform the Prompt pre-fill computation. The pre-fill compute node and the decoding compute node share the same policy pool, which may include policies such as CacheAware, Random, and MinLoad. The CacheAware policy focuses on Prompt prefix cache matching, the MinLoad policy focuses on node task load, and the Random policy randomly selects from candidate nodes.

[0064] In the default implementation, the target decoding compute node selection adopts the MinLoad strategy to reduce the load pressure during the decoding phase; this strategy can also be adjusted to CacheAware or other strategies during deployment via parameters. The decoding compute node selection adopts the CacheAware strategy to improve the Prompt cache reuse rate and reduce the first token latency (TTFT); this strategy is an internal scheduling behavior and can only be replaced by modifying the internal strategy configuration or code.

[0065] To support different strategies at different routing stages, a policy identifier can be carried in the routing or scoring request. When a router requests a score from a decoding compute node, it determines the policy identifier based on the current decoding route configuration, and the decoding compute node selects the appropriate strategy from the shared policy pool based on this identifier for score calculation. During the pre-filling compute node selection process, the decoding compute node can also use a preset pre-filled compute node filtering policy identifier (default is CacheAware, which can be modified via environment variables) to call the corresponding strategy from the shared policy pool to complete the target pre-filled compute node filtering.

[0066] Based on the above embodiments, the generation and transmission of KV caches will be described below: Figure 3 This is a schematic diagram of a key-value (KV) cache generation and transmission process provided in an embodiment of this application. When the target prefill node performs KV cache calculation, it calculates layer by layer according to the transformer layer of the large model. After completing the calculation of the KV cache of each layer, it can transmit the KV cache of that layer to the target decoding node and execute the KV cache calculation of the next layer in parallel.

[0067] When transferring the KV cache obtained from computation, the generated KV cache is temporarily stored in GPU memory (GPU KVCache). KVConnector is the interface layer between the framework (such as vLLM) and the transfer engine. By calling the save_kv_layer() method, the instruction and data handle for saving the current KV cache are passed to the lower-level transfer engine. The KV cache transfer engine (KVTransferEngine) is the actual data transfer executor. Based on the tensor parallel (TP) and pipeline parallel (PP) topology information of the target pre-filled computing node and the target decoding computing node, it directly transfers the KV cache located in GPU memory to the system memory (Host Memory) using technologies such as GPU Direct RDMA, avoiding the overhead of data copying via the CPU. Then, based on the TP / PP information, it determines the KV cache slices to be sent. After the data is ready, a metadata file containing data location, length, TP / PP slice information, etc., is generated. This metadata file is first sent to the target decoding computing node via a high-speed message queue (such as NATS, Redis Stream, ZeroMQ, etc.). It should be noted that the KV Cache can be transmitted between the target decoding computing node and the target pre-filling computing node via the TCP protocol.

[0068] It should be noted that the KV cache transfer engine runs embedded on the compute node. Upon startup, it creates an independent child process, binds to a random TCP port, and registers the IP:Port information with Redis's service discovery list. The main process transfers data handles via POSIX shared memory, while the child process is responsible for pushing the KV cache in the GPU memory to the target decoding compute node's KV cache service via TCP.

[0069] The target decoding compute node's decoding scheduler first receives metadata from the target pre-filled compute node. By parsing this metadata, it can obtain key information such as the size of the upcoming KV cache, the fragmentation method (how to reassemble by TP / PP), and the storage location. Then, it transmits the KV cache payload to the target decoding compute node's system memory. The target decoding compute node's KV cache transfer engine (KVTransferEngine) writes the received data blocks to shared memory based on the information in the metadata, and reorganizes the data according to the local TP / PP topology to conform to the local GPU's read format. The target decoding compute node's interface layer (KVConnector) calls the start_load_kv() method to notify the engine to start loading the KV cache. The KV cache is then loaded from system memory into GPU memory for use in subsequent decoding (autoregressive generation) stages.

[0070] It should be noted that when the target decoding computing node initializes the KV Cache for the first time, it collects the KV Cache tensor information (layer name, shape, number of KV headers, data type, etc.) of all layers of this node. After synchronization between multiple GPU ranks (TP / PP) of the same node, it publishes it to the global channel. The transmission engines of each peer node (pre-filled computing node) automatically register and update, realizing the slice alignment of KV headers across TP configurations. The embodiment of this application can support MLA and standard MHA attention layout.

[0071] Building upon the above embodiments, the front-end layer, core layer, and inference layer communicate via Redis. The core layer's request router subscribes to relevant channels in Redis, detects new requests in real time, reads the request content, and triggers a decentralized routing process. During routing, scoring requests are broadcast to all eligible decoding compute nodes via Redis's Pub / Sub mechanism. Each node independently calculates its score based on its local state and writes it back to Redis. Finally, the router aggregates the results in Redis and selects the target decoding compute node. Decoding compute nodes and pre-filling compute nodes obtain their assigned tasks from Redis and feed back information such as execution status and KV cache transmission progress to Redis for scheduling and monitoring. Furthermore, the online and offline status of all compute nodes, as well as node metadata (such as supported models, node types, load conditions, etc.), are registered and maintained in the Redis database. Other compute nodes can subscribe to Redis and obtain all available compute nodes, i.e., online compute nodes, in real time from the Redis database.

[0072] Furthermore, when a new compute node registers and comes online, its key-value (KV) transport engine starts a separate child process called KVCacheServer, which listens on a random TCP port. After the process starts, it updates the listening address (IP address and port) in Redis. The Redis database then updates the metadata of the newly online compute node. Other compute nodes can discover this compute node in real time through Redis subscription, thus achieving automatic discovery and automatic networking of the KV transport engines of each node.

[0073] This Redis-based communication architecture not only eliminates the configuration complexity and single point of failure issues caused by direct network connections in traditional architectures, but also naturally supports dynamic scaling, concurrent access of multiple instances, and deployment across network environments, greatly improving the system's reliability, flexibility, and operational efficiency.

[0074] In addition to using Redis's communication architecture, NATS, Kafka, or a custom message bus can also be used to replace Redis to achieve the same functionality.

[0075] Figure 4 This is a schematic diagram of a large model inference method provided in an embodiment of this application, such as... Figure 4 As shown, this inference method is implemented based on the large model inference system provided in the above embodiments, and the specific process includes: Step 401: After receiving the user's inference request, the front end sends the inference request to the request router.

[0076] Step 402: The requesting router determines the optimal routing node based on the received inference request. In some embodiments, the requesting router may generate a scoring request based on the inference request. The purpose of this scoring request is to allow the decoding compute nodes to calculate their respective scores. The requesting router may broadcast the scoring request to the decoding compute nodes, with the broadcast address containing the node type and model identifier. This scoring request will be received by the decoding compute nodes whose node type and model identifier match the broadcast address, and then the scoring will be calculated. When calculating the score, dynamic indicators such as the current GPU utilization, KV cache occupancy, and the number of pending requests can be combined to calculate a comprehensive score. After completing the score calculation, the score result is returned to the requesting router, which then determines the target decoding compute node based on the score result.

[0077] Step 403: Request the router to send an inference request to the target decoding node.

[0078] Step 404: After receiving the inference request, the target decoding computing node forwards it to the backend engine for preprocessing according to the interface of the inference backend. Preprocessing operations include parsing the inference request into a standard format that the backend inference engine can recognize; and converting the user's input prompts into a token sequence that the model can process using a word segmenter.

[0079] Step 406: After intercepting the inference request, the decoding scheduler of the target decoding compute node executes the route of the pre-filled compute node asynchronously and synchronously adds it to the request scheduling queue of the backend engine of the target decoding compute node.

[0080] In some embodiments, the target decoding computing node can determine the target pre-filled computing node based on a preset strategy, such as a prefix cache hit strategy or a pre-filled computing node load balancing strategy. For details on the prefix cache hit strategy and the pre-filled computing node load balancing strategy, please refer to the above embodiments, which will not be repeated here.

[0081] Step 407: When the decoding scheduler of the target decoding computing node obtains the inference request from the request scheduling queue, it calculates the number of cache blocks and offset of the external KV cache required for the request.

[0082] Step 408: After determining the target pre-filling computing node, the target decoding computing node sends a KV cache computing task to the target pre-filling computing node through Redis.

[0083] Step 409: The target decoding computing node pre-allocates GPU memory, which is used to store the KV cache obtained by the target pre-filled computing node.

[0084] Step 410: After receiving the KV cache computation task, the target pre-filled computation node adds the KV cache computation task to the scheduling queue of the backend engine of the target pre-filled computation node.

[0085] Step 411: In the KV cache transmission interface, determine the range of KV cache blocks to be transmitted based on the KV cache requirement information of the target decoding computing node.

[0086] Step 412: The pre-fill scheduler of the target pre-fill computing node schedules the KV cache computing task to the pre-fill engine, which then performs the KV cache computing to obtain the KV cache.

[0087] Step 413: The pre-filling engine sends the KV cache to the corresponding target decoding computing node through the KV cache transfer engine.

[0088] Step 414: The KV cache transmission interface in the target decoding computing node checks the cache reception status of all requests waiting for KV cache during each scheduling.

[0089] Step 415: If the target decoding compute node detects that the data from the target pre-filled compute node has been fully received, it will read it from the cache and write it to the specified location of the GPU memory pre-allocated by the decoding scheduler.

[0090] Step 416: Notify the decoding scheduler that the external KV cache for this request is ready. The target decoding computing node performs decoding computation based on the received KV cache and generates a token (streaming or full).

[0091] Step 417: The target decoding computing node returns the generated token to the front end of the request source via Redis.

[0092] This application embodiment achieves efficient inference scheduling under a PD separation architecture by constructing a collaborative workflow of front-end-decoding computing node-prefilling computing node-decoding computing node and combining it with a two-level routing strategy. The first-level routing, using a decentralized scoring mechanism, precisely allocates requests to the optimal decoding node through a global request router, reducing the performance bottleneck and state lag risk of traditional single-point routing. The second-level routing is asynchronously completed within the decoding node by the scheduler, reducing first-word latency by selecting appropriate prefilling computing nodes. This two-level routing collaboration achieves fine-grained awareness and elastic scheduling of computing resources, decouples resource contention between the Prefill and Decode stages, and improves the overall utilization of GPU computing power and memory bandwidth.

[0093] Based on the above embodiments, the decoding computing node can perform the scoring calculation in the following two ways: The first approach involves configuring a scoring calculation strategy within the decoding computing node during large model configuration. In the actual inference process, when the decoding computing node receives a scoring request, it performs scoring calculations based on this pre-configured scoring calculation strategy to obtain the scoring result.

[0094] The second approach involves configuring multiple scoring strategies within the decoding computing node during large model configuration. In the actual inference process, the scoring request sent by the request router to the decoding computing node carries a scoring strategy identifier. Upon receiving the scoring request, the decoding computing node selects the corresponding scoring strategy based on the carried scoring strategy identifier to perform scoring calculation and obtain the scoring result.

[0095] This application embodiment achieves flexible configuration and dynamic selection of routing strategies by carrying a scoring strategy identifier in the scoring request. Different scoring algorithms can be adopted according to actual needs (such as load balancing, cache affinity, minimum latency, etc.) in different scenarios, which improves the adaptability and intelligence of the system. At the same time, each decoding computing node independently performs local scoring according to a unified strategy, which not only ensures the consistency of scoring standards, but also avoids the performance bottlenecks and single point of failure risks caused by centralized strategy scheduling, thus enhancing the overall scalability and reliability of the system.

[0096] Building upon the above embodiments, the KV cache transmission engine of the target pre-filled computing node sends the KV cache to the target decoding computing node through an independent subprocess. This embodiment achieves decoupling of transmission and computation by setting up an independent KV cache transmission engine at the core layer and employing an independent subprocess to perform cross-node transmission of the KV cache. It efficiently exchanges KV tensor data between the inference and transmission processes via POSIX shared memory; uses TCP for cross-node transmission, does not rely on NCCL / GPU direct connections, and is suitable for ordinary Ethernet and heterogeneous clusters; transmission and inference computation can be performed asynchronously.

[0097] Building upon the above embodiments, when the target prefill computation node performs KV cache computation, it computes layer by layer according to the transformer layers of the large model. After completing the computation of each layer's KV cache, it transmits that layer's KV cache to the target decoding computation node and executes the next layer's KV cache computation in parallel. Thus, while the target prefill computation node computes the KV cache layer by layer, each completed layer's cache is sent to the target decoding computation node via an independent KV cache transmission engine, achieving pipelined parallelism of computation and transmission. This layer-by-layer transmission mechanism effectively overlaps the computation time of the Prefill stage and the network transmission time of the KV cache, significantly reducing the first-word latency caused by waiting for all caches to be generated, and improving the overall inference response speed.

[0098] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0099] Furthermore, the units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0100] Furthermore, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0101] In this document, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, without necessarily requiring or implying any such actual relationship or order between these entities or operations.

[0102] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A large-scale model reasoning system, characterized in that, include: The front-end layer is used to receive inference requests sent by the client; The core layer includes request routers and compute nodes; The computing nodes include pre-filled computing nodes and decoding computing nodes; The request router receives the inference request, generates a scoring request based on the inference request, and broadcasts the scoring request to the decoding computing node. The decoding computing node calculates the corresponding scoring result based on the scoring request and returns the scoring result to the requesting router; The requesting router determines the target decoding computing node based on the scoring result and sends the inference request to the target decoding computing node; The inference layer includes the decoding engine, decoding scheduler, pre-filling engine, and pre-filling scheduler; The decoding scheduler obtains the inference request from the target decoding computing node, determines the target pre-filled computing node based on a preset strategy, and sends the KV cache computing task of the inference request to the target pre-filled computing node; The pre-fill scheduler sends the KV cache calculation task to the pre-fill engine, and the pre-fill engine performs pre-fill calculation based on the KV cache calculation task to obtain the KV cache. The decoding engine performs decoding calculations based on the KV cache to generate a token.

2. The system according to claim 1, characterized in that, The core layer also includes a KV cache transmission engine; The KV cache transmission engine sends the KV cache calculated by the target pre-filled computing node to the target decoding computing node through an independent sub-process.

3. The system according to claim 2, characterized in that, The target pre-filled computing node calculates the KV cache layer by layer. After completing the KV cache calculation for one layer, it sends the KV cache of that layer to the target decoding computing node through the KV cache transmission engine.

4. The system according to claim 1, characterized in that, The front-end layer, the core layer, and the inference layer communicate via Redis.

5. The system according to claim 1, characterized in that, The requesting router broadcasts the rating request via Redis; the broadcast address includes the node type and model identifier. The scoring request is received by the decoding computing node that is adapted to the broadcast address.

6. The system according to claim 1, characterized in that, The front-end layer includes a task submitter and at least one client interface; the client sends inference requests through the client interface; the task submitter stores the inference requests in Redis.

7. The system according to claim 4, characterized in that, The Redis database stores node information for registered and online compute nodes; The compute nodes obtain all available compute nodes in real time through Redis subscription.

8. A large-scale model reasoning method, characterized in that, include: The router is requested to generate a scoring request based on the received inference request; The requesting router broadcasts the scoring request to the decoding computing node, so that the decoding computing node that receives the scoring request performs the scoring calculation and returns the scoring result to the requesting router; The requesting router determines the target decoding computing node based on the scoring result and sends the inference request to the target decoding computing node; The target decoding computing node determines the target pre-filling computing node based on a preset strategy, and sends the KV cache computing task corresponding to the inference request to the target pre-filling computing node; The target pre-filling computing node performs pre-filling computing based on the KV cache computing task to obtain the KV cache; The target decoding computing node receives the KV cache and performs decoding calculations based on the KV cache to generate a token.

9. The method according to claim 8, characterized in that, The scoring request includes a scoring strategy identifier; the decoding computing node calculates the corresponding scoring result based on the scoring request, including: The decoding computing node calculates the score based on the scoring strategy identifier and the corresponding scoring strategy to obtain the score result.

10. The method according to claim 8, characterized in that, The target decoding computing node receives the KV buffer, including: The KV cache transmission engine of the target pre-filled computing node sends the KV cache to the target decoding computing node through a separate sub-process; The target decoding computing node receives the KV cache.

11. The method according to claim 8, characterized in that, The KV cache transmission engine of the target pre-filled computing node sends the KV cache to the target decoding computing node through an independent sub-process, including: Each time the target pre-filled computing node obtains a KV cache layer, it sends the KV cache layer to the target decoding computing node through an independent subprocess and executes the next layer of KV cache calculation in parallel.

12. The method according to claim 8, characterized in that, The requesting router broadcasts the scoring request to the decoding computing node, including: The requesting router broadcasts the scoring request to the decoding computing node via Redis.

Images