A restricted block-level kv reuse method and system for RAG reasoning

Abstract

The present application relates to a kind of restricted block level KV multiplexing method and system for RAG reasoning, belong to big language model technical field;The present application divides the search text block into the core area of query key and reusable background area, executes different areas by area perception prefilling path, and reusable background KV object is managed by online runtime;While retaining background text in prompt, it enables repeated non-prefix background block to be safely reused within the clearly defined execution boundary.The present application can be applied to big language model server, RAG intelligent question and answer system and cloud artificial intelligence reasoning platform, in long context, the reasoning scene of high repeat search content, effectively reduce model reasoning calculation cost, improve system response speed and service throughput capacity.

Description

Technical Field

[0001] This invention relates to a restricted block-level key-value reuse method and system for RAG reasoning, belonging to the field of large language model technology. Background Technology

[0002] Online large language model services are shifting from short dialogue prompts to longer contexts and more structured request assembly methods; Retrieval Augmentation (RAG) has been widely used in knowledge-intensive applications; meanwhile, tool calls and agent workflows such as ReAct, Toolformer, and Voyager are increasingly placing retrieval evidence, tool specifications, skill descriptions, and external memory fragments within the same context. In these scenarios, the service system repeatedly processes prompts consisting of multiple text blocks; these text blocks themselves are relatively stable but are reorganized between different requests. As the context length and the number of text blocks increase, the system repeatedly performs pre-filling on prompts with significant content overlap, resulting in a continuously increasing overhead for key-value (KV) computation, writing, and residency.

[0003] Key-value (KV) reuse has become one of the main optimizations for reducing the pre-filling cost of long-hint inference; representative systems such as vLLM and SGLang primarily rely on prefix-based reuse; they identify the longest common prefix across requests and directly reuse the KV representation of that prefix. This design is very effective when reusable content remains within the same preceding context; however, the mechanism fails once overlapping content is reorganized. In online RAGs, tool calls, and agent workflows, retrieval blocks and auxiliary text are often rearranged, inserted, replaced, or pruned only after the request arrives. Therefore, the key issue is not whether the content overlaps, but whether these overlaps still appear in the form of reusable prefixes.

[0004] Non-prefix methods relax this requirement and expand reuse opportunities; CacheBlend restores reuse in non-prefix mismatch scenarios through cache knowledge fusion and selective recomputation. EPIC organizes reusable objects through a location-independent cache path similar to compile / link. These designs address important parts of the problem, but they respectively assume selective recomputation or precomputation of cached objects; in contrast, FlexKV constructs reusable objects directly on the online RAG service path. In online RAG services, valid hints are typically assembled only after the request arrives. This requires that reuse paths be formed directly on the online inference path, without relying on offline cache construction or selective recomputation. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a restricted block-level key-value (KV) reuse method and system for RAG inference. It divides the retrieved text block into a core region of query key information and a reusable background region. Different regions are executed through region-aware pre-filled paths, and reusable background KV objects are managed by the online runtime. While retaining background text in the prompts, it enables the safe reuse of recurring non-prefix background blocks within clearly defined execution boundaries. This invention is based on SGLang. On publicly synchronized RAG workloads, this invention maintains task quality within small quality gaps and achieves lower P99 first-token latency (TTFT) than vLLM and CacheBlend on most model-dataset combinations. In controlled, trajectory calibration request flows containing recurring non-prefix blocks, online cache hits can be accumulated. At 40% and 60% reuse rates, its final window P99 TTFT is reduced by 32.9% and 70.9% compared to SGLang, respectively.

[0006] The technical solution of the present invention is as follows:

[0007] The first aspect of this invention provides a restricted block-level key-value multiplexing method for RAG inference, comprising:

[0008] Step 1: The server receives the retrieval enhancement generation inference request sent by the client, including user query information, system prompt information, and retrieval text block sequence;

[0009] The search text block sequence is encoded to obtain a set of token representations, and the set of token representations is sorted according to the relevance between the set of token representations and the user query information.

[0010] Step 2: Divide the sorted set of token representations into a core area and a background area;

[0011] Step 3: The server performs pre-fill calculations based on the divided core area and background area, and stores the results of the background area pre-fill calculation into the KV cache database; it checks whether there is a corresponding cached object in the KV cache database. If there is, it adjusts the position mapping of the token; otherwise, it performs restricted background path pre-fill calculations.

[0012] Step 4: Perform online management of the KV cache database; merge the cached objects of the reused background area with the generated results of the core area to generate context KV state, use the context KV state to complete the retrieval enhancement generation, and send the results to the client.

[0013] According to a preferred embodiment of the present invention, the server receives a retrieval enhancement generation inference request sent by the client, including user query information, system prompt information, and a sequence of retrieval text blocks; including:

[0014] The server receives a retrieval enhancement generation RAG inference request from the client, as shown below:

[0015] ;

[0016] in, S represents a reasoning request, Q represents a system prompt, and D represents a sequence of retrieved text blocks.

[0017] Specifically, the retrieval text block sequence D uses user query information Q and system prompt information S to search the external knowledge base and obtain a set of candidate documents, as shown below:

[0018] ;

[0019] in, Let m represent the set of candidate documents, and m represent the number of candidate documents. This represents the m-th candidate document;

[0020] Each candidate document is segmented according to a preset length, and boundary processing is performed, i.e., the segmentation position is adjusted, so that each text block maintains semantic integrity, resulting in the segmented text blocks as shown below:

[0021] ;

[0022] in, Let represent the set of text blocks after segmenting the i-th candidate document. This represents the nth text block in the set of i-th text blocks;

[0023] After segmenting all candidate documents, the resulting retrieval text block sequence D={ }

[0024] According to a preferred embodiment of the present invention, the retrieval text block sequence is encoded to obtain a token representation set, and the token representation set is sorted according to its relevance to the user query information; including:

[0025] Each text block set is segmented by a word segmenter. Each text block in the code is segmented and encoded into a discrete token sequence:

[0026] ;

[0027] in, Represents the discrete token sequence of the j-th text block. This represents the r-th token being converted;

[0028] At the same time, record the location index corresponding to each token:

[0029] ;

[0030] in, Represents a set of location indices. This indicates the position of the r-th token within the text block;

[0031] Construct a text block token representation:

[0032] ;

[0033] in, The token representation of the j-th text block. This indicates the number of the j-th text block;

[0034] After converting all text blocks to token representations, we obtain a set of token representations:

[0035] ;

[0036] in, Token represents a set. The token representation of the nth text block;

[0037] The token is calculated as an evaluation value, or relevance, between itself and the user's query information Q, as shown below:

[0038] ;

[0039] in, Represents the evaluation function;

[0040] Sort the evaluation values ​​represented by each Token in descending order to obtain the sorted set of Token representations:

[0041] ;

[0042] in, This indicates that the sorted tokens represent a set. This represents the nth token after sorting.

[0043] According to a preferred embodiment of the present invention, step 2 includes the following specific implementation process:

[0044] The server sets a core ratio η, and selects the top k sorted tokens based on this core ratio as the core region, as shown below:

[0045] ;

[0046] ;

[0047] Where k represents the number of core area tokens, and n represents the number of text blocks. Indicates the core area. This represents the k-th token after sorting.

[0048] The remaining tokens will be used as the background area:

[0049] ;

[0050] in, Indicates the background area. This represents the (k+1)th token after sorting. This represents the nth token after sorting.

[0051] Further optimized, the core ratio η is set to η=0.4;

[0052] According to a preferred embodiment of the present invention, step 3 specifically includes the following steps:

[0053] For the core area, the token representation of the core area and user query information will be used. Input the large language model, which performs pre-filling computation according to the causal attention mechanism of the Transformer network, generates the corresponding Key matrix and Value matrix, and calculates the attention weights between each Token;

[0054] For the background region, the token representation of the background region is input into the large language model. The large language model performs pre-filling calculation according to the causal attention mechanism of the Transformer network to generate the corresponding key matrix and value matrix, which are stored in the KV cache database together with the cache identification information. The cache identification information includes: text block content identifier, system prompt template, region execution rule, and model version.

[0055] The text block content identifier represents the background area; the token represents the generated unique identifier. ;in, () represents a hash function. The j-th token represents the background area. The text content;

[0056] The system prompt template represents the current system prompt message S;

[0057] The region execution rules represent the calculation rules used when generating the KV (Key and Value) matrix, including core proportions and pre-filling calculation methods;

[0058] The model version refers to the current inference model, i.e., the version of the large language model;

[0059] A cache index is generated based on the background region token representation, as shown below:

[0060] );

[0061] in, Indicates the cache index; S indicates a system prompt message. Indicates the rules to be enforced in the region. Indicates the model version;

[0062] Based on the cache index, query the corresponding KV cache object, i.e., the Key-Value matrix, in the KV cache database;

[0063] If the query result exists, the corresponding Key matrix and Value matrix are read, and the cached Key is adjusted by RoPE location encoding based on the position index of the background region Token in the current RAG inference request in the input.

[0064] If the query result does not exist, it is determined that there is no reusable historical KV cache object in the current background area. Restricted background path pre-filling calculation is adopted, including: when performing pre-filling calculation and generating the Key-Value matrix in the Transformer network, restricting the information interaction between tokens in the background area through causal attention mask, so that each token only establishes attention connection with the current token representation and the token representation of the preceding position; and retaining the information interaction between the token representation and the fixed system prompt, that is, the token transformed from the system prompt information, and removing or reducing the information interaction between the token representation of the background area and the token representation corresponding to the user query information.

[0065] After pre-filling the restricted background path, the corresponding Key and Value matrices are generated, and the generated Key and Value matrices and corresponding cache indexes are stored in the KV cache database.

[0066] According to a preferred embodiment of the present invention, step 4 includes the following specific implementation process:

[0067] When the cache space of the KV cache database reaches the preset capacity threshold, the server performs deletion operations on low-value cache objects based on cache access frequency, historical hit count, and resource consumption. This includes deleting cache objects that meet at least one of the following conditions within the KV cache database management cycle: not accessed within a preset time; historical hit count is lower than the preset threshold; resource consumption cache capacity exceeds twice the average cache object capacity, but the hit count is lower than the average level; the corresponding model version, system prompt template, or region execution rule has expired.

[0068] When a cached object in the KV cache database meets all the preset reuse conditions, the cached object is loaded into the current RAG inference request; the preset reuse conditions include:

[0069] The token representation in the KV cache database is consistent with the token representation in the background area;

[0070] The system prompt information in the KV cache database is consistent with the system prompt information of the current RAG inference request;

[0071] The region execution rules in the KV cache database are consistent with the region execution rules corresponding to the current RAG inference request;

[0072] The model version in the KV cache database is consistent with the large language model version of the current RAG inference request;

[0073] The cache creation time, last access time, access count, and current valid status of the KV cache database meet the preset requirements.

[0074] The server concatenates or rearranges the Key and Value of the background region that are cached in the KV cache database with the Key and Value of the core region generated in real time to form a complete context KV state. The context KV state is then used for decoding to complete the RAG inference of the large language model.

[0075] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the restricted block-level KV reuse method for RAG inference described above.

[0076] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the restricted block-level KV multiplexing method for RAG inference described above.

[0077] A second aspect of the present invention provides a restricted block-level key-value multiplexing system for RAG inference, comprising:

[0078] The Token represents the collection acquisition module, which is configured to: receive retrieval enhancement generation inference requests sent by the client, including user query information, system prompt information, and retrieval text block sequences;

[0079] The search text block sequence is encoded to obtain a set of token representations, and the set of token representations is sorted according to the relevance between the set of token representations and the user query information.

[0080] The region division module is configured to divide the sorted set of token representations into a core region and a background region.

[0081] The pre-filling calculation module is configured as follows: the server performs pre-filling calculation based on the divided core area and background area, and stores the result of the background area pre-filling calculation into the KV cache database; it queries whether there is a corresponding cached object in the KV cache database. If it exists, the token position is adjusted by the position mapping; otherwise, the restricted background path pre-filling calculation is performed.

[0082] The result fusion module is configured to: manage the KV cache database online; fuse the cached objects of the reused background area with the generated results of the core area to generate context KV state; use the context KV state to complete the retrieval enhancement generation; and send the results to the client.

[0083] The beneficial effects of this invention are as follows:

[0084] 1. This invention divides the search text blocks in the RAG request into core and background regions, and adopts differentiated reasoning processing for different regions to maintain the complete reasoning path of important texts related to the user query. At the same time, background text blocks that meet the reuse conditions are converted into block-level KV cache objects that can be managed and reused online, thereby reducing the overhead of repeated calculations while ensuring the quality of generation.

[0085] 2. This invention establishes a text block-level KV cache management mechanism to index, match, register, update, and manage the KV-Cache data generated in historical requests. This allows different requests to reuse existing KV cache data based on text block content and execution conditions without maintaining a continuous prefix relationship, thereby improving the utilization efficiency of KV cache resources.

[0086] 3. This invention adopts a region-aware pre-filling mechanism, which selects different calculation paths according to the text region type. For background text blocks that are cached, the corresponding KV cache data is directly loaded. For background text blocks that are not cached, restricted pre-filling calculation is performed, thereby reducing the amount of repeated pre-filling calculation in the large language model inference process, reducing the delay of first token generation, and improving the inference efficiency of the large language model.

[0087] 4. By introducing restricted reuse conditions, this invention uses text block content, system prompt template, region execution rules, and model status as the criteria for KV cache reuse judgment, thereby avoiding the incorrect reuse of KV caches that do not meet the conditions and improving the stability and reliability of the cache reuse process.

[0088] 5. This invention can be applied to large language model servers, RAG intelligent question answering systems, and cloud-based artificial intelligence inference platforms. In inference scenarios with long contexts and highly repetitive retrieval content, it can effectively reduce the computational cost of model inference and improve system response speed and service throughput. Attached Figure Description

[0089] Figure 1 This is an architecture diagram of the restricted block-level KV reuse method for RAG inference according to the present invention;

[0090] Figure 2 This is a schematic diagram of the window-level results under a 20% reuse rate of non-prefixed text blocks for different targets according to the present invention;

[0091] Figure 3 This is a schematic diagram of the window-level results under a 40% reuse rate of different target non-prefixed text blocks in this invention.

[0092] Figure 4 This is a schematic diagram of the window-level results under a 60% reuse rate of different target non-prefixed text blocks in this invention.

[0093] Figure 5 This is a schematic diagram of the average end-to-end throughput of the window under a 20% reuse rate of different target non-prefixed text blocks according to the present invention.

[0094] Figure 6 This is a schematic diagram of the average end-to-end throughput of the window under a 40% reuse rate of different target non-prefixed text blocks in this invention.

[0095] Figure 7 This is a schematic diagram of the average end-to-end throughput of the window under different target non-prefix text block reuse rates of 60% according to the present invention. Detailed Implementation

[0096] The present invention will be further described below with reference to the embodiments and accompanying drawings, but is not limited thereto.

[0097] Example 1

[0098] Terminology Explanation:

[0099] 1. Retrieval Augmentation (RAG): This is a technique that combines information retrieval with large language model generation. By retrieving relevant information from external knowledge bases and using it as context input to the model, it enhances the accuracy, timeliness, and professionalism of the model's answers and reduces generation illusions.

[0100] 2. Key-Value Cache (KV Cache): This is a core memory optimization mechanism in the autoregressive decoding inference process of large language models. When the Transformer network generates text token by token, it caches the attention key and value vectors calculated at each step. When generating new tokens, it is not necessary to repeatedly calculate the KV vectors of historical tokens; attention calculation can be completed simply by concatenating the KV vectors of the new token, thus avoiding redundant calculations and significantly improving inference efficiency. However, as the context length increases, the memory usage of the KV Cache increases linearly, making it the core memory bottleneck for long text inference.

[0101] 3. RoPE Position Encoding: This is a position encoding method based on rotation transformation, used to introduce token position information into the key and query representations during the Transformer attention calculation process, enabling the model to perceive the relative position information between different tokens.

[0102] A restricted block-level key-value reuse method for RAG inference, such as Figure 1 As shown, it includes:

[0103] Step 1: The server receives the retrieval enhancement generation inference request sent by the client, including user query information, system prompt information, and retrieval text block sequence;

[0104] The search text block sequence is encoded to obtain a set of token representations, and the set of token representations is sorted according to the relevance between the set of token representations and the user query information.

[0105] Step 2: Divide the sorted set of token representations into a core area and a background area;

[0106] Step 3: The server performs pre-fill calculations based on the divided core area and background area, and stores the results of the background area pre-fill calculation into the KV cache database; it checks whether there is a corresponding cached object in the KV cache database. If there is, it adjusts the position mapping of the token; otherwise, it performs restricted background path pre-fill calculations.

[0107] Step 4: Perform online management of the KV cache database; merge the cached objects of the reused background area with the generated results of the core area to generate context KV state, use the context KV state to complete the retrieval enhancement generation, and send the results to the client;

[0108] Example 2

[0109] The difference between the restricted block-level KV reuse method for RAG reasoning described in Example 1 and the method described in Example 1 is as follows:

[0110] The server receives a retrieval enhancement generation inference request sent by the client, including user query information, system prompts, and a sequence of retrieval text blocks; including:

[0111] The server receives a retrieval enhancement generation RAG inference request from the client, as shown below:

[0112] ;

[0113] in, S represents a reasoning request, S represents a system prompt (e.g., S: "You are a medical knowledge Q&A assistant, please answer the user's question based on the provided information"), Q represents the user's query information (e.g., Q: "Is metformin suitable for patients with early-stage diabetes?"), and D represents a sequence of retrieved text blocks (e.g., D: D1: Descriptions of diabetes treatment methods in medical papers; D2: Indications and contraindications in drug instructions; D3: Clinical case analysis data).

[0114] Specifically, the retrieved text block sequence D is used to generate a RAG through retrieval enhancement. This RAG then uses user query information Q and system prompt information S to search an external knowledge base, obtaining a candidate document set, as shown below:

[0115] ;

[0116] in, Let m represent the set of candidate documents, and m represent the number of candidate documents. This represents the m-th candidate document; each candidate document includes text content and corresponding document identification information.

[0117] Each candidate document is segmented according to a preset length (dynamically configured based on the size of the large language model's context window, input constraints, and task requirements, such as a preset length of 128 to 2048 tokens). Boundary processing is then performed, adjusting the segmentation positions to ensure that each text block maintains semantic integrity. (For example, when the preset text block length is 128 tokens, if the 128th token happens to be in the middle of a complete sentence, direct segmentation would truncate the sentence's semantics. Therefore, the server can search forward or backward within a preset range for the nearest period, semicolon, paragraph end mark, list boundary, or table row boundary, and adjust the original segmentation position to that semantic boundary. For example, if the 120th token is the end of a sentence, the original 128th token segmentation position can be moved forward to the 120th token, ensuring that the previous text block ends with a complete sentence and the next text block begins with the next complete sentence, thus preserving the semantic and structural integrity of each text block as much as possible.) The resulting segmented text blocks are shown below:

[0118] ;

[0119] in, Let represent the set of text blocks after segmenting the i-th candidate document. This represents the nth text block in the set of i-th text blocks;

[0120] After segmenting all candidate documents, the resulting retrieval text block sequence D={ }

[0121] The retrieved text block sequence is encoded to obtain a token representation set, which is then sorted according to its relevance to the user's query information; this includes:

[0122] Each text block set is segmented by a word segmenter. Each text block in the code is segmented and encoded into a discrete token sequence:

[0123] ;

[0124] in, Represents the discrete token sequence of the j-th text block. This represents the r-th token being converted;

[0125] At the same time, record the location index corresponding to each token:

[0126] ;

[0127] in, Represents a set of location indices. This indicates the position of the r-th token within the text block;

[0128] Construct a text block token representation:

[0129] ;

[0130] in, The token representation of the j-th text block. This indicates the number of the j-th text block;

[0131] After converting all text blocks to token representations, we obtain a set of token representations:

[0132] ;

[0133] in, Token represents a set. The token representation of the nth text block;

[0134] The token is calculated as an evaluation value, or relevance, between itself and the user's query information Q, as shown below:

[0135] ;

[0136] in, Represents the evaluation function (such as BM25 score, vector semantic similarity score);

[0137] Sort the evaluation values ​​represented by each Token in descending order to obtain the sorted set of Token representations:

[0138] ;

[0139] in, This indicates that the sorted tokens represent a set. This represents the nth token after sorting.

[0140] The specific implementation process of step 2 includes:

[0141] The server sets a core ratio η, and selects the top k sorted tokens based on this core ratio as the core region, as shown below:

[0142] ;

[0143] ;

[0144] Where k represents the number of core area tokens, and n represents the number of text blocks. Indicates the core area. This represents the k-th token after sorting.

[0145] The remaining tokens will be used as the background area:

[0146] ;

[0147] in, Indicates the background area. This represents the (k+1)th token after sorting. This represents the nth token after sorting.

[0148] The core area is used to ensure the quality of model generation, while the background area is used to build block-level key-value cache objects that can be reused across requests.

[0149] The core ratio η ranges from 0 to 1, for example, the core ratio η is η=0.4.

[0150] The specific implementation process of step 3 includes:

[0151] For the core area, the token representation of the core area and user query information will be used. Input a large language model (such as Llama-3.1-8B-Instruct or Qwen3-8B), and the large language model performs pre-filling computation according to the causal attention mechanism of the Transformer network to generate the corresponding Key matrix and Value matrix, and calculate the attention weights between each Token;

[0152] For the background region, the token representation of the background region is input into the large language model. The large language model performs pre-filling calculation according to the causal attention mechanism of the Transformer network to generate the corresponding key matrix and value matrix, which are stored in the KV cache database together with the cache identification information. The cache identification information includes: text block content identifier, system prompt template, region execution rule, and model version.

[0153] The text block content identifier represents the background area; the token represents the generated unique identifier. ;in, () represents a hash function. The j-th token represents the background area. The text content used to determine whether the current background block corresponds to a historical cache object;

[0154] The system prompt template represents the current system prompt message S; it is used to ensure that different system prompts do not use the same key-value pairs.

[0155] The region execution rules represent the calculation rules used when generating the KV (Key and Value) matrix, including core proportions and pre-filling calculation methods;

[0156] The model version indicates the current inference model, i.e., the version of the large language model (e.g., Llama-3.1-8B-Instruct).

[0157] A cache index is generated based on the background region token representation, as shown below:

[0158] );

[0159] in, Indicates the cache index; S indicates a system prompt message. Indicates the rules to be enforced in the region. Indicates the model version;

[0160] Based on the cache index, query the corresponding KV cache object, i.e., the Key-Value matrix, in the KV cache database;

[0161] If the query result exists, the corresponding Key matrix and Value matrix are read, and the cached Key is adjusted by RoPE location encoding based on the position index of the background region Token in the current RAG inference request in the input, so that the cached data can be adapted to the current context position.

[0162] If the query result is not found, it is determined that there is no reusable historical KV cache object in the current background area. Restricted background path pre-filling computation is then employed, including: when performing pre-filling computation and generating the Key-Value matrix in the Transformer network, using a causal attention mask to restrict information interaction between tokens in the background area, ensuring that each token only establishes an attention connection with its current token representation and the token representation in its preceding position; retaining information interaction between token representations and fixed system prompts (i.e., tokens transformed from system prompt information), and removing or reducing information interaction between background area token representations and the token representation corresponding to the user's query information; thus preventing the generated KV-Cache from changing with user questions, thereby improving the reusability of the background area KV-Cache across different inference requests.

[0163] After pre-filling the restricted background path, the corresponding Key and Value matrices are generated, and the generated Key and Value matrices and corresponding cache indexes are stored in the KV cache database.

[0164] The specific implementation process of step 4 includes:

[0165] When the cache space of the KV cache database reaches a preset capacity threshold (the preset capacity threshold can be configured according to system hardware resources, for example, set to 70% to 90% of the available GPU cache space, or set to a preset number of KV-Cache objects), the server performs a deletion operation on low-value cache objects based on cache access frequency (the number of times each cache object is queried and loaded within a preset time window recorded during online KV reuse runtime), historical hit status (the number of times and the hit ratio of cache objects in historical inference requests), and resource usage (the monitored video memory space, memory space, and corresponding KV matrix size occupied by cache objects). This includes deleting cache objects that meet at least one of the following conditions within the KV cache database management cycle: not accessed within a preset time (10 minutes to 1 hour); historical hit count is lower than the preset threshold (5 times); resource usage cache capacity exceeds twice the average cache object capacity, but the hit count is lower than the average level; the corresponding model version, system prompt template, or region execution rule has expired.

[0166] When a cached object in the KV cache database meets all the preset reuse conditions, the cached object is loaded into the current RAG inference request; the preset reuse conditions include:

[0167] The token representation in the KV cache database is consistent with the token representation in the background area;

[0168] The system prompt information in the KV cache database is consistent with the system prompt information of the current RAG inference request;

[0169] The region execution rules in the KV cache database are consistent with the region execution rules corresponding to the current RAG inference request;

[0170] The model version in the KV cache database is consistent with the large language model version of the current RAG inference request;

[0171] The cache creation time, last access time, access count, and current valid status of the KV cache database meet the preset requirements.

[0172] The server concatenates or rearranges the Key and Value of the background region that are cached (reused) in the KV cache database with the Key and Value generated in real time in the core region (according to the Transformer level and Token position requirements) to form a complete context KV state. The context KV state is then used for decoding to complete the RAG inference of the large language model.

[0173] Experimental setup:

[0174] (1) Implementation: FlexKV is implemented based on SGLang and includes: a text block orchestration module, a region-aware pre-filling operator, and an online KV reuse runtime. Unless otherwise specified, all methods use the same model weights, tokenizer, dialogue templates, decoding strategies, and batching strategies. This setup allows the evaluation to focus on the differences in reuse mechanisms and runtime design.

[0175] The evaluation platform was a single-node 4-GPU inference server, consisting of 4 × NVIDIA A100 GPUs, 196 GB of host memory, and one Intel Xeon Platinum 8358 CPU. All experiments were run on the same hardware and software stack.

[0176] (2) Datasets: Public workloads include 2WikiMultihopQA, HotpotQA, MuSiQue, Pub-MedQA, and a Macro summarization trajectory. They cover multi-hop question answering, long contextual retrieval question answering, medical question answering, and summarization. HotpotQA employs both Top-10 and Top-50 retrieval settings to cover both short and long evidence budgets. These workloads were used to evaluate the baseline quality and latency behavior of FlexKV on public RAG inputs.

[0177] Non-prefix block reuse workloads were constructed to evaluate FlexKV's block-level key-value reuse capabilities. These workloads controlled the proportion of reusable blocks across requests to 20%, 40%, and 60%, while varying block combinations and positions to avoid forming identical consecutive prefixes. Therefore, they exposed cross-request block reuse opportunities that are difficult to utilize with traditional prefix caching but can be recognized by FlexKV.

[0178] (3) Models: The base models are Llama-3.1-8B-Instruct and Qwen3-8B. They differ in their training formulas and tokenizers, but both use standard dialogue templates and RoPE-style positional encoding. Therefore, they are suitable for testing whether the positional decoupling reuse path of FlexKV can remain effective across the model family.

[0179] (4) Metrics: Simultaneously report latency, throughput, task quality and block hit rate metrics, because the core claim of FlexKV is not unconditional acceleration, but to achieve a limited latency-quality trade-off with background path reuse.

[0180] (5) Baselines: Major latency-quality comparisons use vLLM, SGLang, and CacheBlend. vLLM and SGLang represent prefix-based reuse: vLLM performs hash-based block-level prefix matching, while SGLang uses RadixAttention, maintaining a radix tree at runtime to look up, insert, and evict shared prefixes. CacheBlend represents recomputation-based non-prefix reuse for RAG, reducing TTFT through cached knowledge fusion and selective recomputation; unless otherwise specified, its default 15% recomputation ratio is used. EPIC represents a location-independent cache via a service path similar to compile / link; due to model-related quality anomalies in the environment, the latency discussion in the main text does not depend on EPIC-Llama.

[0181] Top-k selection:

[0182] First, a Top-k sensitivity study was conducted to determine the default core block ratio for subsequent experiments. This study enumerated multiple Top-k candidates under the same model, retrieval results, decoding parameters, and runtime configuration, and simultaneously reported the generation quality and P99 TTFT.

[0183] From a generation quality perspective, increasing the Top-k value moves more highly relevant blocks into the core region and involves them in full attention, thus F1 typically improves with increasing Top-k. The improvement stabilizes around 40%, after which marginal quality gains are limited. In contrast, the P99 TTFT does not exhibit a simple, monotonic pattern. When Top-k is small, more text blocks remain in the background path and enter the cache management pipeline; the overhead of block lookup, registration, movement, and eviction may mask the gains from reduced attention computation. When Top-k is large, more text blocks are processed by full attention, increasing computation and latency. Therefore, neither too small nor too large a Top-k can achieve the optimal joint trade-off between quality and tail latency.

[0184] Taking into account both the stability of the generated quality and the observed P99 TTFT changes, Top-k=40% was used as the default configuration in subsequent experiments. This value is not optimal point-by-point on every dataset, but it provides a balanced operating point that can maintain stable generated quality without significantly increasing inference latency.

[0185] Synchronizing workload results:

[0186] We evaluate the basic behavior of FlexKV on the public RAG dataset under a synchronous workload. In this setting, each worker thread processes its assigned samples sequentially, and requests are not intentionally crafted to share repetitive text blocks. Therefore, the actual non-prefix block reuse rate is low. The purpose of this experiment is not to prove that FlexKV consistently achieves the lowest latency on all public datasets, but rather to examine whether it can reduce unnecessary pre-padding computations while preserving generation quality. To isolate the service mechanism itself, the hardware, software stack, decoding strategy, and batching strategy are fixed, and asynchronous arrival and queuing effects are avoided.

[0187] Table 1 presents the synchronous workload results on the RAG dataset; each system unit reports P99 TTFT / task quality, where P99 is in ms; the quality is F1 on the QA dataset and ROUGE-L on the Macro summary track; CacheBlend uses a 15% recompute configuration.

[0188] Table 1. Results of Synchronization Workload;

[0189]

[0190] Table 2 discloses aggregated system metrics for synchronous workloads. Values ​​are the mean for each model across six datasets; latency is in milliseconds (ms), and throughput is in tokens per second (tokens / s); FlexKV uses Top-k=40%.

[0191] Table 2. Aggregation System Indicators;

[0192]

[0193] Table 2 supplements the per-dataset P99 TTFT / quality table with aggregated end-to-end latency and generation throughput. These publicly synchronized workloads contain almost no artificially constructed non-prefix repetitions, so this table should be considered a compatibility test rather than the results of primary multiplexing stress. In this setup, FlexKV approaches SGLang in TTFT and improves average generation throughput by 30.8% and 19.2% on Llama and Qwen, respectively; end-to-end latency is not always lower, as decoding behavior and output length also have an impact.

[0194] Table 1 shows that under public synchronous workloads, FlexKV primarily reduces the P99 TTFT while maintaining generation quality close to the strongest baseline on most model-dataset combinations, although quality differences are dataset-dependent. Compared to vLLM, FlexKV achieves lower P99 TTFT on 11 out of 12 model-dataset combinations, with an average reduction of 20.6% and a maximum reduction of 34.3%. Compared to CacheBlend, FlexKV reduces P99 TTFT on all 12 combinations, with an average reduction of 37.5% and a maximum reduction of 68.5%. In terms of generation quality, FlexKV's average task metric difference relative to vLLM and CacheBlend is 0.009 and 0.013, respectively; relative to vLLM, the quality decrease is less than 0.03 on 11 combinations, and relative to CacheBlend, the quality decrease is less than 0.03 on 10 combinations, and FlexKV achieves or improves quality on several combinations. This indicates that latency reduction on most public synchronous workloads is not achieved by significantly sacrificing generation quality.

[0195] The dataset-by-dataset results indicate that the benefits of FlexKV are determined more by the workload structure than solely by the dataset identity. On shorter or weakly repetitive public trajectories such as Hotpot-10 and Macro, FlexKV primarily maintains compatibility during runtime: its P99 TTFT is close to the strongest prefix baseline with a small quality gap, but non-prefix KV reuse opportunities are limited. On longer trajectories with heavier pre-padding overhead, such as Hotpot-50, PubMed, and MuSiQue, FlexKV has more room to reduce redundant pre-padding computations, although the final quality depends on whether the selected core block covers the answer evidence. This interpretation connects the public synchronization workload to the controlled non-prefix reuse experiments described below: the benefits of FlexKV should be most pronounced when long context requests contain repetitive text blocks that cannot be aligned to a stable shared prefix.

[0196] Meanwhile, region-aware organization does not necessarily reduce generation quality. Long-sequence sparse attention studies show that when key global connections are preserved and the attention scope is structurally constrained, models can reduce attention costs while maintaining performance on long document tasks. In experiments, FlexKV achieves higher task quality metrics than SGLang on Qwen Hotpot-10, Qwen Macro, Qwen PubMed, and Llama Hotpot-10, with gains ranging from 0.004 to 0.024; ROUGE-L is used for Macro, and F1 is achieved on the QA dataset. It also reaches or surpasses vLLM and CacheBlend on several combinations. One possible explanation is that core regions retain full attention on the main evidence, while less relevant background patches are constrained, thus introducing less noise. These gains are relatively small, therefore they can be seen as evidence of quality preservation rather than a stable quality-enhancing mechanism.

[0197] Non-prefix text block reuse results:

[0198] The target scenario for evaluating FlexKV is: the same text blocks appear repeatedly in long request flows, but do not form consecutive shared prefixes. This scenario comes from RAG service workload research, not from an arbitrarily designed stress pattern. RAGCache observed skewed retrieval locality on MMLU, Natural Questions, HotpotQA, and TriviaQA; under its retrieval settings, the top 3% of documents served 60% of MMLU requests. Cache-Craft further reported block-level duplication in production and benchmark evidence: in its Sys-X and MuSiQue analyses, the top 5% of text blocks were accessed by approximately 60% of requests, compared to approximately 40% on 2WikiMQA; it also found that exact prefix caching only covers a small fraction of production RAG requests. Inspired by these observations, controlled request flows with target text block reuse rates of 20%, 40%, and 60% were constructed to represent low, medium, and high locality mechanisms, respectively, rather than a general constant. The goal of this workload is not to introduce a new RAG benchmark, but to isolate online multiplexing mechanisms under controlled nonprefix locality.

[0199] For each dataset, 1000 requests are sent, and the P99 TTFT and window-level non-prefix block hit rate are reported on every 100 consecutive requests. FlexKV uses the default Top-k=40% configuration selected above and is compared with SGLang, vLLM, and CacheBlend. None of the three baselines maintain a position-decoupled non-prefix text block KV cache, so their non-prefix block hit rate is 0. This setting allows observation of whether the online cache can accumulate reusable text blocks from a cold state and translate it into a stable latency gain relative to the baseline.

[0200] Figure 2 , Figure 3 and Figure 4 The diagram illustrates the cache accumulation process over a long request stream. In the first window, FlexKV's block hit rate is only approximately 0.087–0.088 across the three reuse settings, indicating that the early stages are still dominated by cold cache misses and new block registrations. As more requests arrive, reusable blocks are registered and retained in the hierarchical cache. The window-level block hit rate plateaus after the third window: at 20%, 40%, and 60% reuse rates, the plateau hit rates are approximately 0.273, 0.455, and 0.636, respectively. In contrast, SGLang, vLLM, and CacheBlend all have zero non-prefix block hits throughout the request stream because they cannot reuse the same text block in different locations.

[0201] Figure 2 , Figure 3 and Figure 4 Each row in the table corresponds to a target reuse rate; the left column compares FlexKV with the P99 TTFT of three baselines, and the right column reports the non-prefix block hit rate. Each window contains 100 consecutive requests.

[0202] The tail delay curves illustrate how cumulative hit rate translates into an advantage relative to the baseline. At a 20% reuse rate, FlexKV's first window P99 TTFT is 1251.92 ms, close to SGLang's 1213.00 ms. By the last window, FlexKV's TTFT drops to 800.06 ms, 23.5%, 42.5%, and 48.0% lower than SGLang, vLLM, and CacheBlend, respectively. At a 40% reuse rate, FlexKV's final window P99 TTFT is 710.63 ms, a reduction of 32.9%, 48.4%, and 52.4% relative to the three baselines, respectively. At a 60% reuse rate, the reuse gain is strongest: FlexKV's final window reaches 306.10 ms, a reduction of 70.9%, 77.8%, and 80.2% relative to the three baselines, respectively.

[0203] Figure 5 , Figure 6 and Figure 7 Each point aggregation contract Figure 2 , Figure 3 and Figure 4 One hundred consecutive requests within the same request stream. The y-axis range of each subplot is adjusted separately, so comparisons should be performed within the same multiplexing settings.

[0204] Figure 5 , Figure 6 and Figure 7 The report shows the corresponding throughput curves. The three baselines remain almost flat because they cannot accumulate position-decoupled non-prefix block hits. FlexKV does not immediately dominate in the early windows, especially at 20% and 40% reuse rates: the cache is still cold, and the overhead of lookups, registrations, and admissions may be comparable to the computational savings from limited hits, so the curve fluctuates slightly around the baseline. As more requests arrive, reusable blocks gradually accumulate, and FlexKV gradually surpasses the baseline. This trend is more pronounced at 60% reuse rate because repeated blocks are frequent enough that the reuse benefits outweigh the cache management overhead, resulting in a significant throughput improvement in subsequent windows. This behavior is consistent with... Figure 2 , Figure 3 and Figure 4 Consistency: Continuous non-prefix reuse is most valuable after sufficient reusable background blocks have been accumulated in the online cache.

[0205] Controlled multiplexing request flows are used to isolate online non-prefix block multiplexing behavior, not to define new quality benchmarks. Therefore, Figure 2 , Figure 3 and Figure 4 and Figure 5 , Figure 6 and Figure 7This can be interpreted as system-level evidence of reuse formation, cache accumulation, and latency reduction under controlled non-prefix locality.

[0206] Ablation studies:

[0207] The ablation evidence is mapped to the three modules. The first module, the core / background text block orchestrator, has been examined through the Top-k sensitivity study in Figure 3. This study shows that too small a core region results in a loss of answer quality, while increasing the core proportion improves quality at the cost of a larger full attention region. Therefore, Top-k=40% is used as the default operating point, and the same factor is not repeated here. All rows use Llama-3.1-8B-Instruct, TP4, Top-k=40%, and Hotpot50-sorted workloads. This table should be interpreted as system-level ablation of the main execution paths, rather than fully orthogonal micro-ablation: Top-k orchestration, position correction, and background / core attention layout are semantically coupled, so disabling them independently changes the effective algorithm, not just removes runtime overhead. This table aims to identify the dominant runtime effects under low reuse workloads, rather than fully orthogonally decomposing all FlexKV components. The low hit rate in this sorted workload also means that the table primarily examines cold paths and region-aware prefilling efficiency; while Figure 2 , Figure 3 and Figure 4 The target non-prefix reuse mechanism is evaluated to achieve a higher runtime hit rate by accumulating duplicate blocks.

[0208] Region-Aware Prefill Operator: The region-aware prefill operator implements the restricted attention topology of FlexKV through a block-sparse prefill path. This comparison maintains the same Top-k partitioning, the same online runtime, and the same cache lookup path, while the second line disables the block-sparse prefill implementation. Therefore, it measures a specific execution path coupled within the region-aware operator, rather than removing the entire orchestration mechanism. This comparison also echoes the broader observation that structured sparse attention and IO-aware attention kernels can reduce the cost of long context prefilling without changing the high-level model interface.

[0209] Disabling block-sparse prefill increases the average TTFT from 1827.97 ms to 2468.67 ms, a 35.0% increase. Meanwhile, the block hit rate remains the same at 0.0188, and the F1 score only changes from 0.2567 to 0.2489. This indicates that the latency difference is not caused by different cache hit patterns or significant quality variations. Instead, the results support the role of region-aware prefill operators: once the core / background layout is fixed, block-sparse prefill paths can reduce average prefill overhead while maintaining answer quality.

[0210] Online KV Reuse Runtime: The online reuse runtime is responsible for determining which background blocks should be registered and admitted to the reusable KV pool. This runtime complements prefix-oriented, cache-centric systems such as SGLang and CacheBlend: FlexKV does not assume that reusable content appears with stable prefixes, therefore the runtime must simultaneously avoid under-registering useful blocks and over-registering low-value blocks.

[0211] Disabling the registration variant disables background registration. Its block hit rate drops to 0.0000, and only 195 out of 200 requests are completed. This confirms that online registration is necessary to convert duplicate background blocks into reusable KV entries. The always-register variant aggressively registers candidate objects, achieving the highest hit rate of 0.0203 and the lowest average TTFT of 1612.84 ms in the table. However, it has the lowest throughput among the three online runtime variants at 8.58 tok / s, indicating that blindly maximizing registration introduces runtime management costs.

[0212] The adaptive registration variant maintains almost the same block hit rate as always-registered (0.0193 vs. 0.0203) but boosts throughput to 9.98 tok / s and completes all requests. Its average TTFT is not the lowest, so the conclusion is not that adaptive admission minimizes all latency metrics. Instead, the results support the expected runtime assertion that adaptive registration and admission avoids pursuing caching for the sake of caching objects themselves, preserves useful reuse, and provides a better end-to-end throughput / stability tradeoff for online services. Since registration does not intentionally change the model semantics, the small F1 differences in this table are considered as evaluation variance rather than registration-induced quality gains.

[0213] Experiments show that this constrained reuse mechanism can reduce first-token latency and improve throughput while maintaining a controllable quality tradeoff. On public synchronous workloads, FlexKV approaches SGLang and exhibits a more stable tail latency advantage compared to vLLM and CacheBlend baselines. Its core benefits are more evident in controlled, trajectory-aligned request flows containing repetitive non-prefixed text blocks: as reusable background blocks accumulate in the online cache, FlexKV reduces the final window P99 TTFT by 32.9% relative to SGLang at 40% reuse rate and by 70.9% at 60% reuse rate. These results demonstrate that online location decoupling reuse is effective when reassembling RAG requests retains a sufficiently large reusable background area. More broadly, they show that constrained background path reuse can translate repetitive but non-prefixed overlaps into actual latency and KV efficiency gains in online services.

[0214] Example 3

[0215] A computer device includes a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the restricted block-level KV multiplexing method for RAG inference described in Embodiment 1 or 2.

[0216] Example 4

[0217] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the restricted block-level KV multiplexing method for RAG inference as described in Embodiment 1 or 2.

[0218] Example 5

[0219] A restricted block-level key-value multiplexing system for RAG inference includes:

[0220] The Token represents the collection acquisition module, which is configured to: receive retrieval enhancement generation inference requests sent by the client, including user query information, system prompt information, and retrieval text block sequences;

[0221] The search text block sequence is encoded to obtain a set of token representations, and the set of token representations is sorted according to the relevance between the set of token representations and the user query information.

[0222] The region division module is configured to divide the sorted set of token representations into a core region and a background region.

[0223] The pre-filling calculation module is configured as follows: the server performs pre-filling calculation based on the divided core area and background area, and stores the result of the background area pre-filling calculation into the KV cache database; it queries whether there is a corresponding cached object in the KV cache database. If it exists, the token position is adjusted by the position mapping; otherwise, the restricted background path pre-filling calculation is performed.

[0224] The result fusion module is configured to: manage the KV cache database online; fuse the cached objects of the reused background area with the generated results of the core area to generate context KV state; use the context KV state to complete the retrieval enhancement generation; and send the results to the client.

Claims

1. A restricted block-level key-value reuse method for RAG inference, characterized in that, include: Step 1: The server receives the retrieval enhancement generation inference request sent by the client, including user query information, system prompt information, and retrieval text block sequence; The search text block sequence is encoded to obtain a set of token representations, and the set of token representations is sorted according to the relevance between the set of token representations and the user query information. Step 2: Divide the sorted set of token representations into a core area and a background area; Step 3: The server performs pre-filling calculations based on the divided core and background regions, and stores the results of the background region pre-filling calculations into the KV cache database; Check if a corresponding cached object exists in the KV cache database. If it exists, adjust the position mapping of the Token. Otherwise, perform restricted background path pre-fill calculation. Step 4: Perform online management of the KV cache database; merge the cached objects of the reused background area with the generated results of the core area to generate context KV state, use the context KV state to complete the retrieval enhancement generation, and send the results to the client.

2. The restricted block-level KV reuse method for RAG inference according to claim 1, characterized in that, The server receives a retrieval enhancement generation inference request sent by the client, including user query information, system prompts, and a sequence of retrieval text blocks; including: The server receives a retrieval enhancement generation RAG inference request from the client, as shown below: ； in, S represents a reasoning request, Q represents a system prompt, and D represents a sequence of retrieved text blocks. Specifically, the retrieval text block sequence D uses user query information Q and system prompt information S to search the external knowledge base and obtain a set of candidate documents, as shown below: ； in, Let m represent the set of candidate documents, and m represent the number of candidate documents. This represents the m-th candidate document; Each candidate document is segmented according to a preset length, and boundary processing is performed, i.e., the segmentation position is adjusted, so that each text block maintains semantic integrity, resulting in the segmented text blocks as shown below: ； in, Let represent the set of text blocks after segmenting the i-th candidate document. This represents the nth text block in the set of i-th text blocks; After segmenting all candidate documents, the resulting retrieval text block sequence D={ } 3. The restricted block-level KV reuse method for RAG inference according to claim 2, characterized in that, The retrieved text block sequence is encoded to obtain a token representation set, which is then sorted according to its relevance to the user's query information; this includes: Each text block set is segmented by a word segmenter. Each text block in the code is segmented and encoded into a discrete token sequence: ； in, Represents the discrete token sequence of the j-th text block. This represents the r-th token being converted; At the same time, record the location index corresponding to each token: ； in, Represents a set of location indices. This indicates the position of the r-th token within the text block; Construct a text block token representation: ； in, The token representation of the j-th text block. This indicates the number of the j-th text block; After converting all text blocks to token representations, we obtain a set of token representations: ； in, Token represents a set. The token representation of the nth text block; The token is calculated as an evaluation value, or relevance, between itself and the user's query information Q, as shown below: ； in, Represents the evaluation function; Sort the evaluation values ​​represented by each Token in descending order to obtain the sorted set of Token representations: ； in, This indicates that the sorted tokens represent a set. This represents the nth token after sorting.

4. A restricted block-level KV reuse method for RAG inference according to claim 3, characterized in that, The specific implementation process of step 2 includes: The server sets a core ratio η, and selects the top k sorted tokens based on this core ratio as the core region, as shown below: ； ； Where k represents the number of core area tokens, and n represents the number of text blocks. Indicates the core area. This represents the k-th token after sorting. The remaining tokens will be used as the background area: ； in, Indicates the background area. This represents the (k+1)th token after sorting. This represents the nth token after sorting.

5. A restricted block-level KV reuse method for RAG inference according to claim 4, characterized in that, The core ratio η is set to η=0.

4.

6. A restricted block-level KV reuse method for RAG inference according to claim 5, characterized in that, For the core area, the token representation of the core area and user query information will be used. Input the large language model, which performs pre-filling computation according to the causal attention mechanism of the Transformer network, generates the corresponding Key matrix and Value matrix, and calculates the attention weights between each Token; For the background region, the token representation of the background region is input into the large language model. The large language model performs pre-filling calculation according to the causal attention mechanism of the Transformer network to generate the corresponding key matrix and value matrix, which are stored in the KV cache database together with the cache identification information. The cache identification information includes: text block content identifier, system prompt template, region execution rule, and model version. The text block content identifier represents the background area; the token represents the generated unique identifier. ;in, () represents a hash function. The j-th token represents the background area. The text content; The system prompt template represents the current system prompt message S; The region execution rules represent the calculation rules used when generating the KV (Key and Value) matrix, including core proportions and pre-filling calculation methods; The model version refers to the current inference model, i.e., the version of the large language model; A cache index is generated based on the background region token representation, as shown below: )； in, Indicates the cache index; S indicates a system prompt message. Indicates the rules to be enforced in the region. Indicates the model version; Based on the cache index, query the corresponding KV cache object, i.e., the Key-Value matrix, in the KV cache database; If the query result exists, the corresponding Key matrix and Value matrix are read, and the cached Key is adjusted by RoPE location encoding based on the position index of the background region Token in the current RAG inference request in the input. If the query result does not exist, it is determined that there is no reusable historical KV cache object in the current background area. Restricted background path pre-filling calculation is adopted, including: when performing pre-filling calculation and generating the Key-Value matrix in the Transformer network, restricting the information interaction between tokens in the background area through causal attention mask, so that each token only establishes attention connection with the current token representation and the token representation of the preceding position; and retaining the information interaction between the token representation and the fixed system prompt, that is, the token transformed from the system prompt information, and removing or reducing the information interaction between the token representation of the background area and the token representation corresponding to the user query information. After pre-filling the restricted background path, the corresponding Key and Value matrices are generated, and the generated Key and Value matrices and corresponding cache indexes are stored in the KV cache database.

7. A restricted block-level KV reuse method for RAG inference according to claim 6, characterized in that, The specific implementation process of step 4 includes: When the cache space of the KV cache database reaches the preset capacity threshold, the server performs deletion operations on low-value cache objects based on cache access frequency, historical hit count, and resource consumption. This includes deleting cache objects that meet at least one of the following conditions within the KV cache database management cycle: not accessed within a preset time; historical hit count is lower than the preset threshold; resource consumption cache capacity exceeds twice the average cache object capacity, but the hit count is lower than the average level; the corresponding model version, system prompt template, or region execution rule has expired. When a cached object in the KV cache database meets all the preset reuse conditions, the cached object is loaded into the current RAG inference request; the preset reuse conditions include: The token representation in the KV cache database is consistent with the token representation in the background area; The system prompt information in the KV cache database is consistent with the system prompt information of the current RAG inference request; The region execution rules in the KV cache database are consistent with the region execution rules corresponding to the current RAG inference request; The model version in the KV cache database is consistent with the large language model version of the current RAG inference request; The cache creation time, last access time, access count, and current valid status of the KV cache database meet the preset requirements. The server concatenates or rearranges the Key and Value of the background region that are cached in the KV cache database with the Key and Value of the core region generated in real time to form a complete context KV state. The context KV state is then used for decoding to complete the RAG inference of the large language model.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the restricted block-level KV reuse method for RAG inference as described in any one of claims 1-7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the restricted block-level KV reuse method for RAG inference as described in any one of claims 1-7.

10. A restricted block-level key-value multiplexing system for RAG reasoning, characterized in that, include: The Token represents the collection acquisition module, which is configured to: receive retrieval enhancement generation inference requests sent by the client, including user query information, system prompt information, and retrieval text block sequences; The search text block sequence is encoded to obtain a set of token representations, and the set of token representations is sorted according to the relevance between the set of token representations and the user query information. The region division module is configured to divide the sorted set of token representations into a core region and a background region. The pre-filling calculation module is configured such that the server performs pre-filling calculations based on the divided core region and background region, and stores the results of the background region pre-filling calculations into the KV cache database. Check if a corresponding cached object exists in the KV cache database. If it exists, adjust the position mapping of the Token. Otherwise, perform restricted background path pre-fill calculation. The result fusion module is configured to: manage the KV cache database online; fuse the cached objects of the reused background area with the generated results of the core area to generate context KV state; use the context KV state to complete the retrieval enhancement generation; and send the results to the client.

Images