Method and system for reducing KV storage and loading overhead in retrieval enhancement generation system

Abstract

The invention discloses a method and system for reducing KV storage and loading expenditure in a retrieval enhancement generation system, and the method comprises the steps: S1, compressing a key value cache block of a pre-calculated external document to reduce the storage and loading expenditure of the key value cache block; s2, when the requested key value cache block does not hit the cache, searching and replacing the key value cache block with similar semantics in the cache according to the similarity of retrieval results; and S3, performing comprehensive value evaluation on the key value cache blocks in the cache based on the frequency, the recency and the first retrieval score, and executing cache replacement according to an evaluation result to maximize a cache hit rate. Through an innovative two-stage compression technology, storage occupation and data loading overhead of mass external document pre-calculation key value cache are remarkably reduced. Based on a key value cache reuse strategy of semantic similarity, high-delay disk input / output access is converted into rapid memory access, and the performance bottleneck of data loading is effectively relieved.

Description

Technical Field

[0001] This invention relates to the field of retrieval enhancement generation system optimization, and more specifically to a method and system for reducing the storage and loading overhead of key-value pairs in retrieval enhancement generation systems. Background Technology

[0002] Large Language Models (LLMs) have demonstrated unprecedented potential in handling complex reasoning tasks such as text generation and question answering. However, the inherent illusion problem of LLMs severely limits their application in professional fields such as education, finance, and healthcare. Retrieval-Augmented Generation (RAG) significantly improves the factual consistency of LLM-generated content by utilizing external knowledge bases and has become a standard paradigm for mitigating this problem, widely used. However, the addition of external documents significantly increases the input context, thereby reducing the inference speed of LLMs. This is mainly because the pre-filling stage of LLMs requires traversing the entire input to generate key-value blocks and calculating the attention value between two tokens based on the key-value blocks, which causes the computational complexity to increase quadratically with the input length.

[0003] To address these challenges, current work focuses on improving the inference efficiency of RAG systems. These efforts can be broadly categorized into several types: key-value block cache precomputation and reuse, key-value block cache compression, retrieval efficiency improvement, scheduling strategy optimization, and optimization based on heterogeneous storage systems. Among these, regarding key-value block cache precomputation, many studies aim to reduce computational overhead during inference by generating key-value block caches for retrieved documents in advance. Specifically, this type of work precomputs key-value block caches for retrieved documents and stores them on disk, loading these caches on demand during inference, thus avoiding redundant computation and significantly reducing the computational load in the pre-filling stage. However, this key-value block cache-based paradigm transforms the computational bottleneck into a new and equally severe input / output bottleneck. As the model size increases, the volume of the key-value block cache also expands dramatically, making the overhead of loading data from disk a key performance bottleneck in RAG inference scenarios.

[0004] To address these challenges, a straightforward approach is to introduce a key-value block caching mechanism to avoid repeated disk loads. While some existing work has mitigated this issue by caching intermediate states of key-value blocks in GPU-CPU hierarchical memory, the cache replacement strategies proposed in these works are limited to the relevant features of the retrieved documents during RAG system operation, failing to fully utilize the relationships between retrieved documents, such as the similarity between retrieval scores of different key-value blocks, thus limiting further improvements in cache hit rate. Summary of the Invention

[0005] To address the technical problems mentioned above, this invention achieves a systematic improvement in the efficiency of the retrieval enhancement generation system by coordinating the data flow between the disk, CPU, and GPU.

[0006] To achieve the above objectives, the present invention provides a method for reducing the storage and loading overhead of key-value pairs in a retrieval enhancement generation system, comprising the following steps:

[0007] S1. Compress the key-value cache blocks of pre-computed external documents to reduce their storage and loading overhead;

[0008] S2. When the requested key-value cache block is not found in the cache, search for and replace it with a semantically similar key-value cache block based on the similarity of the search results.

[0009] S3. Based on frequency, recentity, and first retrieval score, perform a comprehensive value assessment of key-value cache blocks in the cache, and perform cache replacement according to the assessment results to maximize cache hit rate.

[0010] Preferably, S1 includes:

[0011] S11. Analyze the data distribution characteristics of the key-value tensor and obtain its numerical heterogeneity information on the dimension axis;

[0012] S12. Based on the numerical heterogeneity information, perform two-stage hybrid compression on the key-value cache block.

[0013] Preferably, S12 includes:

[0014] Based on the numerical heterogeneity information, the key tensor and value tensor are grouped and quantized along the dimension axis to obtain the quantized data and the corresponding scaling factor.

[0015] The quantized data is then subjected to a lossless compression algorithm to obtain the final compressed data.

[0016] Preferably, S2 includes:

[0017] When the key-value cache block ranked kth in the system request is not found in the cache, check whether the key-value cache block ranked k+1th is in the cache;

[0018] If the key-value cache block ranked k+1 is hit in the cache, then the key-value cache block ranked k+1 is used to replace the key-value cache block ranked k for subsequent inference calculations.

[0019] Preferably, S3 includes:

[0020] The overall value score is calculated based on the access frequency, the time of the most recent access, and the relevance score when each key-value cache block is first retrieved.

[0021] Calculate the value density of each key-value cache block based on its comprehensive value score and data block size;

[0022] When the cache space is insufficient, the key-value cache block to be replaced is selected and evicted based on the value density.

[0023] Preferably, the overall value score is calculated using the following formula:

[0024]

[0025] Among them, w r w f w s These are hyperparameter weights used to balance the contributions of each factor. R is the recentity component based on the most recent access time, F is the frequency component based on the access frequency, S is the first retrieval score, and DF is the time decay factor for S.

[0026] Preferably, the frequency component F is calculated using the following logarithmic function:

[0027]

[0028] Wherein, AF is the cumulative number of times the key-value cache block has been accessed since it entered the cache;

[0029] The recentity component R is calculated using the following exponential decay function:

[0030]

[0031] Where Δt represents the time difference between the last time the key-value cache block was retrieved and the current time, λ r This is the recent decay factor.

[0032] The present invention also provides a system for reducing the storage and loading overhead of key-value pairs in a search-enhanced generation system. The system is used to implement the above method and includes: a compression module, a search and replace module, and a replacement module.

[0033] The compression module is used to compress the key-value cache blocks of pre-computed external documents to reduce their storage and loading overhead.

[0034] The search and replace module is used to search for and replace semantically similar key-value cache blocks in the cache when the requested key-value cache block is not found in the cache, based on the similarity of the search results.

[0035] The replacement module is used to comprehensively evaluate the value of key-value cache blocks in the cache based on frequency, recentity, and first retrieval score, and perform cache replacement according to the evaluation results to maximize the cache hit rate.

[0036] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0037] This invention significantly reduces the storage footprint and data loading overhead of pre-compiling key-value (KV) caches for massive amounts of external documents through an innovative two-stage compression technique. A KV cache reuse strategy based on semantic similarity transforms high-latency disk I / O access into fast memory access, effectively avoiding the performance bottleneck of cache misses. Furthermore, the proposed FRS intelligent cache replacement algorithm optimizes cache space utilization through multi-dimensional value evaluation, significantly improving cache hit rates. Attached Figure Description

[0038] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 This is a block diagram of the method model in an embodiment of the present invention. Detailed Implementation

[0040] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0041] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0042] Example 1

[0043] This embodiment provides a method for reducing the storage and loading overhead of key-value pairs in a retrieval enhancement generation system, the steps of which include:

[0044] S1. Compress the key-value cache blocks of pre-computed external documents to reduce their storage and loading overhead.

[0045] Key-value chunks (KV chunks) are a standard technique widely used in LLMs to accelerate long sequence inference. However, their memory consumption is directly proportional to the number of model parameters and the sequence length, posing a significant storage challenge. This embodiment uses LLaMA 2-7B as an example. This model has 32 layers and 32 heads, each with 128 dimensions. If stored using the BF16 data format, with a sequence length of 1024, the required storage space is 32*32*128*2*1024*2=512MB. This problem is particularly prominent in RAG inference scenarios. On the one hand, in the RAG system, this embodiment can pre-compute the KV chunks of external documents, creating conditions for offline KV chunk compression. On the other hand, the number of external knowledge base documents can reach hundreds or even tens of thousands, resulting in storage space requirements reaching TB or higher (assuming 2048 external documents and 1024 tokens per document, then the KV chunk of the external document requires 1TB of storage space), making memory access overhead a bottleneck for inference. In summary, in order to efficiently manage and utilize these pre-computed key-value pairs, offline compression before loading them into the memory system is crucial.

[0046] S11. Analyze the data distribution characteristics of the key-value tensor to obtain its numerical heterogeneity information on the dimension axis.

[0047] This embodiment uses representative models (LLaMA 2-7B and LLaMA 3.2-1B-Instruct) and datasets (NQ and MMLU) as examples, and also uses the representative corpus dataset Wikipedia. A subset is created for each language. In this embodiment, the English subset is used (due to storage limitations, one-sixth of the English subset is used for the experiment) as the source of the experimental corpus. The data analysis results are as follows:

[0048] (1) The numerical volatility of the bond (K) tensor on the dimension axis is much greater than that on other axes.

[0049] This embodiment analyzes the variance of the absolute maximum value across three axes: Dim, Head, and Token. The specific method is as follows: For each KV chunk, this embodiment slices the tensor along a specific axis (such as the Dim axis), calculates the absolute maximum value of each slice, thus obtaining a "list of maximum values," and finally calculates the variance of this list.

[0050] This embodiment replicated the experiment on two models and two datasets. The results show that the variance of the K tensor is particularly pronounced along the Dim axis, reaching as high as 18.02 and 21.57 on LLaMA 2-7B and LLaMA 3.2-1B-Instruct, respectively. In contrast, the variances of the K tensor on other axes, and the variances of the value (V) tensor on all axes, are very small and almost negligible. This result indicates that the numerical range of the K tensor varies significantly across different dimensions, which is a major reason for the poor performance of traditional tensor-level quantization methods. Furthermore, this phenomenon is highly consistent with the functions of K and V in self-attention mechanisms.

[0051] ① The core function of the K tensor is to perform correlation measurement. To achieve high-precision KV matching, its different dimensions are trained to represent highly specific semantic features. The model relies on dramatic numerical fluctuations between dimensions to encode the saliency of these features, thereby maximizing the discriminative power of the features.

[0052] ②The main function of the V tensor is to represent the information to be aggregated. Its numerical distribution must remain relatively uniform to ensure the numerical stability of the subsequent attention-based weighted summation process and avoid distortion of the calculation results due to extreme values ​​in individual dimensions.

[0053] (2) The heterogeneity of the numerical distribution of the KV tensor and the necessity of group quantization

[0054] This embodiment analyzes in detail the heterogeneity of the numerical distribution within the KV tensor. For each KV tensor, the maximum and minimum absolute values ​​of each tensor along the Dim dimension are calculated.

[0055] The results show that the minimum absolute values ​​of both the K tensor and the V tensor are consistently close to 0 across all 128 dimensions. Furthermore, the numerical range of the K tensor exhibits non-uniform fluctuations across different dimensions, showing a distinct peak region; while the V tensor exhibits high-frequency, random fluctuations.

[0056] Traditional tensor quantization, with its single scaling factor (determined by the global maximum), severely compresses the precision of smaller dimension groups, resulting in significant accuracy loss. Therefore, grouped quantization is crucial. By calculating an independent scaling factor for each dimension group, it adapts to the high heterogeneity of the numerical distribution, maximizing the preservation of outlier information while maintaining the quantization precision for the vast majority of ordinary dimension groups.

[0057] S12. Based on the numerical heterogeneity information, perform two-stage hybrid compression on the key-value cache block.

[0058] Based on the data analysis in S11, S12 designed a two-stage hybrid KV chunk compression process, as shown in Table 1. This process first performs efficient lossy quantization on the GPU, and then performs fast lossless compression on the CPU. This is because loading data from disk is less efficient, thus requiring greater data compression, while loading data from the CPU incurs relatively less overhead, making a simple and lossless compression algorithm suitable.

[0059] Table 1

[0060] .

[0061] In the first stage, this embodiment performs symmetric quantization on K and V along the dim dimension, and uses grouped quantization to reduce quantization residuals while preserving the characteristics of extreme points. Quantization is a data compression method that maps a set of continuous floating-point numbers to a finite set of integers. Symmetric quantization uses the origin as the center, and its scaling factor S is determined only by the maximum absolute value. To ensure the universality of using symmetric quantization, this embodiment evaluates the symmetry of the numerical distribution of the KV tensors in different models. This embodiment defines the "symmetry ratio" as follows:

[0062] Symmetry ratio = min(|rmax|, |rmin|) / max(|rmax|, |rmin|)

[0063] Where rmax and rmin represent the maximum and minimum values ​​of the tensor, respectively.

[0064] Experimental data show that the average symmetry ratios of KV tensors in LLaMA 2-7B are 93% and 87%, respectively, while those in LLaMA 3.2-1B-Instruct are 86% and 89%, respectively. This result indicates that the data distribution of KV tensors is highly symmetric, thus making symmetric quantization reasonable and efficient. Compared to asymmetric quantization, symmetric quantization only requires storing one scaling factor, reducing data storage overhead.

[0065] Because the values ​​of the K tensor exhibit extremely high heterogeneity along the Dim axis, traditional tensor-level or head / sequence-level quantization would compress outlier values ​​in the Dim dimension, leading to significant accuracy loss. To address this issue, this embodiment chooses to perform quantization along the dimension axis, calculating an independent scaling factor for each dimension vector.

[0066] Furthermore, data analysis results show a strong correlation between quantization error and the maximum absolute value. Outliers may still exist even within a single-dimensional vector. To handle these local outliers more precisely, this embodiment introduces grouped quantization. Specifically, this embodiment divides each Dim vector into G groups and calculates an independent scaling factor for each group, achieving a good balance between maximizing the preservation of outlier information and minimizing quantization error.

[0067] The first-stage quantization process introduces a large amount of metadata, preventing the compression ratio from reaching the theoretical 2X. Therefore, in the second stage, this embodiment selects a block-based multi-stage compression algorithm. This algorithm first divides the data into blocks to optimize CPU cache utilization, and then sequentially applies a fast preprocessor (such as byte shuffle) and an efficient entropy encoder. While ensuring data accuracy, it effectively removes redundant information from the data and further compresses the KV chunks. This embodiment chose this compression algorithm because the data generated by the first-stage quantization process has an inherent structure suitable for high redundancy: quantization maps a wide range of floating-point numbers to a limited set of 256 integers, which inevitably leads to a large number of floating-point numbers with different original values ​​being mapped to the same integer, resulting in a large number of duplicate values.

[0068] This compression algorithm efficiently utilizes the aforementioned characteristics. In this embodiment, the byte order is first rearranged using a Bitshuffle filter to improve the efficiency of subsequent compressors, and then the zstd codec is used for final compression. This method effectively eliminates redundancy in the quantized data while maintaining extremely high compression / decompression throughput. The performance overhead of decompression will be evaluated in detail in the experimental section.

[0069] S2. When the requested key-value cache block is not found in the cache, search for and use a semantically similar key-value cache block in the cache to replace it based on the similarity of the search results.

[0070] Using caching strategies can improve memory access speed, but cache misses are unavoidable. Each miss triggers a data load from external storage to GPU memory, which is an expensive I / O overhead. To further mitigate the I / O overhead caused by cache misses, this embodiment proposes a similarity-based KV cache reuse strategy. The core idea is to use the similarity between external documents and the relevance between the chunk and the input during the retrieval phase to tolerate cache misses.

[0071] This strategy is based on the phenomenon that adjacent chunks in a search list (such as top-k and top-(k+1)) are usually highly similar semantically and therefore interchangeable. This embodiment quantifies this similarity by calculating the score difference between adjacent ranked chunks: analysis of the score difference distribution between top-k and top-(k+1) shows that the search score difference between top-k and top-(k+1) decreases as k increases.

[0072] This embodiment takes the relevance score difference of LLaMA 2-7B on the NQ test set as an example. The difference in retrieval score decreases as K increases. This small difference in score means high semantic substitutability.

[0073] This embodiment designs the following similarity-based KV cache reuse strategy: when a requested top-k chunk is not found in the cache, the system does not immediately trigger a disk load. Instead, it checks if the top-(k+1) chunk with the closest score is in the cache. If it is, the system uses this semantically highly similar chunk as a replacement for the top-k chunk to complete subsequent inference calculations. This strategy transforms a highly costly disk access into a lower-cost memory access, at the cost of a slight loss of precision. The experimental section will evaluate in detail the significant performance improvement brought by this strategy and quantify its small impact on the final generation quality, thus demonstrating that this is an efficient and feasible optimization scheme in practical applications.

[0074] S3. Based on frequency, recentity, and first retrieval score, perform a comprehensive value assessment of key-value cache blocks in the cache, and perform cache replacement according to the assessment results to maximize cache hit rate.

[0075] After reducing the storage and loading overhead of KV chunks through the two-stage compression scheme in S1, this embodiment further focuses on the issue of data residency in memory. The goal of this step is to design an efficient cache replacement algorithm to maximize the cache hit rate of the RAG inference system.

[0076] Traditional caching algorithms, such as Least Recently Used (LRU), are widely used due to their simplicity and efficiency. The core idea of ​​LRU is to prioritize the eviction of data that has not been accessed for the longest time. However, this strategy, which relies solely on the single dimension of "access time," is suboptimal for RAG scenarios because it ignores the inherent characteristics of document retrieval in RAG inference systems.

[0077] To address this, this embodiment proposes a novel cache eviction strategy—FRS (Frequency, Recency, Score). This strategy breaks away from the limitation of traditional cache replacement algorithms that rely solely on access time, and guides eviction decisions through a more comprehensive value assessment model.

[0078] The core idea of ​​FRS is that cache eviction decisions should be based on the value density of each key-value chunk. When cache space is insufficient, FRS will prioritize eviction of the chunk with the lowest value density. The value density FRS Score is defined as follows:

[0079] .

[0080] RAW_FRS is a comprehensive value scoring function used to quantify the immediate value of a chunk. ChunkSize is the size of the data block. This normalization process ensures that the eviction policy in this embodiment tends to retain the cache item with the highest unit value—that is, the chunk with higher value given the same space requirements.

[0081] The RAW_FRS score evaluates the importance of a key-value chunk using three dimensions: frequency, recency, and first-retrieval score. The formula is as follows:

[0082]

[0083] Where w r w f w s It is a hyperparameter weight used to balance the contributions of each factor.

[0084] The design motivation and calculation method for each component will be explained in detail below:

[0085] The F component is used to reflect the skewness of the access distribution (long-tailed distribution). In RAG inference, not all external documents have an equal probability of being retrieved. Instead, only a small portion of "popular" documents are frequently accessed, while a large number of "unpopular" documents are rarely or never retrieved.

[0086] "Popular" documents, as frequently accessed chunks, should be considered to have high long-term value even if they haven't been accessed recently. This example uses a logarithmic function to quantify their impact:

[0087]

[0088] AF represents the frequency at which the chunk has been retrieved since it entered the cache space. The larger the AF value, the greater the probability that the chunk has been retrieved. Considering that when the AF value is too large, directly using F=AF would overemphasize the importance of some cache entries in this dimension, this embodiment uses a logarithmic function to effectively smooth out the influence of frequency.

[0089] The R component aims to capture temporal locality. The document access patterns of the RAG system are not independent random events, but exhibit significant temporal locality, meaning that once a document is accessed, there is a very high probability that it will be accessed again within a short period of time.

[0090] Similar to LRU, the R component, being the most recently accessed chunk, has a higher probability of being accessed again in the short term. An exponential decay function is used to model it:

[0091]

[0092] Where Δt represents the time difference between the most recent retrieval of the key-value cache block and the current time; the smaller this value, the more important the data. This embodiment uses an exponential decay function to model recentity, thus providing a continuous and smooth measurement method for the recentity of cached items. λ r This is the recent decay factor.

[0093] The S component, being a chunk with a high initial retrieval score, should be assigned a higher initial value. However, this predictive ability diminishes over time. Therefore, this embodiment designs an initial score term with time decay:

[0094]

[0095] S represents the relevance score when the chunk is first retrieved, and DF is a penalty factor that appropriately penalizes cached items that have extremely high initial retrieval scores but are not subsequently retrieved. This represents the time difference between the current time and when the chunk was loaded.

[0096] The method model in this embodiment is as follows: Figure 1 As shown.

[0097] Example 2

[0098] This embodiment also provides a system for reducing the storage and loading overhead of key-value pairs in a retrieval enhancement generation system, including: a compression module, a lookup and replace module, and a replacement module; the compression module is used to compress the key-value cache blocks of pre-computed external documents to reduce their storage and loading overhead; the lookup and replace module is used to search for and replace semantically similar key-value cache blocks in the cache when the requested key-value cache block is not found in the cache, based on the similarity of the search results; the replacement module is used to perform a comprehensive value evaluation of the key-value cache blocks in the cache based on frequency, recentity, and first retrieval score, and perform cache replacement according to the evaluation results to maximize the cache hit rate.

[0099] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A method for reducing the storage and loading overhead of key-value pairs in a retrieval enhancement generation system, characterized by the following steps: include: S1. Compress the key-value cache blocks of pre-computed external documents to reduce their storage and loading overhead; S2. When the requested key-value cache block is not found in the cache, search for and replace it with a semantically similar key-value cache block based on the similarity of the search results. S3. Based on frequency, recentity, and first retrieval score, perform a comprehensive value assessment of key-value cache blocks in the cache, and perform cache replacement according to the assessment results to maximize cache hit rate.

2. The method for reducing KV storage and loading overhead in a retrieval enhancement generation system according to claim 1, characterized in that, S1 includes: S11. Analyze the data distribution characteristics of the key-value tensor and obtain its numerical heterogeneity information on the dimension axis; S12. Based on the numerical heterogeneity information, perform two-stage hybrid compression on the key-value cache block.

3. The method for reducing KV storage and loading overhead in a retrieval enhancement generation system according to claim 2, characterized in that, S12 includes: Based on the numerical heterogeneity information, the key tensor and value tensor are grouped and quantized along the dimension axis to obtain the quantized data and the corresponding scaling factor. The quantized data is then subjected to a lossless compression algorithm to obtain the final compressed data.

4. The method for reducing KV storage and loading overhead in a retrieval enhancement generation system according to claim 1, characterized in that, S2 includes: When the key-value cache block ranked kth in the system request is not found in the cache, check whether the key-value cache block ranked k+1th is in the cache; If the key-value cache block ranked k+1 is hit in the cache, then the key-value cache block ranked k+1 is used to replace the key-value cache block ranked k for subsequent inference calculations.

5. The method for reducing KV storage and loading overhead in a retrieval enhancement generation system according to claim 1, characterized in that, S3 includes: The overall value score is calculated based on the access frequency, the time of the most recent access, and the relevance score when each key-value cache block is first retrieved. Calculate the value density of each key-value cache block based on its comprehensive value score and data block size; When the cache space is insufficient, the key-value cache block to be replaced is selected and evicted based on the value density.

6. The method for reducing KV storage and loading overhead in a retrieval enhancement generation system according to claim 5, characterized in that, The overall value score is calculated using the following formula: Among them, w r w f w s These are hyperparameter weights used to balance the contributions of each factor. R is the recentity component based on the most recent access time, F is the frequency component based on the access frequency, S is the first retrieval score, and DF is the time decay factor for S.

7. The method for reducing KV storage and loading overhead in a retrieval enhancement generation system according to claim 6, characterized in that, The frequency component F is calculated using the following logarithmic function: Wherein, AF is the cumulative number of times the key-value cache block has been accessed since it entered the cache; The recentity component R is calculated using the following exponential decay function: Where Δt represents the time difference between the last time the key-value cache block was retrieved and the current time, λ r This is the recent decay factor.

8. A system for reducing the storage and loading overhead of key-value pairs in a retrieval enhancement generation system, the system being used to implement the method according to any one of claims 1-7, characterized in that, include: Compression module, find and replace module, and replacement module; The compression module is used to compress the key-value cache blocks of pre-computed external documents to reduce their storage and loading overhead. The search and replace module is used to search for and replace semantically similar key-value cache blocks in the cache when the requested key-value cache block is not found in the cache, based on the similarity of the search results. The replacement module is used to comprehensively evaluate the value of key-value cache blocks in the cache based on frequency, recentity, and first retrieval score, and perform cache replacement according to the evaluation results to maximize the cache hit rate.

Images