KV Cache compression and hierarchical management method and device for RAG acceleration and medium

Abstract

The invention discloses a KV Cache compression and hierarchical management method and device for RAG acceleration and a medium, and belongs to the technical field of large model reasoning acceleration. In order to solve the problems that in an existing RAG KV Cache management technology, storage occupation is too large, and long-term importance and instant accessibility of data cannot be considered at the same time, the invention provides a hierarchical management strategy, and according to the method, a multi-dimensional popularity vector containing global popularity, session popularity and context popularity is defined for a KV Cache block. The method is characterized in that the compression precision of a cache block is independently determined by using global popularity so as to balance fidelity and performance; and meanwhile, the session heat and the context heat are used for independently determining the placement positions in heterogeneous hierarchies such as a GPU (Graphics Processing Unit), a CPU (Central Processing Unit) fixed memory and a CPU paging memory. According to the method, placement is guided through the real-time dimension, compression is guided through the importance dimension, context intelligent prefetching is achieved, and the reasoning performance of an RAG system, especially in a session and in a context pursuit scene is remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of natural language processing, in particular to the technical field of large language model (LLM). More specifically, the present application relates to a system optimization method for the inference process of a large language model, in particular to a method, device and medium for KV (Key-Value) cache management, compression and hierarchical management in a retrieval-augmented generation (RAG) system. BACKGROUND

[0002] Retrieval-augmented generation (RAG) is a key technology in the current field of large language models (LLM). It enhances the contextual understanding ability of LLM by retrieving relevant document chunks from an external knowledge base, thereby reducing the "hallucination" of the model while providing more accurate and timely answers.

[0003] The inference process of RAG mainly consists of two stages: prefill and decode. In the prefill stage, the system needs to process all retrieved document chunks at once. In this step, the LLM calculates the Key and Value tensors required for the attention mechanism for each Token in each document chunk, i.e., the KV Cache.

[0004] The core value of the KV Cache lies in accelerating the subsequent decoding stage. When the RAG system generates answers autoregressively, the LLM needs to review all previously processed contexts (i.e., retrieved document chunks) for each new Token generated. If the KV Cache of these document chunks is retained, the system does not need to recalculate their Key and Value at each decoding step, thereby greatly improving the inference speed of RAG and significantly reducing the generation delay.

[0005] To achieve this acceleration, a common strategy is to precompute and store the entire knowledge base as a KV Cache. However, this poses a serious challenge: the memory overhead of the KV Cache is extremely large. The total size of the KV Cache of a complete knowledge base after precomputation can reach TB level. However, the capacity of the GPU high-bandwidth memory (GPU HBM) necessary for LLM computation is extremely limited (usually only a few tens of GB). Therefore, how to efficiently manage this huge amount of data and limited high-speed memory has become a bottleneck for the performance of RAG systems.

[0006] To address this challenge, the existing technology mainly adopts a strategy of combining hierarchical storage (using GPU HBM, CPU fixed memory, CPU pageable memory, and disk) and data compression (such as INT8, GSE-8). In managing data migration and compression, the existing technology usually relies on a single hotness indicator to make decisions. This strategy assigns "hot" data (frequently accessed) to both high-precision compression formats and high-speed memory locations, and assigns "cold" data to low-precision (or high-compression ratio) formats and low-speed memory locations (CPU memory).

[0007] The present application aims to solve the defects brought by this single hotness strategy that "couples" compression decisions with placement decisions:

[0008] Resource mismatch: The existing technology cannot distinguish between "global importance" and "immediate session demand". A data block that is hot in global access but temporarily useless in the current specific user session may be incorrectly assigned high precision and continuously occupy valuable GPU HBM resources.

[0009] High latency penalty: A globally cold but crucial data block for the current user session query may be incorrectly placed in CPU memory and use a high compression ratio format. When the user needs it, the system must bear the expensive data transfer overhead and decompression delay, leading to a significant decline in user experience.

[0010] Lack of predictability: The existing technology is mostly reactive and cannot intelligently predict users' "follow-up" behavior. When a user asks a follow-up question related to the previous query, the relevant KV cache may have been evicted, causing the system to need to be loaded from disk again, resulting in repeated I / O overhead.

[0011] Therefore, there is an urgent need for a more intelligent KV cache management method that can decouple data compression decisions from placement decisions and dynamically predict user demand based on context to address the above-mentioned resource mismatch, high latency, and repeated loading problems. SUMMARY

[0012] To clearly describe the technical solutions of the present application, first, the system environment, key data structures, and core measurement indicators involved are formally defined.

[0013] KV Cache Block

[0014] The basic data unit handled by the present application is the KV cache block, denoted as C i . Each cache block is an atomic unit that contains the Key and Value tensors required for Large Language Model (LLM) attention mechanism computation. Its structure can be defined as a tuple: C i = (D i , Mi ), wherein:

[0015] D i is the data payload (Payload), i.e. the actual Key-Value tensor data.

[0016] M i is the metadata (Metadata), which is the core of the present application to realize intelligent management. M i contains at least the following fields:

[0017] Unique identifier (ID): used to uniquely identify the cache block.

[0018] Current compression format (Compression Format): records the compression method currently used by the block, such as INT8, GSE-8, etc.

[0019] Current memory location (Memory Location): records where the block is currently stored in the memory hierarchy.

[0020] Multi-dimensional hotness vector (Multi-Dimensional Hotness Vector): this is the core data structure of the present application, denoted as It replaces the single hotness score in the prior art and is defined as:

[0021]

[0022] where H global,i , H session,i and H context,i represent global hotness, session hotness and context hotness, respectively.

[0023] Heterogeneous memory hierarchy

[0024] The present application runs in a heterogeneous memory hierarchy containing multiple storage media, each level having different performance characteristics, which can be formally described as:

[0025] Capacity Cap (T): the maximum amount of data that the level can accommodate.

[0026] Access latency L acc (T): the average time to read data from the level.

[0027] Transmission bandwidth B xfer (T source , T dest):The rate of transferring data between different levels.

[0028] The memory levels defined by the system are ranked from high to low in terms of access speed:

[0029] GPU High Bandwidth Memory (GPU HBM): The lowest access latency and the highest bandwidth, but the most limited capacity, is the necessary place to perform LLM calculation.

[0030] CPU Pinned Memory (CPU Pinned Memory): Also known as page-locked memory, it will not be swapped to disk by the operating system. Its key advantage is that it can be read by the GPU at high speed through Direct Memory Access (DMA), which is an ideal high-speed staging area for GPU HBM.

[0031] CPU Pageable Memory (CPU Pageable Memory): The regular system main memory has a larger capacity, but it may be swapped out to disk by the operating system, and the transfer to the GPU needs to be copied to the pinned memory first, introducing additional overhead.

[0032] Disk Storage (Disk Storage): The slowest access speed, but the largest capacity, is used for persistent storage of the complete KV cache knowledge base.

[0033] Formal definition of multi-dimensional heat index

[0034] The present invention decomposes the definition of "heat" in multiple dimensions to more accurately depict the importance of data blocks in different time scales and context environments.

[0035] Global Heat (H global ): This index reflects the importance and access frequency of a KV cache block C i in the entire system over a long period of time. Unlike the simple cumulative count in the prior art, the present invention uses Exponentially Weighted Moving Average (EWMA) to calculate it, giving higher weight to recent access, thus better adapting to the dynamic changes of global access patterns. Its update formula is:

[0036]

[0037] where, is the indicator function (1 for access, 0 otherwise), t represents the current time step, and α ∈ (0, 1) is a smoothing factor (or called decay factor) to control the influence of historical data. This calculation method avoids the excessive accumulation of historical access, making the global heat score more sensitive to the current global trend.

[0038] Session Heat (H session) : This metric is used to capture how often a KV cache chunk C i is accessed by a particular user u in the current session.

[0039] Session Definition: A session is defined as a series of consecutive queries from the same user identifier u within a pre-set time window (e.g., 15 minutes).

[0040] Calculation: H session,i,u is a simple count of how many times user u has accessed C i in the current session window. When the session times out or ends, this score for this user will be reset to zero.

[0041] H session,i,u = the number of times user u has accessed C i in the current session

[0042] Contextual Hotness (H context ) : This metric is a predictive, short-term prefetch signal designed to address scenarios with contextual relevance, such as follow-up questions.

[0043] Definition Origin: When the system performs a retrieval operation for a user query Q t , it results in a relevant set of chunks S retrieved = {C1, C2,..., C k}. However, in generating the answer A t , the LLM might only use a subset of these chunks. Those "contextual leftover" chunks that were retrieved but not ultimately used in generating the answer, i.e., the set S context = S retrieved \S used , are considered to have high contextual hotness.

[0044] Calculation: For any chunk C context that belongs to S i , its contextual hotness is immediately set to a high initial value H max . This hotness value is exponentially decayed over subsequent time steps or interactions until it is either reactivated or decays to zero.

[0045]

[0046] where β ∈ (0, 1) is a decay factor. This mechanism formalizes the intuition that "subsequent questions are likely to be related to the broader context of the initial query," enabling the system to proactively cache these potentially relevant chunks in high-speed memory without indefinitely occupying valuable resources for their non-use.

[0047] The implementation process of the method will be described in detail through the following steps.

[0048] Step one: system initialization

[0049] At system startup, all KV cache blocks C i in the knowledge base are initialized.

[0050] 1. Set the multi-dimensional hotness vector i of all blocks C to zero vector.

[0051] 2. Set all memory cache levels (GPU HBM, CPU fixed memory, CPU pageable memory) to empty state.

[0052] 3. The compressed version of all KV cache blocks is stored on disk.

[0053] Step two: query reception and hotness update

[0054] When the system receives an input query Q t from a user u, the following operations are performed:

[0055] 1. Retrieval: the system performs retrieval operation to identify a set of KV cache blocks S t related to Q retrieved from the knowledge base.

[0056] 2. Hotness update:

[0057] For each retrieved block C i ∈ S retrieved :

[0058] Update its global hotness using the EWMA formula:

[0059] Increase its session hotness corresponding to user u: H session,i,u ← H session,i,u + 1.

[0060] For all blocks not retrieved

[0061] Decay its global hotness:

[0062] If it has active context hotness, decay:

[0063] Step three: decoupled placement and compression strategy engine

[0064] For each block C retrieved in S i, the system will update its hotness vector A two-stage policy engine is applied to make decisions. This process is fundamentally different from the single decision process in prior art that couples compression and placement together.

[0065] 3.1 Precision Selection Policy

[0066] This decision is mainly dominated by the global hotness H global,i . The core idea of this policy is: the more important a block is globally, the more it should be compressed in a format that preserves its numerical precision to the maximum extent, to ensure the inference quality of most tasks.

[0067] Policy basis: The policy is based on empirical analysis of the characteristics of different compression formats. As shown by existing research, the root mean square error (RMSE) of the INT8 format is the lowest, i.e., the precision loss is the smallest; and the decompression time of the GSE-8 format is the shortest [1].

[0068] Policy rule example: By setting thresholds θ global,high and θ global,low , the following rules can be defined:

[0069] If H global,i > θ global,high : the target compression format is set to INT8. This applies to the most important and most frequently accessed blocks to ensure the highest calculation fidelity.

[0070] If θ global,low < H global,i ≤ θ global,high : the target compression format is set to FP8-E4M3. This applies to blocks of medium importance, achieving a good balance between precision and performance.

[0071] If H global,i ≤ θ global,low : the target compression format is set to GSE-8. This applies to globally cold data, when it is occasionally accessed, the extremely fast decompression speed can minimize the delay of single access.

[0072] 3.2 Memory Placement Policy

[0073] This decision is mainly dominated by the immediacy dimensions (H session,i,u and H context,i ). The core idea of this policy is: the degree of immediate demand for data determines which speed level of memory it should be placed in.

[0074] Policy rationale: High session hotness implies "current immediate need", must be placed in the GPUHBM with the lowest access latency. High context hotness implies "next round likely need", suitable for placement in the prefetching / buffering area (i.e. CPU pinned memory) that can be quickly transferred to the GPU.

[0075] Multi-dimensional policy decision matrix: The following table clearly demonstrates the decision logic employed by the present invention, intuitively embodying the decoupling of "importance" and "immediacy".

[0076] With this decision matrix, the present invention is able to make far more intelligent and dynamic placement decisions than the prior art. For example, for a "globally cold, session hot" chunk, the prior art would place it in slow memory due to its low global hotness, resulting in degraded user experience. The present invention, however, recognizes its high session hotness and decisively places it in the GPUHBM, dynamically responding to the specific needs of the user's session.

[0077]

[0078] Step 4: Cache management and execution

[0079] For each chunk C retrieved in the cache S i , the system performs the following management operations according to the decision from Step 3:

[0080] 1. Obtain decision: Obtain its target compression format F target and target memory level T target from the policy engine.

[0081] 2. State check: Check the current location T i and current format F current of chunk C current .

[0082] 3. Perform action:

[0083] Cache miss: If C i is not in any memory level (i.e. T current is disk), load from disk.

[0084] Format conversion: If F current ≠ F target , perform format conversion. This may require first decompression from F current , then compression to F target .

[0085] Memory migration: If T current ≠ T target , perform memory migration operation to move chunk Ci Move to T target This process may involve multiple jumps, such as from disk to pageable memory, then to fixed memory, and finally to GPUHBM.

[0086] 4. Eviction Handling: When eviction is requested from the target level T... target Insert block C i This will cause its capacity Cap(T) target When an overflow occurs, an eviction mechanism must be triggered. The system will select a "victim" C. victim Evicting is then performed. The eviction strategy is no longer a simple LRU (Least Recently Used) but is based on multi-dimensional popularity, prioritizing the eviction of blocks with the lowest immediacy. For example, a comprehensive immediacy score f(H) can be defined. session H context The block with the lowest score is chosen as the sacrifice.

[0087] Step 5: Reasoning and Generation

[0088] When all S retrieved After all the blocks generated in this process have been placed in their target locations according to the strategy (especially GPUHBM) and converted to the target format, LLM performs forward propagation and attention calculation to generate a pair of queries Q. t Answer A t .

[0089] Step Six: Post-processing and Context Update

[0090] After the generation process is complete, the system performs the following key post-processing steps:

[0091] 1. Identify used blocks: Accurately identify blocks used in the generation of A t The actual set of blocks used in the process

[0092] 2. Update context heat: For the "context remaining" set S context =S retrieved \S used Each block C in j :

[0093] Its context heat H context,j Set to a higher initial value H max .

[0094] Recall the strategy engine (step 3). Based on the decision matrix, these blocks are likely to be moved to CPU fixed memory for efficient prefetching of the user's next query, thus preparing for potential follow-up questions.

[0095] Through the circulation of the above six steps, the method and system proposed by the application can realize dynamic, refined and efficient management of the KV cache in the RAG system, and significantly improve the reasoning performance and resource utilization of the system. BRIEF DESCRIPTION OF DRAWINGS

[0096] Figure 1 A system flowchart of the application.

[0097] Figure 2 A multi-dimensional hotness cache system architecture diagram of the application.

[0098] Figure 3 A multi-dimensional hotness placement strategy decision example of the application. DETAILED DESCRIPTION

[0099] In order to make the purpose, technical solutions and advantages of the application clearer and more apparent, the technical solutions of the application will be described in detail below in combination with specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the application and do not limit the application.

[0100] The core of the application is a KV Cache compression and hierarchical management method, device and medium for RAG acceleration. The method manages hierarchical content by introducing KV Cache cache management and multi-dimensional hotness vector, thereby realizing decoupling decision of "importance" (determining compression precision) and "immediacy" (determining memory location) of the cache block.

[0101] Step 1: System environment definition and initialization

[0102] The application is executed in a heterogeneous memory system, which is the basis for implementing the hierarchical cache strategy of the application.

[0103] 1.1 Hardware environment definition

[0104] The system hardware environment at least includes:

[0105] GPU (Graphics Processing Unit): used for neural network computation of large language model (LLM). Its on-board memory is defined as the highest speed memory level:

[0106] GPU high bandwidth memory (GPU HBM): lowest access delay, highest bandwidth.

[0107] CPU (Central Processing Unit): used for system scheduling, retrieval, data preprocessing and cache management logic. The main memory controlled by it is divided into two levels:

[0108] CPU Pinned Memory (CPU Pinned Memory): Also known as page-locked memory. This memory area is not swapped to disk by the operating system and can be accessed asynchronously at high speed by the GPU via Direct Memory Access (DMA), making it an ideal high-speed temporary storage area for GPU HBM. Its capacity is denoted as Cap. PIN .

[0109] CPU Pageable Memory: Regular system main memory. Capacity (Cap) PAGE The memory is relatively large, but it may be swapped out by the OS, and it needs to be copied to fixed memory before being transferred to the GPU, which introduces additional overhead [1].

[0110] Disk storage: such as SSDs or HDDs. Slowest access speed, but with virtually unlimited capacity, used for persistent storage of complete knowledge base key-value caches.

[0111] 1.2 Data Structure Definition: Multidimensional Popularity Key-Value Cache Block

[0112] The basic data unit managed by this invention is the KV cache block C. i During system initialization, each block C in the knowledge base is... i Create a metadata structure M i The metadata M i Include:

[0113] ID i : The unique identifier of the block.

[0114] Location i : Current storage location (e.g., Disk, Pageable, Pinned, HBM).

[0115] Precision i : Current compression format (e.g., BF16, INT8, GSE-8).

[0116] Multidimensional heat vector. This is the core of this invention, defined as:

[0117]

[0118] 1.3 Initialization State

[0119] When the system starts up:

[0120] Clear all memory cache levels (HBM, Pinned, Pageable).

[0121] Will Initialize to a zero vector.

[0122] All KV cache chunks C i of data D i are stored on disk.

[0123] Step 2: Offline pre-computation of knowledge base and global hotness profiling

[0124] This step is executed before the system provides online services, to build the complete KV cache knowledge base and obtain the initial global hotness distribution.

[0125] 2.1 KV cache pre-computation

[0126] The invention first processes all documents in the external knowledge base:

[0127] 1. Each document is split into fixed-length chunks (Chunks), e.g. 512 tokens.

[0128] 2. Each chunk C i is input into the LLM, performing a forward pass, computing and extracting all its Transformer layer Key and Value tensors.

[0129] 3. These Key-Value tensors (i.e. KV cache) are serialized, stored as data D i on disk. At this point, a general, lossless or near-lossless format can be used for storage.

[0130] After this step, the disk has the complete KV cache of all document chunks in the knowledge base.

[0131] 2.2 Global hotness (H global ) profiling

[0132] H global aims to measure the importance of a chunk C i in the system-wide, long-term, user-agnostic sense.

[0133] 1. Use a representative query dataset (e.g. TriviaQA [1]) to "warm up" or "stress test" the system.

[0134] 2. For each query Q in the dataset, perform the standard RAG retrieval process to find the relevant set of KV cache chunks S retrieved .

[0135] 3. For each chunk C retrieved in S i , update its global hotness count.

[0136] 4. In this embodiment, we use Exponentially Weighted Moving Average (EWMA) to compute H global to dynamically reflect the hotspot changes. Its update formula is:

[0137]

[0138] where, is an indicator function (1 for access, 0 otherwise), t is the time step (or batch) of the query, and a is a small smoothing factor (e.g., a = 0.05) to control the history weight.

[0139] This step (or continuously executed in system running) will provide a relatively stable H i score for each C global,i . Step three: offline compression based on global hotness (precision decision)

[0140] This step embodies the first part of the invention to decouple "importance" from "location": using H global to decide the compression precision. This is a background task that can be executed asynchronously periodically.

[0141] 1. Set the global hotness threshold. In this embodiment, we do not use a fixed value, but use a percentage as the threshold,

[0142] 2. Traverse all KV cache blocks C i :

[0143] 3. Decision 1 (global hot block): if H global,i is within the range of global,high .

[0144] Decision: The block C i is judged as global important data.

[0145] Action: The system sets its target compression precision Precision target,i to INT8. INT8 has extremely low precision loss (RMSE), which can ensure that the generation quality of LLM is not affected when frequently accessed.

[0146] 4. Decision 2 (global cold block): if H global,i is within the range of global,low .

[0147] Decision: The block C i is judged as global cold data.

[0148] Action: The system sets its target compression precision Precision target,iSet to GSE-8. GSE-8 (Group Shared Exponent - 8 bits) is slightly less accurate but extremely fast to decompress [1], making it ideal for cold data to minimize single load latency when "occasionally" accessed.

[0149] 5. Decision 3 (Global Warm Block): If C i is between the above two.

[0150] Decision: The block C i is moderately hot.

[0151] Action: The system sets its target compression precision Precision target,i to FP8 E4M3, a compromise between accuracy and performance.

[0152] 6. The system stores this target precision Precision target,i in the block's metadata M i . The actual compression conversion operation can be performed offline at this time or deferred until the block is first loaded (Step Five) "on the fly".

[0153] Step Four: Online Query Processing and Multidimensional Warmth Update

[0154] When the system is officially in service, the following online RAG process will be executed.

[0155] 1. Define a Session

[0156] The system must be able to distinguish between different users' sessions. A session Session u is defined as a series of queries from a particular user u. This embodiment employs a time-based sliding window:

[0157] Set the session timeout window T session , for example, 10 minutes.

[0158] If the user u has no new queries within the T session interval, his session Session u ends, and all related H session,·,u scores will be automatically cleared.

[0159] 2. Receive Query and Retrieve

[0160] The system receives a query Q t from user u at time t.

[0161] The RAG retriever encodes Q t and retrieves the k most relevant document blocks from the vector index library.

[0162] The set of KV cache blocks corresponding to these k document blocks is denoted as:

[0163] 3. Multidimensional heat vector update

[0164] This is the core online logic of the present invention.

[0165] Update the retrieved block: For

[0166] (Follow the EWMA formula in step 2.2).

[0167] (in the current session) u (Accumulate the access count within).

[0168] Decaying unvisited blocks: for

[0169] (Global heat passively decays.)

[0170] (Contextual relevance actively decays). β is a decay factor; for example, β = 0.5 means that contextual relevance is reduced by half in each round of querying.

[0171] Step 5: Cache placement and loading based on multi-dimensional popularity (location decision)

[0172] This step embodies the second part of the invention's decoupling of "importance" from "position": using H session and H context Determine the memory location.

[0173] for Each block C in i The system checks its current location. i :

[0174] 1. Cache Hit

[0175] Scenario 1: If Location i =GPU HBM.

[0176] Action: No action required. Data is already ready at the highest speed position.

[0177] Scenario 2: If Location i =CPU PinnedMemory.

[0178] Action: Trigger an asynchronous DMA transfer, transferring C iCopy from pinned memory to GPU HBM.

[0179] Case 3: If Location i = CPU Pageable Memory.

[0180] Action: Trigger a synchronous copy from C i from Pageable memory to Pinned memory, and (or simultaneously) trigger a DMA transfer to GPU HBM.

[0181] Cache Promotion: In both Case 2 and Case 3, since C i is accessed, its H session has increased (Step 4.3). The system should re-evaluate its location. If its H session has become high enough (e.g., > 1), then this block C i should reside in GPU HBM after this use, instead of being swapped out immediately.

[0182] 2. Cache Miss

[0183] If Location i = Disk, the system must load C i from disk and decide its target memory hierarchy.

[0184] Get Precision Decision: The system reads from metadata M i the target compression precision Precision global determined by H target,i (in Step 3, e.g., INT8 or GSE-8).

[0185] Get Location Decision: The system makes an on-the-fly location decision based on current H session and H context scores:

[0186] Set on-the-fly threshold θ session,high = 1 (i.e., accessed more than 1 time in the session) and θ context,high > 0.

[0187] Decision 1 (Session Hot): If Target Level L target : GPU HBM. Because this data is urgently needed in the current session.

[0188] Decision 2 (Context Hot): If and Target Level L target: CPU PinnedMemory. This indicates that the block is not urgently needed in the current round, but is a "context surplus" from the previous round, and is placed in the high-speed scratchpad as a prefetch block.

[0189] Decision 3 (Cold Conversation & Cold Context): If and

[0190] Target level L target : CPU Pageable Memory. This block is only globally important (as defined by H). global (Decision), but there is no immediate need, so it can be placed in a large capacity main memory.

[0191] 3. Loading and Conversion:

[0192] Read raw data from disk D i .

[0193] "On-the-fly" translates this into target precision. target,i (For example, BF16→INT8 or BF16→GSE-8).

[0194] The converted data D i' Store in target level L target .

[0195] 5.3 Cache Eviction

[0196] When L target Capacity (e.g., GPU HBM) HBM When a block is full, one or more blocks must be evicted. The eviction strategy of this invention is also based on multidimensional heat, rather than a simple LRU.

[0197] Evicting from HBM: Prioritize eviction of GPU HBM. seeeion Block C with the lowest score j .

[0198] Evicting Pinned: Prioritize eviction of CPU-Pinned memory from H. context Block C with the lowest score (or whose score has decayed to near 0) k .

[0199] Evicting Pageables: Prioritize eviction of CPU Pageables from memory. global Block C with the lowest score l .

[0200] This design ensures that the most immediacy-critical data is always kept in the fastest memory.

[0201] Step six: LLM inference and context hotness update

[0202] 6.1 Inference execution

[0203] When all the chunks have been loaded into the GPU HBM according to the strategy, the LLM executes a Prefill phase, computing the attention between the query Q t and .

[0204] The LLM generates the first token, and then continues the autoregressive generation until the full answer A t is produced.

[0205] 6.2 Context hotness (H context ) activation

[0206] During the generation of A t , the attention mechanism of the LLM can determine which chunks C i in are "really used" (i.e., get high attention weights), and which chunks are "unused" (retrieved but not attended to).

[0207] 1. Identify the "used" set

[0208] 2. Identify the "context remaining" set

[0209] 3. Activate H context : for those chunks that were retrieved but not used to generate A t : set their context hotness to a higher initial value: (e.g., set to 1.0).

[0210] 4. Trigger prefetch: after H context is activated, the system immediately re-applies the placement strategy of Step five to C j .

[0211] At this point, H session,j,u is low (because it was not used), but H context,j is high.

[0212] According to decision 2 of Step 5.2, this chunk C j will be automatically placed into the CPU PinnedMemory.

[0213] Effect: This mechanism enables intelligent prefetching. The system predicts that the user is likely to ask a follow-up question Q t+1 in the next round that is related to Q t but At The content not covered (i.e.) The system processes the blocks in the database and preloads them into the high-speed temporary storage area (PinnedMemory), thereby achieving extremely low loading latency in the next round of queries.

[0214] Step 7: Parameter Setting Reference

[0215] To achieve the present invention, those skilled in the art can refer to the following parameter configurations (specific values ​​can be optimized through experiments):

[0216] 1. Popularity parameters:

[0217] H global Smoothing factor α: 0.05

[0218] H context Decay factor β: 0.5 (halving in each round)

[0219] H context Initial activation value H context,init :1.0

[0220] 2. Session parameters:

[0221] Session timeout T session 600 seconds (10 minutes)

[0222] 3. Threshold parameter:

[0223] Global heat θ global,high H global Rank > 90%

[0224] Global heat θ global,low H global Rank <20%

[0225] Conversation popularity θ session,high H session Count ≥ 1 (or ≥ 2, depending on the strategy)

[0226] Context heat θ context,high H context Count > 0.1

[0227] 4. Compression format:

[0228] H global High: INT8

[0229] H global Chinese: FP8(E4M3)

[0230] H global Low: GSE-8

[0231] 4. Memory capacity (example value, depends on hardware):

[0232] Cap HBM : allocate space that can accommodate T GPU = 5% of total block number

[0233] Cap PIN : allocate space that can accommodate T PIN = 10% of total block number

[0234] Tap PAGE : allocate space that can accommodate T PAGE = 10% of total block number

[0235] By performing the above steps one to seven, the multi-dimensional heat-based hierarchical cache management method described in the present application is completely implemented.

[0236] The method successfully decouples the compression precision of the KV cache block (determined by long-term global heat) and its memory location (determined by immediate session and context heat), making the cache management strategy much more intelligent, dynamic and efficient than the prior art, especially significantly optimizing the context follow-up and intra-session repeated query scenarios in the RAG system.

Claims

1. A method, device and medium for KV Cache compression and hierarchical management for RAG acceleration, characterized in that: S1, one or more KV cache blocks (C i ) in the knowledge base are defined as a multi-dimensional hotness vector The vector contains at least: global hotness (H global,i ), session hotness (H session,i ) and context hotness (H context,i ); S2, in response to a query (Q t ) received by the system, retrieving a relevant set of cache blocks (S retrieved ) and updating the multi-dimensional hotness vector of the cache blocks in the set. S3, determining a target compression accuracy of the cache block based on the global hotness (H global,i ) S4, determining a target memory location of the cache block in a heterogeneous memory hierarchy based on the session hotness (H session,i ) and the context hotness (H context,i ). S5. Based on the target compression accuracy and target memory location, performing format conversion or memory migration operation on the cache block.

2. The method of claim 1, wherein, The global hotness (H global,i ) in the S1 is calculated by an exponentially weighted moving average (EWMA) to reflect the long-period access frequency of the cache block, and its update formula is: wherein is an indicator function, t is a time step, and a is a smoothing factor.

3. The method of claim 1, wherein, The session hotness (H session,i ) in S1 is calculated as the number of accesses to the cache block (C i ) by a particular user (u) in the current session within a pre-set session time window, and is reset to zero after the session expires or ends.

4. The method of claim 1, wherein, The updating of the context hotness (H context,i ) comprises: After the system generates an answer (A t ) based on the search set (S retrieved ), a subset of cache blocks (S used ) that were actually used in the generation process is identified. determining a set of "context remaining" blocks S that were retrieved but not used context = S retrieved \ S used ; For each cache block in the "context remaining" block set (S context ), its context hotness (H context ) is activated to a preset initial value (H max or H context,init ).

5. The method of claim 4, wherein, The context hotness (H context,i ) is exponentially decayed by a decay factor (β) at subsequent time steps when not activated, with the update formula:

6. The method of claim 1 or 2, wherein, The compression accuracy determination step based on global hotness includes: If the H global,i above a global high heat threshold (Θ global,high , the target compression precision is set to a high-precision reserved format. If the H global,i below a global low-heat threshold (Θ global,low ), the target compression accuracy is set to a high-decompression-speed format.

7. The method of claim 1, 3 or 4, wherein, The heterogeneous memory hierarchy at least includes GPU high bandwidth memory (GPU HBM), CPU pinned memory (CPU Pinned Memory) and CPU pageable memory (CPU Pageable Memory) from high to low in access speed; and the memory location determination step based on session hotness and context hotness includes: If H session,i is determined to be high, the target memory location is set to GPU HBM; If H session,i is determined to be low and H context,i is determined to be high, the target memory location is set to CPU pinned memory as a prefetch staging area; If H session,i is determined to be low and H context,i is determined to be low, the target memory location is set to CPU pageable memory or evicted from memory.

8. The method according to claim 1 or 7, characterized in that, When migrating the cache block to the target memory location causes capacity overflow, triggering an eviction mechanism, which preferentially evicts the cache block with the lowest immediacy based on the multi-dimensional hotness vector; wherein, upon eviction from the GPU HBM, the Hs are prioritized session the lowest cache block; H is preferred when evicting from CPU pinned memory context the lowest cache block. 9.A method, device and medium for KVCache compression and hierarchical management for RAG acceleration. Comprising: a metadata management module for creating and maintaining a multi-dimensional hotness vector for one or more KV cache blocks (C i ) in the knowledge base The vector contains at least: global hotness (H global,i ), session hotness (H session,i ) and context hotness (H context,i ); a hotness updating module configured to update the multi-dimensional hotness vector of the cache block in the set when the system responds to the query (Q t ) and retrieves the relevant set of cache blocks (S retrieved ) A policy decision engine configured to: determining a target compression accuracy of the cache block based on the global hotness (H global,i ) determining a target memory location for the cache block in a heterogeneous memory hierarchy based on the session hotness (H session,i ) and the context hotness (H context,i ). A cache execution module configured to perform format conversion and / or memory migration operation on the cache block according to the target compression accuracy and target memory location determined by the policy decision engine.

10. A computer device, comprising: Comprising: A memory and a processor, which are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the method of any one of claims 1-9.

11. A computer readable storage medium, characterized in that, The computer readable storage medium stores computer instructions for making a computer execute the method of any one of claims 1-9.

Images