Context-dependent perception based key-value cache compression method, system, terminal and medium

Abstract

The application discloses a key value cache compression method and system based on context-dependent perception, a terminal and a medium, and belongs to the technical field of artificial intelligence reasoning optimization and computer storage management. The method divides and compresses units for long contexts before future queries arrive and performs causal pre-filling. After online collection of received attention statistics representing subsequent access degree of tokens and output attention entropy representing context-dependent specificity of the tokens, the application is fused into a reserved score according to a monotonic rule after scale normalization. The application allocates reserved quotas according to a global cache budget, screens high-score tokens, freezes the tokens in the original order, and generates query-independent reusable compressed cache prefixes. The application significantly reduces cache occupancy and latency, improves multi-query reuse capability, and balances reasoning accuracy and compatibility.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence inference optimization and computer storage management technology, and in particular to a context-dependent awareness-based key-value caching compression method, system, terminal and medium. Background Technology

[0002] During autoregressive inference, Transformer models typically calculate and save the key and value states for each historical token during the pre-filling stage. These states are repeatedly read in the subsequent generation stage to avoid recalculating the entire historical context. These states are commonly referred to as the Key-Value Cache (KV cache). The storage size of the KV cache increases approximately linearly with the context length, the number of model layers, and the number of KV heads. When the context expands to tens of thousands or even longer tokens, the KV cache consumes a significant amount of GPU memory or system memory, increasing storage read / write overhead, first-token latency, and per-token generation latency.

[0003] Existing key-value caching and compression techniques typically remove some key-value states based on attention weights obtained from historical tokens, local observation windows, key vector differences, context reconstruction results, or session fragment relevance. While such techniques can reduce cache size for single queries or with fixed hints, they struggle to pre-build and reliably reuse the results of future queries or end-of-line observation windows when future queries have not yet arrived or the same historical context needs to be reused by multiple different queries.

[0004] On the other hand, when retaining tokens solely based on the attention they receive from subsequent accesses, it's easy to select frequently accessed but less informationally distinctive delimiters, stop words, or format tokens. Conversely, retaining tokens solely based on their forward focus makes it impossible to determine whether the token will actually be used in subsequent contexts. Therefore, it's necessary to characterize both the subsequent access demands of historical tokens and the contextual dependencies in their representation, even under unknown future query conditions, and to generate a reusable compressed key-value cache within a fixed cache budget.

[0005] Typical existing key-value (KV) cache compression techniques include SnapKV, KeyDiff, KVzip, and EpiCache. SnapKV utilizes the observation window at the end of the cue to statistically analyze attention and selects important historical positions based on the attention head. This approach requires no training and can directly compress the KV cache, but its position selection depends on the observation content at the end of the cue; when future questions have not yet arrived or the same historical context requires answers to multiple different questions, the selection results obtained from the observation window are difficult to reuse stably.

[0006] KeyDiff selects representative key states based on the differences or similarities between key vectors. It can control the cache size without explicitly relying on the attention matrix. However, the differences in key vectors mainly reflect the diversity of representations. It does not directly measure the degree to which a historical word is continuously accessed by subsequent contexts, nor does it jointly model the degree of dependence of the word on its predecessor context.

[0007] KVzip estimates the importance of key-value pairs by leveraging the ability to reconstruct the original context using compressed KV caches. It is a query-independent cache compression scheme that can support multi-query reuse. However, its scoring requires an additional reconstruction process, resulting in higher cache construction costs. Furthermore, it does not directly utilize the bidirectional attention statistics naturally generated during causal pre-filling to distinguish between future access needs and information specificity.

[0008] EpiCache performs block-based pre-filling and cache compression on a session or episode basis, limiting the peak cache size during long dialogue processing and retaining topic-related information through hierarchical budget management. Its core focus is on session fragment management and layer-sensitive budget allocation; it has not yet jointly utilized the receive attention ("subsequent positions point to this term") and the output attention entropy ("this term points to a visible predecessor") to select on the same candidate lexical.

[0009] Overall, the existing technology has the following technical defects: (1) When the future query is unknown, it is difficult to build a compressed cache in advance by relying on specific queries or the end observation window of the prompt, and the cache result is also difficult to be stably reused by multiple subsequent queries; (2) Using only the receiving attention is easy to retain high-frequency but low-information lexical units; using only the representation difference or local rules may delete long-distance facts that are still valuable across sessions, resulting in inaccurate identification of important historical information under limited budget; (3) Completely saving the attention matrix or uniformly compressing it after the full context prefilling will generate high temporary memory and computational overhead, which is not conducive to processing ultra-long dialogues, documents and agent trajectories; (4) Different sessions, semantic segments or fixed-length segments have different lengths. If a fixed ratio or simple average budget is used, it is easy to have problems such as short segments having no reserved quotas, long segments occupying excessive budget, or the total amount deviating from the target after rounding; (5) If the original order, position index and cache constraints of different attention layers are not maintained after compression, the causal position relationship may be destroyed, resulting in the compressed cache not being able to be directly connected to subsequent autoregressive inference.

[0010] Therefore, existing technologies still need improvement and development. Summary of the Invention

[0011] To address the aforementioned shortcomings of existing technologies, this invention provides a context-dependent-aware key-value caching compression method, system, terminal, and medium. The technical solution adopted by this invention is as follows: In a first aspect, the present invention provides a context-dependent-aware key-value caching compression method, the method comprising: Obtain the long context input to be processed, the protection prefix, and the global cache budget; divide the historical tokens after the protection prefix into at least one compression unit; and always retain the key state and value state corresponding to the protection prefix. Before future queries arrive, causal pre-filling is performed for each compression unit. During the pre-filling process, the received attention statistics and output attention entropy are determined. The received attention statistics are used to characterize the degree to which each candidate historical token in the corresponding compression unit is accessed by subsequent visible locations, and the output attention entropy is used to characterize the degree of dependence concentration of each candidate historical token on the visible predecessor context. Within the same global attention layer and the same compression unit, the received attention statistics and the output attention entropy are scaled and normalized respectively, and then fused according to preset rules to obtain the retention score of each candidate historical token. Based on the global cache budget, a number of reserved tokens are allocated to each compression unit. Within each compression unit, target candidate historical tokens are selected. The key and value states corresponding to the target candidate historical tokens are retained and frozen according to their original position order. The key and value states of unselected candidate historical tokens are released to form a compressed key-value cache prefix. The target candidate historical tokens are the candidate historical tokens with the highest retention score and whose quantity is equal to the number of reserved tokens within each compression unit. When a future query arrives, the compressed key-value cache prefix is ​​copied and appended with the token sequence corresponding to the future query, and autoregressive decoding is performed to generate an answer.

[0012] In one implementation, the historical tokens following the protection prefix are divided into at least one compression unit, including: The historical tokens following the protection prefix are divided into at least one compression unit according to dialogue sessions, semantic segments, document chapters, tool call trajectories, or fixed-length token blocks.

[0013] In one implementation, the process of determining the received attention statistics includes: For the first in the compression unit The candidate historical token will be located in the . Each candidate historical token and the query position visible within the current statistical scope is divided into at least one query block; For each query block, all visible query positions and all query headers pointing to the first... The attention weights of the candidate historical tokens are averaged. The average results of each query block are summed, averaged, or the maximum value is calculated to obtain the result of the first query block. The received attention statistics for each candidate historical token, wherein the attention weight is the attention probability value after causal masking and Softmax normalization.

[0014] In one implementation, the process of determining the output attention entropy includes: For the first in the compression unit The candidate historical token, obtain the first The candidate historical tokens are a set of strictly precedent tokens accessible according to a causal mask, wherein the strictly precedent token set excludes the first candidate token from the visible position. Each candidate historical token itself; The first The original attention weights of each candidate historical token on the strict predecessor token set are renormalized, the Shannon entropy of the normalized attention distribution is calculated, and normalized by the logarithm of the number of query heads and the size of the strict predecessor token set to obtain the output attention entropy with a value ranging from 0 to 1.

[0015] In one implementation, the scale normalization process employs minimum-maximum normalization with the addition of a positive stable term, mapping the received attention statistics and the output attention entropy to a numerical range of 0 to 1, respectively. The preset rule is that the retention score does not decrease monotonically with the received attention statistics and does not increase monotonically with the output attention entropy. The fusion is expressed in a product form as: Retention score = Normalized receive attention statistic × (1 - Normalized output attention entropy).

[0016] In one implementation, the method for allocating a number of reserved tokens to each compression unit is represented as follows:

[0017] in, To be assigned to the The number of tokens to be retained per compression unit The number of global target cache tokens, including the protection prefix. For the first The number of historical candidate tokens within each compression unit. To protect the total number of input tokens for the prefix and all compression units, The number of protection prefix tokens that are always retained. This indicates the rounding up operation.

[0018] In one implementation, the compressed key-value cache prefix resides in GPU memory, CPU memory, local storage, or a distributed cache system and is loaded on demand when a future query arrives. When the same compressed key-value cache prefix corresponds to at least two different future queries, the compressed key-value cache prefix is ​​copied and the corresponding query's token sequence is appended to each query, and the decoding and answer generation are completed independently.

[0019] Secondly, embodiments of the present invention also provide a context-dependent awareness key-value cache compression system, wherein the system is used to implement the steps of the context-dependent awareness key-value cache compression method described in any of the above solutions, and the system includes: The output processing module is used to obtain the long context input to be processed, the protection prefix and the global cache budget, divide the historical tokens after the protection prefix into at least one compression unit, and always retain the key state and value state corresponding to the protection prefix. The attention and statistical calculation module is used to perform causal pre-filling for each compression unit before future queries arrive. During the pre-filling process, the received attention statistics and output attention entropy are determined. The received attention statistics are used to characterize the degree to which each candidate historical token in the corresponding compression unit is accessed by the subsequent visible location, and the output attention entropy is used to characterize the degree of dependence concentration of each candidate historical token on the visible predecessor context. The scoring fusion module is used to perform scale normalization on the received attention statistics and the output attention entropy within the same global attention layer and the same compression unit, and then fuse them according to preset rules to obtain the retention score of each candidate historical token. The cache building module is used to allocate the number of reserved tokens to each compression unit according to the global cache budget, select target candidate historical tokens in each compression unit, retain the key state and value state corresponding to the target candidate historical tokens in the original position order and freeze them, release the key state and value state of the unselected candidate historical tokens, and form a compressed key-value cache prefix. The target candidate historical tokens are the candidate historical tokens with the highest retention score in each compression unit and whose quantity is the number of reserved tokens. The answer generation module is used to copy the compressed key-value cache prefix and append the token sequence corresponding to the future query when a future query arrives, and then perform autoregressive decoding to generate an answer.

[0020] Thirdly, embodiments of the present invention also provide a terminal, wherein the terminal includes a memory, a processor, and a context-dependent awareness key-value cache compression program stored in the memory and executable on the processor. When the processor executes the context-dependent awareness key-value cache compression program, it implements the steps of the context-dependent awareness key-value cache compression method of any of the above schemes.

[0021] Fourthly, embodiments of the present invention also provide a computer-readable storage medium, wherein a context-dependent awareness key-value cache compression program is stored on the computer-readable storage medium, and the context-dependent awareness key-value cache compression program implements the steps of the context-dependent awareness key-value cache compression method described in any of the above schemes on the computer-readable storage medium.

[0022] Beneficial Effects: Compared with existing technologies, this invention provides a context-dependent-aware key-value cache compression method. First, it obtains the long context input to be processed, the protection prefix, and the global cache budget. The historical tokens after the protection prefix are divided into at least one compression unit, and the key and value states corresponding to the protection prefix are always preserved. Next, before future queries arrive, causal pre-filling is performed for each compression unit. During pre-filling, received attention statistics and output attention entropy are determined. The received attention statistics characterize the degree to which each candidate historical token within the corresponding compression unit is accessed by subsequent visible locations, and the output attention entropy characterizes the degree of dependency concentration of each candidate historical token on the visible preceding context. Then, within the same global attention layer and the same compression unit, the received attention statistics and the output attention entropy are scale-normalized respectively, and then fused according to preset rules to obtain the retention score of each candidate historical token. Next, based on the global cache budget, a number of reserved tokens are allocated to each compression unit. Within each compression unit, target candidate historical tokens are selected, and their key and value states are retained and frozen in their original positional order. The key and value states of unselected candidate historical tokens are released, forming a compressed key-value cache prefix. The target candidate historical tokens are the candidate historical tokens with the highest retention score within each compression unit, and their number equals the reserved token quota. Finally, when a future query arrives, the compressed key-value cache prefix is ​​copied and appended with the token sequence corresponding to the future query. Autoregressive decoding is then performed to generate an answer.

[0023] This invention, under the condition of unknown future queries, simultaneously characterizes the subsequent access needs and contextual dependency features of historical tokens, generates a reusable compressed KV cache within a fixed cache budget, and balances cache compression rate, inference accuracy and multi-query reuse capability, while being compatible with mainstream Transformer model structures and inference backends. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of the modules and data flow of the software used to implement the context-dependent awareness key-value caching compression method of this invention.

[0025] Figure 2 This is a flowchart of a preferred embodiment of the context-dependent-aware key-value caching compression method of the present invention.

[0026] Figure 3 This is a schematic diagram illustrating the technical route of the context-dependent-aware key-value caching compression method according to an embodiment of the present invention.

[0027] Figure 4 This is a schematic diagram of two types of attention statistics and token selection in the context-dependent awareness-based key-value caching compression method of this invention.

[0028] Figure 5 This is a schematic diagram illustrating the progressive selection, order preservation, and freezing of compression units in the context-dependent awareness key-value caching compression method of this invention.

[0029] Figure 6 This is a technical framework diagram of a context-dependent-aware key-value caching compression system according to an embodiment of the present invention.

[0030] Figure 7 A schematic diagram of a terminal provided in an embodiment of the present invention. Detailed Implementation

[0031] To make the objectives, technical solutions, and effects of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0032] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content, operations, or steps, nor does it require execution in the described order. For example, some operations or steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.

[0033] It should be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0034] It should be understood that, in order to clearly describe the technical solutions of the embodiments of the present invention, the terms "first" and "second" are used in the embodiments of the present invention to distinguish identical or similar items with essentially the same function and effect. For example, "first control information" and "second control information" are only used to distinguish different control information and do not limit their order.

[0035] Those skilled in the art will understand that the words "first" and "second" do not limit the quantity or the order of execution, and that the words "first" and "second" do not necessarily imply that they are different.

[0036] It should also be understood that the term “and / or” as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0037] The method of this embodiment can be applied to a terminal, which can be a mobile phone, tablet, computer, or other intelligent electronic device. In practical applications, the method of this embodiment can be implemented through preset software, such as... Figure 1As shown, the software consists of five parts: configuration and command line, data and hint construction, model and attention acquisition, scoring budget and compression, and inference evaluation and reporting. The configuration and command line section is responsible for reading the runtime configuration, verifying parameters such as cache budget and compression units, and starting the experiment. The data and hint construction section is responsible for loading baseline data, constructing long context hints, and aligning segments according to session, semantic segment, or fixed-length token blocks. The model and attention acquisition section is responsible for loading the Transformer model, obtaining or reconstructing attention statistics online during causal pre-filling, and adapting to execution backends such as regular attention and Flash Attention. Flash Attention is an efficient, memory-optimized attention mechanism that significantly accelerates the attention computation of the Transformer model by reducing the number of GPU (Graphics Processing Unit) memory accesses. The scoring budget and compression section is responsible for statistical scaling, budget allocation, score retention calculation, Top-K (i.e., finding the K largest (or smallest) elements from n numbers) position selection, and synchronous pruning and freezing of key and value states. The inference evaluation and reporting section reuses the formed compressed cache prefix to complete subsequent queries, metric evaluation, and result summarization. Each part transmits data in the order of runtime configuration, context, cache status, compression prefix, and prediction evaluation results.

[0038] Specifically, such as Figure 2 As shown in the figure, the context-dependent-aware key-value cache compression method of this embodiment specifically includes the following steps: Step S100: Obtain the long context input to be processed, the protection prefix, and the global cache budget; divide the historical tokens after the protection prefix into at least one compression unit; and always retain the key state and value state corresponding to the protection prefix.

[0039] Combination Figure 3 As shown in the diagram, this embodiment first obtains the long context input X to be processed, the protection prefix P, and the global cache budget K. The protection prefix P may include globally effective content such as system prompts, role settings, document descriptions, or common session information. Its corresponding key-value (KV) states are always fully preserved and do not participate in compression or deletion, ensuring the stability of the model's basic behavior and output format. Historical tokens following the protection prefix are divided into m compression units S1 to Sm according to session, semantic segment, document section, tool call trajectory, or fixed-length lexical blocks. The advantage of dividing compression units by semantic units is that they can be processed and frozen unit by unit, without simultaneously loading all KV states and attention matrices of the complete context, resulting in lower peak memory usage. It also avoids interference between lexical units across semantic segments, improving the accuracy of key information recognition. In other scenarios, compression units may not be divided, and statistics and selection can be performed uniformly on all candidate historical tokens.

[0040] Step S200: Before the future query arrives, perform causal pre-filling for each compression unit. During the pre-filling process, determine the received attention statistics and the output attention entropy. The received attention statistics are used to characterize the degree to which each candidate historical token in the corresponding compression unit is accessed by the subsequent visible location, and the output attention entropy is used to characterize the degree of dependence concentration of each candidate historical token on the visible predecessor context.

[0041] In this embodiment, as Figure 3 As shown in the diagram, causal pre-filling is essentially a Transformer block-by-block forward computation process that follows causal attention constraints. The entire process only inputs existing historical context (i.e., the protection prefix and candidate historical tokens), without involving any future query content. It leverages the inherent characteristic of causal attention—that "the current position can only see itself and its predecessors"—to simultaneously accumulate and receive attention online while calculating attention layer by layer and generating key and value states. and output attention entropy There is no need to wait for future queries to arrive, nor is there a need to save the complete attention matrix.

[0042] Specifically, in this embodiment, for each compression unit, causal pre-filling is performed before the future query arrives to obtain the query, key, Softmax normalized term, or equivalent attention probability of the selected attention layer; without changing the original output of the model and the KV update path, the received attention statistics are collected online. and output attention entropy This process does not require waiting for all context pre-filling to complete, nor does it require saving the complete attention matrix. It only needs to synchronously update the statistics as the pre-filling flows through each word, resulting in low computational and memory overhead. In this embodiment, the received attention statistics are used to characterize the degree to which candidate historical words are accessed by subsequent visible positions. The higher the value, the higher the probability that the word will be reused in subsequent inference, and the more worthwhile it is to retain. Specifically, for the first... The first in the global attention layer The candidate historical token will be located in the . The query positions visible within the current statistical scope after each candidate historical token are divided into one or more query blocks. Next, for each query block, all visible query positions and all query headers pointing to the first... The attention weights of each candidate historical token are averaged. Then, the average results of each query block are summed, averaged, or the maximum value is calculated to obtain the result of the first query block. The received attention statistics for each candidate historical token, wherein the attention weight is the attention probability value after causal masking and Softmax normalization.

[0043] This embodiment is set To query the number of headers, Let the causal mask and the attention weights after Softmax processing represent the attention weights. Then the received attention statistics can be calculated according to equation (1): (1) In equation (1), For global attention layer index; This is the index of the candidate historical tokens within the current compression unit; For querying the block index; For the first The query block contains the data located at the 1st query block. The set of query locations visible within the current statistical scope after each candidate historical token. | represents the number of query locations in this set; for The query location index; For querying the header index; To query the total number of headers; Indicates the first Layer Query position in the query header Pointing to the Attention weights of candidate historical tokens after causal masking and Softmax normalization; Indicates the first The candidate historical tokens in the first The layer's attention statistics.

[0044] Equation (1) first checks all visible query positions and all query headers pointing to the first query position within each query block. The attention weights of each candidate historical token are averaged, and then the average results for each query block are summed. Besides summing, the statistics for each query block can also be averaged or maximized depending on implementation needs: summing is suitable for global evaluation in long contexts, averaging is suitable for scenarios with large differences in query block length, and maximizing is suitable for highlighting the impact of key query positions. Furthermore, corrections can be made based on the actual causal exposure counts obtained for each key position to further improve statistical accuracy.

[0045] Receiving attention only explains the first Whether a candidate historical token is used subsequently cannot be determined by whether its representation is formed from a small amount of key preceding information. Therefore, this embodiment introduces output attention entropy to characterize the degree of dependence of a lexical unit on its preceding context, indirectly reflecting its information specificity. Let... For the first The first in the layer The set of strict predecessor tokens that can be accessed by a causal mask or sliding window mask, i.e., excluding the first candidate historical token from the visible position. The set obtained after the candidate historical tokens themselves; then the set of the first... The original attention weights of each candidate historical token on the strict predecessor token set are renormalized. The Shannon entropy of the normalized attention distribution is calculated and normalized by the logarithm of the number of query heads and the size of the strict predecessor token set, resulting in an output attention entropy ranging from 0 to 1. The normalized weights are calculated according to equation (2): (2) In equation (2), and All are indexes of the visible predecessor tokens; Indicates the predecessor position for calculating the normalized weights. Used for traversal All positions in the expression are included, and the denominator is calculated. This represents the original attention weights before renormalization; This represents the attention weight renormalized after excluding the i-th candidate historical token itself, and its sum across the set of visible predecessor tokens is 1. When the number of visible predecessors is greater than 1, the output attention entropy is obtained by averaging over the query header and normalizing by the logarithm of the set size, resulting in an entropy ranging from 0 to 1. As shown in equation (3): (3) In equation (3), Indicates the first The candidate historical tokens in the first The normalized output attention entropy of the layer; To query the total number of headers; The number of tokens in the set of visible predecessor tokens; For querying the header index; Index of the visible predecessor token; Let be the renormalized attention weight obtained from equation (2); log represents the natural logarithm, and it is agreed that the product term corresponding to an attention weight of 0 is 0. The calculation result of equation (3) is used to measure the first... The degree of dispersion of attention among candidate historical tokens at their visible predecessor positions. When the number of visible predecessors is less than or equal to 1, the attention entropy will be output. Set to 0; output attention entropy A smaller value indicates that attention is more focused on a small number of predecessor positions, and the token representation has a stronger degree of context-dependent concentration. This embodiment will use 1... or Other monotonically decreasing transformations are used as information-specific signals.

[0046] In other implementations, information specificity can also be achieved using Rényi entropy, attention concentration, maximum attention weight, effective support set size, or their monotonic transformations. Different metrics have varying computational complexity and discriminative power: attention concentration is faster to calculate and suitable for edge deployments; normalized Shannon entropy offers higher discriminative power and is suitable for high-precision server-side scenarios. Excluding the lexical itself, if the number of visible predecessors is less than or equal to 1, the entropy can be set to 0.

[0047] Step S300: Within the same global attention layer and the same compression unit, the received attention statistics and the output attention entropy are scaled and normalized respectively, and then fused according to preset rules to obtain the retention score of each candidate historical token.

[0048] In this embodiment, after local scaling of the two types of statistics, they are fused according to the preset rule that the retention score does not decrease when the received attention increases and the retention score does not increase when the output entropy increases, to obtain the retention score of each candidate historical token. .

[0049] like Figure 4 As shown, the left side illustrates the accumulated attention to each historical word at multiple subsequent visible positions, illustrating the received attention. The right side illustrates the output attention entropy by showing the attention distribution from candidate lexical units to visible predecessors. The bottom shows the retention scores obtained after scaling and fusion. And perform the Top-K selection process.

[0050] To avoid numerical scale differences between different attention layers and compression units, received attention is processed separately within the same layer and the same compression unit. and output attention entropy In this embodiment, the scale normalization process involves adding a positively stable term. Minimum-maximum normalization: (4) In equation (4), This indicates the statistical type to be normalized, and can take the first value. Layer's receiving attention Or output attention entropy ; For the first The statistical value corresponding to each candidate historical token; Within the same attention layer and the same compression unit, the first Statistical values ​​corresponding to each candidate historical token Iterate through all candidate tokens within this unit; and These represent the minimum and maximum values ​​in the set of statistical values, respectively. A positive stability term is set up to prevent the denominator from being zero; Indicates the first The statistical values ​​of each candidate historical token are normalized by the minimum-maximum value. The calculation result of equation (4) maps the statistics of different dimensions and value ranges to an approximate range of 0 to 1, which is used to obtain the normalized value of the received attention and the normalized value of the output attention entropy, respectively. During fusion, the product form is adopted, which is expressed as: Retention score = Normalized received attention statistics × (1 - Normalized output attention entropy). Specifically, in this embodiment, the retention score of the candidate historical token is calculated according to equation (5): (5) In equation (5), Indicates the first The candidate historical tokens in the first The retention score of the layer; This represents the normalized value of the received attention after processing by equation (4); This represents the normalized value of the output attention entropy after processing by equation (4); the value in square brackets... This is a low-entropy information specificity factor. The product calculation result of Equation (5) gives tokens with higher receiving attention and lower output entropy a higher retention priority. In addition to the product form, other embodiments may use weighted sum, geometric mean, proportional function, gating function or ranking fusion, as long as the score is kept relative to the receiving attention. Monotonic and non-decreasing, attention entropy to output It is monotonically non-increasing. Multi-level results can be selected layer by layer, or the mean, sum, or maximum value can be calculated across layers after normalization within each layer. Figure 4 This section comprehensively illustrates the received attention statistics, output attention entropy, fusion score, and Top-K selection process. The left side shows the accumulated attention of each historical token across multiple subsequently visible positions, illustrating the received attention. The right side illustrates the output attention entropy by showing the attention distribution of candidate tokens pointing to visible predecessors. The bottom shows the retention scores obtained after scaling and fusion. And perform the Top-K selection process. Figure 4 The median values ​​are used only to illustrate the calculation and selection mechanisms and are not considered as experimental results.

[0051] Step S400: Allocate the number of reserved tokens to each compression unit according to the global cache budget, select target candidate historical tokens in each compression unit, retain the key state and value state corresponding to the target candidate historical tokens in the original position order and freeze them, release the key state and value state of the unselected candidate historical tokens, and form a compressed key-value cache prefix. The target candidate historical tokens are the candidate historical tokens with the highest retention score in each compression unit and whose quantity is equal to the number of reserved tokens.

[0052] This embodiment calculates the number of retained tokens for each compression unit based on the global cache budget. Within each compression unit, it selects the candidate historical tokens with the highest retention score and a quantity equal to the number of retained tokens. These tokens are then sorted in ascending order of their original positions, and their corresponding key and value states are retained. Unselected key and value states are released. After the current compression unit is processed, its retained state is frozen and no longer participates in the statistics and eviction of subsequent units. The process continues until all compression units are processed, ultimately forming a compressed key-value cache prefix that is independent of specific queries.

[0053] Let the global cache budget be The number of protection prefix tokens that are always retained is , No. Each compression unit contains Given N historical candidate tokens, the total number of input tokens is N. A unit budget consistent with the current software experimental implementation is: (6) In equation (6), For compression unit index; To be assigned to the The number of tokens to be retained per compression unit; For the first The number of historical candidate tokens within each compression unit; The number of global target cache tokens, including the protection prefix; The number of protection prefix tokens that are always retained; To protect the total number of input tokens for the prefix and all compression units; This indicates that the real number result within the parentheses will be rounded up; The operation guarantees that each non-empty compression unit is allocated at least one reserved token quota; The calculation guarantees that the number of reserved tokens allocated will not exceed the number of candidate tokens in that unit. Greater than and When the result is greater than 0, the calculation result of equation (6) is calculated according to the first... The remaining budget is allocated proportionally to the length of each unit across all compressible tokens. Due to rounding up, the sum of the budgets for multiple units may slightly exceed the total budget. If the deployment requires a strict total budget, the actual budget can be calculated first, and then the remaining amount can be supplemented or recovered based on the decimal surplus, while limiting the unit budget to 0 to 1. Between, so that the final total does not exceed In other implementations, the number of reserved tokens can be allocated based on the total importance of the unit, layer sensitivity, or preset priority, to adapt to the different levels of importance that different businesses place on different segments.

[0054] After determining the number of tokens to be retained in each compression unit, the token with the highest retention score is selected within each compression unit. The system obtains target candidate historical tokens at each position and then sorts them in ascending order of their original positions. For each attention layer, the key tensor and value tensor are pruned using the same position index; the original absolute position or equivalent position mapping of the tokens is preserved, and the remaining tokens are not renumbered. This embodiment fully preserves the causal positional relationship, and the compressed KV cache can be directly integrated into the existing autoregressive decoding process without modifying the model's inference logic or violating the positional constraints of causal attention, exhibiting extremely strong compatibility. After completing a compression unit, the selected key state and value state are frozen and will not be recompressed during subsequent unit statistics and release. This embodiment significantly reduces peak GPU memory usage by only retaining the complete KV and statistics of the current unit, while frozen units only retain the compressed state, making it particularly suitable for processing ultra-long time sequence scenarios such as ultra-long documents, cross-day dialogues, and long agent trajectories. Figure 5 The process of sequentially processing and freezing three compression units, S1, S2, and S3, is shown. The protection prefix is ​​always retained, and the green position indicates the currently selected or frozen token. After all compression units are completed, the retained positions are spliced ​​together in the original order to form the compressed dense KV slot.

[0055] Step S500: When a future query arrives, copy the compressed key-value cache prefix and append the Token sequence corresponding to the future query, and perform autoregressive decoding to generate an answer.

[0056] When a future query arrives, copy the frozen compressed key-value cache prefix and append different token sequences to generate the answer, such as... Figure 3In this embodiment, answers to future questions Q1, Q2, and Q3 are generated. A single compressed key-value cache prefix can serve one or more future queries, avoiding repeated pre-filling and re-compression for each future query. The compressed key-value cache prefix resides in GPU memory, CPU memory, local storage, or a distributed caching system, and is loaded on demand when a future query arrives. When the same compressed key-value cache prefix corresponds to at least two different future queries, the compressed key-value cache prefix is ​​copied and appended with the token sequence of the corresponding query, and decoding and answer generation are completed independently.

[0057] Furthermore, the context-dependent key-value caching compression method of this embodiment can be adapted to various Transformer attention architectures: for layers employing global attention, attention is received as described above. Output attention entropy and retained fractions Execution location selection: For native local attention layers (such as sliding window attention), the sliding window FIFO cache can be maintained, avoiding redundant score calculations for positions outside the window, thus ensuring compatibility with various model structures with local windows. For multi-query attention or grouped query attention, multiple query heads sharing the same KV header can first aggregate the received attention and output entropy, then share the same set of reserved positions, adapting to various mainstream large model structures and reducing computational load. Regarding precision design, statistics should preferably use floating-point FP32 accumulation to ensure computational accuracy; model states and KV caches can maintain floating-point FP16, BF16, or quantized formats, balancing accuracy and storage efficiency.

[0058] This embodiment compares the method of the present invention with existing compression methods under the same cache budget and evaluation protocol. Table 1 shows the model average results for three main benchmarks; different benchmarks use their own standard indicators, therefore, there is no horizontal comparison of values ​​between benchmarks.

[0059] Table 1

[0060] Table 1 shows LoCoMo (Long-Context Conversational Memory Benchmark), a benchmark for evaluating long-conversation memory in large language model agents. It specifically tests the model's ability to remember and extract historical information in long dialogues and multi-turn agent trajectory scenarios, with a core focus on the accuracy of factual memory within long contexts. Token F1 is a word-level F1 score, a standard quantitative indicator for information extraction tasks. It simultaneously measures the model's accuracy and recall: first, the model output and standard answer are decomposed into tokens, the proportion of matching tokens is calculated, and finally, an F1 score in the range of 0-100 is obtained. A higher score indicates that the model can still accurately extract target information from long contexts after compression and caching, with less loss of key information. Compared to the EpiCache method, which is also optimized for dialogue scenarios, the Token F1 score of this invention increases from 25.65 to 28.99, an improvement of 3.34, indicating that under the same cache budget, this invention retains more key factual information and has better accuracy in long-conversation memory.

[0061] RealTalk is a long-term dialogue dataset for real-world scenarios, containing 21 days of real human dialogue data. It recreates the semantic distribution, topic transitions, and conversation structure of real-life long-term conversations, more closely reflecting actual long-term dialogue business scenarios. GPT scoring is a subjective quality assessment based on a large model, typically using high-order models such as GPT-4 as judges. It comprehensively scores the model output from multiple dimensions, including correctness, coherence, relevance, and fluency, more closely resembling human perception of response quality. Compared to the EpiCache method, the GPT score of this invention increased from 29.68 to 33.57, an improvement of 3.89, indicating that the compressed cache of this invention not only retains discrete factual information but also better maintains the overall semantic coherence of the dialogue, resulting in higher overall response quality.

[0062] LongBench is an industry-standard benchmark for bilingual, multi-task long-context understanding, covering dozens of sub-tasks such as long document question answering, summarizing, information retrieval, and multi-hop reasoning. It comprehensively measures the model's overall performance across different types of long-context tasks, making it a universal evaluation standard for long-context models. Macro Average is a multi-task evaluation aggregation method: it first calculates the score for each sub-task in LongBench separately, then directly takes the arithmetic mean of all sub-task scores. It treats all tasks equally, unaffected by differences in task sample size, and fairly reflects the model's overall performance across various long-context scenarios. Compared to the KeyDiff method based on key vector differences, the macro average of this invention improves from 27.28 to 28.48, an improvement of 1.20, indicating that the method is not only optimized for dialogue scenarios but also possesses information preservation advantages in general long document and multi-task scenarios, demonstrating greater versatility.

[0063] Table 2 compares the efficiency of cache size, inference latency, and GPU memory usage, used to verify the benefits of this invention in terms of resource overhead and inference speed. The test baseline is the native scheme of full KV prefix reuse (i.e., without compression, retaining all historical KV caches), and the comparison scheme is the one where the target of this invention is to compress the cached tokens to approximately 8K. Test settings: average historical length 21958 tokens (approximately 21.96K), cache is built only once, TTFT (Time To First Token, referring to the time from sending a request to receiving the first token) and TPOT (Time Per Output Token, referring to the average time to generate each subsequent token) are query phase indicators after reusing the built cache, corresponding to a typical long context scenario of one-time construction and multiple reuses.

[0064] Table 2

[0065] Based on the above embodiments, the present invention also provides a context-dependent-aware key-value caching compression system, which is used to implement the steps in the above method embodiments. Figure 6As shown, the system includes: an output processing module 10, an attention and statistical calculation module 20, a scoring fusion module 30, a cache construction module 40, and an answer generation module 50. Specifically, the output processing module 10 is used to acquire the long context input to be processed, the protection prefix, and the global cache budget, and divide the historical tokens after the protection prefix into at least one compression unit, always retaining the key state and value state corresponding to the protection prefix. The attention and statistical calculation module 20 is used to perform causal pre-filling for each compression unit before future queries arrive, determining the received attention statistics and output attention entropy during the pre-filling process. The received attention statistics characterize the degree to which each candidate historical token within the corresponding compression unit is accessed by subsequent visible locations, and the output attention entropy characterizes the degree of dependence concentration of each candidate historical token on the visible preceding context. The scoring fusion module 30 is used to perform scale normalization processing on the received attention statistics and the output attention entropy within the same global attention layer and the same compression unit, and then fuse them according to preset rules to obtain the retention score of each candidate historical token. The cache construction module 40 is used to allocate a number of reserved tokens to each compression unit according to the global cache budget, select target candidate historical tokens within each compression unit, retain the key and value states corresponding to the target candidate historical tokens in their original position order and freeze them, and release the key and value states of unselected candidate historical tokens to form a compressed key-value cache prefix. The target candidate historical tokens are the candidate historical tokens with the highest retention score and whose quantity is equal to the number of reserved tokens within each compression unit. The answer generation module 50 is used to copy the compressed key-value cache prefix and append the token sequence corresponding to the future query when a future query arrives, and perform autoregressive decoding to generate an answer.

[0066] The principles of each module in the context-dependent-aware key-value caching compression system embodiment are the same as the implementation process of each step in the above method embodiment, and will not be elaborated further here.

[0067] Based on the above embodiments, the present invention also provides a terminal, the principle block diagram of which can be as follows: Figure 7 As shown. The terminal may include one or more processors 100 ( Figure 7(Only one is shown in the diagram), memory 101, and computer program 102 stored in memory 101 and executable on one or more processors 100. For example, a context-dependent awareness key-value cache compression program. When one or more processors 100 execute computer program 102, they can implement the various steps in the context-dependent awareness key-value cache compression method embodiment. Alternatively, when one or more processors 100 execute computer program 102, they can implement the functions of various modules / units in the context-dependent awareness key-value cache compression apparatus embodiment, which is not limited here.

[0068] In one embodiment, the processor 100 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0069] In one embodiment, memory 101 can be an internal storage unit of the terminal, such as a hard disk or RAM. Memory 101 can also be an external storage terminal of the terminal, such as a plug-in hard disk, smart media card (SM), secure digital card (SD), flash card, etc., all equipped on the terminal. Furthermore, memory 101 can include both internal and external storage units. Memory 101 is used to store computer programs and other programs and data required by the terminal. Memory 101 can also be used to temporarily store data that has been output or will be output.

[0070] Those skilled in the art will understand that Figure 7 The block diagram shown is merely a partial structural diagram related to the present invention and does not constitute a limitation on the terminal to which the present invention is applied. A specific terminal may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0071] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), direct memory bus RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0072] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A context-dependent-aware key-value caching compression method, characterized in that, The method includes: Obtain the long context input to be processed, the protection prefix, and the global cache budget; divide the historical tokens after the protection prefix into at least one compression unit; and always retain the key state and value state corresponding to the protection prefix. Before future queries arrive, causal pre-filling is performed for each compression unit. During the pre-filling process, the received attention statistics and output attention entropy are determined. The received attention statistics are used to characterize the degree to which each candidate historical token in the corresponding compression unit is accessed by subsequent visible locations, and the output attention entropy is used to characterize the degree of dependence concentration of each candidate historical token on the visible predecessor context. Within the same global attention layer and the same compression unit, the received attention statistics and the output attention entropy are scaled and normalized respectively, and then fused according to preset rules to obtain the retention score of each candidate historical token. Based on the global cache budget, a number of reserved tokens are allocated to each compression unit. Within each compression unit, target candidate historical tokens are selected. The key and value states corresponding to the target candidate historical tokens are retained and frozen according to their original position order. The key and value states of unselected candidate historical tokens are released to form a compressed key-value cache prefix. The target candidate historical tokens are the candidate historical tokens with the highest retention score and whose quantity is equal to the number of reserved tokens within each compression unit. When a future query arrives, the compressed key-value cache prefix is ​​copied and appended with the token sequence corresponding to the future query, and autoregressive decoding is performed to generate an answer.

2. The context-dependent awareness-based key-value caching compression method according to claim 1, characterized in that, Divide the historical tokens following the protection prefix into at least one compressed unit, including: The historical tokens following the protection prefix are divided into at least one compression unit according to dialogue sessions, semantic segments, document chapters, tool call trajectories, or fixed-length token blocks.

3. The context-dependent awareness-based key-value caching compression method according to claim 1, characterized in that, The process of determining the received attention statistics includes: For the first in the compression unit The candidate historical token will be located in the . Each candidate historical token and the query position visible within the current statistical scope is divided into at least one query block; For each query block, all visible query positions and all query headers pointing to the first... The attention weights of the candidate historical tokens are averaged. The average results of each query block are summed, averaged, or the maximum value is calculated to obtain the result of the first query block. The received attention statistics for each candidate historical token, wherein the attention weight is the attention probability value after causal masking and Softmax normalization.

4. The context-dependent awareness-based key-value caching compression method according to claim 1, characterized in that, The process of determining the output attention entropy includes: For the first in the compression unit The candidate historical token, obtain the first The candidate historical tokens are a set of strictly precedent tokens accessible according to a causal mask, wherein the strictly precedent token set excludes the first candidate token from the visible position. Each candidate historical token itself; The first The original attention weights of each candidate historical token on the strict predecessor token set are renormalized, the Shannon entropy of the normalized attention distribution is calculated, and normalized by the logarithm of the number of query heads and the size of the strict predecessor token set to obtain the output attention entropy with a value ranging from 0 to 1.

5. The context-dependent awareness-based key-value caching compression method according to claim 1, characterized in that, The scaling normalization process employs minimum-maximum normalization with the addition of a positive stable term, mapping the received attention statistics and the output attention entropy to a numerical range of 0 to 1, respectively. The preset rule is that the retention score does not decrease monotonically with the received attention statistics and does not increase monotonically with the output attention entropy. The fusion is expressed in a product form as: Retention score = Normalized receive attention statistic × (1 - Normalized output attention entropy).

6. The context-dependent-aware key-value caching compression method according to claim 1, characterized in that, The method for allocating the number of reserved tokens to each compression unit is expressed as follows: in, To be assigned to the The number of tokens to be retained per compression unit The number of global target cache tokens, including the protection prefix. For the first The number of historical candidate tokens within each compression unit. To protect the total number of input tokens for the prefix and all compression units, The number of protection prefix tokens that are always retained. This indicates the rounding up operation.

7. The context-dependent awareness-based key-value caching compression method according to claim 1, characterized in that, The compressed key-value cache prefix resides in GPU memory, CPU memory, local storage, or a distributed cache system, and is loaded on demand when a future query arrives. When the same compressed key-value cache prefix corresponds to at least two different future queries, the compressed key-value cache prefix is ​​copied and the corresponding query's token sequence is appended to each query, and the decoding and answer generation are completed independently.

8. A context-dependent-aware key-value caching compression system, characterized in that, The system is used to implement the steps of the context-dependent-aware key-value caching compression method according to any one of claims 1-7, the system comprising: The output processing module is used to obtain the long context input to be processed, the protection prefix and the global cache budget, divide the historical tokens after the protection prefix into at least one compression unit, and always retain the key state and value state corresponding to the protection prefix. The attention and statistical calculation module is used to perform causal pre-filling for each compression unit before future queries arrive. During the pre-filling process, the received attention statistics and output attention entropy are determined. The received attention statistics are used to characterize the degree to which each candidate historical token in the corresponding compression unit is accessed by the subsequent visible location, and the output attention entropy is used to characterize the degree of dependence concentration of each candidate historical token on the visible predecessor context. The scoring fusion module is used to perform scale normalization on the received attention statistics and the output attention entropy within the same global attention layer and the same compression unit, and then fuse them according to preset rules to obtain the retention score of each candidate historical token. The cache building module is used to allocate the number of reserved tokens to each compression unit according to the global cache budget, select target candidate historical tokens in each compression unit, retain the key state and value state corresponding to the target candidate historical tokens in the original position order and freeze them, release the key state and value state of the unselected candidate historical tokens, and form a compressed key-value cache prefix. The target candidate historical tokens are the candidate historical tokens with the highest retention score in each compression unit and whose quantity is the number of reserved tokens. The answer generation module is used to copy the compressed key-value cache prefix and append the token sequence corresponding to the future query when a future query arrives, and then perform autoregressive decoding to generate an answer.

9. A terminal, characterized in that, The terminal includes a memory, a processor, and a context-dependent key-value cache compression program stored in the memory and executable on the processor. When the processor executes the context-dependent key-value cache compression program, it implements the steps of the context-dependent key-value cache compression method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a context-dependent key-value cache compression program, which implements the steps of the context-dependent key-value cache compression method as described in any one of claims 1-7 on the computer-readable storage medium.

Images