An activity-aware KV cache compression method for multi-turn long text question answering

Abstract

The application discloses a kind of active perception KV cache compression methods for multi-round long text question and answer. For the existing method relies on static importance or fixed rule, it is difficult to adapt to the problem of history Token access focus migration caused by current problem change, the Key / Value state of history Token and attention weight are generated by obtaining multi-round question and answer context and current question first;Second, the attention access intensity of current query to history Token is defined as access focus, the importance score is updated based on attention weight, and the activity score is calculated by exponential moving average;Then, under the constraint of cache budget, the importance score and activity score are combined to divide reserved, candidate fusion and candidate discarded Token;Finally, the Key state similarity is fused according to the candidate Token, the Token that meets the condition is discarded, and only the reserved, fused and undiscarded Token is written back to KV cache, to reduce cache Token entry, realize KV cache compression.

Description

Technical Field

[0001] This invention belongs to the field of large language model inference optimization and cache management technology. Specifically, it relates to an activity-aware KV cache compression method for multi-turn long text question answering, which is used to dynamically manage the historical token cache state in multi-turn long text question answering to improve the utilization efficiency of KV cache and the controllability of inference. Background Technology

[0002] With the widespread application of large language models in scenarios such as long text understanding, multi-turn question answering, document retrieval, and intelligent assistants, the demand for continuous utilization of contextual information during model inference is constantly increasing. In large language models based on the Transformer architecture, the attention mechanism typically needs to access the key and value states of historical tokens when generating each new token. To avoid repeatedly calculating the entire historical context during autoregressive generation, existing inference frameworks usually adopt a key-value caching mechanism to save the key and value states already calculated in each network layer and reuse them directly in subsequent inference steps. Key-value caching can reduce redundant calculations and improve inference efficiency, but as the length of the input context, the number of historical dialogue turns, and the number of generation steps increase, the key and value states stored in the cache will continue to grow, consuming a large amount of GPU memory resources and limiting the deployment cost, inference throughput, and scalability of long-context inference. Therefore, how to reasonably compress the key-value cache while preserving as much effective contextual information as possible has become an important problem in the optimization of large language model inference.

[0003] To address the issue of ever-increasing key-value (KV) cache size, existing technologies have proposed several methods for cache selection, eviction, and fusion. Some methods determine the importance of historical KV caches based on their cumulative attention scores, retaining tokens with higher scores and deleting those with lower scores. Other methods merge similar cached content based on the similarity or redundancy between token representations to reduce cache length. Still others employ fixed windows, head and tail retention, or preset budgets to prune historical KV caches. These methods can achieve some compression in general long text generation or single-round long contextual reasoning, but their core reliance is often on historical cumulative importance, fixed position rules, or static redundancy relationships, typically assuming that the cached value of historical tokens is relatively stable during inference. Existing methods may retain tokens that are no longer relevant to the current problem due to their high historical cumulative scores, while prematurely deleting or merging tokens with low historical cumulative scores that are still needed in the current problem. Therefore, cache compression methods that solely rely on static importance or static redundancy cannot accurately reflect the state changes of historical tokens at different inference stages.

[0004] In multi-turn long-text question answering scenarios, the above problems are even more pronounced. These tasks typically involve lengthy contextual text, continuous historical question-and-answer information, and constantly changing current questions. The text regions that the model focuses on shift as the user's questions change across different turns. Some historical tokens might have been key evidence in previous turns, but gradually lose their current access value in subsequent turns; conversely, some tokens with low historical cumulative scores might become relevant to the model again due to new questions. Therefore, the token caching value in multi-turn long-text question answering is not fixed, but is simultaneously influenced by long-term importance, recent access status, and representation redundancy. Existing KV caching compression methods lack characterization of recent query access trends and token activity status, making it difficult to dynamically schedule historical caches based on the current question-and-answer turn. Therefore, it is necessary to propose a KV caching compression method for multi-turn long-text question answering that further considers the recent activity level of tokens in addition to historical importance and redundancy, to support the retention, fusion, and discarding of historical tokens in a way that better aligns with current inference needs. Summary of the Invention

[0005] To address the aforementioned issues, this invention discloses an activity-aware key-value (KV) cache compression method for multi-turn long text question answering. During the inference process of a large language model, the method simultaneously maintains the importance score, activity score, and fusion weight of historical tokens, and combines the key state similarity to determine the redundancy relationship between historical tokens. Thus, under the preset cache budget constraint, historical tokens are retained, similar tokens are fused, and optional discarding is performed, enabling the compressed KV cache to better reflect the contextual access requirements under the current question-answering round.

[0006] To achieve the above objectives, the technical solution of the present invention is as follows:

[0007] An activity-aware key-value caching and compression method for multi-turn long text question answering includes the following steps:

[0008] Step S1: Obtain the context text, historical question-answering information and current question in the multi-round long text question-answering task, and input them into the large language model; during the forward propagation of the large language model, obtain the Key state, Value state, attention weight information and historical Token state information corresponding to each network layer. The historical Token state information includes importance score, activity score and fusion weight.

[0009] Step S2: Update the importance score of historical tokens based on the attention weight information; wherein, the attention weights generated in the current inference step are aggregated, and the aggregation result is accumulated or decayed with the historical importance score.

[0010] Step S3: Update the activity score of historical tokens based on the attention access intensity of the current query vector to historical tokens; wherein, the activity score is jointly determined by the historical activity score and the current attention access intensity according to the time decay method;

[0011] Step S4: Calculate the comprehensive ranking score based on the importance score and the activity score, and calculate the redundancy based on the key state similarity; according to the comprehensive ranking score and the redundancy, divide the historical tokens into a set of retained tokens, a set of candidate fusion tokens, and a set of candidate discarded tokens.

[0012] Step S5: Perform similar token fusion processing on the tokens in the candidate fusion token set, perform discard processing on the tokens in the candidate discard token set that meet the preset discard conditions, and write the processed Key status, Value status, importance score, activity score and fusion weight back to the KV cache to obtain the compressed KV cache.

[0013] Furthermore, the specific method for step S1 is as follows:

[0014] First, the context text, historical question-and-answer information, and current question from the multi-round long text question-answering task are obtained and concatenated according to a preset input format to form the model input text for the current round. Then, the model input text is converted into an input token sequence using the tokenizer corresponding to the large language model, denoted as:

[0015] ;

[0016] in, This represents the input token sequence, where n represents the number of input tokens. This represents the i-th input token.

[0017] After the input token sequence is fed into the large language model, forward propagation computation is performed in each attention layer of the large language model. For the l-th network layer, its input hidden state is denoted as... The corresponding query status, key status, and value status are represented as follows:

[0018] ;

[0019] ;

[0020] ;

[0021] in, , , These represent the query state, key state, and value state of the l-th network layer, respectively. , , These represent the corresponding linear projection parameters. Attention weights are calculated based on the query state and the Key state; the calculation process can be represented as follows:

[0022] ;

[0023] in, This represents the attention weight information corresponding to the l-th network layer and the h-th attention head. The dimension representing the key's state.

[0024] After completing the forward propagation computation, the key and value states corresponding to each historical token are saved as KV cache content, and state information is established for each historical token. The state information includes importance score, activity score, and fusion weight. For tokens newly added in the current forward computation, their importance score and activity score can be initialized according to the current attention weight information, or set to a preset initial value; the fusion weight is set to a preset initial value.

[0025] Therefore, the state information of the i-th historical token can be represented as:

[0026] ;

[0027] in, This represents the state information of the i-th historical token. Indicates the key status. Indicates the Value state. Indicates importance score, Indicates activity score, This represents the fusion weight. Finally, the above information is stored according to the network layer, attention head, and token position to form the historical token state information of each network layer, providing basic data for subsequent importance updates, activity updates, and KV cache compression.

[0028] Furthermore, the specific method for step S2 is as follows:

[0029] The importance score is used to characterize the long-term contribution of historical tokens to the model output during the current and historical inference processes. During the forward propagation of the large language model, the current query position accesses historical tokens through an attention mechanism, so attention weight information can be used to measure the influence of different historical tokens on the current inference result.

[0030] In the t-th inference step, for the i-th historical token under the l-th network layer and the h-th attention head, the attention weights of each query position in the current forward computation process for that historical token are obtained, and the attention weights are summed or weighted by decay to obtain the importance increment of that historical token in the current inference step. The formula for calculating the importance increment is:

[0031] ;

[0032] in, This represents the importance increment of the i-th historical token under the l-th network layer and the h-th attention head in the t-th inference step; This represents the attention weight of the q-th query position to the i-th historical token during the current forward computation process; This represents the attenuation weight corresponding to the q-th query position; Q represents the number of query positions in the current forward computation process. When query position attenuation is not used, The value is 1; when using query position attenuation, The attenuation factor is determined according to the query position order.

[0033] After obtaining the importance increment for the current inference step, the importance score of the historical token in the previous inference step is read, and the importance score of the previous inference step is accumulated or decayed with the current importance increment to obtain the updated importance score. The update formula for the importance score is:

[0034] ;

[0035] in, This represents the importance score of the i-th historical token under the l-th network layer and the h-th attention head in the t-th inference step; The importance score represents the score in the previous reasoning step; γ represents the historical decay coefficient. This represents the importance increment obtained in the current inference step. When historical decay is not used, γ has a value of 1; when historical decay is used, γ is determined according to preset parameters.

[0036] After the update, the importance score is written into the historical token state information according to the network layer, attention head, and token position, and is associated with the corresponding token's key state, value state, activity score, and fusion weight. Through this method, if a historical token receives continuous attention from the current or subsequent queries across multiple inference steps, its importance score will accumulate; if a historical token is not accessed by the attention mechanism for a long period, its importance score will gradually decrease under the decay mechanism. Therefore, in the subsequent KV cache compression process, historical tokens with high long-term contributions can be prioritized for retention, while reducing the cache usage of low-contribution tokens.

[0037] Furthermore, the specific method for step S3 is as follows:

[0038] The activity score is used to characterize the recent access status of historical tokens in the current inference stage. Unlike the importance score, which mainly reflects the cumulative contribution of historical tokens in a longer inference process, the activity score focuses more on whether historical tokens are still being accessed by the current question or query, thus characterizing the dynamic changes of the model's focus area in multi-turn long text question answering.

[0039] In the t-th inference step, for the i-th historical token under the l-th network layer and the h-th attention head, the attention weight of the current query vector for that historical token is obtained and used as the attention access information in the current inference step. When the current forward computation process contains only one query position, the attention weight of that query position for the i-th historical token is directly used as the current attention access intensity; when the current forward computation process contains multiple query positions, the attention weights corresponding to multiple query positions are averaged and aggregated to obtain the current attention access intensity, and its calculation formula is:

[0040] ;

[0041] in, This represents the current attention access strength of the i-th historical token under the l-th network layer and the h-th attention head in the t-th inference step; represents the attention weight of the q-th query position to the i-th historical token; Q represents the number of query positions in the forward computation process.

[0042] After obtaining the current attention access intensity, the activity score of the historical token in the previous inference step is read and updated using an exponential moving average method to obtain the activity score in the current inference step. The calculation formula is as follows:

[0043] ;

[0044] in, This represents the activity score of the i-th historical token under the h-th attention head in the l-th network layer during the t-th inference step. This represents the activity score in the previous reasoning step; Indicates the current intensity of attention access; The value range is from 0 to 1.

[0045] when When the value is large, the activity score changes more smoothly and can retain the access status over a longer period of time; when When the value is small, the activity score is more sensitive to changes in current query access and can reflect the shift in the current issue's focus area more quickly. For tokens newly added in the current forward computation process, their current attention access intensity can be used as the initial activity score, or their activity score can be set to a preset initial value.

[0046] After the update, the activity score is written into the historical token state information according to network layer, attention head, and token position, and is associated with the corresponding token's key state, value state, importance score, and fusion weight. Through this method, if a historical token is continuously accessed by the current query in the most recent inference step, its activity score remains at a high level; if a historical token has not been accessed by the current query for a long time, its activity score will gradually decrease. Therefore, in the subsequent KV cache compression process, the situation of retaining outdated tokens for a long time solely based on historical cumulative importance can be reduced, and the risk of currently active historical tokens being prematurely fused or discarded can be mitigated.

[0047] Furthermore, the specific method for step S4 is as follows:

[0048] After obtaining the importance and activity scores of historical tokens, to avoid relying solely on the cumulative results of a single attention head, which could lead to the long-term retention of some historically high-scoring but currently inactive tokens, this invention uses both scores as the basis for caching and sorting. Specifically, the importance and activity scores of historical tokens under the same network layer and the same attention head are normalized, and the comprehensive ranking score is calculated according to the following formula:

[0049] ;

[0050] in, This represents the overall ranking score. This represents the importance score after normalization. This represents the normalized activity score. This represents the activity weighting coefficient. This comprehensive ranking method ensures that tokens that consistently contribute significantly to the model's output and those frequently accessed in the current question-and-answer round receive higher rankings, thus making the cached results more consistent with the dynamic focus changes in multi-round long-text question-and-answer sessions.

[0051] Under cache budget constraints, historical tokens with higher overall ranking scores are prioritized for retention; the remaining tokens enter the candidate compression range. For candidate compression tokens, this invention further utilizes key state similarity to determine whether there is a representation redundancy relationship between them and other tokens, and the calculation formula is as follows:

[0052] ;

[0053] in, This represents the key state similarity between the i-th historical token and the j-th historical token. and These represent the corresponding Key states. If the candidate Token and the target Token have a high similarity, it indicates that they have strong redundancy in the attention retrieval space and can enter the candidate fusion process; if the candidate Token has a low overall ranking and insufficient recent activity, it enters the candidate discard process. Therefore, historical Tokens are no longer simply deleted according to scores, but are distinguished according to three different states: "should be retained," "can be merged," and "can be discarded."

[0054] Furthermore, the specific method for step S5 is as follows:

[0055] For candidate fusion tokens, this invention does not directly delete their cached content. Instead, it performs a weighted fusion with target tokens that have high key-state similarity, in order to reduce cache length while retaining some of their semantic contribution. If the i-th historical token and the j-th target token satisfy the redundancy judgment condition, then their key and value states are updated according to the fusion weight:

[0056] ;

[0057] ;

[0058] in, and These represent the merged Key and Value states, respectively. and These represent the fusion weights of the two tokens. After fusion, the fusion weights are updated to...

[0059] ;

[0060] Importance scores can be updated by summing or taking the maximum value, and activity scores can be updated by taking the maximum value, so that the merged token continues to carry the state contribution of the merged token.

[0061] For candidate discard tokens, their key state, value state, and related state information are only removed when token discarding processing is enabled and the preset discarding conditions are met. Subsequently, the retained tokens, the merged tokens, and the tokens that were not discarded are reorganized into a compressed KV cache, which serves as the historical cache input for subsequent inference steps. Through the above processing, this invention can reduce the accidental deletion of currently active information while compressing the size of the KV cache, and retain the effective semantic contribution of some redundant tokens.

[0062] The beneficial effects of this invention are as follows:

[0063] 1) This invention introduces an activity score into the KV caching compression process. By analyzing the attention intensity of the current query vector to historical tokens and the time decay update method, it characterizes the recent access status of historical tokens in the current inference stage. Compared with caching compression methods that rely solely on historical cumulative attention scores, this invention can better reflect the dynamic migration of the model's attention area as the question changes in multi-turn long text question answering, reducing the situation where historically high-scoring but currently inactive tokens are retained for a long time.

[0064] 2) This invention combines importance score, activity score, and key state similarity to perform retention, candidate fusion, and candidate discarding of historical tokens, enabling the KV cache compression process to simultaneously consider the long-term contribution, recent access status, and representation redundancy of historical tokens. This avoids cache scheduling bias caused by a single ranking metric and improves the rationality of historical token retention, fusion, and discarding.

[0065] 3) This invention uses similar token fusion and optional token discarding mechanisms to control the size of the KV cache while preserving the effective semantic information and state contribution of historical tokens as much as possible. At the same time, this method does not require changing the main structure of the large language model, nor does it rely on additional training processes. It can complete online updates and cache compression based on attention weights and historical token state information during the inference process, making it easy to embed into the existing large language model inference process. Attached Figure Description

[0066] Figure 1 This is a system flowchart of the present invention;

[0067] Figure 2 This is a flowchart illustrating the historical token status update process of this invention.

[0068] Figure 3 This is a flowchart detailing the KV cache compression of the present invention. Detailed Implementation

[0069] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the present invention.

[0070] This embodiment uses a multi-turn long text question answering task as the application scenario and the Qwen series 7B large language model as the base model for illustration. The large language model employs a Transformer-based decoder structure. During inference, it uses a key-value cache to store the key and value states of historical tokens in each attention layer, allowing for the reuse of historical context information in subsequent generation processes. It should be noted that this invention is not limited to the Qwen series model and can also be applied to other large language models that employ attention mechanisms and key-value caching mechanisms.

[0071] like Figure 1 As shown, the activity-aware KV cache compression method in this embodiment includes input construction, forward propagation, historical token state initialization, importance score update, activity score update, comprehensive sorting and dynamic distribution, similar token fusion, optional token discarding, and writing back the compressed KV cache. The specific steps are as follows.

[0072] Step S1: Input construction and historical token state initialization.

[0073] This embodiment uses a multi-round long text question-answering task as an example. First, the long text content to be answered, the historical question-answering records generated in the current session, and the current user's question are read, and then concatenated into the model input text according to a preset template. The preset template may include a contextual text area, a historical question-answering area, and a current question area, enabling the model to simultaneously utilize the original long text information, previous question-answering information, and the current query intent during the current round of reasoning.

[0074] After constructing the input text, the tokenizer corresponding to the basic large language model is called to encode the input text, obtaining model input tensors such as input_ids and attention_mask, which are then fed into the Qwen series 7B large language model for forward propagation.

[0075] During the forward propagation of the model, this embodiment obtains the corresponding query state, key state, and value state in each attention layer. The state generation process of the l-th network layer can be represented as follows:

[0076] ;

[0077] ;

[0078] ;

[0079] Then, attention weight information is calculated based on the query status and Key status:

[0080] ;

[0081] After obtaining the Key state, Value state, and attention weight information corresponding to each attention head in each layer, the Key state and Value state are written into the model's `past_key_values` or equivalent KV cache structure. Unlike ordinary KV caches that only store Key and Value states, this embodiment synchronously maintains an importance score, activity score, and fusion weight for each historical token in the cache structure, so that the state record of the i-th historical token is represented as:

[0082] ;

[0083] Among them, the importance score is used to measure the long-term contribution of historical tokens, the activity score is used to measure the recent access status of historical tokens in the current inference stage, and the fusion weight is used for weighted calculation when similar tokens are fused in the future.

[0084] During the first round of forward propagation or when a new token is first written to the cache, the fusion weight of the new token is initialized to 1. When multiple tokens are fused, the fusion weights of each token are added together to form the new fusion weight of the fused token. When the fused token participates in fusion again, it continues to accumulate and update in the same way. The fusion weight does not have a preset upper limit; its value is determined by the cumulative number of tokens participating in fusion. The importance score can be initialized based on the aggregation result of the current attention weight, or it can be set to 0 and then accumulated and updated in subsequent inference steps. The activity score can be initialized based on the attention access intensity of the token in the current query, or it can be set to a preset initial value. After initialization, the key state, value state, and additional state information are stored according to the correspondence between network layers, attention heads, and token positions, forming a historical token state cache that can be continuously updated in subsequent inference steps.

[0085] Through the above operations, this step completes the construction of multi-round long text question-answering input, the acquisition of model forward propagation information, and the initialization of historical token state information, providing a data foundation for subsequent importance score updates, activity score updates, and dynamic compression of KV cache.

[0086] Step S2: Importance score update.

[0087] After initializing the historical token state in step S1, this embodiment updates the importance score of the historical tokens based on the attention weight information obtained during the model's forward propagation. For example... Figure 2As shown, the importance score corresponds to pos_score in the cache state in the program implementation, which is used to record the degree to which historical tokens are accessed by the attention mechanism and contribute cumulatively during the inference process.

[0088] Specifically, during the attention calculation process, the current query state and historical key states are used to calculate attention weights, which are then aggregated along the query position dimension to obtain the importance increment corresponding to the current step. When the current forward propagation process involves multiple query positions, the attention weights for different query positions can be weighted and aggregated according to a preset attenuation method. The calculation formula is as follows:

[0089] ;

[0090] in, This represents the importance increment of the i-th historical token under the h-th attention head in the l-th network layer during the t-th inference step. This represents the attention weight of the q-th query position for that historical token. This indicates the decay weight corresponding to the query position.

[0091] In this embodiment, query position decay is controlled by the shrink_factor parameter, which is preferably set to 0.98. This moderately attenuates the attention contribution of earlier query positions, allowing attention information closer to the current inference stage to have a more direct impact on importance updates. When query position decay is not needed, this parameter can also be set to 1.

[0092] During the decoding phase, when historical token states are already stored in the KV cache, this embodiment first performs a decay process on the importance score stored in the previous inference step, and then adds the importance increment obtained in the current inference step. The update method is as follows:

[0093] ;

[0094] in, This indicates the updated importance score. This represents the importance score saved in the previous reasoning step. This corresponds to the shrink_factor in this embodiment. According to the current implementation, The preferred value is 0.98.

[0095] During the initial long text input or pre-filling stage, the model calculates an initial attention score based on the current input sequence and writes it into the cache as the importance score of historical tokens. In subsequent generation or multiple rounds of question-answering inference, new attention scores are continuously added to pos_score. If the same historical token is repeatedly accessed by the current query in multiple inference steps, its importance score will gradually increase; if a token has not been accessed for a long time, its importance score will relatively decrease under the decay mechanism.

[0096] The new importance score is written to the historical token state cache according to the network layer, attention head, and token position, and kept synchronized with the corresponding key state, value state, activity score, and fusion weight. In the subsequent comprehensive ranking process, this importance score serves as a measure of the long-term contribution of historical tokens. For tokens that are repeatedly accessed by the attention mechanism in multiple rounds of question answering or continuous generation, their importance score will continue to accumulate, while tokens that have not been accessed for a long time will gradually reduce their ranking advantage due to the decay mechanism.

[0097] Step S3: Update activity score.

[0098] After updating the importance score, this embodiment further updates the activity score of the historical token based on the attention intensity of the current query to the historical token. The activity score corresponds to pos_activity in the cache state in the program implementation, and is used to record the access trend of the historical token in recent inference steps, thereby reflecting whether the token is still within the effective attention range of the current question-and-answer round.

[0099] Specifically, in the attention calculation process, this embodiment generates the current access strength based on the attention weight of the historical tokens at the current query position. When the current forward propagation process includes multiple query positions, the attention weights at multiple query positions are averaged to obtain the access strength of the current step; when the current forward propagation process includes only one query position, the attention weight corresponding to that query position is directly taken as the current access strength, and its calculation formula can be expressed as:

[0100] ;

[0101] in, This represents the current access strength of the i-th historical token under the h-th attention head in the l-th network layer during the t-th inference step. This represents the attention weight of the q-th query position for that historical token.

[0102] After obtaining the current access intensity, this embodiment uses an exponential moving average method to update the activity score, and the update formula is as follows:

[0103] ;

[0104] in, This indicates the updated activity score. This represents the activity score saved in the previous reasoning step. This represents the activity decay coefficient over time.

[0105] In this embodiment, The corresponding program parameter ema_lambda is preferably set to 0.9. This setting allows the activity score to retain some historical access trends while also updating based on current query access. When a historical token is continuously accessed in recent inference steps, its pos_activity remains at a high level; when a historical token has not been accessed by the current query for a long time, its pos_activity will gradually decrease over time.

[0106] For the first round of input or the first addition of a new token to the KV cache, this embodiment uses the per_step_activity obtained during the current forward propagation as the initial activity score of the token, and writes it into the historical token state cache according to the network layer, attention head, and token position. In subsequent generation or multi-round question answering processes, after each new attention calculation, the pos_activity of existing historical tokens is updated according to the current per_step_activity.

[0107] The new activity score reflects the recent access status of historical tokens in the current inference stage. When user questions change or the model's focus area shifts, the activity score can be adjusted according to recent attention access trends, thus providing a basis for judgments related to the current round for subsequent dynamic traffic routing.

[0108] Step S4: Combine sorting with dynamic distribution of historical tokens.

[0109] After updating the importance score and activity score, this embodiment dynamically distributes historical tokens based on the cache budget. For example... Figure 3As shown, specifically, when KV cache compression needs to be performed, the actual cache budget is first calculated based on the current cache length and the preset cache ratio. In this embodiment, cache_budget can be set to 0.2, meaning the compressed KV cache length is approximately 20% of the current cache length; cache_tail can be set to 0.5 to retain the tail tokens near the current generation position in the cache budget; and cache_dense can be set to 0.02 to determine the size of the subsequent fusion region.

[0110] For non-tail-end historical tokens, this embodiment reads their importance score (pos_score) and activity score (pos_activity) respectively, and normalizes both. Then, the normalized importance score and activity score are weighted and summed to obtain a comprehensive ranking score:

[0111] ;

[0112] in, This represents the importance score of the i-th historical token. This represents the activity score of the i-th historical token. This represents the activity weighting coefficient. In this embodiment, η corresponds to the program parameter activity_high_cap_ratio, which can be set to 0.5 to control the influence of the activity score on the overall ranking score.

[0113] When activity_mode is set to ratio, this embodiment does not use highly active tokens as an absolute retention condition. Instead, it uses activity as a dynamic weighting factor in the overall ranking, giving tokens that have been accessed recently a higher priority in the ranking. Subsequently, the non-tail historical tokens are sorted in descending order according to the overall ranking score. The tokens that rank high and meet the cache budget requirements are divided into a set of retained tokens, which, together with the tail tokens, are used as the directly retained part in the compressed KV cache.

[0114] For historical tokens not included in the retained token set, this embodiment further selects a subset of tokens as candidate compression tokens. The number of candidate compression tokens can be determined jointly by `cache_dense` and `residual_multiplier`; `residual_multiplier` can be set to 20 to expand the candidate range for subsequent fusion judgment. For candidate compression tokens, their suitability for similar token fusion will be determined based on key state similarity; tokens with low overall ranking and low activity will be subject to candidate discard judgment. Thus, this step initially divides historical tokens into retained tokens, candidate fusion tokens, and candidate discard tokens, providing input for subsequent similarity fusion, optional discarding, and KV cache rearrangement.

[0115] Step S5: Similar Token Fusion, Optional Discarding, and KV Cache Write-back.

[0116] After completing the dynamic distribution of historical tokens in step S4, this embodiment performs similar token fusion processing on candidate fused tokens, and when the token discarding function is enabled, discarding processing is performed on candidate tokens that meet the discarding conditions.

[0117] For candidate fusion tokens, this embodiment first determines the fusion target based on the key state similarity. Specifically, the key state of the candidate token is read, and the cosine similarity is calculated with the key state of the token in the fusion target region. When the key state similarity between the two is not lower than a preset redundancy threshold, the corresponding token is determined as the fusion target token. In this embodiment, the similarity measurement method is cosine, and the redundancy threshold can be set to 0.9.

[0118] For the i-th candidate token and the j-th target token that meet the fusion conditions, read their corresponding Key state, Value state, and fusion weight, and perform weighted fusion according to the fusion weight:

[0119] ;

[0120] ;

[0121] in, and These represent the merged Key and Value states, respectively. and These represent the fusion weights corresponding to the two tokens. After fusion, the fusion weights will be updated to:

[0122] ;

[0123] Simultaneously, the importance score and activity score are updated synchronously. In this embodiment, the importance score corresponds to pos_score, which is updated by default using the sum method, that is, adding the importance scores of the merged token and the target token; the activity score corresponds to pos_activity, which is updated by taking the maximum value to retain the higher of the two recent access states. After completing the above updates, the merged Key state, Value state, merge weight, importance score, and activity score are written back to the cache location corresponding to the merged target token.

[0124] For candidate discardable tokens, this embodiment sets token discarding as an optional process. By default, enable_drop is set to False, and the system primarily achieves cache compression through the fusion of similar tokens. When enable_drop is set to True, the system determines the proportion of discardable tokens based on drop_score_ratio, which can be set to 0.01, and prioritizes tokens with lower importance and activity scores as discardable targets. For tokens that meet the discarding conditions, their key state, value state, and corresponding state information are no longer retained in the compressed KV cache.

[0125] Finally, the retained tokens, the merged tokens, and the tokens that were not discarded are reorganized according to their compressed positions to form a new KV cache structure. This cache structure synchronously stores the key state, value state, pos_weight, pos_score, and pos_activity, serving as historical cache input for subsequent inference steps or the next round of question answering. Thus, this embodiment, while controlling the KV cache length, preserves the semantic contribution and state information of candidate merged tokens, enabling the KV cache compression process to adapt to dynamic focus changes in multi-round long text question answering.

[0126] It should be noted that the above content merely illustrates the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. For those skilled in the art, various improvements and modifications can be made without departing from the principle of the present invention, and all such improvements and modifications fall within the scope of protection of the claims of the present invention.

Claims

1. An activity-aware key-value caching and compression method for multi-turn long text question answering, characterized in that, Includes the following steps: Step S1: Obtain the context text, historical question-answering information and current question in the multi-round long text question-answering task, and input them into the large language model; during the forward propagation of the large language model, obtain the Key state, Value state, attention weight information and historical Token state information corresponding to each network layer. The historical Token state information includes importance score, activity score and fusion weight. Step S2: Update the importance score of historical tokens based on the attention weight information; wherein, the attention weights generated in the current inference step are aggregated, and the aggregation result is accumulated or decayed with the historical importance score. Step S3: Update the activity score of historical tokens based on the attention access intensity of the current query vector to historical tokens; wherein, the activity score is jointly determined by the historical activity score and the current attention access intensity according to the time decay method; Step S4: After normalizing the importance score and activity score, perform a weighted summation to obtain the comprehensive ranking score, and calculate the redundancy based on the key state similarity; according to the comprehensive ranking score and the redundancy, divide the historical tokens into a set of retained tokens, a set of candidate fusion tokens, and a set of candidate discarded tokens. Step S5: Perform similar token fusion processing on the tokens in the candidate fusion token set, perform discard processing on the tokens in the candidate discard token set that meet the preset discard conditions, and write the processed Key status, Value status, importance score, activity score and fusion weight back to the KV cache to obtain the compressed KV cache.

2. The method according to claim 1, characterized in that, Step S1 specifically includes: The context text, historical question-and-answer information, and current question are converted into an input token sequence and input into the large language model. In each attention layer of the large language model, a query state, key state, and value state are generated based on the input token sequence, and attention weight information is calculated from the query state and key state. The key state and value state are saved as KV cache content for the corresponding token. The importance score and activity score of historical tokens are initialized according to the attention weight information, and the fusion weight of historical tokens is set to a preset initial value. The key state, value state, importance score, activity score, and fusion weight are stored according to the token position to form the historical token state information of each network layer.

3. The method according to claim 2, characterized in that, The current increment of the importance score is obtained in the following way: In the h-th attention head of the l-th network layer, obtain the attention weights of each query position for the i-th historical token during the current forward computation process; The importance increment of the i-th historical token under the l-th network layer and h-th attention head is obtained by summing or attenuating the attention weights corresponding to each query position. The calculation formula is as follows: ； Where α represents the attention weight of the corresponding query position to the historical token, β represents the decay weight corresponding to the query position, and Q represents the number of query positions in the current forward calculation process; when query position decay is not used, β is 1; when query position decay is used, β is determined by the preset decay factor according to the query position order. The attention access intensity is used to characterize the immediate access status of historical tokens in the current inference step; The importance increment is used as the update input for the importance score of the historical token in the current forward computation process, which represents the cumulative attention contribution of the historical token in the current forward computation process, and is stored accordingly according to the network layer, attention head and token position.

4. The method according to claim 3, characterized in that, The importance score is updated and maintained in the following manner: In the t-th inference step, read the importance score of the i-th historical token under the l-th network layer and the h-th attention head in the previous inference step; Multiply the importance score from the previous reasoning step by the historical decay coefficient, and add it to the importance increment to obtain the importance score for the t-th reasoning step. The calculation formula is as follows: ； Where I represents the importance score, ΔI represents the importance increment, and γ represents the historical decay coefficient; when historical decay is not used, γ is 1; when historical decay is used, γ is determined by preset parameters. The updated importance score is written into the state information of the historical token according to the network layer, attention head, and token position.

5. The method according to claim 4, characterized in that, The activity score is updated in the following manner: In the t-th inference step, obtain the attention weight of the current query vector to the i-th historical token under the l-th network layer and the h-th attention head, and use it as the attention access information of the historical token in the current inference step; When the current forward computation process involves multiple query positions, the attention access intensity of the i-th historical token in the current inference step is obtained by arithmetically averaging the attention weights corresponding to each query position without query position attenuation. When the current forward computation process contains only one query position, the attention weight corresponding to that query position is used as the attention access strength of the i-th historical token in the current inference step. Read the activity score of the i-th historical token in the previous inference step, and fuse the activity score of the previous inference step with the attention access intensity in the current inference step using an exponential moving average method to obtain the activity score of the t-th inference step. The calculation formula is as follows: ； in, This represents the activity score of the i-th historical token under the h-th attention head in the l-th network layer during the t-th inference step. λ represents the activity score in the previous reasoning step; λ represents the activity time decay coefficient, and the value of λ ranges from 0 to 1. The activity score is used to characterize the access trend of historical tokens in recent inference steps; When a historical token is a newly added token in the current forward computation process, the current attention access intensity corresponding to that token is used as its initial activity score, or its activity score is set to a preset initial value. The updated activity score is written into the state information of the historical token according to the network layer, attention head, and token position, and then associated with the key state, value state, importance score, and fusion weight of the corresponding token.

6. The method according to claim 5, characterized in that, The process of allocating the comprehensive ranking score and historical tokens in step S4 is determined as follows: Obtain the importance score and activity score of each historical token under the l-th network layer and the h-th attention head; normalize the importance scores of each historical token under the same network layer and the same attention head to obtain the normalized importance score; The activity scores of historical tokens under the same network layer and the same attention are normalized to obtain normalized activity scores. The weighted sum of the normalized importance score and the normalized activity score yields the comprehensive ranking score of historical tokens, calculated using the following formula: ； Wherein, S represents the overall ranking score, I represents the normalized importance score, A represents the normalized activity score, and η represents the activity weight coefficient, with η ranging from 0 to 1; the normalized importance score is used to characterize the long-term cumulative contribution of historical tokens, and the normalized activity score is used to characterize the recent access trend of historical tokens. The historical tokens are sorted in descending order according to the comprehensive ranking score, and the number of tokens to be retained is determined according to the preset cache budget. Historical tokens that meet the retention criteria based on their comprehensive ranking scores are assigned to a retention token set. Retention processing is performed on the historical tokens in the retention token set to maintain the corresponding relationship between their key status, value status, importance score, activity score, and fusion weight, and these are written into the compressed KV cache. Historical tokens that did not meet the retention criteria were divided into a set of candidate compressed tokens; The historical tokens in the candidate compressed token set are further divided: historical tokens that meet the preset fusion conditions are assigned to the candidate fusion token set, and historical tokens that meet the preset discard conditions are assigned to the candidate discard token set. The preset fusion conditions include at least the historical tokens not meeting the retention conditions and the key state similarity meeting the redundancy judgment conditions; the preset discard conditions include at least the historical tokens not meeting the retention conditions, the comprehensive ranking score being lower than the preset ranking threshold, and the activity score being lower than the preset activity threshold. The reserved token set, the candidate fusion token set, and the candidate discard token set are used as inputs for subsequent KV cache retention, similar token fusion, and token discarding processes, respectively.

7. The method according to claim 6, characterized in that, The similar token fusion process described in step S5 is performed as follows: Obtain the Key status, Value status, Importance score, Activity score, and Fusion weight of the i-th historical Token in the candidate Fusion Token set, and select the Token to be matched from the Fusion Target Area corresponding to the Reserved Token set or the Candidate Fusion Token set; The key state similarity is calculated based on the key state of the i-th historical token and the key state of the token to be matched, and the key state similarity is used as the redundancy between the i-th historical token and the token to be matched. The calculation formula is as follows: ； Where R represents the key state similarity, K represents the key state corresponding to the token, and i and j represent the position numbers of different historical tokens, respectively. When the key state similarity between the i-th historical token and the j-th historical token meets the preset redundancy threshold, the j-th historical token is determined as the fusion target token of the i-th historical token. Read the fusion weights corresponding to the i-th and j-th historical tokens, and perform a weighted fusion of their Key and Value states based on these fusion weights to obtain the fused Key and Value states. The calculation formula is as follows: ； ； Where K represents the Key state, V represents the Value state, W represents the fusion weight, and Kij represents the fused Key state. This represents the merged Value state. and These represent the fusion weights corresponding to the i-th and j-th historical tokens, respectively. Add the fusion weights corresponding to the i-th historical token and the j-th historical token to obtain the fusion weight; sum the importance scores corresponding to the i-th historical token and the j-th historical token to obtain the fusion importance score; take the maximum value of the activity scores corresponding to the i-th historical token and the j-th historical token to obtain the fusion activity score. The merged Key state, Value state, Importance score, Activity score, and Merging weight are written to the KV cache location corresponding to the merged target Token, and the original KV cache content of the merged Token is no longer retained separately in the compressed KV cache.

8. The method according to claim 6, characterized in that, The discarding process of the candidate discard token set and the construction process of the compressed KV cache are performed in the following manner: Obtain the importance score, activity score, comprehensive ranking score, and fusion weight of each historical token in the candidate discard token set; When token discarding is enabled, determine whether the candidate discard tokens meet the preset discarding conditions; The preset discard conditions include: the overall ranking score of the candidate discard token is lower than the preset ranking threshold, and the activity score of the candidate discard token is lower than the preset activity threshold. For historical tokens that meet the preset discard conditions, their corresponding Key status, Value status, Importance score, Activity score, and Fusion weight will no longer be retained in the compressed KV cache. For historical tokens that do not meet the preset discard conditions, they are reclassified into the candidate fusion token set or the retained token set, and processed in accordance with the manner described in claim 6 or claim 7; When the current KV cache length exceeds the preset cache budget, the retention process, similar token fusion process, and discard process are triggered; when the current KV cache length does not exceed the preset cache budget, the current KV cache content is maintained, and the status information of historical tokens continues to be updated. The KV cache content in the retained Token set, the KV cache content obtained by fusing similar Tokens in the candidate fused Token set, and the historical Token state information that was not discarded are rearranged according to the Token position or the compressed position to form a compressed KV cache. The compressed KV cache is used as the historical cache input for subsequent inference steps of the large language model.

Images