Key-value cache data pruning processing method and device

Abstract

The application provides a key-value cache data pruning processing method and device. On one hand, the method first determines the attention score of a token; determines the activation importance of the token according to the value of each dimension in the value vector corresponding to the token; determines the integration score of the token based on the attention score and the activation importance; sorts the integration scores of multiple tokens, and determines the key-value data corresponding to N tokens to be cached according to the sorting result. On the other hand, the method also proposes to determine the cache level of each word element in the prompt word based on the semantic attribute of the word element, and configure different pruning rates for different cache levels to retain the KV data of the token with a higher security level in the prompt word. The method can solve the problem of significant distortion of the output caused by pruning based on the importance of the token determined by the attention score, and improve the safety and controllability of the model output.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method and apparatus for key-value cache data pruning. Background Technology

[0002] Key (K) and value (V) data caching involves caching already computed key and value vectors during the execution of a large model to avoid redundant computation. This caching is typically done in the video random access memory (VRAM) of the graphics processing unit (GPU). The amount of KV data grows linearly with the context length. To alleviate the capacity bottleneck of VRAM and other memory, a KV caching pruning strategy is employed. This strategy removes some KV data corresponding to tokens while retaining key information, thereby reducing memory usage and computational costs. Current KV caching pruning strategies only use attention scores to determine token importance, pruning KV data for tokens with relatively low attention scores. This can lead to the removal of KV data for tokens containing high-intensity feature signals, causing significant distortion in the output of the corresponding feature channels and reducing model performance. Furthermore, current KV data caching ignores the security levels of different tokens within the prompt words, leaving the security and controllability of the large model's output content needing improvement. Summary of the Invention

[0003] This invention provides a key-value cache data pruning method and apparatus to solve the problem of significant output distortion caused by using only attention scores to determine token importance for pruning, and to improve the security of KV data caching through a prompt word caching level mechanism.

[0004] This invention provides a key-value cache data pruning method, which includes: determining the attention score of the i-th lexical unit included in the prompt word relative to the observation window; the prompt word includes M lexical units, 1≤i≤M; the attention score is used to capture the relevance of the i-th lexical unit to the context; determining the activation importance of the i-th lexical unit based on the values ​​of each dimension in the value vector corresponding to the i-th lexical unit; the activation importance is used to capture the feature signal intensity of the value vector corresponding to the i-th lexical unit in each feature channel; determining the integrated score of the i-th lexical unit based on the attention score and the activation importance; sorting the multiple integrated scores corresponding to multiple lexical units, and determining N lexical units from the multiple lexical units according to the sorting result; caching the key-value data corresponding to the N lexical units; N<M.

[0005] In one embodiment, determining the activation importance of the i-th word based on the values ​​of each dimension in the value vector corresponding to the i-th word includes: selecting the value with the largest absolute value as the activation importance of the i-th word based on the values ​​of each dimension in the value vector corresponding to the i-th word.

[0006] In one embodiment, the method is applied to a GQA architecture; determining the attention score of the i-th lexical element included in the prompt word relative to the observation window includes: aggregating the attention scores of the i-th lexical elements corresponding to the G query heads within the same group relative to the observation window to obtain a first attention score; G≥2; determining the integration score of the i-th lexical element based on the attention score and activation importance includes: determining the integration score of the i-th lexical element based on the first attention score and activation importance; sorting the multiple integration scores corresponding to multiple lexical elements includes: sorting the multiple lexical elements within the same group based on the integration score. Alternatively, the method is applied to an MHA architecture; determining the attention score of the i-th lexical element included in the prompt word relative to the observation window includes: determining a second attention score of the i-th lexical element corresponding to each query head relative to the observation window; determining the integration score of the i-th lexical element based on the attention score and activation importance includes: determining the integration score of the i-th lexical element based on the second attention score and activation importance; sorting the multiple integration scores corresponding to multiple lexical elements includes: sorting the multiple lexical elements corresponding to a single head based on the integration score.

[0007] In one embodiment, before sorting the multiple integrated scores corresponding to multiple lexical units, the method further includes: determining the cache level of the i-th lexical unit based on its semantic attributes, and configuring different pruning rates for different cache levels.

[0008] In one embodiment, the semantic attributes include at least two of the following: system messages, key user instructions, and ordinary dialogue history. Determining the cache level corresponding to the i-th lexical unit based on its semantic attributes includes: determining that the i-th lexical unit has a first cache level based on its semantic attribute being a system message; configuring a first pruning rate for the first cache level; determining that the i-th lexical unit has a second cache level based on its semantic attribute being a key user instruction; configuring a second pruning rate for the second cache level; determining that the i-th lexical unit has a third cache level based on its semantic attribute being ordinary dialogue history; configuring a third pruning rate for the third cache level; wherein the first pruning rate < the second pruning rate < the third pruning rate.

[0009] In one embodiment, the semantic attribute further includes tool dialogue history; determining the cache level corresponding to the i-th word based on the semantic attribute of the i-th word includes: determining that the i-th word has a fourth cache level based on the semantic attribute of the i-th word being tool dialogue history; configuring a fourth pruning rate for the fourth cache level; and the third pruning rate < the fourth pruning rate.

[0010] In one embodiment, multiple integration scores corresponding to multiple lexical units are sorted, and N lexical units are determined from the multiple lexical units based on the sorting results. This includes: sorting multiple lexical units at the same cache level in descending order of integration scores, and selecting the top N lexical units from the multiple lexical units based on the pruning rate configured for the same cache level.

[0011] In one embodiment, the first pruning rate is 0; the second pruning rate is 0.7; and the third pruning rate is 0.9.

[0012] In one embodiment, the large model includes multiple attention heads, and each attention head includes multiple attention layers; the method further includes: in the multiple attention layers, at a preset number of layers interval, redetermining the lexical index set once; the lexical index set is the union of the different lexical index sets corresponding to different cache levels and the lexical index set corresponding to the observation window.

[0013] In one embodiment, after determining the integrated score of the i-th word element and before sorting the multiple integrated scores corresponding to multiple word elements, the method further includes: performing average pooling smoothing on the score sequence composed of the integrated scores of multiple word elements.

[0014] This invention also provides a key-value cache data pruning method, which includes: determining the cache level of each lexical unit based on the semantic attributes of each lexical unit among the M lexical units included in the prompt word; wherein different cache levels correspond to different pruning rates; the different cache levels represent different security levels; when the security level represented by the first cache level is higher than the security level represented by the second cache level, the first pruning rate corresponding to the first cache level is less than the second pruning rate corresponding to the second cache level; selecting N' lexical units from multiple lexical units of the same cache level according to the pruning rate; caching the key-value data corresponding to the N' lexical units; N' < M.

[0015] In one embodiment, semantic attributes include at least two of the following: system messages, key user instructions, general dialogue history, and tool dialogue history.

[0016] This invention also proposes a key-value cache data pruning processing device, which includes: an attention score determination module, used to determine the attention score of the i-th lexical element included in the prompt word relative to the observation window; the prompt word includes multiple lexical elements, i≥1; the attention score is used to capture the relevance of the i-th lexical element to the context; an activation importance determination module, used to determine the activation importance of the i-th lexical element based on the values ​​of each dimension in the value vector corresponding to the i-th lexical element; the activation importance is used to capture the feature signal intensity of the value vector corresponding to the i-th lexical element in each feature channel; a comprehensive scoring module, used to determine the integrated score of the i-th lexical element based on the attention score and the activation importance; and a first pruning module, used to sort the multiple integrated scores corresponding to the multiple lexical elements, determine N lexical elements from the multiple lexical elements according to the sorting result, and cache the key-value data corresponding to the N lexical elements; N<M.

[0017] This invention also provides a key-value cache data pruning processing device, which includes: a level division module, used to determine the cache level of each word based on the semantic attributes of each word in the M words included in the prompt word; wherein different cache levels correspond to different pruning rates; different cache levels represent different security levels; when the security level represented by the first cache level is higher than the security level represented by the second cache level, the first pruning rate corresponding to the first cache level is less than the second pruning rate corresponding to the second cache level; and a second pruning module, used to select N' words from multiple words in the same cache level according to the pruning rate; and cache the key-value data corresponding to the N' words; where N' < M.

[0018] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any of the key-value cache data pruning methods described above.

[0019] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the key-value cache data pruning methods described above.

[0020] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements any of the key-value cache data pruning methods described above.

[0021] The key-value cache data pruning method and apparatus provided by this invention represent the relevance of the i-th token to the context by the attention score of the i-th token relative to the observation window. This invention has discovered the impact of large activation values ​​in the token's value vector on the model's output quality. Pruning tokens with large activation values ​​may lead to a decrease in the output quality of large models or the overall performance of agents. Therefore, this invention introduces activation importance as an indicator to represent the importance of tokens. Activation importance is determined based on the magnitude of each dimension in the value vector. Activation importance can capture the feature signal intensity of the token's value vector in each feature channel. Combining the attention score and activation importance, a comprehensive integrated score is obtained. Using this integrated score to comprehensively evaluate the importance of tokens, compared to pruning strategies that rely solely on attention scores, can prevent the pruning of tokens with large activation values, allowing some tokens to be retained due to their high activation importance. This preserves the important feature signals of the corresponding feature channels, saving memory storage space and reducing the impact of pruning on the performance of large models or agents. On the other hand, this invention also proposes a hierarchical key-value data pruning method. For different prompt words with different security levels, they are divided into different caching levels according to semantic attributes, and different pruning rates are configured. The higher the security level, the lower the pruning rate and the more key-value data is retained, so that tokens with higher security levels can be retained. This controls the output of the large model to conform to the security prompt words, thus satisfying security requirements while performing key-value data pruning. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0023] Figure 1 This is one of the flowcharts illustrating the key-value cache data pruning method (large activation value) provided in this embodiment of the invention; Figure 2 This is one of the flowcharts illustrating the key-value cache data pruning method provided in this embodiment of the invention. Figure 3 This is a flowchart illustrating one embodiment of the key-value cache data pruning method provided in this invention. Figure 4 This is the second schematic diagram of the process architecture of the key-value cache data pruning processing method provided in the embodiment of the present invention; Figure 5 This is the second flowchart of the key-value cache data pruning method (hierarchical pruning) provided in the embodiment of the present invention; Figure 6a This is a diagram showing the performance score comparison of the Llama-2-7B model based on the LongBench benchmark test set, applying the methods and related technologies proposed in this invention, as well as the fullKV performance score. Figure 6b This is a schematic diagram showing the accuracy comparison of applying the proposed method and related technologies of this invention to the Llama-3.1-8B model based on the LongBench v2 test set, as well as the fullKV method. Figure 7 This is a performance comparison diagram of applying the method of this invention and related technologies to the Mistral-7B model architecture of GQA architecture; Figure 8 This is a schematic diagram comparing an embodiment of the key-value cache data pruning processing method provided in this invention with related technologies to illustrate technical differences. Figure 9 This is a schematic diagram of the module structure of the key-value cache data pruning processing device provided in an embodiment of the present invention; Figure 10 A schematic diagram of the structure of the electronic device provided in the embodiment of the present invention.

[0024] Figure label: 901: Attention Score Determination Module; 902: Activation Importance Determination Module; 903: Comprehensive Scoring Module; 904: Pruning Module; 1010: Processor; 1020: Communication interface; 1030: Memory; 1040: Communication bus. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0026] The amount of key-value (KV) cache data increases linearly with the context length of the input sequence. Taking a 16GB RTX (Ray Tracing TeXture) 4080 graphics card as an example, when running the large model Llama-2-7B (Large Language Model Meta AI 2 - 7 Billion parameters), the model parameters themselves occupy about 14GB, leaving only about 2GB of GPU memory for the KV cache data. This can only process about 256 tokens of context. In real-world scenarios such as long documents and multi-turn dialogues, the GPU memory is severely insufficient, limiting batch processing throughput. Increasing the GPU memory capacity would significantly increase costs, leading to high inference costs. Pruning the KV cache data can alleviate the problem of insufficient GPU memory capacity to some extent.

[0027] Key-value (KV) cache pruning methods in related technologies, such as H2O (Heavy-Hitter Oracle) and SnapKV (Snap Key-Value cache), rely solely on attention scores. H2O retains the tokens that contribute the most to the attention score and the tokens within the observation window, discarding the rest. SnapKV selects important KV positions for compression by observing the window at the end of the prompt words and clustering them for each attention head. During their research, the inventors discovered that these two pruning schemes, when determining the importance of a token, rely solely on the token's attention score, ignoring the magnitude information of the value vector itself. This may lead to the omission of key feature signals. Large activation values ​​in the value vector (i.e., the V in the KV data to be cached) directly determine the intensity of the feature signal output by the model. If tokens containing large activation values ​​are pruned, the output of the corresponding feature channel will be significantly distorted.

[0028] Furthermore, the related technology SnapKV uses a per-head pruning strategy, which is incompatible with the GQA (Grouped-QueryAttention) architecture. GQA divides query headers into several groups, and multiple query headers within each group share a set of key-value data, causing per-head pruning to conflict with the shared cache structure.

[0029] Furthermore, different semantic parts of the input sequence have different levels of importance. For example, the prompts input during the pre-filling stage generally include system prompts (system messages) and user prompts, which may specifically include at least one of the following: role / identity, task / instruction, context / background information, input data, output format constraints, examples, etc. The key-value cache data pruning method in related technologies adopts the same pruning strategy for different parts of the prompts, lacking the ability to perceive the importance of different instructions in the prompts. It treats the system prompts (system messages) that need to be followed as the same as ordinary historical messages (such as meaningless chat history conversations), causing the model or agent to focus on responding to instructions or tasks that do not require special attention, resulting in a decrease in compliance with the user's key instructions.

[0030] In view of this, embodiments of the present invention propose a key-value cache data pruning method (hereinafter referred to as the LevelKV method) and apparatus, which can solve the above problems. The LevelKV method adopts at least one of the following improvements: Improvement 1, a comprehensive importance scoring mechanism based on activation importance and attention score; Improvement 2, group awareness, adopting a group-level aggregation voting mechanism, compatible with GQA pruning strategy, and compatible with MHA (Multi-HeadAttention) pruning strategy when the number of query heads in the group is reduced to 1; Improvement 3, adopting a cache level mechanism (LevelDict), sensing the cache level of tokens in prompt words or input sequences, and configuring different pruning efficiencies for different cache levels. The LevelKV pruning scheme proposed in the embodiments of the present invention is compatible with existing frameworks, requires no training, is plug-and-play, and natively supports MHA and GQA.

[0031] The method proposed in this invention can be applied to various large models and / or agents employing attention mechanisms to prune and cache the key-value data generated by the large model or agent during operation. Specifically, it can be applied to large language models (LLMs) and / or multimodal large language models (MLLMs) and agents based on the transformer architecture, such as one or more of the following large models or agents: GPT (Generative Pre-trained Transformer) series, LLaMA (Large Language Model Meta AI) series, Qwen, Gemma, Mistral; LLaVA (Large Language and Vision Assistant), Qwen-VL / Qwen2.5-VL (Qwen Visual Language), InternVL (InternVision-Language) series, Phi-3.5-Vision, Mobile VLM (Mobile Vision-Language Model); Claude Code, codex, Open Code, SWE-agent, Open Hands, Hermes Agent, ToolOrchestra, Latent Mesh.

[0032] The key-value cache data pruning method and apparatus of the present invention are described in detail below. Figure 1 This is one of the flowcharts illustrating the key-value cache data pruning method provided by the present invention, such as... Figure 1 As shown, the specific process may include the following: Step 101: Determine the attention score of the i-th word element included in the cue word relative to the observation window.

[0033] The cue words consist of M tokens, where 1 ≤ i ≤ M. The attention score is used to capture the relevance of the i-th token to the context.

[0034] The observation window refers to a continuous sequence of tokens at the end of the input sequence. It is a fixed observation range. The large model uses the tokens within this range as query vectors to evaluate the importance of all historical tokens preceding this token.

[0035] For example, the attention score of the i-th token relative to the observation window is calculated as follows (1): (1); Let represent the attention score of the i-th token relative to the observation window. h This represents the attention head index, where i represents the token index in the input sequence.t W represents the token index within the observation window. obs [ h,i,t [] indicates that within the observation window, the i-th token in the header h corresponds to the i-th token in the observation window. t Attention weights for each token.

[0036] Step 102: Determine the activation importance of the i-th token based on the values ​​of each dimension in the value vector corresponding to the i-th token.

[0037] Activation importance is used to capture the feature signal intensity of the value vector corresponding to the i-th token in each feature channel. In other words, activation importance characterizes whether there is a large activation value in the token's value vector (V vector). If a large activation value exists, the activation importance of this token is determined based on the large activation value. In some embodiments, the activation importance of the i-th token is determined according to the values ​​of each dimension in the value vector corresponding to the i-th token. Specifically, it can be that the value with the largest absolute value in each dimension of the value vector corresponding to the i-th token is selected as the activation importance of the i-th token.

[0038] A large activation value refers to the value of an element in the V vector whose magnitude is significantly greater than the mean of all other elements. Specifically, it can be the value whose magnitude is greater than the sum of the mean and multiple standard deviations of all elements in the V vector. For example, if the V vector has 256 or 128 dimensions, meaning it has 256 or 128 elements, and each dimension or element corresponds to a feature channel, then if a feature channel has a large activation value, the subsequent neurons in that feature channel will be activated.

[0039] For example, the activation importance calculation formula is as follows (2): (2); in, This represents the activation importance of the i-th token, where i represents the token index, max represents finding the maximum value, and d represents the dimension of the V vector. This represents the magnitude (value) of the d-th dimension in the V vector of the i-th token.

[0040] For example, for the value vector V∈R DD represents the dimension of the V vector. Assuming D=256 or D=128, each dimension can be regarded as a feature channel. A few dimensions have large activation values ​​with amplitudes much larger than the mean. For example, V=[0.2, -0.1, 0.15, 6.8, …0.3, -0.2, 0.1, 0.05], where 6.8 is a large activation value much larger than the mean. This V vector corresponds to the activation importance of the token. For example, consider V = [0.12, -0.08, 0.21, -0.15, 0.18, ..., -4.5, ... 3.8, ..., 5.2, ..., -0.11, 0.09, 0.14, -0.07, 0.06]. In this 256-dimensional array, most values ​​fall between [-0.3, 0.3], with a mean μ ≈ 0.05. However, a few dimensions exhibit values ​​significantly higher than the mean, representing large activation values: -4.5, 3.8, and 5.2. Taking the largest absolute value, 5.2, as the activation importance of this token, results in pruning of the V vector containing these large activation values. Pruning this V vector would cause the loss of signals from feature channels with high response, leading to output distortion.

[0041] For vector V containing two or more large activation values, such as V = [0.12, -0.08, 0.21, -0.15, 0.18, ..., -4.5, ..., 3.8, ..., 5.2, ..., -0.11, 0.09, 0.14, -0.07, 0.06], the average or weighted sum of the two or more large activation values ​​-4.5, 3.8, and 5.2 can be used as the activation importance of the i-th token.

[0042] Step 103: Determine the integrated score of the i-th token based on the attention score and activation importance.

[0043] For example, in some embodiments, the integration score of the i-th token is determined based on the attention score and activation importance, which may be by calculating the product of the attention score and activation importance, for example, by calculating the integration score according to the following formula (3): (3); This represents the integrated score of the i-th token. Used to capture the relevance of the i-th token to the context. Used to capture the characteristic signal strength of the i-th token.

[0044] It should be noted that the above calculation formula is only an example. According to the above explanation, other implementation methods can be obtained. For example, the above formula (1) can be changed to find the maximum or minimum value (the large activation value with the symbol ﹣). The obtained activation importance is allowed to be negative. The absolute value of the total activation importance is then calculated in formula (3), or the absolute value is calculated based on the product obtained to obtain the integrated score.

[0045] In other embodiments, determining the integrated score of the i-th token based on the attention score and activation importance can be achieved by weighted summation of the attention score and activation importance. For example, the weights corresponding to the attention score and activation importance are hyperparameters, and the integrated score is obtained by weighted summation of the attention score and activation importance.

[0046] Based on steps 101 to 103 above, the integrated score of each token in the input sequence can be obtained.

[0047] Step 104: Sort the multiple integrated scores corresponding to multiple tokens, determine N tokens from the multiple tokens based on the sorting results, and cache the key-value data corresponding to the N tokens.

[0048] Remove the key-value data corresponding to tokens other than the N tokens in the prompt.

[0049] N < M, and both N and M are positive integers.

[0050] Sort the multiple integration scores corresponding to multiple tokens. Based on the sorting results, determine N tokens from the multiple tokens. Specifically, this can be done by sorting the integration scores of multiple tokens in descending order of integration score and selecting the top N tokens; or by sorting the integration scores of multiple tokens in ascending order of integration score and selecting the bottom N tokens. The determined N tokens are the tokens for which KV data needs to be retained, and record the indices of the N tokens in the input sequence.

[0051] Figure 1The illustrated embodiment, based on the attention score used to evaluate the importance of tokens, introduces activation importance. A comprehensive integrated score is obtained for each token in the input sequence based on the attention score and activation importance. Multiple tokens are then sorted based on this integrated score. N tokens that need to be cached are determined from the sorted tokens. Tokens containing large activation values ​​are retained due to their high activation importance, allowing the neurons in subsequent layers of the corresponding feature channel to be activated, thus preserving the important information of that feature channel. Compared to pruning strategies such as H2O and SnapKV, this approach saves GPU memory while significantly reducing the impact of pruning on the performance of large models, maintaining the stability of large model inference performance.

[0052] In the MHA architecture, each query header h has independent key-value cache data: ; In GQA (Grouped Query Attention), G query headers share the same set of key-value cache data: .

[0053] H2O or SnapKV's pruning strategy uses per-head pruning, selecting different subsets of tokens for different heads, which directly conflicts with GQA's shared cache structure.

[0054] In some embodiments, to address the incompatibility between the head-based pruning strategy and GQA, this invention proposes a group-level solution for LevelKV, which employs a group voting mechanism. For example, the attention scores S1 of all query heads within the same group are aggregated as shown in equation (4): (4); G indicates that the same group includes G query headers, L obs The length of the observation window is represented by t∈L. obs This indicates that the current token is the t-th token in the observation window. This represents the attention score after aggregation within the same group.

[0055] In this way, when ranking, the integrated score is ranked only within the same group, and a uniform importance (i.e., integrated score) ranking is obtained within each group.

[0056] Thus, pruning is performed at the group level, naturally compatible with the GQA architecture. When G=1, it naturally degenerates into MHA, meaning the pruning method proposed in this embodiment is compatible with the MHA architecture.

[0057] In the LongBench benchmark test, the aforementioned large activation value evaluation mechanism combined with LevelKV was used to prune the KV data generated by the Mistral-7B (GQA) model to evaluate the model performance. This benchmark covers various long text tasks such as single / multi-document question answering, summary generation, and few-shot learning. Experimental results show that the method of this embodiment achieves an average accuracy of 30.58%, which is 14.96% higher than the current state-of-the-art method (26.60%). This fully verifies the effectiveness of the GQA-compatible pruning mechanism proposed in this embodiment and the superiority of large activation value awareness in long text compression scenarios.

[0058] Based on the above exemplary description, for a GQA architecture where G query headers (G≥2) share the same set of key-value cache data, the attention score of the i-th token included in the prompt word relative to the observation window is determined. Specifically, this can be done by determining the first attention score of the i-th token corresponding to the h-th query header within the same group relative to the observation window, where 1≤h≤G. For example, the first attention score is calculated according to formula (1). Next, according to formula (4), the first attention scores of the i-th token corresponding to the G query headers within the same group relative to the observation window are aggregated to obtain the first attention score. Based on the first attention score and activation importance, the integration score of the i-th token is determined. Correspondingly, in step 104, the multiple integration scores corresponding to multiple tokens are sorted. Specifically, this can be done by sorting all tokens within the same group based on the integration score, thus ensuring compatibility with GQA pruning strategies.

[0059] For an MHA architecture that configures independent key-value cache data for a single query head, determine the attention score of the i-th token included in the prompt word relative to the observation window. Specifically, determine the second attention score of the i-th token corresponding to each query head relative to the observation window. For example, calculate the attention score of each token inside each query head according to the above calculation formula (1). Based on the second attention score and activation importance, determine the integration score of the i-th token. Then, based on the integration score, sort the multiple tokens corresponding to a single head. In this way, the head-pruning strategy can be compatible.

[0060] To address the issue that some related technologies treat system prompts and ordinary historical messages equally, leading to a decrease in user compliance with critical instructions, this invention proposes an instruction-level retention mechanism called LevelDict. This mechanism configures a permission level dictionary and allows users to customize the levels (caching levels) of multiple parts included in custom prompt words.

[0061] For example, tokens in the prompt can be assigned to different cache levels based on their semantic attributes, and different pruning rates can be configured for different cache levels: The first cache level L0 can include (system messages) system prompts, security rules, etc., and is configured with a first pruning rate r0=0; The second cache level L1 can include important messages such as key user instructions, and is configured with a second pruning rate r1 = 0.7; The third cache level, L2, can include general conversation history (normal conversation history) and is configured with a third pruning rate r3 = 0.9.

[0062] The caching level represents the importance of a token within the prompt, and is determined based on the semantic attributes of each token within the prompt. Semantic attributes refer to which part of the prompt the token semantically belongs to. For example, semantic attributes may include at least two of the following: system messages, key user commands, and general conversation history. In other embodiments, semantic attributes may further include at least one of the following: role, output format constraints, example, etc.

[0063] The above pruning rate configuration is just an example. The higher the cache level (the higher the importance), the lower the corresponding pruning rate, in order to retain more important features. The lower the cache level (the lower the importance), the higher the corresponding pruning rate, in order to remove more valueless token KV data that occupies GPU memory.

[0064] Based on the above example, before sorting the multiple integrated scores corresponding to multiple tokens, optionally, the cache level corresponding to the i-th token can be determined based on the semantic attributes of the i-th token included in the prompt word. Different cache levels correspond to different pruning rates. For example, specifically, based on the semantic attribute of the i-th token included in the prompt word being a system message, the i-th token can be determined to correspond to a first cache level, and the first cache level corresponds to a first pruning rate; based on the semantic attribute of the i-th token being a key user instruction, the i-th token can be determined to correspond to a second cache level, and the second cache level corresponds to a second pruning rate; based on the semantic attribute of the i-th token being ordinary dialogue history, the i-th token can be determined to correspond to a third cache level, and the third cache level corresponds to a third pruning rate. The first pruning rate < the second pruning rate < the third pruning rate.

[0065] During their research, the inventors discovered that removing key-value pairs (KV) from low-level caches could actually improve the model's compliance with high-privilege instructions (i.e., critical user instructions) and reduce the interference effect of invalid instructions.

[0066] It should be noted that multiple tokens within the observation window are not pruned, meaning the pruning rate is 0. For example, if the observation window takes the last 32 tokens of the input sequence, and Lobs = 32 tokens, these 32 tokens within the observation window will never be pruned to ensure local contextual coherence.

[0067] In this way, zero-pruning protection is applied to important information such as system messages at cache level L0, ensuring that important information is preserved intact and preventing the accidental deletion of important security rules. Furthermore, tokens at different cache levels can be pruned independently according to their respective cache levels. Multiple tokens at the same cache level are integrated, scored, and sorted, and the Top-N tokens are selected, allowing the total budget for existing storage space to be flexibly allocated among different cache levels.

[0068] Specifically, in some embodiments, the M tokens at level j are sorted in descending order of their integration scores based on the integration score, and the top N tokens at level j are selected from the multiple tokens at level j according to the pruning rate corresponding to level j, to obtain the token index set corresponding to level j; j≥1.

[0069] Alternatively, based on the integration score, sort the M tokens at level j in ascending order of integration score, and select the last N tokens from the multiple tokens at level j according to the pruning rate corresponding to level j, to obtain the token index set corresponding to level j.

[0070] For example, suppose the number of tokens at level j is P, then N = P × (1 - r) j ); r j This represents the pruning rate corresponding to level j.

[0071] It should be noted that the method proposed in this embodiment supports cross-layer index reuse. The large model includes multiple attention layers, where each attention layer includes attention computation, which involves operations using at least one of Q-vectors, K-vectors, and V-vectors. In this embodiment, for attention layers other than the first layer, the index set of the first layer can be directly reused to reduce the computational overhead of repeatedly calculating the activation magnitude and attention score of each token in subsequent layers.

[0072] The following is a specific example. For instance... Figure 2 As shown, this specific embodiment can be based on Figure 2 The process architecture shown is implemented.

[0073] This specific embodiment uses a GQA-compatible architecture as an example. In the pre-filling stage, the attention weight matrix within the observation window is first calculated for each attention layer. According to the grouping of the GQA architecture, the attention scores of the tokens in each group are aggregated to obtain the attention score S1. The maximum absolute value of the Value vector for each token is taken to obtain the activation magnitude (i.e., activation importance) S2. The product of the attention score and the activation importance is calculated to obtain the integrated score value of each token. Optionally, this specific embodiment uses a 1-dimensional average pooling smoothing method to smooth the integrated scores of multiple tokens in the input sequence to reduce noise. Then, the multiple integrated scores within the same group are sorted. Next, following the LevelDict mechanism, the caching level of each token in the input sequence is determined, and the corresponding pruning rate is determined based on the caching level. For example, the pruning rate for system messages is 0, so no pruning is needed. The pruning rate for critical user instructions is 0.7, so based on the pruning rate, 30% of the tokens with the semantic attribute of critical user instructions are retained, and the Top-N tokens are selected. The index set corresponding to this caching level is recorded. The index sets of the tokens retained at each caching level are concatenated to obtain the first-level index set. Subsequent layers directly select the K and V vectors of the tokens at the corresponding index positions according to the first-level index set for caching, without the need for re-scoring. In the first step (time step) of the decoding stage, the KV data pruned in the pre-filling stage can be read directly.

[0074] Specifically, this specific embodiment may include the following process: Step 301: Calculate the attention weight of the i-th token relative to the observation window.

[0075] (5); Where Wobs represents the attention weight matrix of the observation window; softmax represents the normalized exponential function; Q obs represents the Query matrix of the observation window, K T Represents the transpose of the complete key matrix. This represents the scaling factor, which is the square root of the dimension of the Key vector, to prevent the gradient from entering the saturation region due to excessively large dot product values. d k The dimension of a single attention head is indicated, specifically the dimension of the Key vector or Query vector, which is typically 128 in Llama-2-7B. L Indicates the length of the complete sequence, which is the total number of tokens for the prompt (input Prompt); L obs represents the length of the observation window; The dimension is G×L obs A tensor of ×L.

[0076] Step 302: Perform group aggregation to aggregate the attention scores of multiple query heads within the same group to obtain the aggregated attention score.

[0077] For example, the attention score is obtained by performing aggregation according to the above calculation formula (4).

[0078] Step 303: Determine the activation threshold corresponding to the i-th token.

[0079] For example, the activation magnitude of the i-th token is calculated according to the above formula (2).

[0080] Step 304: Calculate the product of the activation magnitude and attention score of each token to obtain the integrated score.

[0081] Step 305: Smooth the integrated scoring sequence.

[0082] A specific implementation of smoothing can be a one-dimensional average pooling smoothing process. Assuming the input sequence contains M tokens, the above steps yield M integrated scores for the M tokens, forming a score sequence. The scores of individual tokens may contain noise; for example, a token might receive an abnormally high score due to contextual fluctuations. Smoothing ensures that each token's score references the scores of its neighboring tokens, thus making the importance ranking more stable and robust.

[0083] For example, 1D average pooling smoothing is performed, using a kernel width... z For the integrated score of the i-th token in the scoring sequence, take the scores before and after the i-th integrated score. z The average of / 2 neighbors (including itself) is used as the new integrated score for the i-th position, thus obtaining a smoothed score sequence.

[0084] The above-described 1D average pooling smoothing method is merely an example. In other embodiments, Gaussian smoothing or median filtering can be used instead of the above-described 1D average pooling smoothing.

[0085] Step 306: Sort the smoothed score sequence according to the integrated score.

[0086] In this specific embodiment, the scores are sorted in descending order of integration score.

[0087] Step 307: Determine the cache level corresponding to the i-th token based on the semantic attributes of the i-th token.

[0088] For example, if the i-th token is a token in a system message, then the semantic attribute of the i-th token is system message, and it has a first caching level. If the i-th token is a token in a critical user instruction, then the semantic attribute of the i-th token is critical user instruction, and it has a second caching level. If the i-th token is a token in the ordinary dialogue history, then the semantic attribute of the i-th token is ordinary dialogue history, and it has a third caching level.

[0089] In practical applications, semantic attributes are not limited to the system messages, key user commands, and ordinary dialogue history mentioned above. They can also include other semantic attributes, such as role settings, output format constraints, and examples. The caching levels of various semantic attributes and the pruning rates corresponding to each caching level can be specifically set by the user based on the relative importance of each part during actual runtime and the storage capacity of the video memory.

[0090] Step 308: Sort multiple tokens at the same cache level in descending order of integration score, and select the top N tokens from the multiple tokens based on the pruning rate configured for the same cache level.

[0091] For each cache level ∈{L0,L1,L2}, calculate the retention rate. (6); Indicates cache level The set of token indices selected for retention; TopN represents the operation of selecting the N tokens with the highest integration scores; This represents the score sequence after smoothing. [tokens∈ ] indicates that the cache level is A subset of the token's rating; Indicates cache level The pruning rate; · Indicates rounding up, | | indicates the cache level The total number of tokens in the system.

[0092] For example, for each token in a system message with a cache level of L0, if the first pruning rate r0=0 is used, no pruning is required, and the key-value data of all tokens in the system message are retained.

[0093] For each token in an important message with cache level L1, pruning is performed using a second pruning rate r1=0.7, with a retention rate of 1-pruning rate=0.3. The tokens in the important message are sorted in descending order of integration score, and only the top 30% of N tokens are retained. For example, if an important message contains 528 tokens, the key-value data of the top 158 tokens are retained and cached in video memory, while the key-value data of the bottom 370 tokens are pruned and do not need to be cached in video memory.

[0094] For each token in the general dialogue history with a cache level of L2, pruning is performed using a second pruning rate r1=0.9, and the retention rate = 1 - pruning rate = 0.1. The multiple tokens in the general dialogue history are sorted in descending order of integration score, and only the top 10% of N tokens are retained. For example, assuming that the general dialogue history includes 1000 tokens, the key-value data of the top 100 tokens are retained and cached in video memory, while the key-value data of the bottom 900 tokens are pruned and do not need to be cached in video memory.

[0095] Step 309: Determine the set of first-layer indices corresponding to the first layer among multiple attention layers.

[0096] Specifically, at the first layer of multiple attention layers, the set of token indices for each level of cache is determined, and the set of token indices for multiple tokens within the observation window is determined. Then, the union of the set of token indices for each level and the set of token indices for the observation window is calculated to obtain the first-layer index set.

[0097] Concatenate the index sets corresponding to each cache level and the index sets corresponding to the observation window: (7); (8); Represents the first-level index set; This represents the set of indices corresponding to cache level L0. This represents the set of indices corresponding to cache level L1. This represents the set of indices corresponding to cache level L2. Indicates the observation window L obs The corresponding set of indices; K pruned This represents the pruned Key matrix; V pruned This represents the Value matrix after pruning.

[0098] This indicates retrieving the key matrix by its first-level index set. The token ID in the data is used to select the key vector of the corresponding token and cache it.

[0099] This indicates retrieving values ​​from the Value matrix according to the first-level index set. The token ID in the data is used to select the value vector of the corresponding token and cache it.

[0100] Step 310: In multiple attention layers, at each preset number of layers, redetermine the lexical index set once.

[0101] For example, in multiple attention layers, the term index set is recalculated every 6 layers. The large model includes multiple attention heads, and each attention head includes multiple attention layers. For example, in the MHA architecture, the large model includes multiple attention heads, and each attention head includes multiple attention layers. For example, an attention layer may include 30 layers. The term index set can be recalculated every 6 layers without needing to redetermine which tokens' key-value data to retain. For non-head layers in the 6 layers, steps 301-309 above can be skipped, and the term index set of the head layer in the 6 layers can be directly reused. In this way, the 6 layers share the same term index set.

[0102] The lexical index set is the union of the different lexical index sets corresponding to different cache levels and the lexical index set corresponding to the observation window.

[0103] It should be noted that in the MHA or GQA architecture, the last layer of each attention head needs to recalculate the lexical index set.

[0104] It should be noted that, Figure 2 The architecture shown is merely an exemplary one, and does not depict steps such as calculating an integration score based on attention scores and activation importance. Other architectures may be used in other embodiments. Figure 2 This is not intended to be a necessary limitation on the methods presented in the embodiments of the invention.

[0105] The improvements one, two, and three proposed in this embodiment of the invention can be implemented individually or in combination, as described above. Figure 3 The illustrated embodiment is an example of a combined implementation. It should be noted that improvements one and two can be applied to the pre-filling stage, and improvement three can be applied to either the pre-filling stage or the decoding stage, for example, as... Figure 4 As shown, Figure 4The diagram illustrates an exemplary architecture where Improvement Points 1 and 2 are applied to the pre-filling stage, and Improvement Point 3 is applied to the decoding stage. In the pre-filling stage, under the GQA architecture, attention scores from multiple query headers within a group are aggregated to obtain an attention score S1 and an activation importance S2 for each token. The product of the attention score and activation importance is calculated to obtain an integrated score. Tokens are sorted based on this integrated score, and the key-value data of the top-N tokens are selected for caching according to a uniform pruning rate. The input sequence for the pre-filling stage is the prompt words. An initial key-value cache is calculated based on the prompt words, and the first pruning is performed. In the decoding stage, the input sequence includes the prompt words, and a new token is generated at each step. For each new token generated, its key-value data is appended to the end of the cache, and then the Top-N selection by cache level is re-executed. Because the addition of new tokens changes the sequence length and context importance distribution, the retained token set needs to be dynamically updated.

[0106] like Figure 5 As shown, based on improvement point two, this embodiment of the invention also provides a key-value cache data pruning method based on instruction hierarchy pruning, which includes: Step 501: Determine the caching level of each word based on the semantic attributes of each word in the M words included in the prompt word.

[0107] Different cache levels correspond to different pruning rates, and different cache levels represent different security levels. When the security level represented by the first cache level is higher than the security level represented by the second cache level, the first pruning rate corresponding to the first cache level is less than the second pruning rate corresponding to the second cache level.

[0108] Semantic attributes include at least two of the following: system messages, key user commands, general dialogue history, and tool dialogue history.

[0109] For example, when the application object is a large model, the semantic attributes include at least two of the following: system messages, key user commands, and general dialogue history. For instance, determining the cache level corresponding to the i-th lexical based on its semantic attributes can specifically include: Step 5011: Based on the semantic attribute of the i-th word being a system message, determine that the i-th word has a first cache level; configure the first cache level with a first pruning rate; Step 5012: Based on the semantic attribute of the i-th word as a key user instruction, determine that the i-th word has a second cache level; configure the second cache level with a second pruning rate; Step 5013: Based on the semantic attribute of the i-th word being ordinary dialogue history, determine that the i-th word has a third cache level; configure the third cache level with a third pruning rate; First pruning rate < Second pruning rate < Third pruning rate.

[0110] When the application target is an intelligent agent, the prompt word further includes the tool dialogue history, and correspondingly, the semantic attribute further includes the tool dialogue history. Based on the semantic attribute of the i-th word element, the cache level corresponding to the i-th word element is determined, which also includes determining that the i-th word element has a fourth cache level based on the semantic attribute of the i-th word element being tool dialogue history. The fourth cache level is configured with a fourth pruning rate; the third pruning rate < the fourth pruning rate.

[0111] It's important to note that the caching level can be determined based on the semantic attributes of each lexical unit and the security level of its actual semantic representation. For example, if the user instruction contains instructions with a higher security level—such as a prompt that includes both system instructions (system prompts) and user instructions (user prompts) containing phrases like "Do not output content unsuitable for children" or "Do not output content related to tobacco and alcohol"—then the caching level for the user instructions can be increased, even higher than that of the system prompts, to ensure that important information is not lost, thereby improving the security of the model's or agent's output. In other words, the caching level of each lexical unit in the prompt is determined comprehensively based on the semantic attributes represented by that lexical unit and the specific semantic meaning it represents.

[0112] Step 502: Select N' lexical units from multiple lexical units at the same cache level according to the pruning rate; cache the key-value data corresponding to the N' lexical units.

[0113] N' < M, where N' is an integer. N = N' or N ≠ N'.

[0114] like Figure 6a As shown, Figure 6a This demonstrates the implementation using the LongBench benchmark suite. Figure 1 or Figure 3 Example: Performance comparison experimental data table of Llama-2-7B model. Figure 6aIn the KV Size, 256 or 128 indicates that the number of key or value vectors after pruning is 256 or 128, which is the maximum number of tokens that the KV cache space can hold. The numbers 30.12, 27.53, etc., are the average scores (e.g., F1 score or accuracy) of various pruning strategies on the LongBench benchmark. A higher score indicates a stronger ability of the model to maintain its original performance after KV cache pruning. FullKV (baseline) represents the case without compression (i.e., no pruning), which preserves the complete KV cache. The corresponding score of 30.12 is the upper bound of performance sought by various pruning methods. The pruning schemes proposed in this embodiment of the invention scored 29.78 and 28.93 points respectively, very close to the performance upper bound.

[0115] As can be seen, on the Llama-2-7B model, the LevelKV method proposed in this embodiment of the invention achieves higher performance scores than related methods (SnapKV, H2O, Streaming LLM) regardless of whether the KV size is 256 or 128, and is very close to the baseline of uncompressed FullKV. This indicates that the pruning strategy of this embodiment of the invention can better preserve information that is crucial for understanding long texts while compressing KV cache data.

[0116] Figure 6b The example demonstrates the implementation using the LongBench v2 test suite. Figure 5 The example shown is a performance comparison experiment data table of the Llama-3.1-8B model. The values ​​27.04%, 28.50%, and 30.82% represent the accuracy. FullKV also represents the uncompressed (unpruned) case. The present invention achieved an accuracy of 30.82%, which even surpasses the uncompressed FullKV.

[0117] Figure 7 The data shown indicates that on the Mistral-7B (GQA architecture), when the KV cache is compressed to a maximum of 128, the scores of related methods (H2O / SnapKV / AdaKV) all drop to 26.60%, while the scores proposed in this embodiment of the invention are significantly higher. Figure 1 or Figure 5 The LevelKV method in the illustrated embodiment improves performance by 30.58% with its innovative pruning strategy, representing a relative improvement of approximately 15%, fully demonstrating the superiority of LevelKV in extreme compression scenarios.

[0118] from Figure 6a and Figure 7As can be seen, when KV Size=128 (only 1 / 16 of the baseline), LevelKV still outperforms Streaming LLM (2048 tokens) by about 22.9%, and leads other pruning methods by about 4 percentage points in the GQA model. It can be seen that the Value activation awareness + cache level retention mechanism (hierarchical pruning) adopted in the embodiment of the present invention jointly contribute to the technical effect of performance improvement.

[0119] like Figure 8 As shown, Figure 8 This illustrates the technical differences between the LevelKV method proposed in this invention and methods in related technologies. Figure 8 It can be seen that the method proposed in this invention is the only one that simultaneously satisfies all four characteristics: value cache awareness, GQA compatibility, instruction hierarchy awareness, and no need for training. It is plug-and-play and requires no fine-tuning or retraining. It outperforms other methods by about 4 percentage points on mainstream GQA models (Mistral, Llama-3).

[0120] Based on the above exemplary description, the LevelKV method proposed in this embodiment of the invention adopts an overall KV importance scoring method (S=S1×S2), and for the first time incorporates large activation values ​​of the Value cache into the KV pruning decision. By fusing the product of attention importance and activation magnitude, more accurate token retention is achieved.

[0121] Furthermore, the LevelKV method proposed in this invention employs a group-level pruning mechanism that is natively compatible with group query attention. It aggregates importance scores at the GQA group level, resolving compatibility conflicts between existing methods and the GQA architecture for the first time, and uniformly supporting both MHA and GQA, two mainstream attention architectures.

[0122] This invention further proposes a hierarchical awareness pruning framework based on LevelDict. By using a user-defined permission level dictionary, it achieves zero-pruning protection for system instructions and discovers the counterintuitive phenomenon that hierarchical pruning can improve the compliance of high-privilege instructions.

[0123] Furthermore, the method proposed in this embodiment of the invention supports a cross-layer index reuse mechanism, which allows non-first-layer elements to directly reuse the selection index of the previous layer's head, reducing the overhead of repeatedly calculating attention scores and activation importance, and reducing additional latency in the pre-filling stage while maintaining performance.

[0124] In summary, the LevelKV method proposed in this invention is a complete hierarchical KV caching and pruning framework, achieving breakthroughs in the following three aspects: In terms of accuracy: Through S1×S2 integrated scoring, it even surpasses the uncompressed baseline on LongBench v2; In terms of compatibility: Group-level GQA pruning mechanism, natively supports current mainstream GQA model architectures; In terms of reliability: The LevelDict cache level retention mechanism improves system instruction compliance by approximately 14%.

[0125] In practical applications of this technology, long-context inference is a core bottleneck in deploying large models. Key-value (KV) cache compression (pruning) has widespread technical demand and application value. The method proposed in this invention requires no training and can be directly integrated into existing inference frameworks (vLLM, etc.), demonstrating its practicality. Employing the LevelDict framework, it possesses scalability, extending to more layers and real-world application scenarios. In short, the LevelKV method proposed in this invention achieves better inference performance with less caching.

[0126] The key-value cache data pruning processing apparatus provided in the embodiments of the present invention is described below. The key-value cache data pruning processing apparatus described below can be referred to in correspondence with the key-value cache data pruning processing method described above.

[0127] like Figure 9 As shown in the embodiment of the present invention, a key-value cache data pruning processing device is proposed, the device comprising: Attention score determination module 901 is used to determine the attention score of the i-th token included in the prompt word relative to the observation window.

[0128] The activation importance determination module 902 is used to determine the activation importance of the i-th token based on the values ​​of each dimension in the value vector corresponding to the i-th token.

[0129] The integrated scoring module 903 is used to determine the integrated score of the i-th token based on the attention score and activation importance.

[0130] The pruning module 904 is used to sort multiple integrated scores corresponding to multiple tokens, determine N tokens from the multiple tokens based on the sorting results, and cache the key-value data corresponding to the N tokens.

[0131] On the other hand, embodiments of the present invention also provide a key-value cache data pruning processing device, the device comprising: a level division module, used to determine the cache level of each word based on the semantic attributes of each word in the M words included in the prompt word; wherein, different cache levels correspond to different pruning rates; different cache levels represent different security levels; when the security level represented by the first cache level is higher than the security level represented by the second cache level, the first pruning rate corresponding to the first cache level is less than the second pruning rate corresponding to the second cache level; The second pruning module is used to select N' lexical units from multiple lexical units at the same cache level according to the pruning rate; cache the key-value data corresponding to the N' lexical units; N' < M.

[0132] Figure 10 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 10 As shown, the electronic device may include a processor 1010, a communications interface 1020, a memory 1030, and a communication bus 1040, wherein the processor 1010, the communications interface 1020, and the memory 1030 communicate with each other via the communication bus 1040. In some embodiments, the processor 1010 may include the processor 1010 of any of the above embodiments, for example, the processor 1010 may be the processor 1010 of any of the above embodiments, or the processor 1010 may be a processor array composed of the processors 1010 of any of the above embodiments.

[0133] Furthermore, the logical instructions in the aforementioned memory 1030 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0134] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the key-value cache data pruning processing method provided by the above methods, the method including: Determine the attention score of the i-th token relative to the observation window, which is included in the cue words; the cue words include M tokens, 1≤i≤M; the attention score is used to capture the relevance of the i-th token to the context; determine the activation importance of the i-th token based on the values ​​of each dimension in the value vector corresponding to the i-th token; the activation importance is used to capture the feature signal intensity of the value vector corresponding to the i-th token in each feature channel; determine the integrated score of the i-th token based on the attention score and activation importance; sort the multiple integrated scores corresponding to multiple tokens, and determine N tokens from the multiple tokens based on the sorting results; cache the key-value data corresponding to the N tokens; N<M.

[0135] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the key-value cache data pruning processing method provided by the above methods, the method comprising: Determine the attention score of the i-th token relative to the observation window, which is included in the cue words; the cue words include M tokens, 1≤i≤M; the attention score is used to capture the relevance of the i-th token to the context; determine the activation importance of the i-th token based on the values ​​of each dimension in the value vector corresponding to the i-th token; the activation importance is used to capture the feature signal intensity of the value vector corresponding to the i-th token in each feature channel; determine the integrated score of the i-th token based on the attention score and activation importance; sort the multiple integrated scores corresponding to multiple tokens, and determine N tokens from the multiple tokens based on the sorting results; cache the key-value data corresponding to the N tokens; N<M.

[0136] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0137] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0138] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for pruning key-value cache data, characterized in that, The method includes: Determine the attention score of the i-th lexical unit included in the cue word relative to the observation window; the cue word includes M lexical units, 1≤i≤M; the attention score is used to capture the relevance of the i-th lexical unit to the context; The activation importance of the i-th word is determined based on the values ​​of each dimension in the value vector corresponding to the i-th word; the activation importance is used to capture the feature signal intensity of the value vector corresponding to the i-th word in each feature channel. Based on the attention score and the activation importance, determine the integrated score of the i-th word element; The integrated scores corresponding to multiple lexical units are sorted, and N lexical units are determined from the multiple lexical units based on the sorting results; the key-value data corresponding to the N lexical units are cached; N < M.

2. The method according to claim 1, characterized in that, Based on the values ​​of each dimension in the value vector corresponding to the i-th word, the activation importance of the i-th word is determined, including: Based on the values ​​of each dimension in the value vector corresponding to the i-th word, the value with the largest absolute value is selected as the activation importance of the i-th word.

3. The method according to claim 1 or 2, characterized in that, The method is applied to the GQA architecture; Determine the attention score of the i-th lexical unit included in the cue word relative to the observation window, including: The attention scores of the i-th word corresponding to each of the G query heads in the same group relative to the observation window are aggregated to obtain the first attention score; G≥2; Based on the attention score and the activation importance, the integrated score for the i-th lexical unit is determined, including: Based on the first attention score and the activation importance, determine the integrated score of the i-th word element; The integrated scores corresponding to multiple lexical units are ranked, including: Based on the integrated score, multiple word units within the same group are ranked; Alternatively, the method can be applied to an MHA architecture; Determine the attention score of the i-th lexical unit included in the cue word relative to the observation window, including: Determine the second attention score of the i-th term corresponding to each query head relative to the observation window; Based on the attention score and the activation importance, the integrated score for the i-th lexical unit is determined, including: Based on the second attention score and the activation importance, the integrated score of the i-th word is determined; The integrated scores corresponding to multiple lexical units are ranked, including: Based on the integrated score, the multiple lexical units corresponding to a single head are sorted.

4. The method according to claim 1, characterized in that, Before ranking the multiple integrated scores corresponding to multiple lexical units, the method further includes: Based on the semantic attributes of the i-th lexical unit, the cache level of the i-th lexical unit is determined, and different pruning rates are configured for different cache levels.

5. The method according to claim 4, characterized in that, The semantic attributes include at least two of the following: system messages, key user commands, and ordinary dialogue history; Based on the semantic attributes of the i-th lexical unit, the cache level corresponding to the i-th lexical unit is determined, including: Based on the semantic attribute of the i-th word being a system message, it is determined that the i-th word has a first cache level; the first cache level is configured with a first pruning rate; Based on the semantic attribute of the i-th word as a key user instruction, it is determined that the i-th word has a second cache level; the second cache level is configured with a second pruning rate; Based on the semantic attribute of the i-th word element being ordinary dialogue history, it is determined that the i-th word element has a third cache level; the third cache level is configured with a third pruning rate; The first pruning rate < the second pruning rate < the third pruning rate.

6. The method according to claim 5, characterized in that, The semantic attributes also include tool dialogue history; Based on the semantic attributes of the i-th lexical unit, the cache level corresponding to the i-th lexical unit is determined, including: Based on the semantic attribute of the i-th word as the tool dialogue history, it is determined that the i-th word has a fourth cache level; the fourth cache level is configured with a fourth pruning rate; The third pruning rate is less than the fourth pruning rate.

7. The method according to any one of claims 4-6, characterized in that, The integrated scores corresponding to multiple lexical units are ranked, and N lexical units are determined from the multiple lexical units based on the ranking results, including: Multiple terms at the same cache level are sorted in descending order of integration score, and the top N terms are selected from the multiple terms based on the pruning rate configured for the same cache level.

8. The method according to claim 1 or 4, characterized in that, The large model includes multiple attention heads, and each attention head includes multiple attention layers; the method further includes: In the multiple attention layers, the lexical index set is redefined every preset number of layers; The term index set is the union of the different term index sets corresponding to different cache levels and the term index set corresponding to the observation window.

9. A method for pruning key-value cache data, characterized in that, The method includes: Based on the semantic attributes of each of the M tokens included in the prompt word, the caching level of each token is determined; wherein, different caching levels correspond to different pruning rates; the different caching levels represent different security levels; when the security level represented by the first caching level is higher than the security level represented by the second caching level, the first pruning rate corresponding to the first caching level is less than the second pruning rate corresponding to the second caching level. From multiple lexical units at the same cache level, select N' lexical units according to the pruning rate; cache the key-value data corresponding to the N' lexical units; N' < M.

10. The method according to claim 9, characterized in that, The semantic attributes include at least two of the following: system messages, key user commands, general dialogue history, and tool dialogue history.

11. A key-value cache data pruning processing device, characterized in that, The device includes: An attention score determination module is used to determine the attention score of the i-th lexical unit included in the prompt word relative to the observation window; the prompt word includes multiple lexical units, i≥1; the attention score is used to capture the relevance of the i-th lexical unit to the context; The activation importance determination module is used to determine the activation importance of the i-th word based on the values ​​of each dimension in the value vector corresponding to the i-th word; the activation importance is used to capture the feature signal intensity of the value vector corresponding to the i-th word in each feature channel; The comprehensive scoring module is used to determine the integrated score of the i-th word element based on the attention score and the activation importance. The first pruning module is used to sort multiple integrated scores corresponding to multiple lexical units, determine N lexical units from the multiple lexical units according to the sorting results, and cache the key-value data corresponding to the N lexical units; N < M.

12. A key-value cache data pruning processing device, characterized in that, The device includes: The level division module is used to determine the caching level of each word based on the semantic attributes of each word in the M words included in the prompt word; wherein, different caching levels correspond to different pruning rates; the different caching levels represent different security levels; when the security level represented by the first caching level is higher than the security level represented by the second caching level, the first pruning rate corresponding to the first caching level is less than the second pruning rate corresponding to the second caching level. The second pruning module is used to select N' lexical units from multiple lexical units at the same cache level according to the pruning rate; and to cache the key-value data corresponding to the N' lexical units; N' < M.

Images