A fixed-capacity KV cache management method, device and equipment based on multi-dimensional dynamic importance evaluation and a storage medium

Abstract

The application provides a fixed-capacity KV cache management method and device based on multi-dimensional dynamic importance evaluation, equipment and storage medium, aiming at the problem that the reserved token in the fixed-capacity KV cache is difficult to cover the actual required history token generated subsequently, the scores of the attention accumulation dimension, the position dimension and the semantic particularity dimension of each token in the cache are weighted and fused to obtain a comprehensive importance score, and the comprehensive importance score is dynamically updated with the reasoning process; when the number of tokens in the cache reaches the upper limit of the capacity, the comprehensive importance score is used to determine and eliminate the token to be eliminated, and space is created for the newly generated token. Therefore, the retention or elimination of the token is no longer dependent on single historical attention or position information, but dynamically measured by comprehensively considering the historical performance, position value and semantic uniqueness of the token, so that the truly key historical token can be continuously retained in the fixed capacity, thereby relieving the problem of the decline of the output quality with the growth of the context.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to a fixed-capacity KV cache management method, apparatus, device, and storage medium based on multidimensional dynamic importance assessment. Background Technology

[0002] As the context length supported by large language models expands from thousands of tokens to 128k or even longer, Transformer models need to use key-value caches (KV caches) to store the key and value matrices of historical tokens in autoregressive inference to avoid redundant calculations. Since the KV cache usage grows linearly with the sequence length, on hardware with limited GPU memory, the cache capacity cannot expand synchronously with the context. Therefore, the industry generally adopts a fixed-capacity KV cache and evicts some tokens according to certain criteria when the cache is full.

[0003] Among the existing fixed-capacity KV cache eviction schemes, one type of scheme sorts the tokens based on the attention weights accumulated by each historical token in the generation steps that have occurred, and evicts the token with the lowest accumulated score when the cache is full; another type of scheme makes a selection based on the position of the token in the sequence, only permanently retaining a few tokens at the head of the sequence and maintaining a fixed-size nearest neighbor sliding window, discarding tokens that fall out of the window.

[0004] It should be noted that the above-mentioned long context reasoning has the following technical problems: under the premise that the cache capacity is strictly limited by hardware and cannot be expanded with the growth of the context, the set of tokens retained in the fixed-capacity cache according to the existing eviction criteria is difficult to stably cover the historical tokens actually relied upon by subsequent generation steps. This causes the model output quality to degrade unexpectedly with the increase of the context length. Moreover, this degradation cannot be eliminated by increasing the cache capacity (limited by the hardware memory limit) or by simply adjusting parameters such as the number of retained tokens or the sliding window size.

[0005] In view of the above, this application is hereby submitted. Summary of the Invention

[0006] This invention discloses a fixed-capacity KV cache management method, apparatus, device, and storage medium based on multidimensional dynamic importance assessment, aiming to solve the technical problem that the tokens retained in the fixed-capacity KV cache are difficult to stably cover the historical tokens actually needed for subsequent generation, resulting in a degradation of the model output quality.

[0007] The first embodiment of this invention provides a fixed-capacity key-value cache management method based on multidimensional dynamic importance assessment, applied to the autoregressive inference process of a large language model based on the Transformer architecture. The method includes: For each token stored in the fixed-capacity KV cache of the large language model, the following operations are performed: the weighted fusion of the token's attention accumulation dimension score, position dimension score, and semantic specificity dimension score to obtain the token's comprehensive importance score; and the comprehensive importance score of each token in the fixed-capacity KV cache is dynamically updated as the autoregressive inference process proceeds; wherein, the attention accumulation dimension score reflects the accumulation of attention weights obtained by the token in the historical generation steps, the position dimension score reflects the token's relative position in the sequence, and the semantic specificity dimension score reflects the token's semantic uniqueness relative to its neighboring tokens; When the number of tokens in the fixed-capacity KV cache reaches its capacity limit, tokens to be evicted are determined from the fixed-capacity KV cache based on the comprehensive importance score and evicted to free up storage space for newly generated tokens. Determining tokens to be evicted includes: not evicting tokens within their keep-alive period, and determining tokens to be evicted based on a double quantile strategy for tokens not within their keep-alive period; wherein the token... Whether a device is in the keep-alive period is determined by the keep-alive period state function:

[0008]

[0009] The set of tokens to be phased out is as follows:

[0010] The potential value is:

[0011] In the formula, For token At any moment The keep-alive state function, when its value is 1, indicates that the token... Within the survival period, For indicator functions, For token The moment when the fixed-capacity KV buffer is entered. To ensure the length of the survival period, This represents the average of the overall importance scores within the cache. This represents the minimum overall importance score within the cache. The set of tokens within the fixed-capacity KV cache. , These are the lower and upper limits of the survival period, respectively. This is the scaling factor; A set of tokens to be phased out. , These are the 20th and 80th percentiles of the overall importance score within the cache, respectively. For token Potential value The potential value of newly added tokens, This is the current query vector. For token The key vector, For the sigmoid function, This is the balance coefficient; To determine the overall importance score, To prevent positive numbers with a denominator of zero.

[0012] The second embodiment of the present invention provides a fixed-capacity key-value cache management device based on multidimensional dynamic importance assessment, applied to the autoregressive inference process of a large language model based on the Transformer architecture. The device includes: The importance assessment unit performs the following operations for each token stored in the fixed-capacity KV cache of the large language model: weightedly fusing the token's attention accumulation dimension score, position dimension score, and semantic specificity dimension score to obtain the token's comprehensive importance score; and dynamically updating the comprehensive importance score of each token in the fixed-capacity KV cache as the autoregressive inference process proceeds; wherein, the attention accumulation dimension score reflects the accumulation of attention weights obtained by the token in the historical generation steps, the position dimension score reflects the token's relative position in the sequence, and the semantic specificity dimension score reflects the token's semantic uniqueness relative to its neighboring tokens; A cache eviction unit is used to determine and evict tokens from the fixed-capacity KV cache based on the comprehensive importance score when the number of tokens in the cache reaches its capacity limit, in order to free up storage space for newly generated tokens. Determining the tokens to be evicted includes: not evicting tokens within their keep-alive period, and determining the tokens to be evicted for tokens not within their keep-alive period using a double quantile strategy; wherein the token... Whether a device is in the keep-alive period is determined by the keep-alive period state function:

[0013]

[0014] The set of tokens to be phased out is as follows:

[0015] The potential value is:

[0016] In the formula, For token At any moment The keep-alive state function, when its value is 1, indicates that the token... Within the survival period, For indicator functions, For token The moment when the fixed-capacity KV buffer is entered. To ensure the length of the survival period, This represents the average of the overall importance scores within the cache. This represents the minimum overall importance score within the cache. The set of tokens within the fixed-capacity KV cache. , These are the lower and upper limits of the survival period, respectively. This is the scaling factor; A set of tokens to be phased out. , These are the 20th and 80th percentiles of the overall importance score within the cache, respectively. For token Potential value The potential value of newly added tokens, This is the current query vector. For token The key vector, For the sigmoid function, This is the balance coefficient; To determine the overall importance score, To prevent positive numbers with a denominator of zero.

[0017] The third embodiment of the present invention provides a fixed-capacity KV cache management device based on multidimensional dynamic importance assessment, including a memory and a processor. The memory stores a computer program, which can be executed by the processor to implement a fixed-capacity KV cache management method based on multidimensional dynamic importance assessment as described in any of the above embodiments.

[0018] The fourth embodiment of the present invention provides a computer-readable storage medium storing a computer program, which can be executed by the processor of the device where the computer-readable storage medium is located, to implement a fixed-capacity KV cache management method based on multidimensional dynamic importance assessment as described in any of the above embodiments.

[0019] Based on the fixed-capacity KV cache management method, apparatus, device, and storage medium provided by this invention, which addresses the problem that the tokens retained in the fixed-capacity KV cache cannot cover the historical tokens actually needed for subsequent generation, a comprehensive importance score is obtained by weighting and fusing the scores of attention accumulation dimension, position dimension, and semantic specificity dimension of each token in the cache, and this score is dynamically updated during the inference process. When the number of tokens in the cache reaches the capacity limit, tokens to be evicted are determined and removed based on this comprehensive importance score, freeing up space for newly generated tokens. Therefore, the retention or removal of tokens no longer depends on a single historical attention or position information, but rather on a dynamic measurement of their historical performance, positional value, and semantic uniqueness, ensuring that truly critical historical tokens are continuously retained within the fixed capacity, thereby alleviating the problem of output quality deterioration with increasing context. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating a fixed-capacity KV cache management method based on multidimensional dynamic importance assessment provided in the first embodiment of the present invention. Figure 2 This is a schematic diagram of a fixed-capacity KV cache management device based on multidimensional dynamic importance assessment provided in the second embodiment of the present invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] To better understand the technical solution of the present invention, the embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0023] This invention discloses a fixed-capacity KV cache management method, apparatus, device, and storage medium based on multidimensional dynamic importance assessment, aiming to solve the technical problem that the tokens retained in the fixed-capacity KV cache are difficult to stably cover the historical tokens actually needed for subsequent generation, resulting in a degradation of the model output quality.

[0024] Please see Figure 1 The first embodiment of the present invention provides a fixed-capacity KV cache management method based on multidimensional dynamic importance assessment, applied to the autoregressive inference process of a large language model based on the Transformer architecture. This method can be executed by a management device (hereinafter referred to as a system), specifically by one or more processors within the management device, to at least implement the following steps: S101, for each token stored in the fixed-capacity KV cache of the large language model, perform the following operations: weightedly fuse the token's attention accumulation dimension score, position dimension score, and semantic specificity dimension score to obtain the token's comprehensive importance score; and dynamically update the comprehensive importance score of each token in the fixed-capacity KV cache as the autoregressive inference process proceeds; wherein, the attention accumulation dimension score reflects the accumulation of attention weights obtained by the token in the historical generation steps, the position dimension score reflects the token's relative position in the sequence, and the semantic specificity dimension score reflects the token's semantic uniqueness relative to its neighboring tokens; In this embodiment, at each generation time of the autoregressive inference of the large language model Execution: Each time the model generates a new token, that is, for each token stored in the current fixed-capacity KV cache... Calculate the scores for their attention accumulation dimension respectively. Score in the location dimension Score of semantic specificity dimension Then, the three factors are weighted and combined to obtain the overall importance score of the token. The above scores are recalculated in the same way at each subsequent generation time, thereby realizing the dynamic update of the overall importance score of each token in the cache.

[0025] The score of the attention accumulation dimension is used to characterize the token. The attention accumulated in each historical generation step is calculated using the following formula:

[0026] In the formula For the first token generation steps The attention weights obtained This is a configurable time decay factor; the formula uses an exponential decay method to sum the attention gained from each historical step in a weighted manner, with the contribution of the generation step closer to the current time being greater, thus making... It can reflect both the long-term historical value of the token and highlight its recent level of attention.

[0027] The location dimension score is used to characterize the token. The intrinsic value of a position in the sequence is calculated using the following formula:

[0028] In the formula This is the total length of the current sequence. The probability density function is the standard normal distribution. , , , , To assign weights to the head, tail, and middle of the sequence respectively, this embodiment can take... , , , , These are the exponential decay scales for the head and tail terms, respectively. This is the normalization factor; its first term varies with the position number. The first term decreases as the distance from the end of the sequence increases, giving higher weight to tokens near the beginning of the sequence to preserve the initial information. The second term decreases as the distance from the end of the sequence increases, giving higher weight to tokens near the current generation position. The third term uses a normal distribution to characterize the relative value of tokens in the middle. The three terms are weighted and then normalized to ensure that tokens at the beginning, end, and middle all receive a reasonable positional value assessment, avoiding the limitation of existing schemes that only retain the beginning and end while discarding information in the middle.

[0029] The score of the semantic specificity dimension is used to characterize the token. Its semantic uniqueness relative to its local neighborhood is calculated as follows:

[0030] In the formula For token In the hidden state of the current layer, For token In the hidden state of the current layer, For token Centered on, size is The sliding window in this embodiment can be adopted. , For window The mean vector of the hidden states of each token. To prevent positive constants with a denominator of zero, the numerator of this expression is token. The distance between the hidden state and its neighborhood mean is calculated, with the denominator representing the dispersion of the hidden state relative to the mean within that neighborhood. Dividing the two values ​​yields the deviation after neighborhood dispersion standardization. The larger the value, the stronger the token. The more unique the semantics within its local context and the less likely it is to be replaced by neighboring tokens, the higher the priority it should be when being evicted.

[0031] The token is obtained by weighting and combining the scores from the three dimensions mentioned above. The overall importance score is used. As a further preferred implementation, to improve the robustness of the importance assessment, a score for a cross-layer consistency dimension can also be introduced. When both parties participate in the integration process, the overall importance score is calculated using the following formula:

[0032] The score for the cross-layer consistency dimension is:

[0033] In the formula For token In the The hidden state of the layer For token In the +1 layer hidden state, The number of layers in the large language model. For cosine similarity, , , , These are adjustable weighting coefficients for four dimensions, with values ​​ranging from [0,1]. The sum of the four coefficients can be configured to be 1. By calculating the token The average cosine similarity between hidden states in adjacent layers measures the stability of the representation across different layers. The more consistent the representations in adjacent layers, the more stable and reliable the semantic information carried by the token, and the more worthwhile it is to retain. Thus, the overall importance score of a token no longer depends on a single historical attention or location information, but is measured by comprehensively considering its historical performance, location value, semantic uniqueness, and even cross-layer stability.

[0034] At each generation time As new tokens are generated and the current sequence length increases... As the value increases, the model regains the attention weights of each token in the current step. and the hidden states of each layer, based on which... , , as well as Update them one by one and re-weight them to obtain the results. This ensures that the overall importance score of each token in the cache dynamically reflects its latest importance throughout the reasoning process, providing an evaluation basis for determining tokens to be phased out based on the overall importance score.

[0035] S102, when the number of tokens in the fixed-capacity KV cache reaches its capacity limit, the tokens to be evicted are determined from the fixed-capacity KV cache according to the comprehensive importance score and evicted to free up storage space for newly generated tokens.

[0036] In this embodiment, the autoregressive inference process continues to execute: when the number of tokens in the fixed-capacity KV cache reaches its capacity limit. At that time, based on the comprehensive importance score that is dynamically updated, the tokens to be eliminated are identified from the cache and eliminated, freeing up storage space for newly generated tokens.

[0037] When determining tokens to be phased out, this implementation first sets a keep-alive period for newly added tokens to protect them, preventing them from being phased out due to low initial overall importance scores and unrealized potential value. Specifically, tokens... Whether a device is in the keep-alive period is determined by the keep-alive period state function:

[0038] In the formula For token When entering the cache, For indicator functions, when The time represents the token Those still within the survival period will not be eliminated in this round; the survival period length is calculated using the following formula:

[0039] in and These are the mean and minimum values ​​of the overall importance score currently in the cache. , The lower and upper limits of the survival period length (which can be taken in this embodiment) , ), Scaling factor (can be taken as follows) ), To prevent positive numbers with a denominator of zero, this formula makes the keep-alive period length adaptively change with the distribution of scores in the cache. The higher the overall score of the cache relative to its lowest score, the longer the observation window is given to the new token.

[0040] For those not within the survival period (i.e. For the tokens in this implementation, a double quantile strategy is used to determine the tokens to be evicted: the 20th percentile of the overall importance score of all tokens currently in the cache is used. and 80th percentile As two threshold levels, high and low; for those with a comprehensive importance score lower than... Tokens with significantly low importance are directly eliminated; tokens with an overall importance score between [a certain threshold] are eliminated. and Tokens in between are not immediately eliminated, but their potential values ​​are further compared. Potential value of newly added tokens A token is only eliminated if its potential value is less than the potential value of the new token, to avoid mistakenly eliminating tokens with moderate current scores but still strong relevance to the current query. The potential value is calculated using the following formula:

[0041] In the formula This is the current query vector. For token The key vector, For the sigmoid function, The balance coefficient (can be taken as follows) This formula strikes a balance between the token's existing overall importance score and its immediate relevance to the current query. Therefore, the set of tokens to be phased out can be represented as:

[0042] To reduce the computational overhead of the aforementioned elimination decision, this implementation leverages the long-tail distribution of importance scores, focusing on maintaining only the small subset of tokens with the lowest scores, rather than sorting all tokens each time. Specifically, a min-heap is maintained, storing only the tokens with the lowest overall importance score in the cache. The index and score of each token. When the cache is full and needs to be evicted, directly retrieve the top element of the min-heap (i.e., the element with the lowest current score, whose score usually falls within the top of the cache). The following tokens are designated as pending replacements, eliminating the need to traverse the entire cache. When the overall importance score of a token is updated in step S1, or when a token is replaced or swapped in / out, the min-heap is adjusted accordingly to maintain its heap property. Thus, a single replacement only requires processing tokens of size [size missing]. The smaller heap was adjusted to avoid global sorting of the full cache.

[0043] For inputs of very long documents, during the pre-filling phase of the autoregressive inference, the input sequence length is... It may be far greater than the capacity limit. At this point, the most valuable tokens need to be selected from the input tokens to construct the initial content of the cache. This implementation utilizes the complete attention matrix obtained in the pre-filling phase for each input token. Calculate initial importance to avoid the loss of key information caused by existing schemes that can only perform random or uniform downsampling at this stage due to a lack of attention feedback. Specifically, first calculate the importance of each token. The importance of being noticed With proactive attention to importance :

[0044]

[0045] in Describing the token As the object of attention, it represents the weighted level of attention it receives in the attention of each subsequent token. tokens in the attention matrix For token Attention weights (under causal masking) (when it is 0) For the attenuation parameter (which can be taken as follows) ); Describing the token As the subject of attention, the degree of focus on actively exerting attention on each of the preceding tokens For the attenuation parameter (which can be taken as follows) The two scores are then standardized and fused to obtain the initial importance score:

[0046] In the formula , and , Sequences , The mean and standard deviation, For the sigmoid function, The balance coefficient (can be taken as follows) ), Marginal gain coefficient (can be taken as...) ), For token The amount of semantic information, For token To conditionally predict the probability distribution of the next word, This represents the global word frequency distribution. Finally, tokens are selected based on the chosen metrics and added to the cache.

[0047] in For the head Individual and tail A mandatory retention set consisting of tokens. This is an indicator function; it takes the value 1 if the condition within the parentheses is true, and 0 otherwise. For the sigmoid function, To select the threshold, choose the initial importance score that results in the total number of selected tokens being exactly [value missing]. The High quantile: This means forcibly retaining the key tokens at the beginning and end of the sequence, and selecting the token with the highest initial importance score from the remaining tokens to fill the cache, thereby building an initial cache with the maximum information content before decoding begins, as the starting state for the elimination decision.

[0048] Furthermore, to address the issue of some tokens being unimportant at the beginning but becoming crucial in subsequent generation (i.e., "late-maturing" tokens), and being permanently lost once evicted, this implementation also includes a cache exchange mechanism. Specifically, an external candidate pool is maintained for evicted tokens. The candidate pool is stored in CPU memory rather than GPU memory, and its capacity is... The tokens are fixed and eliminated according to the Least Recently Used (LRU) strategy. The key and value vectors of the tokens in the pool are compressed using quantization (e.g., 8-bit quantization), resulting in minimal memory usage and no GPU memory consumption. During the generation process, the candidate pool is... Each candidate token Calculate the wake-up score:

[0049] In the formula For token The last time I left the cache, The size of the time window (can be set to...) ), For the first step token The obtained attention weight, The attenuation factor (can be taken as) ), This is a normalization factor; the wake-up score measures the degree to which a candidate token has regained attention within the most recent time window. When a candidate token exists in the candidate pool, its wake-up score is... Minimum of the overall importance score in the current cache The difference exceeds the exchange threshold (Recommended) When this happens, the candidate token is swapped into the cache, and at the same time... The corresponding token is exchanged to the candidate pool. Thus maintaining cache capacity Under the premise of keeping things fixed, tokens that were once eliminated but later became important again can be returned to the cache. To reduce overhead, the calculation of the wake-up score and the exchange of the cache are not performed every step, but are triggered once every preset number of generation steps (such as every 100 steps). When the token is swapped in, it is transmitted asynchronously via PCIe and performed in parallel with the model calculation, so that the latency introduced by the exchange is very small.

[0050] This embodiment was tested on the LongBench mainstream ultra-long text dataset. Compared with the traditional fixed window elimination algorithm and the H2O single attention elimination algorithm, this embodiment relies on a collaborative mechanism of four-dimensional importance joint evaluation, adaptive keep-alive period, potential value hierarchical elimination and dynamic replacement of CPU candidate pool. The perplexity of long text generation is significantly improved, and the retention effect of key semantic tokens is significantly better than the comparison scheme. With the addition of min-heap accelerated filtering and PCIe asynchronous exchange design, the increase in inference time caused by the additional computation is controllable, and the overall inference efficiency is not significantly degraded. It effectively improves the defect that the model output quality continuously deteriorates as the text length increases in ultra-long context scenarios.

[0051] Please see Figure 2 The second embodiment of the present invention provides a fixed-capacity KV cache management device based on multidimensional dynamic importance assessment, applied to the autoregressive inference process of a large language model based on the Transformer architecture. The device includes: Importance evaluation unit 201 is used to perform the following operations for each token stored in the fixed-capacity KV cache of the large language model: weightedly fuse the token's attention accumulation dimension score, position dimension score, and semantic specificity dimension score to obtain the token's comprehensive importance score; and dynamically update the comprehensive importance score of each token in the fixed-capacity KV cache as the autoregressive inference process proceeds; wherein, the attention accumulation dimension score reflects the accumulation of attention weights obtained by the token in the historical generation steps, the position dimension score reflects the token's relative position in the sequence, and the semantic specificity dimension score reflects the token's semantic uniqueness relative to its neighboring tokens; The cache eviction unit 202 is used to determine the tokens to be evicted from the fixed-capacity KV cache and evict them according to the comprehensive importance score when the number of tokens in the fixed-capacity KV cache reaches its capacity limit, so as to free up storage space for newly generated tokens.

[0052] The third embodiment of the present invention provides a fixed-capacity KV cache management device based on multidimensional dynamic importance assessment, including a memory and a processor. The memory stores a computer program, which can be executed by the processor to implement a fixed-capacity KV cache management method based on multidimensional dynamic importance assessment as described in any of the above embodiments.

[0053] The fourth embodiment of the present invention provides a computer-readable storage medium storing a computer program, which can be executed by the processor of the device where the computer-readable storage medium is located, to implement a fixed-capacity KV cache management method based on multidimensional dynamic importance assessment as described in any of the above embodiments.

[0054] Based on the fixed-capacity KV cache management method, apparatus, device, and storage medium provided by this invention, which addresses the problem that the tokens retained in the fixed-capacity KV cache cannot cover the historical tokens actually needed for subsequent generation, a comprehensive importance score is obtained by weighting and fusing the scores of attention accumulation dimension, position dimension, and semantic specificity dimension of each token in the cache, and this score is dynamically updated during the inference process. When the number of tokens in the cache reaches the capacity limit, tokens to be evicted are determined and removed based on this comprehensive importance score, freeing up space for newly generated tokens. Therefore, the retention or removal of tokens no longer depends on a single historical attention or position information, but rather on a dynamic measurement of their historical performance, positional value, and semantic uniqueness, ensuring that truly critical historical tokens are continuously retained within the fixed capacity, thereby alleviating the problem of output quality deterioration with increasing context.

[0055] Exemplary examples show that the computer program described in the third and fourth embodiments of the present invention can be divided into one or more modules, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in implementing a fixed-capacity KV cache management device based on multidimensional dynamic importance assessment. For example, the apparatus described in the second embodiment of the present invention.

[0056] The processor referred to can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. This processor is the control center of the fixed-capacity KV cache management method based on multi-dimensional dynamic importance assessment, and it connects various parts of the method using various interfaces and lines.

[0057] The memory can be used to store the computer program and / or modules. The processor, by running or executing the computer program and / or modules stored in the memory, and by calling the data stored in the memory, implements various functions of a fixed-capacity KV cache management method based on multi-dimensional dynamic importance assessment. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, text conversion function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, text message data, etc.). In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0058] If the implemented module is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0059] It should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0060] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A fixed-capacity key-value cache management method based on multidimensional dynamic importance assessment, applied to the autoregressive inference process of a large language model based on the Transformer architecture, characterized in that... The method includes: For each token stored in the fixed-capacity KV cache of the large language model, the following operations are performed: the weighted fusion of the token's attention accumulation dimension score, position dimension score, and semantic specificity dimension score to obtain the token's comprehensive importance score; and the comprehensive importance score of each token in the fixed-capacity KV cache is dynamically updated as the autoregressive inference process proceeds; wherein, the attention accumulation dimension score reflects the accumulation of attention weights obtained by the token in the historical generation steps, the position dimension score reflects the token's relative position in the sequence, and the semantic specificity dimension score reflects the token's semantic uniqueness relative to its neighboring tokens; When the number of tokens in the fixed-capacity KV cache reaches its capacity limit, tokens to be evicted are determined from the fixed-capacity KV cache based on the comprehensive importance score and evicted to free up storage space for newly generated tokens. Determining tokens to be evicted includes: not evicting tokens within their keep-alive period, and determining tokens to be evicted based on a double quantile strategy for tokens not within their keep-alive period; wherein the token... Whether a device is in the keep-alive period is determined by the keep-alive period state function: The set of tokens to be phased out is as follows: The potential value is: In the formula, For token At any moment The keep-alive state function, when its value is 1, indicates that the token... Within the survival period, For indicator functions, For token The moment when the fixed-capacity KV buffer is entered. To ensure the length of the survival period, This represents the average of the overall importance scores within the cache. This represents the minimum overall importance score within the cache. The set of tokens within the fixed-capacity KV cache. , These are the lower and upper limits of the survival period, respectively. This is the scaling factor; A set of tokens to be phased out. , These are the 20th and 80th percentiles of the overall importance score within the cache, respectively. For token Potential value The potential value of newly added tokens, This is the current query vector. For token The key vector, For the sigmoid function, This is the balance coefficient; To determine the overall importance score, To prevent positive numbers with a denominator of zero.

2. The fixed-capacity KV cache management method based on multidimensional dynamic importance assessment according to claim 1, characterized in that, The expressions for the scores of the attention accumulation dimension, the position dimension, and the semantic specificity dimension are as follows: In the formula, , , They are tokens At any moment The scores for attention accumulation, location, and semantic specificity dimensions; For the first token generation steps The obtained attention weight, This is the time decay factor; This is the total length of the current sequence. , These are the normal terms. The mean and standard deviation, , , These are the weights for the head, tail, and middle items, respectively. , These are the exponential decay scales for the head and tail terms, respectively. Normalization factor; For token In the hidden state of the current layer, For token Centered on, size is Sliding window, The number of tokens in the window. This is the mean vector of the hidden states within the window. It is the L2 norm. To prevent positive numbers with a denominator of zero, For token In the hidden state of the current layer, It is the probability density function of the standard normal distribution.

3. The fixed-capacity KV cache management method based on multidimensional dynamic importance assessment according to claim 1, characterized in that, The weighted fusion also includes incorporating scores from a cross-layer consistency dimension into the fusion process. The expression for the comprehensive importance score is: The score for the cross-layer consistency dimension is as follows: In the formula, For token At any moment The overall importance score; , , These are the scores for the attention accumulation dimension, the position dimension, and the semantic specificity dimension, respectively. The score for the cross-layer consistency dimension; For token In the The hidden state of the layer The number of layers in the large language model. Cosine similarity; , , , These are the weighting coefficients for the four dimensions. For token In the +1 layer hidden state.

4. The fixed-capacity KV cache management method based on multidimensional dynamic importance assessment according to claim 1, characterized in that, The step of determining and eliminating tokens to be evicted from the fixed-capacity KV cache based on the comprehensive importance score specifically involves: Maintain a min-heap, which stores the indices and overall importance scores of a preset number of tokens with the lowest overall importance scores in the fixed-capacity KV cache; when a token to be evicted is determined, take the top element of the min-heap, and maintain the min-heap after the overall importance score of the token is updated or the token is evicted.

5. A fixed-capacity KV cache management method based on multidimensional dynamic importance assessment according to claim 1, characterized in that, In the pre-filling stage of the autoregressive inference, the attention matrix obtained in this stage is used for each input token. The importance of computation Actively focus on importance and initial importance score And according to the selected indicators Select a token and fill it into the fixed-capacity KV cache. The expressions are as follows: In the formula, For the token in the attention matrix For token Attention weights Input the sequence length for the pre-padding stage; , They are respectively , The attenuation parameter; , and , Sequences , The mean and standard deviation; For balance coefficient, This is the marginal gain coefficient; For semantic information content, When true, the token Selected for initial cache To enforce the retention of the token set, , These represent the number of header and tail tokens that are forcibly retained, respectively. To select the threshold, choose from the initial importance scores that result in a total number of selected tokens equal to the capacity limit. The High quantiles This is an indicator function; it takes the value 1 if the condition within the parentheses is true, and 0 otherwise. This is the sigmoid function.

6. A fixed-capacity KV cache management method based on multidimensional dynamic importance assessment according to claim 1, characterized in that, This also includes cache swapping, specifically: Maintain an external candidate pool for expired tokens. The external candidate pool Stored in CPU memory, its capacity The external candidate pool is fixed and evicts candidates according to the LRU policy. The key vector and value vector of the token are compressed and stored in a quantized manner; for the external candidate pool Candidate tokens Calculate the wake-up score: When the external candidate pool There is a wake-up score. The minimum overall importance score within the fixed-capacity KV cache The difference exceeds the exchange threshold When a candidate token is found, the candidate token is swapped into the fixed-capacity KV cache, and... The corresponding token is swapped out to the external candidate pool. The calculation of the large language model is performed in parallel with the calculation of the wake-up score via PCIe in an asynchronous transmission mode, and the calculation of the wake-up score and the cache exchange are triggered once every preset number of generation steps. In the formula, Candidate token At any moment wake-up score As the normalization factor, For token The last time the fixed-capacity KV cache was left, The size of the time window. For the first token generation steps The obtained attention weight, This is the attenuation factor.

7. A fixed-capacity key-value cache management device based on multidimensional dynamic importance assessment, applied to the autoregressive inference process of a large language model based on the Transformer architecture, characterized in that... The device includes: The importance assessment unit performs the following operations for each token stored in the fixed-capacity KV cache of the large language model: weightedly fusing the token's attention accumulation dimension score, position dimension score, and semantic specificity dimension score to obtain the token's comprehensive importance score; and dynamically updating the comprehensive importance score of each token in the fixed-capacity KV cache as the autoregressive inference process proceeds; wherein, the attention accumulation dimension score reflects the accumulation of attention weights obtained by the token in the historical generation steps, the position dimension score reflects the token's relative position in the sequence, and the semantic specificity dimension score reflects the token's semantic uniqueness relative to its neighboring tokens; A cache eviction unit is used to determine and evict tokens from the fixed-capacity KV cache based on the comprehensive importance score when the number of tokens in the cache reaches its capacity limit, in order to free up storage space for newly generated tokens. Determining the tokens to be evicted includes: not evicting tokens within their keep-alive period, and determining the tokens to be evicted for tokens not within their keep-alive period using a double quantile strategy; wherein the token... Whether a device is in the keep-alive period is determined by the keep-alive period state function: The set of tokens to be phased out is as follows: The potential value is: In the formula, For token At any moment The keep-alive state function, when its value is 1, indicates that the token... Within the survival period, For indicator functions, For token The moment when the fixed-capacity KV buffer is entered. To ensure the length of the survival period, This represents the average of the overall importance scores within the cache. This represents the minimum overall importance score within the cache. The set of tokens within the fixed-capacity KV cache. , These are the lower and upper limits of the survival period, respectively. This is the scaling factor; A set of tokens to be phased out. , These are the 20th and 80th percentiles of the overall importance score within the cache, respectively. For token Potential value The potential value of newly added tokens, This is the current query vector. For token The key vector, For the sigmoid function, This is the balance coefficient; To determine the overall importance score, To prevent positive numbers with a denominator of zero.

8. A fixed-capacity key-value cache management device based on multidimensional dynamic importance assessment, characterized in that, The system includes a memory and a processor. The memory stores a computer program that can be executed by the processor to implement a fixed-capacity KV cache management method based on multidimensional dynamic importance assessment as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The device contains a computer program that can be executed by a processor of the device where the computer-readable storage medium is located, to implement a fixed-capacity KV cache management method based on multidimensional dynamic importance assessment as described in any one of claims 1 to 6.

Images