Key-value cache pruning method and device, equipment and storage medium

Abstract

The application discloses a key-value cache pruning method and device, equipment and a storage medium, and relates to the technical field of model optimization. The method comprises the following steps: dividing a key matrix of a key-value cache in a large language model into a plurality of sub-blocks, each of which contains a plurality of intra-block tokens; extracting a block feature vector of each sub-block; if the cumulative size of newly generated tokens in the decoding stage of the large language model reaches the size of a single sub-block, calculating a block score of each sub-block according to the block feature vector and a current window query feature vector; selecting a plurality of target blocks with the lowest scores from each sub-block according to the block scores; and expelling all intra-block tokens of the target blocks from the key-value cache to complete pruning of the key-value cache. The above-mentioned method of the application alleviates the problem that the high-bandwidth memory access burden linearly increases with the text length in the prior art, which needs to calculate the score with all historical tokens every time decoding, and improves the overall efficiency of key-value cache pruning in the long sequence decoding process.

Description

Technical Field

[0001] This application relates to the field of model optimization technology, and in particular to a key-value cache pruning method, apparatus, device and storage medium. Background Technology

[0002] In the inference process of a large language model based on the Transformer architecture, the decoding phase is a process of generating new tokens one by one. For each new token generated, the model needs to use the key-value cache (KV cache) corresponding to all previously generated tokens to calculate the attention score. As the sequence length increases, the storage space occupied by the key-value cache and the memory access overhead increase linearly, leading to increased inference latency and memory pressure.

[0003] To alleviate these problems, related technologies propose pruning strategies to retain key-value pairs that make significant semantic contributions to subsequent generation and actively remove less important parts. For example, some strategies maintain a cumulative attention score for each word, evaluate the importance of each word in real time during each decoding iteration, and perform pruning operations word by word. However, to calculate the score for each word, the model must calculate the score with all historical words in each decoding iteration. This drastically increases the memory access burden of high-bandwidth memory (HBM) under long text and high concurrency conditions, and the decoding time increases linearly with the length of the text.

[0004] Therefore, how to reduce the memory access burden of high-bandwidth memory during key-value cache pruning to improve the overall efficiency of long sequence decoding is a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0005] The main objective of this application is to provide a key-value cache pruning method, apparatus, device, and storage medium, aiming to solve the technical problem of how to reduce the memory access burden of high-bandwidth memory during key-value cache pruning, so as to improve the overall efficiency of long sequence decoding.

[0006] To achieve the above objectives, this application provides a key-value cache pruning method, which includes the following steps: The key matrix of the key-value cache in the large language model is divided into multiple sub-blocks, and each sub-block contains several intra-block lexical units; Extract the block feature vector of each sub-block, and the block feature vector is used to characterize the original semantic features of all intra-block words in the corresponding sub-block; If the cumulative size of newly generated lexical units in the decoding stage of the large language model reaches the size of a single sub-block, then the block score of each sub-block is calculated based on the feature vector of each block and the current window query feature vector; Based on the scores of each block, select several target blocks with the lowest scores from each sub-block, and evict all intra-block terms of the target blocks from the key-value cache to complete the pruning of the key-value cache.

[0007] In one embodiment, the step of extracting the block feature vector of each of the sub-blocks includes: Obtain the original key vectors in the key matrix that have not undergone rotation position encoding transformation; The block average vector and the block maximum absolute value deviation vector of each sub-block are calculated based on the original key vectors of all intra-block words in each sub-block. The block average vector is used to characterize the core semantics of the corresponding sub-block, and the block maximum absolute value deviation vector is used to characterize the feature looseness of the corresponding sub-block. The block average value vector and the block maximum absolute value deviation vector are jointly determined as the block feature vector of each sub-block.

[0008] In one embodiment, the step of calculating the block score of each sub-block based on the feature vector of each block and the current window query feature vector includes: Obtain the current query semantic aggregation vector and the current query semantic radius vector corresponding to the current window query feature vector. The current query semantic aggregation vector is used to characterize the semantic center of all query vectors accumulated in the current decoding stage within the query window. The current query semantic radius vector is used to characterize the magnitude of deviation of all query vectors in the current sub-block window from the semantic center. Based on the logical position of each sub-block, the block position parameters of each sub-block are determined. Based on the block position parameters, the current query semantic aggregation vector and the current query semantic radius vector are subjected to inverse rotation position encoding transformation to obtain the inverse rotation query semantic aggregation vector and the inverse rotation query semantic radius vector. The block score of each sub-block is calculated based on the block average value vector, the block maximum absolute value deviation vector, the query semantic aggregation vector after inverse rotation, and the query semantic radius vector after inverse rotation.

[0009] In one embodiment, the step of calculating the block score of each sub-block based on the block average vector, the block maximum absolute value deviation vector, the query semantic aggregation vector after inverse rotation, and the query semantic radius vector after inverse rotation includes: The semantic center score of each sub-block is determined by the inner product of the block average vector of each sub-block and the query semantic aggregation vector after inverse rotation. The semantic looseness score of each sub-block is determined by the inner product of the maximum absolute value deviation vector of each sub-block and the magnitude of the query semantic radius vector after inverse rotation. The semantic center score is added to the semantic loose score to obtain the block score of each sub-block. The block score is used to characterize the upper limit of the overall attention score of the current window query feature vector to the sub-block.

[0010] In one embodiment, the step of dividing the key matrix of the key-value cache in the large language model into multiple sub-blocks includes: Determine all intra-block lexical units contained in the key matrix of the key-value cache in the large language model; According to the logical order of the terms within each block in the key matrix, starting from the first term within the block, the key matrix is ​​divided into multiple sub-blocks until all terms within the blocks are assigned to their corresponding sub-blocks.

[0011] In one embodiment, the step of selecting a plurality of target blocks with the lowest scores from the sub-blocks based on the scores of each of the blocks includes: The number of blocks to be evicted is determined based on the preset pruning ratio and the total number of sub-blocks contained in the key-value cache; The scores of each block are sorted, and the sub-block with the lowest number of blocks to be evicted is selected as the target block based on the sorting results.

[0012] In one embodiment, the step of evicting all intra-block terms of the target block from the key-value cache includes: Obtain the memory space occupied by each target block in the key-value cache, and generate a memory release instruction corresponding to each target block based on each memory space; The memory space occupied by each target block is released by the memory release instructions, and the key-value cache is dynamically maintained by logical forward replenishment so as to evict all intra-block terms of the target block from the key-value cache.

[0013] Furthermore, to achieve the above objectives, this application also proposes a key-value cache pruning device, which includes: The sub-block partitioning module is used to divide the key matrix of the key-value cache in the large language model into multiple sub-blocks, each of which contains several intra-block lexical units; The vector extraction module is used to extract the block feature vector of each sub-block, and the block feature vector is used to represent the original semantic features of all intra-block words in the corresponding sub-block; The scoring module is used to calculate the block score of each sub-block based on the feature vector of each block and the query feature vector of the current window if the cumulative size of the newly generated lexical units in the decoding stage of the large language model reaches the size of a single sub-block. The pruning execution module is used to select several target blocks with the lowest scores from each of the sub-blocks based on the scores of each block, and to evict all intra-block terms of the target blocks from the key-value cache to complete the pruning of the key-value cache.

[0014] In addition, to achieve the above objectives, this application also proposes a key-value cache pruning device, the device comprising: a memory, a processor, and a key-value cache pruning program stored in the memory and executable on the processor, the key-value cache pruning program being configured to implement the steps of the key-value cache pruning method as described above.

[0015] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a key-value cache pruning program is stored, wherein when the key-value cache pruning program is executed by a processor, it implements the steps of the key-value cache pruning method as described above.

[0016] In addition, to achieve the above objectives, the present invention also provides a computer program product, the computer program product including a key-value cache pruning program, which, when executed by a processor, implements the steps of the key-value cache pruning method as described above.

[0017] One or more technical solutions proposed in this application have at least the following technical effects: The key-value cache pruning method of this application includes: dividing the key matrix of the key-value cache in a large language model into multiple sub-blocks, each sub-block containing several intra-block lexical units; extracting the block feature vector of each sub-block, the block feature vector being used to characterize the original semantic features of all intra-block lexical units in the corresponding sub-block; if the cumulative size of newly generated lexical units in the decoding stage of the large language model reaches the size of a single sub-block, then calculating the block score of each sub-block based on the block feature vector and the current window query feature vector; selecting several target blocks with the lowest scores from each sub-block based on the block scores, and evicting all intra-block lexical units of the target blocks from the key-value cache to complete the pruning of the key-value cache.

[0018] This application divides the key matrix of the key-value cache in a large language model into multiple sub-blocks and extracts the block feature vector of each sub-block, thereby elevating the pruning operation of the key-value cache from the word level to the block level. The block feature vector can represent the original semantic features of all words within the corresponding sub-block. Based on this, the block score calculation based on the block feature vector and the current window query feature vector is triggered only when the cumulative size of newly generated words reaches the size of a single sub-block, and the lowest-order block is pruned according to the block score. This block-level pruning mechanism reduces the frequency of score calculation through word accumulation triggering, and reduces the computational granularity involved in a single pruning through block-level eviction. It aggregates the memory access operations of the original word-by-word calculation and word-by-word eviction into block-level operations, thereby effectively reducing the overall number and amount of memory accesses in high-bandwidth memory during decoding, and alleviating the cumulative decoding latency caused by dense memory access in long sequence scenarios. Thus, this application alleviates the problem in the prior art where the high-bandwidth memory access burden caused by calculating the score with all historical words for each decoding operation increases linearly with the text length, and improves the overall efficiency of key-value cache pruning in the long sequence decoding process. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating the first embodiment of the key-value cache pruning method of this application; Figure 2 This is a flowchart illustrating the second embodiment of the key-value cache pruning method of this application; Figure 3 This is a flowchart illustrating the third embodiment of the key-value cache pruning method of this application; Figure 4 This is a structural block diagram of the first embodiment of the key-value cache pruning device of this application; Figure 5 This is a schematic diagram of the key-value cache pruning device of this application.

[0022] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0023] To make the above-mentioned objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0024] This application provides a key-value cache pruning method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the key-value caching pruning method of this application.

[0025] In this embodiment, the key-value cache pruning method includes the following steps: Step S1: Divide the key matrix of the key-value cache in the large language model into multiple sub-blocks, each of which contains several intra-block lexical units.

[0026] It should be noted that the execution subject of this application embodiment can be a key-value cache pruning device capable of executing the key-value cache pruning method of this application, and this embodiment does not limit this. The following uses a key-value cache pruning device (hereinafter referred to as pruning device) as an example to describe this embodiment and the following embodiments.

[0027] Understandably, the aforementioned key matrix can refer to the set of key vectors contained in the key-value cache (KV Cache) of a large language model. This key matrix is ​​typically stored as a two-dimensional tensor in High-Bandwidth Memory (HBM), where each row corresponds to the key vector of a historical token. The number of rows equals the length of the currently generated sequence, and the number of columns equals the dimension of a single attention head in the model. The aforementioned sub-block can refer to the data block obtained by dividing the key matrix into blocks according to the token dimension. Each sub-block contains key vectors corresponding to several consecutive token positions; for example, each sub-block can contain key vectors of 16 or 32 tokens. The aforementioned intra-block tokens can refer to the individual tokens belonging to the same sub-block. Each token corresponds to a row of key vectors in the key matrix, and the logical positions of intra-block tokens in the original sequence are consecutive.

[0028] In its implementation, the pruning device retrieves the key matrix stored in the current key-value cache. This key matrix contains key vectors corresponding to all historical lexical units from the start of the sequence to the current generated position. The pruning device then divides the key matrix into multiple non-overlapping sub-blocks according to the row dimension, i.e., the logical order of each lexical unit in the sequence. Each sub-block contains several consecutive rows of key vectors, with each row corresponding to a lexical unit within the block. There is no overlap or intersection between lexical units contained in different sub-blocks. After partitioning, the pruning device records the logical position range of the lexical units within each sub-block, as well as the start and end row indices of each sub-block in the key matrix. This information is used by subsequent processes to perform feature extraction and pruning decisions based on sub-blocks.

[0029] Step S2: Extract the block feature vector of each sub-block, which is used to characterize the original semantic features of all intra-block lexical units within the corresponding sub-block.

[0030] It should be noted that the aforementioned block feature vector can refer to a vector representation used to characterize the overall semantic attributes of a sub-block, reflecting the common semantic information of the group of lexical units contained in the corresponding sub-block. The aforementioned raw semantic features can refer to the semantic information carried by the original values ​​in the lexical key vectors without Rotary Position Embedding (RoPE) transformation. Raw semantic features do not include the positional information of the lexical units in the sequence; they only reflect the textual semantic content of the lexical units themselves. Within a sub-block, the raw semantic features of all lexical units within the block collectively constitute the block feature vector of that sub-block, used to evaluate the importance of that sub-block in attention calculation in subsequent steps.

[0031] It should be understood that for each sub-block, the pruning device can read the key vectors corresponding to all the intra-block tokens contained in that sub-block from the key-value cache, and read the original key vectors without rotational position encoding transformation. Since the original key vectors only carry the textual semantic information of the tokens themselves and do not contain relative position information, the block feature vectors extracted based on these original key vectors can truly reflect the pure semantic commonality of all tokens within the sub-block. The pruning device determines the feature vectors that can summarize the overall semantic attributes of the sub-block based on the distribution of all original key vectors within each sub-block.

[0032] Step S3: If the cumulative size of newly generated lexical units in the decoding stage of the large language model reaches the size of a single sub-block, then calculate the block score of each sub-block based on the feature vector of each block and the current window query feature vector.

[0033] It should be noted that the aforementioned decoding stage can refer to the stage where the large language model generates new lexical units one by one during the autoregressive generation process. In the decoding stage, the model generates one new lexical unit per forward propagation. The newly generated lexical unit is added to the end of the already generated sequence and participates in the attention calculation of subsequent generation processes. The cumulative size of the newly generated lexical units can refer to the total number of newly generated lexical units added to the key-value cache from the last pruning operation to the current decoding time. The size of a single sub-block can refer to the pre-defined number of lexical units contained in each sub-block, which is a fixed positive integer, such as 16 or 32 lexical units. The current window query feature vector can be composed of a semantic aggregation vector and a semantic aggregation radius vector, used to calculate the attention score with each key vector in the key-value cache. The block score can refer to the upper limit of the overall attention score of the current window query feature vector for a single sub-block; the lower the block score, the greater the likelihood that the sub-block will be selected as the expulsion target in subsequent pruning operations.

[0034] In its implementation, the pruning device maintains a cumulative counter that records the number of newly generated terms written to the key-value cache since the last sub-block pruning operation. Each time a new term is generated during the decoding phase, the pruning device increments the cumulative counter. It then compares the current counter value with a pre-defined sub-block size. If the counter value is less than the sub-block size, the condition for triggering pruning calculation has not yet been met, and the pruning device does not perform block score calculation, continuing to wait for new terms to be generated in subsequent decoding iterations. If the cumulative counter value reaches the sub-block size, the block score calculation process is triggered. The pruning device reads the block feature vectors of each sub-block from the metadata buffer and obtains the current window query feature vector. Based on these vectors, it calculates a corresponding block score for each sub-block. After the block score calculation is complete, the pruning device resets the cumulative counter to zero and waits for the next round of counting. This calculation method, which uses the cumulative size of newly generated lexical units reaching the size of a single sub-block as the trigger condition, ensures that block score calculation is only executed after the cumulative increment reaches the preset granularity, rather than being forcibly executed in every decoding iteration.

[0035] Step S4: Select several target blocks with the lowest scores from each of the sub-blocks based on the scores of each block, and evict all intra-block terms of the target blocks from the key-value cache to complete the pruning of the key-value cache.

[0036] It should be noted that the target block mentioned above can refer to the sub-block selected as the object to be evicted after being sorted according to the block score. All intra-block terms contained in the target block in the key-value cache will be removed as a whole. Here, eviction can refer to releasing the storage space occupied by the target block from the key-value cache and making that part of the storage space no longer participate in subsequent attention calculations and memory access operations; pruning can refer to actively removing some sub-blocks in the key-value cache that are considered less important, in order to reduce the storage capacity occupied by the key-value cache and the memory access overhead.

[0037] In its implementation, the pruning device obtains the block scores for all child blocks and sorts them in ascending order of score. Then, it determines the number of child blocks to be evicted based on a preset pruning strategy. For example, it might prune one child block per operation, or determine the number of evicted blocks based on a preset proportion of the total number of child blocks in the current key-value cache. Next, based on the sorting results, it selects several child blocks with the lowest block scores as target blocks. It then obtains the starting address and length information of the memory space occupied by each target block in the key-value cache and calls a memory release operation to free this memory space, ensuring that it no longer stores any key vector data.

[0038] This embodiment divides the key matrix of the key-value cache in the large language model into multiple sub-blocks and extracts the block feature vector of each sub-block. This elevates the pruning operation of the key-value cache from the word level to the block level, and the block feature vector can represent the original semantic features of all words in the corresponding sub-block. Based on this, block score calculation based on the block feature vector and the current window query feature vector is triggered only when the cumulative size of newly generated words reaches the size of a single sub-block. The lowest-order block is then pruned according to the block score. This block-level pruning mechanism reduces the frequency of score calculation through word accumulation and reduces the computational granularity involved in a single pruning operation through block-level eviction. It aggregates the memory access operations of the original word-by-word calculation and word-by-word eviction into block-level operations, thereby effectively reducing the overall number and amount of memory accesses in high-bandwidth memory during decoding, and alleviating the cumulative decoding latency caused by intensive memory access in long sequence scenarios. Therefore, this embodiment alleviates the problem in existing technologies where the high-bandwidth memory access burden increases linearly with text length due to the need to calculate scores with all historical words for each decoding operation, improving the overall efficiency of key-value cache pruning in long sequence decoding.

[0039] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to the above embodiment can be referred to the above description, and will not be repeated hereafter. On this basis, a second embodiment of the key-value cache pruning method of this application is proposed, please refer to... Figure 2 , Figure 2 This is a flowchart illustrating the second embodiment of the key-value caching pruning method of this application.

[0040] Furthermore, in order to improve the accuracy of pruning decisions in determining semantic importance, thereby avoiding the accidental deletion of key semantic blocks due to feature distortion and effectively maintaining the semantic integrity of the pruned key-value cache, step S2 may include: Step S21: Obtain the original key vectors in the key matrix that have not undergone rotation position encoding transformation.

[0041] It should be noted that the Rotation Position Encoding (RoPE) transformation refers to the encoding operation in large language models used to inject relative positional information into word vectors. Typically, RoPE is used to apply a rotation matrix related to the sequence position to the key vector and query vector, so that the transformed vector carries the positional information of the words in the sequence. The original key vector mentioned above can refer to the key vector in the key matrix that has not yet undergone the RoPE transformation. This original key vector only contains the textual semantic information of the word itself and does not contain sequence positional information; its value is directly generated by the linear transformation during the model's forward propagation.

[0042] It should be understood that during the construction of the key-value cache in a large language model, the key vector undergoes two stages from generation to writing to high-bandwidth memory: the first stage generates the original key vector without rotation-position encoding transformation; the second stage applies rotation-position encoding transformation to the original key vector and writes the transformed key vector to high-bandwidth memory. The pruning device reads the original key vector from an on-chip register or cache after the original key vector is generated but before the rotation-position encoding transformation is performed. After obtaining the original key vector, the pruning device associates it with the corresponding lexical index and stores it in the metadata buffer for use in subsequent block feature vector extraction operations.

[0043] Step S22: Calculate the block average vector and the block maximum absolute value deviation vector of each sub-block based on the original key vectors of all intra-block lexical units in each sub-block. The block average vector is used to characterize the core semantics of the corresponding sub-block, and the block maximum absolute value deviation vector is used to characterize the feature looseness of the corresponding sub-block.

[0044] It should be noted that the aforementioned block average vector can refer to the vector obtained by calculating the arithmetic mean of the original key vectors of all words within a sub-block element-wise along the vector dimension. The value of each dimension in this vector represents the center value of all words within the sub-block along that dimension, used to characterize the common semantic center of the group of words contained in the entire sub-block. The aforementioned core semantics can refer to the shared textual semantic connotation of all words within the sub-block, which is centrally reflected by the block average vector. The aforementioned block maximum absolute value deviation vector can refer to the vector obtained by calculating the maximum absolute value of each word's deviation from the mean of that dimension along the vector dimension, used to characterize the degree of dispersion of the semantic distribution of words within the sub-block. The aforementioned feature looseness can refer to the degree of dispersion of the semantic features of each word within the sub-block relative to the semantic center represented by the block average vector, quantitatively measured by the block maximum absolute value deviation vector.

[0045] In its implementation, assuming a sub-block contains N tokens, and the original key vector of each token has a dimension of D, the pruning device first traverses all N original key vectors within the sub-block. For each dimension d (d=1,2,...,D), it calculates the sum of the values ​​of the N tokens in that dimension, and divides this sum by N to obtain the average value for that dimension. After traversing all D dimensions, the pruning device obtains a D-dimensional block average vector, where the value in the d-th dimension is the average value of all tokens within the sub-block in that d-th dimension. Next, the pruning device calculates the maximum absolute value deviation vector based on the block average vector. For each dimension d, the pruning device calculates the absolute difference between the value of the original key vector of each token within the sub-block in that dimension and the value of the block average vector in that d-th dimension, and takes the maximum value among all N absolute differences as the value of the maximum absolute value deviation vector in that d-th dimension. After traversing all D dimensions, the pruning device obtains a D-dimensional block maximum absolute value deviation vector.

[0046] In one feasible implementation, the block average vector can be updated based on the following formula, which uses a moving average method to update the block average vector, gradually integrating the original key vectors of new lexical units into the core semantic expression of the sub-block: ; in, This represents the updated block average vector. This represents the block average vector before the update. This represents the newly generated original key vector. This is a token counter, representing the number of tokens currently contained in the sub-block.

[0047] The block maximum absolute value deviation vector can be updated based on the following formula, which shifts the old block maximum absolute value deviation vector to compensate for the offset of the block average vector, and then compares it with the deviation of the original key vector of the new word relative to the new block average vector, taking the maximum value in each dimension to obtain the updated block maximum absolute value deviation vector: ; in, This represents the updated block maximum absolute value deviation vector. This represents the block average vector before the update. This represents the operation of taking the maximum value of two vectors one dimension at a time.

[0048] Step S23: The block average value vector and the block maximum absolute value deviation vector are jointly determined as the block feature vector of each sub-block.

[0049] In this implementation, when extracting the block feature vectors of each sub-block, the original key vectors in the key matrix that have not undergone rotational position encoding transformation are obtained. Based on the original key vectors of all words within each sub-block, the block average vector and the block maximum absolute value deviation vector are calculated. Since the original key vectors have not yet been superimposed with relative position information, their values ​​only reflect the semantic content of the words themselves. Therefore, the block average vector calculated based on the original key vectors can accurately characterize the core semantics of the sub-block, and the block maximum absolute value deviation vector can objectively measure the degree of dispersion of the semantics of each word within the sub-block relative to the core semantics. By jointly determining the block average vector and the block maximum absolute value deviation vector as the block feature vector, the block feature vector is freed from phase shift interference caused by rotational position encoding, enabling it to characterize the importance of each sub-block in the key-value cache with pure semantic information. This feature extraction method based on the original key vectors ensures the authenticity and stability of the semantic features upon which the block score calculation depends, thereby improving the accuracy of pruning decisions in judging semantic importance, avoiding the accidental deletion of key semantic blocks due to feature distortion, and effectively maintaining the semantic integrity of the pruned key-value cache.

[0050] Furthermore, to ensure the rationality of pruning decisions at both the semantic and positional levels, thereby reducing the key-value cache access overhead while maintaining the attention mechanism's sensitivity to sequence structure, step S3 may include: Step S31: Obtain the current query semantic aggregation vector and the current query semantic radius vector corresponding to the current window query feature vector. The current query semantic aggregation vector is used to represent the semantic center of all query vectors accumulated in the current decoding stage within the query window. The current query semantic radius vector is used to represent the magnitude of deviation of all query vectors in the current sub-block window from the semantic center.

[0051] It should be noted that the aforementioned current query semantic aggregation vector can refer to the element-wise cumulative sum vector of all query vectors corresponding to new tokens generated during the current decoding phase, from the calculation of the most recent trigger block score to the current moment. This vector represents the semantic center position of all query vectors within the cumulative query window. The aforementioned current query semantic radius vector can refer to the maximum cumulative magnitude vector of the deviation of all query vectors corresponding to new tokens generated during the current decoding phase from the calculation of the most recent trigger block score to the current moment. This vector represents the degree of dispersion of all query vectors within the cumulative query window around the semantic center.

[0052] In one feasible implementation, the aforementioned current query semantic aggregation vector can be represented as: ; in, This represents the semantic aggregation vector of the current query. This represents the t-th query vector within the window, and 16 represents an example of the size of a single sub-block (i.e., the number of tokens contained in a single sub-block).

[0053] The semantic radius vector of the current query above can be represented as: ; in, This represents the semantic radius vector of the current query.

[0054] In its implementation, the pruning device maintains the current query semantic aggregation vector and the current query semantic radius vector in each iteration of the decoding phase. The pruning device allocates storage space for these two vectors in the metadata buffer and initializes them to zero vectors each time a block score is triggered and the cumulative counter is reset to zero. In subsequent decoding iterations, whenever a new term is generated and its corresponding query vector is calculated, the pruning device reads the old values ​​of the current query semantic aggregation vector and the current query semantic radius vector from the metadata buffer and updates them using the query vector corresponding to the new term. Specifically, the pruning device adds the old value of the current query semantic aggregation vector element-wise to the query vector corresponding to the new term to obtain the updated current query semantic aggregation vector; for the current query semantic radius vector, the pruning device calculates the element-wise absolute difference between the query vector corresponding to the new term and the updated current query semantic aggregation vector, compares each dimension of the absolute difference with the corresponding dimension of the old value of the current query semantic radius vector, and takes the maximum value to obtain the updated current query semantic radius vector.

[0055] Step S32: Determine the block position parameters of each sub-block according to the logical position of each sub-block, and perform reverse rotation position encoding transformation on the current query semantic aggregation vector and the current query semantic radius vector according to the block position parameters to obtain the reverse rotated query semantic aggregation vector and the reverse rotated query semantic radius vector.

[0056] It should be noted that the aforementioned logical position can refer to the position occupied by the sub-block in the key-value cache according to the sequence order. This position is determined by the index of the first word in the sub-block within the entire sequence. For example, if the index of the first word in the first sub-block is 0 in the entire sequence, then the logical position of the sub-block is 0; if the index of the first word in the second sub-block is 16 in the entire sequence, then the logical position of the sub-block is 1. The aforementioned block position parameter can refer to the rotation angle parameter calculated based on the logical position of the sub-block. This parameter is used for the matrix construction of the inverse rotation position encoding transformation. Specifically, it can be represented as the product of the logical position of the sub-block and the size of a single sub-block. For example, the block position parameter is equal to the logical position of the sub-block multiplied by the size of a single sub-block. This parameter directly corresponds to the cumulative rotation angle value at the position of the first word in the block during rotation position encoding.

[0057] It should also be noted that the reverse rotation position encoding transformation in this step rotates the current query semantic aggregation vector and the current query semantic radius vector from their current positions back to coordinate positions aligned with the original key vector of the sub-block.

[0058] In its implementation, the pruning device first obtains the logical position of the sub-block whose block score needs to be calculated. This logical position, initialized during the pre-filling stage, is equal to the index of the first term in the entire sequence divided by the size of a single sub-block. After pruning during the decoding stage, it is dynamically maintained through logical forward supplementation. Then, the logical position of the sub-block is multiplied by the size of the single sub-block to obtain the block position parameter. An inverse rotation matrix is ​​constructed based on this parameter, with the rotation angle being the negative of the forward rotation angle corresponding to the block position parameter. Next, the pruning device reads the current query semantic aggregation vector from the metadata buffer and multiplies it by the inverse rotation matrix to obtain the inversely rotated query semantic aggregation vector. Simultaneously, the pruning device reads the current query semantic radius vector from the metadata buffer and multiplies it by the same inverse rotation matrix to obtain the inversely rotated query semantic radius vector.

[0059] In one feasible implementation, the query semantic aggregation vector after inverse rotation can be obtained based on the following formula: ; in, This represents the query semantic aggregation vector after reverse rotation. The rotation matrix represents the inverse rotation position encoding transformation. Indicates the block position parameter.

[0060] The query semantic radius vector after inverse rotation can be obtained based on the following formula: ; in, This represents the query semantic radius vector after reverse rotation.

[0061] Step S33: Calculate the block score of each sub-block based on the block average value vector, the block maximum absolute value deviation vector, the query semantic aggregation vector after inverse rotation, and the query semantic radius vector after inverse rotation.

[0062] In this implementation, the current query semantic aggregation vector and the current query semantic radius vector are obtained to represent the semantic distribution within the cumulative query window. Then, block position parameters are determined based on the logical position of each sub-block. The current query semantic aggregation vector and the current query semantic radius vector are then subjected to inverse rotation position encoding transformation, ensuring that the transformed query vector and the original key features of each sub-block are in the same coordinate system. This preserves the relative distance information between the query and sub-blocks when calculating block scores, allowing the block score to simultaneously reflect semantic similarity and sequence positional relationships. Based on this, the inversely rotated query semantic aggregation vector and the inversely rotated query semantic radius vector are combined with the block average vector and the maximum absolute value deviation vector of each sub-block to calculate the block score. This integrates multi-dimensional information such as the core semantics of the sub-block, feature looseness, and the query semantic center and radius, making the block score a more comprehensive and accurate assessment of the importance of the sub-blocks. This block score calculation method, which considers relative distance, ensures the dual rationality of pruning decisions at both the semantic and positional levels, reducing key-value cache access overhead while maintaining the sensitivity of the attention mechanism to sequence structure.

[0063] Furthermore, in order to ensure that the block score can reliably reflect the upper bound of the importance of sub-blocks without requiring precise calculation of the true attention score for each word, thereby reducing computational complexity while ensuring the safety of pruning decisions, step S33 may include: Step S331: Determine the semantic center score of each sub-block based on the inner product of the block average vector of each sub-block and the query semantic aggregation vector after inverse rotation.

[0064] It should be noted that the semantic center score mentioned above can be used to quantitatively measure the degree of matching between the current cumulative query semantic center and the core semantics of the sub-block. The higher the value, the closer the query semantic center is to the core semantics of the sub-block.

[0065] Step S332: Determine the semantic looseness score of each sub-block based on the inner product of the maximum absolute value deviation vector of each sub-block and the modulus of the query semantic radius vector after inverse rotation.

[0066] It should be noted that the above semantic looseness score can be used to quantitatively measure the coupling contribution between the looseness of features within a sub-block and the current query radius. The higher the value, the greater the impact of the dispersion of lexical features within the sub-block on the upper limit of the attention score under the current query radius.

[0067] Step S333: Add the semantic center score to the semantic loose score to obtain the block score of each sub-block. The block score is used to characterize the upper limit of the overall attention score of the current window query feature vector to the sub-block.

[0068] It should be understood that the block score is used to characterize the upper limit of the overall attention score of the current window query feature vector to the sub-block. That is, the maximum value of the true attention score of the current window query feature vector and all the sub-block's internal words will not exceed the block score. Therefore, when performing pruning operations based on the block score, the sub-block with the lower block score is indeed in a lower importance range in terms of the true attention score, thus ensuring the safety of the pruning decision.

[0069] In one feasible implementation, the above block score can be calculated based on the following formula: ; in, This indicates the block score.

[0070] In this implementation, the semantic center score is obtained by taking the inner product of the block average vector of each sub-block and the query semantic aggregation vector after inverse rotation. This reflects the degree of fit between the current query semantic center and the core semantics of the sub-block. Simultaneously, the semantic looseness score is obtained by taking the inner product of the maximum absolute value deviation vector of each sub-block and the magnitude of the query semantic radius vector after inverse rotation. This reflects the contribution of the semantic dispersion within the sub-block to the overall score. The block score obtained by adding the semantic center score and the semantic looseness score is used to characterize the upper limit of the overall attention score of the current window query feature vector to the sub-block. This means that in subsequent pruning decisions, when performing expulsion operations based on this block score, the actual attention score of the pruned sub-block under the current query will not exceed this upper limit, thus ensuring that the expelled sub-block is indeed in a lower importance range in terms of attention weight. This method of aggregating the contributions of multiple lexical units within a sub-block into two dimensions—semantic center and looseness—to calculate the overall score upper limit allows the block score to reliably reflect the upper bound of the sub-block's importance without requiring precise calculation of the true attention score for each lexical unit. This reduces computational complexity while ensuring the safety of pruning decisions.

[0071] Based on the first and / or second embodiments of this application, the content that is the same as or similar to the first and second embodiments described above in the third embodiment of this application can be referred to the above description and will not be repeated hereafter. On this basis, a third embodiment of the key-value cache pruning method of this application is proposed. Please refer to... Figure 3 , Figure 3 This is a flowchart illustrating the third embodiment of the key-value caching pruning method of this application.

[0072] Furthermore, in order to maintain the consistency of the memory allocator's managed space, thereby reducing the fragmentation of the key-value cache caused by repeated word-by-word pruning, step S1 may include: Step S11: Determine all intra-block lexical units contained in the key matrix of the key-value cache in the large language model.

[0073] It should be understood that when the pruning device performs the sub-block partitioning process, it first determines all intra-block tokens contained in the key matrix. During the pre-filling stage, the key matrix undergoes three processes from generation to storage in high-bandwidth memory: first, the model forward propagates to generate the original key vectors without rotational position encoding transformation; second, rotational position encoding transformation is applied to the original key vectors; and finally, the transformed key vectors are written to the high-bandwidth memory using a write operator. At the start of the partitioning operation, the pruning device reads the total number of currently stored key vectors from the metadata in the key-value cache, i.e., the total number of rows in the key matrix. This total number of rows equals the total number of tokens generated from the beginning of the sequence to the current decoding time; these tokens are all intra-block tokens contained in the key matrix.

[0074] Step S12: According to the logical order of each word in the block in the key matrix, starting from the first word in the block, divide the key matrix into multiple sub-blocks until all words in the block are divided into their corresponding sub-blocks.

[0075] It should be understood that the pruning device obtains the total number of rows in the key matrix and the preset size of a single sub-block. Starting from the first word in the block with the highest logical position, it groups the rows of the key matrix in a continuous order. For each sub-block, starting from the current word position, it continuously selects a number of words equal to the size of the single sub-block and assigns them to the same sub-block. Then it moves to the next word position and continues the division until all words are assigned to their corresponding sub-blocks. During the division process, the logical positions of the words within each sub-block in the original sequence are continuous and non-overlapping. Different sub-blocks are arranged sequentially without intersection or overlap. When the total number of rows in the key matrix is ​​not an integer multiple of the size of a single sub-block, the number of words in the last sub-block can be less than the preset size of a single sub-block.

[0076] In this implementation, when partitioning the key matrix of the key-value cache, the key matrix is ​​divided into multiple sub-blocks sequentially, starting from the first sub-block, according to the logical position of the terms within each block in the key matrix, until all terms within each block are assigned to their corresponding sub-blocks. This sequential partitioning method ensures that the terms contained in each sub-block are logically continuous and do not overlap, allowing each sub-block to occupy a contiguous address range in physical memory, avoiding address dispersion problems caused by cross-range, skip-grouping. During subsequent decoding stages, when loading the key matrix for attention calculation or pruning operations, the contiguous address range facilitates the graphics processor initiating merged memory access requests, fully utilizing the burst transfer characteristics of high-bandwidth memory, reducing the number of fragmented memory accesses and address translation overhead caused by address discontinuities, and improving the locality of memory access. Meanwhile, since the sub-block boundary is consistent with the actual arrangement order of the words in the sequence, the size of the remaining hole after releasing memory in units of sub-blocks during pruning is aligned with the size of the sub-block. This helps maintain the consistency of the memory allocator's managed space, thereby reducing the fragmentation of the key-value cache caused by repeated word-by-word pruning and simplifying the complexity of memory management.

[0077] Furthermore, in order to provide an adjustable management method for key-value cache capacity control while ensuring pruning efficiency, thereby preventing the loss of critical information due to blind eviction, step S4 may include: Step S41: Determine the number of blocks to be evicted based on the preset pruning ratio and the total number of sub-blocks contained in the key-value cache.

[0078] It should be noted that the aforementioned preset pruning ratio can refer to a pre-set hyperparameter used to control the proportion of the number of child blocks to be evicted in a single pruning operation relative to the total number of child blocks in the key-value cache. This ratio is a value between 0 and 1, for example, it can be set to 0.2 or 0.25. The aforementioned number of blocks to be evicted can refer to the number of target blocks to be selected in this pruning operation, calculated based on the preset pruning ratio and the total number of child blocks, used to limit the number of child blocks with the lowest score selected from all child blocks.

[0079] It should be understood that in this solution, since the pruning operation is performed on a sub-block basis rather than on a word basis, the pruning ratio directly affects the total number of sub-blocks, making the memory release of a single pruning operation predictable and controllable. At the same time, the pruning intensity can be flexibly configured according to the actual application scenario.

[0080] Step S42: Sort the scores of each block, and select the sub-block with the lowest number of blocks to be evicted from each sub-block according to the sorting result as the target block.

[0081] It should be noted that after determining the number of blocks to be evicted, the pruning device can read all child blocks and their corresponding block scores from the metadata buffer. The pruning device sorts all child blocks in ascending order of their block scores, obtaining a sorting result. The child block at the beginning of this sorting result is the one with the lowest block score. Then, based on the sorting result, the pruning device selects, starting from the beginning of the sorted list, a number of child blocks equal to the number of blocks to be evicted as target blocks.

[0082] In this implementation, the number of blocks to be evicted is determined based on a preset pruning ratio and the total number of sub-blocks contained in the key-value cache. Then, the scores of each block are sorted, and the sub-block with the lowest score and the largest number of blocks to be evicted is selected as the target block based on the sorting results. This allows the pruning intensity to be flexibly configured according to the actual application scenario, avoiding insufficient memory release or over-pruning due to too many or too few evictions in a single instance. Furthermore, based on the block score sorting results, several sub-blocks with the lowest scores are selected to ensure that the selected target blocks have the lowest attention score upper limit among all sub-blocks, meaning these sub-blocks are considered to contribute the least to subsequent generation in the current decoding stage. By constraining the number of evictions with a preset ratio and combining this with the sorting and selection of the lowest-scoring blocks, the memory release amount for each pruning is predictable and the pruning targets are clearly defined. This ensures pruning efficiency while providing an adjustable management method for key-value cache capacity control, preventing the loss of critical information due to indiscriminate evictions.

[0083] Furthermore, in order to alleviate the problems of storage space fragmentation and logical address discretization caused by word-by-word eviction, thereby improving the read / write efficiency and maintainability of the pruned key-value cache, step S4 may further include: Step S43: Obtain the memory space occupied by each target block in the key-value cache, and generate a memory release instruction corresponding to each target block based on the memory space.

[0084] It should be noted that the aforementioned memory space can refer to the physical storage area in the key-value cache used to store the key vector data and related metadata information corresponding to all words within a sub-block. This storage area is defined by its starting address and length and is located in high-bandwidth memory or graphics processor memory. The aforementioned memory release instruction can refer to an instruction or function call issued by the pruning device to the operating system or graphics processor runtime library to release a specified memory space. This instruction contains the starting address and length information of the memory space to be released, and is used to instruct the underlying memory management module to mark the corresponding physical storage area as reusable.

[0085] In its implementation, the pruning device can read the starting address and length information of the memory space occupied by each target block from the metadata of the key-value cache. The starting address is the address of the first byte of the target block in video memory, and the length is the total number of bytes of all the word key vector data contained in the target block. For each target block, the pruning device encapsulates the read starting address and length information into a memory release instruction. This instruction instructs the underlying memory management module to release the storage area starting at that address and with that length. The pruning device generates an independent memory release instruction for each target block and stores all generated memory release instructions in the instruction cache, arranged according to the logical position of the target blocks, for subsequent eviction operations.

[0086] Step S44: Release the memory space occupied by each target block through the memory release instructions, and dynamically maintain the key-value cache through logical forward replenishment, so as to evict all intra-block terms of the target block from the key-value cache.

[0087] It should be noted that the aforementioned logical forward replenishment refers to the position adjustment operation performed after some sub-blocks in the key-value cache are evicted, in order to maintain the logical continuity of the remaining sub-blocks and their consistency with the physical storage order. Specifically, it involves shifting the logical positions of all remaining sub-blocks after the evicted sub-block forward by one unit, so that the logical positions of the remaining sub-blocks are renumbered as a continuously increasing integer sequence. The aforementioned dynamic maintenance refers to the process by which the key-value cache updates its own metadata information based on the eviction result after each pruning operation, including updating the number of remaining sub-blocks, the logical position of each remaining sub-block, and the mapping relationship of the starting addresses of each remaining sub-block in physical memory.

[0088] It should be understood that when the pruning device executes a memory release instruction, it marks the memory space occupied by the corresponding target block as reusable, preventing that memory space from storing any key vector data. After all the memory space of the target blocks has been released, the pruning device performs a logical forward replacement operation on the key-value cache, shifting the logical positions of all remaining sub-blocks after the evicted sub-block forward. This renumbers the logical positions of the remaining sub-blocks as a continuously increasing integer sequence starting from zero, ensuring that the calculation of block position parameters during subsequent decoding can still be performed correctly based on consecutive logical positions. The pruning device synchronously updates the metadata of the key-value cache, including the total number of remaining sub-blocks, the mapping relationship between the logical positions of each remaining sub-block and the physical memory addresses, and the number of tokens contained in each remaining sub-block. After completing the above physical memory release and logical position replacement, the pruning device evicts all in-block tokens of the target block from the key-value cache as a whole, and the freed storage space can be used for writing newly generated tokens.

[0089] In its implementation, this embodiment first obtains the memory space occupied by each target block and generates corresponding memory release instructions when evicting target blocks from the key-value cache. Then, it actually releases the memory space occupied by each target block through these instructions, while dynamically maintaining the key-value cache through logical forward supplementation. This scheme maintains the logical positional relationship of the key-value cache while performing physical memory release, ensuring that the logical order of the remaining sub-blocks after pruning remains continuous, avoiding logical position breaks or misalignments caused by simply releasing physical space. The logical forward supplementation maintenance mechanism ensures that the calculation of block position parameters and the reverse rotation position encoding transformation can be performed based on the correct logical position during subsequent decoding, thereby maintaining the accuracy of the relative distance information on which the block score calculation depends. By releasing continuous memory ranges as a whole on a sub-block basis, combined with the sequential supplementation of logical positions, the data integrity and access consistency of the pruned key-value cache are guaranteed at both the physical and logical levels. This alleviates the storage space fragmentation and logical address discretization problems caused by word-by-word eviction, improving the read / write efficiency and maintainability of the pruned key-value cache.

[0090] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the key-value cache pruning method of this application. Any simple modifications based on this technical concept are within the scope of protection of this application. All actions in this application involving the acquisition of relevant authorized data are performed in accordance with the relevant data protection regulations of the country where the application is located and with authorization from the owner of the relevant device.

[0091] This application also provides a key-value cache pruning device. Please refer to... Figure 4 , Figure 4 This is a structural block diagram of the first embodiment of the key-value caching pruning device of this application.

[0092] like Figure 4 As shown, the key-value cache pruning device proposed in this application includes: The sub-block partitioning module 401 is used to divide the key matrix of the key-value cache in the large language model into multiple sub-blocks, each of which contains several intra-block lexical units; The vector extraction module 402 is used to extract the block feature vector of each sub-block, and the block feature vector is used to characterize the original semantic features of all intra-block words in the corresponding sub-block; The scoring calculation module 403 is used to calculate the block score of each sub-block based on the feature vector of each block and the query feature vector of the current window if the cumulative size of the newly generated word units in the decoding stage of the large language model reaches the size of a single sub-block. The pruning execution module 404 is used to select several target blocks with the lowest scores from each of the sub-blocks based on the scores of each block, and to evict all intra-block words of the target blocks from the key-value cache to complete the pruning of the key-value cache.

[0093] This embodiment divides the key matrix of the key-value cache in the large language model into multiple sub-blocks and extracts the block feature vector of each sub-block. This elevates the pruning operation of the key-value cache from the word level to the block level, and the block feature vector can represent the original semantic features of all words in the corresponding sub-block. Based on this, block score calculation based on the block feature vector and the current window query feature vector is triggered only when the cumulative size of newly generated words reaches the size of a single sub-block. The lowest-order block is then pruned according to the block score. This block-level pruning mechanism reduces the frequency of score calculation through word accumulation and reduces the computational granularity involved in a single pruning operation through block-level eviction. It aggregates the memory access operations of the original word-by-word calculation and word-by-word eviction into block-level operations, thereby effectively reducing the overall number and amount of memory accesses in high-bandwidth memory during decoding, and alleviating the cumulative decoding latency caused by intensive memory access in long sequence scenarios. Therefore, this embodiment alleviates the problem in existing technologies where the high-bandwidth memory access burden increases linearly with text length due to the need to calculate scores with all historical words for each decoding operation, improving the overall efficiency of key-value cache pruning in long sequence decoding.

[0094] Based on the first embodiment of the key-value cache pruning device described in this application, a second embodiment of the key-value cache pruning device of this application is proposed.

[0095] In this embodiment, as one implementation method, the vector extraction module 402 is further used to obtain the original key vectors in the key matrix that have not undergone rotation position encoding transformation; calculate the block average vector and the block maximum absolute value deviation vector of each sub-block based on the original key vectors of all intra-block words in each sub-block, wherein the block average vector is used to characterize the core semantics of the corresponding sub-block, and the block maximum absolute value deviation vector is used to characterize the feature looseness of the corresponding sub-block; and determine the block average vector and the block maximum absolute value deviation vector together as the block feature vector of each sub-block.

[0096] In one implementation, the score calculation module 403 is further configured to obtain the current query semantic aggregation vector and the current query semantic radius vector corresponding to the current window query feature vector. The current query semantic aggregation vector is used to characterize the semantic center of all query vectors accumulated in the query window during the current decoding stage, and the current query semantic radius vector is used to characterize the magnitude of deviation of all query vectors in the current sub-block window from the semantic center. The module determines the block position parameters of each sub-block based on its logical position, and performs inverse rotation position encoding transformation on the current query semantic aggregation vector and the current query semantic radius vector based on the block position parameters to obtain the inversely rotated query semantic aggregation vector and the inversely rotated query semantic radius vector. Finally, the module calculates the block score of each sub-block based on the block average value vector, the block maximum absolute value deviation vector, the inversely rotated query semantic aggregation vector, and the inversely rotated query semantic radius vector.

[0097] In one implementation, the score calculation module 403 is further configured to determine the semantic center score of each sub-block based on the inner product of the block average vector of each sub-block and the query semantic aggregation vector after inverse rotation; determine the semantic looseness score of each sub-block based on the inner product of the block maximum absolute value deviation vector of each sub-block and the magnitude of the query semantic radius vector after inverse rotation; and add the semantic center score and the semantic looseness score to obtain the block score of each sub-block, wherein the block score is used to characterize the upper limit of the overall attention score of the current window query feature vector to the sub-block.

[0098] In one implementation, the sub-block partitioning module 401 is further configured to determine all intra-block lexical units contained in the key matrix of the key-value cache in the large language model; and to partition the key matrix into multiple sub-blocks sequentially, starting from the first intra-block lexical unit, according to the logical order of each intra-block lexical unit in the key matrix, until all intra-block lexical units are partitioned into their corresponding sub-blocks.

[0099] In one implementation, the pruning execution module 404 is further configured to determine the number of blocks to be evicted based on a preset pruning ratio and the total number of sub-blocks contained in the key-value cache; sort the scores of each block; and select the sub-block with the lowest block score and the number of blocks to be evicted from among the sub-blocks as the target block based on the sorting result.

[0100] In one implementation, the pruning execution module 404 is further configured to obtain the memory space occupied by each target block in the key-value cache, and generate a memory release instruction corresponding to each target block according to each memory space; release the memory space occupied by each target block through each memory release instruction, and dynamically maintain the key-value cache through logical forward replenishment, so as to evict all intra-block terms of the target block from the key-value cache.

[0101] The key-value cache pruning apparatus provided in this application, employing the key-value cache pruning method in the above embodiments, can solve the technical problem of how to reduce the memory access burden of high-bandwidth memory during key-value cache pruning, thereby improving the overall efficiency of long sequence decoding. Compared with the prior art, the beneficial effects of the key-value cache pruning apparatus provided in this application are the same as those of the key-value cache pruning method provided in the above embodiments, and other technical features in the key-value cache pruning apparatus are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0102] This application provides a key-value cache pruning device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the key-value cache pruning method in the first embodiment described above.

[0103] The following reference Figure 5 The diagram illustrates a structural schematic suitable for implementing the key-value cache pruning device in the embodiments of this application. The key-value cache pruning device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 5 The key-value cache pruning device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0104] like Figure 5As shown, the key-value cache pruning device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to a program stored in read-only memory 1002 or a program loaded from storage device 1003 into random access memory 1004. Random access memory 1004 also stores various programs and data required for the operation of the key-value cache pruning device. The processing unit 1001, read-only memory 1002, and random access memory 1004 are interconnected via bus 1005. Input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to input / output interface 1006: input devices 1007 including, for example, touch screens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the key-value cache pruning device to communicate wirelessly or wiredly with other devices to exchange data. While the figures show key-value cache pruning devices with various systems, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0105] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0106] The key-value cache pruning device provided in this application, employing the key-value cache pruning method in the above embodiments, can solve the technical problem of how to reduce the memory access burden of high-bandwidth memory during key-value cache pruning, thereby improving the overall efficiency of long sequence decoding. Compared with the prior art, the beneficial effects of the key-value cache pruning device provided in this application are the same as those of the key-value cache pruning method provided in the above embodiments, and other technical features in this key-value cache pruning device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0107] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0108] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0109] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the key-value cache pruning method in the above embodiments.

[0110] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0111] The aforementioned computer-readable storage medium may be included in the key-value cache pruning device; or it may exist independently and not be assembled into the key-value cache pruning device.

[0112] The aforementioned computer-readable storage medium carries one or more programs that, when executed by a key-value caching pruning device, enable the key-value caching pruning device to write computer program code for performing the operations of this application in one or more programming languages ​​or a combination thereof. These programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++; and also include conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network, such as a local area network (LAN) or a wide area network (WAN), or connected to an external computer (e.g., via the Internet using an Internet service provider).

[0113] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0114] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0115] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described key-value cache pruning method. This solves the technical problem of how to reduce the memory access burden of high-bandwidth memory during key-value cache pruning, thereby improving the overall efficiency of long sequence decoding. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the key-value cache pruning method provided in the above embodiments, and will not be repeated here.

[0116] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the key-value cache pruning method described above.

[0117] The computer program product provided in this application solves the technical problem of how to reduce the memory access burden of high-bandwidth memory during key-value cache pruning, thereby improving the overall efficiency of long sequence decoding. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the key-value cache pruning method provided in the above embodiments, and will not be repeated here.

[0118] The above description is only a part of the embodiments of this application and does not limit the scope of protection of this application. All equivalent structural transformations made under the technical concept of this application and using the content of this application specification and drawings, or direct / indirect applications in other related technical fields, are included in the scope of protection of this application.

Claims

1. A key-value cache pruning method, characterized in that, The method includes the following steps: The key matrix of the key-value cache in the large language model is divided into multiple sub-blocks, and each sub-block contains several intra-block lexical units; Extract the block feature vector of each sub-block, and the block feature vector is used to characterize the original semantic features of all intra-block words in the corresponding sub-block; If the cumulative size of newly generated lexical units in the decoding stage of the large language model reaches the size of a single sub-block, then the block score of each sub-block is calculated based on the feature vector of each block and the current window query feature vector; Based on the scores of each block, select several target blocks with the lowest scores from each sub-block, and evict all intra-block terms of the target blocks from the key-value cache to complete the pruning of the key-value cache.

2. The key-value cache pruning method as described in claim 1, characterized in that, The step of extracting the block feature vector of each of the sub-blocks includes: Obtain the original key vectors in the key matrix that have not undergone rotation position encoding transformation; The block average vector and the block maximum absolute value deviation vector of each sub-block are calculated based on the original key vectors of all intra-block words in each sub-block. The block average vector is used to characterize the core semantics of the corresponding sub-block, and the block maximum absolute value deviation vector is used to characterize the feature looseness of the corresponding sub-block. The block average value vector and the block maximum absolute value deviation vector are jointly determined as the block feature vector of each sub-block.

3. The key-value cache pruning method as described in claim 1, characterized in that, The step of calculating the block score of each sub-block based on the feature vector of each block and the current window query feature vector includes: Obtain the current query semantic aggregation vector and the current query semantic radius vector corresponding to the current window query feature vector. The current query semantic aggregation vector is used to characterize the semantic center of all query vectors accumulated in the current decoding stage within the query window. The current query semantic radius vector is used to characterize the magnitude of deviation of all query vectors in the current sub-block window from the semantic center. Based on the logical position of each sub-block, the block position parameters of each sub-block are determined. Based on the block position parameters, the current query semantic aggregation vector and the current query semantic radius vector are subjected to inverse rotation position encoding transformation to obtain the inverse rotation query semantic aggregation vector and the inverse rotation query semantic radius vector. The block score of each sub-block is calculated based on the block average value vector, the block maximum absolute value deviation vector, the query semantic aggregation vector after inverse rotation, and the query semantic radius vector after inverse rotation.

4. The key-value cache pruning method as described in claim 3, characterized in that, The step of calculating the block score of each sub-block based on the block average vector, the block maximum absolute value deviation vector, the query semantic aggregation vector after inverse rotation, and the query semantic radius vector after inverse rotation includes: The semantic center score of each sub-block is determined by the inner product of the block average vector of each sub-block and the query semantic aggregation vector after inverse rotation. The semantic looseness score of each sub-block is determined by the inner product of the maximum absolute value deviation vector of each sub-block and the magnitude of the query semantic radius vector after inverse rotation. The semantic center score is added to the semantic loose score to obtain the block score of each sub-block. The block score is used to characterize the upper limit of the overall attention score of the current window query feature vector to the sub-block.

5. The key-value cache pruning method as described in claim 1, characterized in that, The step of dividing the key matrix of the key-value cache in the large language model into multiple sub-blocks includes: Determine all intra-block lexical units contained in the key matrix of the key-value cache in the large language model; According to the logical order of the terms within each block in the key matrix, starting from the first term within the block, the key matrix is ​​divided into multiple sub-blocks until all terms within the blocks are assigned to their corresponding sub-blocks.

6. The key-value cache pruning method as described in claim 1, characterized in that, The step of selecting the lowest-scoring target blocks from the sub-blocks based on the scores of each block includes: The number of blocks to be evicted is determined based on the preset pruning ratio and the total number of sub-blocks contained in the key-value cache; The scores of each block are sorted, and the sub-block with the lowest number of blocks to be evicted is selected as the target block based on the sorting results.

7. The key-value cache pruning method as described in claim 1, characterized in that, The step of evicting all intra-block terms of the target block from the key-value cache includes: Obtain the memory space occupied by each target block in the key-value cache, and generate a memory release instruction corresponding to each target block based on each memory space; The memory space occupied by each target block is released by the memory release instructions, and the key-value cache is dynamically maintained by logical forward replenishment so as to evict all intra-block terms of the target block from the key-value cache.

8. A key-value cache pruning device, characterized in that, The key-value cache pruning device includes: The sub-block partitioning module is used to divide the key matrix of the key-value cache in the large language model into multiple sub-blocks, each of which contains several intra-block lexical units; The vector extraction module is used to extract the block feature vector of each sub-block, and the block feature vector is used to represent the original semantic features of all intra-block words in the corresponding sub-block; The scoring module is used to calculate the block score of each sub-block based on the feature vector of each block and the query feature vector of the current window if the cumulative size of the newly generated lexical units in the decoding stage of the large language model reaches the size of a single sub-block. The pruning execution module is used to select several target blocks with the lowest scores from each of the sub-blocks based on the scores of each block, and to evict all intra-block terms of the target blocks from the key-value cache to complete the pruning of the key-value cache.

9. A key-value cache pruning device, characterized in that, The device includes: a memory, a processor, and a key-value cache pruning program stored in the memory and executable on the processor, the key-value cache pruning program being configured to implement the steps of the key-value cache pruning method as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and the storage medium stores a key-value cache pruning program, which, when executed by a processor, implements the steps of the key-value cache pruning method as described in any one of claims 1 to 7.

Images