Key-value cache allocation method, electronic device, and computer program

Abstract

The application discloses a key value cache allocation method and electronic equipment and a computer program, relates to the technical field of data processing, and calculates an importance vector by comprehensively considering the attention vector of a word element and the reference number thereof, so that the importance of the word element can be evaluated from two dimensions of the attention weight and the reference number of the word element. A feature vector is constructed based on the importance vector, and an affinity matrix is constructed in combination with the reference number to perform clustering processing, so that reasonable classification can be realized according to the cache characteristics of the word element. Then, a differentiated cache budget vector is allocated to different clusters based on the importance vector, and space compression allocation is performed, so that the storage resources can be accurately allocated to the cache with high importance.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a key-value cache allocation method, electronic device, and computer program. Background Technology

[0002] When handling massive user requests, large-scale model inference frameworks need to manage a large number of key-value (KV) caches for these requests. However, a large number of KV caches consume significant storage space and transmission bandwidth. Therefore, it is necessary to allocate KV caches to reduce storage space consumption. The KV cache allocation methods in related technologies mainly use the attention weight (AW) generated during user request processing as an evaluation indicator of the importance of KV caches. This may eliminate some KV caches with moderate AW values ​​that can be reused multiple times, resulting in a decrease in the reuse rate of KV caches and thus a decrease in data processing efficiency.

[0003] Therefore, how to improve the reuse rate of KV cache while reducing the storage space occupied by KV cache, and thus improve data processing efficiency, is an urgent problem to be solved. Summary of the Invention

[0004] This application provides a key-value cache allocation method, electronic device, and computer program to at least solve the problem in related technologies of how to improve the reuse rate of key-value cache and thus improve data processing efficiency while reducing the storage space occupied by key-value cache.

[0005] This application provides a method for allocating key-value caches, including:

[0006] Based on the attention vector generated by each of the at least one word elements on at least one attention layer of the preset model, and the citation count of each of the at least one word elements in the preset model, determine the importance vector of each of the at least one word elements on at least one attention layer.

[0007] Construct feature vectors for at least one word at at least one attention layer based on the importance vectors, and construct an affinity matrix based on the feature vectors and the citation count;

[0008] Based on the affinity matrix, the caches of at least one terminology on at least one attention layer are clustered to obtain at least one cache cluster. Based on the importance vector and the layer identifier of at least one attention layer, a corresponding cache budget vector is assigned to each of the at least one cache cluster.

[0009] Based on the cache budget vector, the cache space is compressed and allocated for each of at least one terminology on at least one attention layer.

[0010] This application also provides a key-value cache allocation device, including:

[0011] The determining unit is used to determine the importance vector of at least one word element on at least one attention layer based on the attention vector generated by each of at least one word element on at least one attention layer of the preset model, and the citation number of each of at least one word element in the preset model.

[0012] The first construction unit is used to construct feature vectors for at least one word element on at least one attention layer based on the importance vector;

[0013] The second building unit is used to construct the affinity matrix based on the eigenvectors and the reference count;

[0014] Clustering unit, used to cluster the caches of at least one word on at least one attention layer according to the affinity matrix, to obtain at least one cache cluster;

[0015] The first allocation unit is used to allocate a corresponding cache budget vector to at least one cache cluster based on the importance vector and the layer identifier of at least one attention layer.

[0016] The second allocation unit is used to perform cache space compression allocation for at least one terminology on at least one attention layer based on the cache budget vector.

[0017] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for implementing any of the above-described key-value cache allocation methods when executing the computer program.

[0018] This application also provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the steps of any of the above-described key-value cache allocation methods.

[0019] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described key-value cache allocation methods.

[0020] The key-value cache allocation method, electronic device, and computer program of this application calculate an importance vector by combining the attention vector and citation count of key-value units (KV units). This allows for the evaluation of the importance of key-value units from two dimensions: attention weight and citation count. Based on the importance vector, a feature vector is constructed, and an affinity matrix is ​​built using citation counts for clustering. This enables reasonable classification based on the caching characteristics of key-value units. Then, differentiated cache budget vectors are allocated to different clusters based on the importance vector, and space compression is performed, allowing storage resources to be accurately allocated to high-importance caches. Therefore, this method solves the technical problem of improving KV cache reuse rate while reducing the storage space occupied by the KV cache, thereby improving data processing efficiency. It achieves the technical effect of effectively compressing the storage space occupied by the key-value cache while increasing cache reuse rate, thus improving data processing efficiency. Attached Figure Description

[0021] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 A flowchart illustrating a key-value cache allocation method provided in an embodiment of this application;

[0023] Figure 2 This application provides a schematic diagram of a clustering process for key-value caching.

[0024] Figure 3 A schematic diagram of the structure of a key-value cache allocation system provided in an embodiment of this application;

[0025] Figure 4 A schematic diagram of a key-value cache allocation device provided in an embodiment of this application;

[0026] Figure 5 This is a schematic diagram of another key-value cache allocation device provided in an embodiment of this application. Detailed Implementation

[0027] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0028] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0029] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0030] This application discloses a key-value cache allocation method, which is applicable at least to the large language model inference service process. It aims to optimize the use of key-value cache space through multi-dimensional cache importance assessment and cluster allocation, thereby reducing storage resource consumption and improving the throughput and response speed of the inference service while ensuring model inference performance.

[0031] Figure 1 This document provides a flowchart illustrating a key-value cache allocation method as an embodiment of this application. The method is described in detail below, taking into account the execution flow of the key-value cache allocation method.

[0032] like Figure 1 As shown, the allocation method for this key-value cache includes:

[0033] Step 101: Determine the importance vector of each of the at least one word element on at least one attention layer based on the attention vector generated by each of the at least one word element on at least one attention layer of the preset model, and the citation count of each of the at least one word element in the preset model.

[0034] In the embodiments of this application, a lexical unit refers to the smallest unit when a pre-defined model (e.g., a large language model) processes text. It typically corresponds to a word or sub-word fragment in the input text (data to be processed) and is converted into a numerical representation for computation during model inference. The pre-defined model refers to a large language model based on the Transformer architecture, such as a Generative Pre-trained Transformer (GPT) or Gemini, which contains multiple attention layers. Each attention layer is a component in the pre-defined model that performs self-attention computation to generate association weights between lexical units. Different attention layers are located in different positions (shallow or deep) within the pre-defined model, and their impact on semantic understanding and result generation also varies.

[0035] Attention vectors are numerical vectors representing the importance of lexical units, generated during the self-attention computation process in the attention layer. They are typically derived from the attention weight output and reflect the degree of attention the current lexical unit pays to other lexical units. Citation counts refer to the number of times each lexical unit's key-value cache is used during the use of the predefined model, reflecting the shared value and reuse potential of that lexical unit's key-value (KV) cache.

[0036] The importance vector is determined by combining the attention vector with the citation count. For example, the attention vector is multiplied by the square root of the citation count to obtain an important vector that comprehensively measures the importance and reusability (repeated use) of the key-value cache. This ensures that the importance evaluation of the key-value cache considers not only its intrinsic attention contribution but also its external reuse frequency (frequency of repeated use), making the evaluation of the key-value cache of the lexical more comprehensive.

[0037] Step 102: Construct feature vectors for at least one word element on at least one attention layer based on the importance vector, and construct an affinity matrix based on the feature vectors and the citation count.

[0038] In the embodiments of this application, the feature vector is a multi-dimensional data point used to represent the KV caching characteristics of the word in different attention layers. The feature vector includes at least the importance vector, the index identifier of the word in the vocabulary of the preset model (vocabulary index), and the layer number of the attention layer (layer identifier).

[0039] The feature vectors are constructed to capture the multidimensional attributes of the key-value caches for cluster analysis. The affinity matrix is ​​a symmetric matrix where each element represents the degree of similarity between different tokens across key-value caches at different attention layers.

[0040] The affinity matrix is ​​calculated based on the cosine similarity between eigenvectors, weighted by the influence of citation counts. For example, by dividing the dot product of two eigenvectors by their magnitude and then multiplying by the geometric mean of citation counts, the similarity between frequently reused key-value caches is enhanced. The construction of the affinity matrix quantifies the features and reuse relationships of the key-value caches, providing a foundation for subsequent clustering.

[0041] Step 103: Based on the affinity matrix, cluster the caches of at least one terminology on at least one attention layer to obtain at least one cache cluster. Based on the importance vector and the layer identifier of at least one attention layer, assign a corresponding cache budget vector to each of the at least one cache cluster.

[0042] In embodiments of this application, the caching of at least one lexical unit on at least one attention layer refers to the KV cache of each lexical unit on each attention layer.

[0043] Regarding clustering, for example, spectral clustering is used to transform the affinity matrix, and then KV caches are grouped into multiple cache clusters through eigenvalue decomposition and the K-means algorithm. Each cache cluster contains a set of KV caches with similar characteristics. The layer identifier is a unique number for the attention layer, used to distinguish different attention layers in the preset model.

[0044] Based on clustering, and combining the statistical characteristics of importance vectors (such as mean or standard deviation) and the distribution of layer identifiers, a cache budget vector is assigned to each cache cluster. The cache budget vector defines the cache space that the corresponding cache cluster can occupy, usually expressed as the number of cache granularity parts.

[0045] The budget allocation process can be achieved through data-driven modeling, for example, by using neural network models to predict the optimal budget value based on clustering features, ensuring that importance and inter-layer differences are reasonably balanced.

[0046] Step 104: Based on the cache budget vector, perform cache space compression allocation for the cache of at least one word on at least one attention layer.

[0047] In the embodiments of this application, the key-value cache is filtered and its storage is adjusted according to the cache budget vector, so as to reduce storage usage while ensuring inference performance.

[0048] Cache space compression allocation refers to the process of selectively retaining or evicting key-value (KV) caches based on budget vectors. For example, for each cache cluster, only the KV caches with the highest importance vectors are retained, while the remaining KV caches are discarded or swapped to secondary storage devices. By dynamically adjusting cache usage, efficient utilization of storage space is achieved while minimizing the impact on model inference performance.

[0049] This application overcomes the limitation of relying solely on attention weights for cache allocation by introducing citation count and multi-dimensional feature analysis, making cache management more intelligent and efficient. Comprehensive evaluation of importance vectors improves the accuracy of cache reuse, affinity matrices and clustering ensure the rationality of cache allocation, and cache space compression allocation based on budget vectors optimizes the utilization of storage resources. Overall, this application can significantly reduce cache usage and improve the throughput and response speed of large model inference services while maintaining model output quality.

[0050] In one possible implementation of this application embodiment, when constructing the feature vector of each of the at least one word element on at least one attention layer, the following methods can be used, but are not limited to: obtaining the word index corresponding to each of the at least one word element in the preset model dictionary corresponding to the preset model; constructing the feature vector of each of the at least one word element on at least one attention layer based on the importance vector, word index and layer identifier of each of the at least one word element on at least one attention layer, wherein the layer identifier is the layer number of each of the at least one attention layer in the preset model.

[0051] In the embodiments of this application, the construction of feature vectors aims to transform the KV cache attributes of each lexical unit on the attention layer into structured data.

[0052] The pre-defined model dictionary is a lexical mapping dictionary used by the pre-defined model during training and inference. It stores all the lexical units (tokens) that the pre-defined model can recognize and their corresponding unique identifiers, serving as a bridge between the lexical units and the model's internal data processing. The construction of the pre-defined model dictionary needs to consider the linguistic characteristics and semantic requirements of the model's training data, ensuring that it covers all kinds of text elements that may appear in user requests, including common words, professional terms, and special symbols.

[0053] The lexicon index is a unique numerical or character identifier for a lexicon in the pre-defined model dictionary. Each lexicon corresponds to a unique lexicon index in the pre-defined model dictionary. Its function is to distinguish lexicons with different semantics. Even if two lexicons are similar in text form or produce similar importance vectors on the same attention layer, their semantic differences can be accurately identified through the lexicon index.

[0054] Layer identifiers are unique layer numbers within a predefined model, used to distinguish attention calculations across different attention layers. For example, they are numbered sequentially from the input layer to the output layer. Constructing a feature vector requires combining these three elements into a multi-dimensional vector, typically represented in vector form. For example, for each token... In the first The feature vector of the attention layer can be defined as follows: ,in, For token In the The importance vector of the layer's key-value cache, For token The word index in the preset model dictionary, The layer number is the layer identifier.

[0055] This combination captures the multidimensional characteristics of the KV cache: the importance vector reflects the dynamic value of the KV cache, the vocabulary index reflects the static semantic attributes of the words, and the layer identifier indicates the model depth where the KV cache is located, thus ensuring that the feature vector fully covers the key dimensions of the cache.

[0056] Furthermore, if the feature vectors of all words have been constructed on all attention layers, it can be represented as follows: if the preset model has a total of Attention layer, currently processing a total of Each token. Then the above Group data, which consists of the feature vectors of at least one word unit on at least one attention layer, can be represented as a set of feature vectors. ,

[0057] The feature vectors constructed in this application not only encode the importance information of the KV cache but also incorporate lexical identities and inter-layer positional relationships, enabling subsequent affinity matrix calculations and clustering processes to more accurately identify the similarities between KV caches. For example, the vocabulary index helps distinguish the semantic differences between different lexical terms, while layer identifiers allow the model to differentiate the contribution differences between shallow and deep caches. This improves the fine-grainedness of cache management, provides a reliable data foundation for budget allocation, ultimately optimizes the efficiency of cache space utilization, and enhances the stability and performance of large model inference.

[0058] In one possible implementation of this application embodiment, when constructing an affinity matrix based on feature vectors and citation counts, the following methods can be used, but are not limited to: filtering feature vectors in descending order of importance to obtain at least one target feature vector and at least one target word corresponding to each target feature vector; performing data calculation processing based on the first target feature vector, the first target citation count, the second target feature vector, and the second target citation count to obtain matrix numerical elements, wherein the first target feature vector and the second target feature vector are any target feature vectors, the first target citation count is the citation count of the first target word corresponding to the first target feature vector, and the second target citation count is the citation count of the second target word corresponding to the second target feature vector; until at least one target feature vector has undergone data calculation processing to obtain at least one matrix numerical element, and constructing an affinity matrix based on the at least one matrix numerical element.

[0059] In the embodiments of this application, the feature vectors are sorted in descending order according to the magnitude of their corresponding importance vectors, and then the feature vectors with larger importance vectors are selected as the target feature vectors. Descending order filtering is a sorting and selection operation, that is, arranging the feature vectors from largest to smallest according to the value of the importance vectors, and truncating the top few. ... The selection criteria can be customized or determined based on actual computing resources and accuracy requirements. For example, selecting the feature vectors with the highest importance vectors can ensure that subsequent calculations focus on critical caches. The target feature vectors obtained through selection represent a subset of features from the high-importance caches. Each target feature vector corresponds to a target word, that is, at least one word whose importance vector is relatively high.

[0060] Next, the elements of the affinity matrix are calculated based on the target feature vectors and their corresponding citation counts. The affinity matrix is ​​a symmetric matrix, and its element values ​​represent the degree of similarity between different target feature vectors; the higher the similarity, the larger the value. The calculation process involves pairing any two target feature vectors. The first target feature vector and the second target feature vector are any two vectors selected from the target feature vectors. They may come from different target words or different target attention layers, or they may come from the same target word or the same target attention layer. That is, the first target feature vector and the second target feature vector may be different or the same.

[0061] The first target citation count is the citation count of the first target term corresponding to the first target feature vector, and the second target citation count is the citation count of the second target term corresponding to the second target feature vector. Data computation processing refers to the process of generating matrix numerical elements through mathematical operations, including but not limited to calculating the similarity between two target feature vectors and weighting the impact of citation counts. For example, calculating the cosine similarity between the first and second target feature vectors, i.e., their dot product divided by their respective magnitudes, and then multiplied by the geometric mean of the first and second target citation counts (e.g., the product of square roots), yields the matrix numerical elements. These matrix numerical elements not only capture the semantic and structural similarity between feature vectors but also enhance the association weights between frequently reused KV caches through citation counts, making the affinity matrix more reflective of the actual value of the cache.

[0062] The above calculation process is repeated until all target eigenvectors have been processed, generating a complete set of matrix numerical elements. Finally, all matrix numerical elements are arranged according to the order of the eigenvectors to construct the affinity matrix. For example: if the target eigenvectors have If there are 1, then the affinity matrix is ​​1. × The matrix, where the first... Line 1 The elements of the column correspond to the first The and the first The affinity matrix is ​​calculated by assigning similarity values ​​to the target feature vectors. Once constructed, the affinity matrix will be used for subsequent spectral clustering analysis, thereby achieving efficient grouping of the KV cache.

[0063] Specifically, the construction of the affinity matrix can also be achieved in, but is not limited to, the following ways: when When the dimension is too large (i.e., the preset model has a total of Attention layer, currently processing a total of (If all tokens are allocated to a KV cache, it will consume a large amount of storage space; therefore, only one token can be used.) highest Construct an affinity matrix from the target eigenvectors, and then... lower They are grouped into a separate category.

[0064] When constructing the affinity matrix, it can be calculated using, but is not limited to, formula (1):

[0065] Formula (1)

[0066] Wherein, it represents the token. In the Target feature vector of layer and token In the Target feature vector of layer Using the similarity between the two and the number of citations of the first target Second target citation count , build Affinity matrix of dimension It should be noted that if and They are respectively The The and the first There are elements, each corresponding to an affinity matrix. The Rows or columns and the first Rows or columns. Then That is, the affinity matrix The Line 1 The numerical elements of a column matrix.

[0067] The affinity matrix construction method in this application effectively reduces the number of feature vectors involved in the calculation by filtering feature vectors in descending order of importance vectors, thereby reducing the dimension and computational complexity of the affinity matrix and avoiding unnecessary consumption of computational resources. At the same time, by incorporating the reference count into the calculation of matrix numerical elements, the affinity matrix can not only reflect feature similarity but also reflect the reuse value of KV cache, enhancing the impact of KV cache reuse in similarity measurement, making KV cache allocation more in line with actual inference scenarios, and ultimately improving the utilization efficiency of cache space and model inference performance.

[0068] In one possible implementation of this application embodiment, regarding the process of clustering at least one lexical unit's cache at at least one attention layer based on the affinity matrix, this application embodiment also provides a schematic flowchart of key-value cache clustering processing, such as... Figure 2 As shown, it includes:

[0069] Step 201: Calculate the diagonal matrix corresponding to the affinity matrix, and perform matrix construction processing based on the diagonal matrix and the affinity matrix to obtain the constructed matrix.

[0070] In the embodiments of this application, the diagonal matrix is ​​a square matrix derived from the affinity matrix, where all off-diagonal elements are zero, and each element on the diagonal is the sum of all elements in the corresponding row of the affinity matrix. That is, for the first element of the affinity matrix... The row, the first row of the diagonal matrix The diagonal element (i.e., the first element of the diagonal matrix) (the nth element) is the nth element of the affinity matrix The sum of the numerical elements of a row matrix.

[0071] Matrix construction refers to the process of combining an affinity matrix and a diagonal matrix into a new matrix through mathematical operations. This includes, but is not limited to, multiplying the diagonal matrix by the affinity matrix after taking the inverse square root of the diagonal matrix. For example, calculating the negative 1 / 2 power of the diagonal matrix (i.e., taking the reciprocal of the square root of each diagonal element), and then multiplying it by the affinity matrix on the left and right to obtain the construction matrix. The construction matrix is ​​usually called the Laplace matrix in spectral clustering.

[0072] The construction of the construction matrix can be achieved through, but is not limited to, formula (2):

[0073] Formula (2)

[0074] in, To construct a matrix, It is a diagonal matrix. For the affinity matrix, the first element in the diagonal matrix is... Each element is an affinity matrix. The The sum of the elements in the row.

[0075] Step 202: Perform singular value decomposition on the construction matrix using a preset decomposition algorithm to obtain at least one singular value vector in the construction matrix and the singular value corresponding to each singular value vector.

[0076] In the embodiments of this application, the preset decomposition algorithm refers to a singular value decomposition (SVD) or eigenvalue decomposition (EVD) algorithm, such as the Singular Value Decomposition (SVD) algorithm, which is used to decompose a matrix into a series of eigenvalues. SVD decomposition is the process of decomposing the constructed matrix into the product of three matrices: a left singular vector matrix, a singular value diagonal matrix, and a right singular vector matrix. This process yields singular value vectors and singular values. The singular value vectors are the eigenvectors obtained after decomposition, representing the main directions of the data, while the singular values ​​are scalars representing the importance of each direction. In spectral clustering, the singular value vectors capture the latent structure of the KV cache data, while the singular values ​​quantify the contribution of each direction to the overall variance, thus providing a basis for subsequent screening.

[0077] Step 203: Filter at least one singular value vector in descending order according to the size of the singular value to obtain at least one target singular value vector, and concatenate the at least one target singular value vector column by column to obtain a singular value matrix.

[0078] In the embodiments of this application, descending sorting involves sorting the singular value vectors from largest to smallest and selecting the top-ranked vectors. vectors, where The settings can be customized or determined based on the number of clusters and accuracy requirements. For example, selecting a few singular value vectors with the largest singular values ​​to retain the main information. The target singular value vector is the singular value vector retained after filtering, representing the most representative component in the constructed matrix.

[0079] Concatenation is the process of combining multiple singular value vectors into a matrix in column order. For example: The singular value vectors of the target are arranged as column vectors to obtain the singular value matrix. ∈ ,in, The number of target feature vectors, Let be the number of singular value vectors. Each row of the singular value matrix corresponds to the coordinates of the KV cache in space corresponding to a target word, providing a low-dimensional representation for clustering.

[0080] Specifically, the construction of singular value matrices can also be achieved through, but is not limited to, the following methods: finding the construction matrix. of The singular value vector with the largest singular value. And concatenate them column by column as follows:

[0081] Step 204: Standardize the singular value matrix by row to obtain the matrix to be clustered, and then perform clustering on the matrix to be clustered using a preset clustering algorithm to obtain at least one cache cluster.

[0082] In the embodiments of this application, since the magnitude of the projected values ​​of different rows in the singular value matrix varies greatly (e.g., the projected values ​​of some rows are generally large, while those of others are small), direct clustering will cause the large-magnitude projected values ​​to dominate the results and mask the influence of the small-magnitude but important projected values. Therefore, standardization processing is required first.

[0083] Standardization refers to normalizing each row of the singular value matrix, making the magnitude of each row vector equal to 1. For example, this can be achieved by calculating the 2-norm of each row element and then dividing each element by the norm value. This ensures comparability between different rows, eliminates the influence of unit dimensions, and generates the matrix to be clustered.

[0084] The pre-defined clustering algorithm is an iterative clustering method, such as the k-means algorithm, which minimizes intra-cluster distances by grouping data points into multiple categories. The clustering process treats each row of the matrix to be clustered as a data point and groups them according to their Euclidean distance or similarity. The final output consists of multiple cache clusters, each containing a set of KV caches of target words with similar features across multiple target attention layers. The clustering results reflect the natural grouping of the KV caches, providing a structural basis for subsequent budget allocation.

[0085] Specifically, the construction of the clustering matrix can also be achieved in, but is not limited to, the following ways:

[0086] singular value matrix Normalized into clustering matrix based on row units Then the clustering matrix The Line 1 The elements of a column can be defined by, but are not limited to, formula (3):

[0087] Formula (3)

[0088] in, Singular value matrix No. Line 1 The elements of the column.

[0089] Next clustering matrix Each row is treated as a data point, and the data is clustered using the k-means algorithm. Classes can then be further categorized by data type. The mean values, from highest to lowest, are referred to as category 1 to category 2. kind.

[0090] This application corrects data bias through matrix construction, extracts key features through singular value decomposition, eliminates magnitude interference through standardization, and achieves efficient classification through clustering algorithms. The final cache clustering accurately reflects the attributes and value differences of key-value caches, laying the foundation for subsequent differentiated budget allocation. This allows cache space compression allocation to selectively retain high-value caches and optimize low-value caches, reducing storage consumption while maximizing throughput and response speed for large model inference.

[0091] In one possible implementation of this application embodiment, when calculating the diagonal matrix corresponding to the affinity matrix, the following methods can be used, but are not limited to: summing the matrix numerical elements of each row in the affinity matrix to obtain the combined numerical elements corresponding to each row of the affinity matrix; and constructing the diagonal matrix corresponding to the affinity matrix based on the combined numerical elements.

[0092] In the embodiments of this application, the calculation of the diagonal matrix can quantify the total association strength of each KV cache sample in the overall similarity relationship.

[0093] Calculating a diagonal matrix requires summing all the elements in each row of the affinity matrix. Summation involves adding all elements of a given row to obtain a scalar value. For the first row of the affinity matrix... The first row is the row whose all matrix elements are summed. The result is called the merged element. This merged element represents the first row. The higher the overall similarity strength between a key-value cache and all other key-value caches, the more central the key-value cache is in the overall cache relationship network, and the stronger its correlation. For example, if the key-value cache corresponding to a certain row is reused by multiple requests and its features are highly similar to other key-value caches, then the combined numerical value of that row will be significantly higher.

[0094] After obtaining the merged numerical elements of all rows, construct a diagonal matrix based on the merged numerical elements. All off-diagonal elements of the diagonal matrix are zero, while the diagonal elements are filled sequentially by the merged numerical elements.

[0095] Specifically, the construction process includes, but is not limited to, placing the first merged numerical element in the first row and first column of the matrix, the second merged numerical element in the second row and second column, and so on, until all diagonal positions are filled. The resulting diagonal matrix has each diagonal element directly corresponding to the total association strength of the corresponding row in the affinity matrix, while the off-diagonal elements remain zero, thus forming a normalized matrix containing only rows and information.

[0096] This application constructs a diagonal matrix, which is then used to standardize the affinity matrix during subsequent spectral clustering, eliminating differences in association strength between different samples and improving the effectiveness of similarity comparison. This ensures that cache clustering analysis can accurately capture the inherent structure of the data, laying a mathematical foundation for the rational allocation of cache budgets, and ultimately improving the efficiency of key-value cache management and model inference performance.

[0097] In one possible implementation of this application embodiment, when performing clustering processing on the clustering matrix to be clustered using a preset clustering algorithm, it can be implemented in the following ways, but not limited to: determining the row data of the matrix to be clustered as the data to be clustered, wherein the row data is used to indicate the caching of at least one target word on at least one target attention layer, and the at least one target attention layer is the attention layer selected from at least one attention layer during the construction of the matrix to be clustered; performing clustering processing on at least one data to be clustered using a preset clustering algorithm to obtain at least one cache cluster, wherein each cache cluster contains a cache of a target word on at least one target attention layer.

[0098] In the embodiments of this application, the purpose of clustering is to group KV caches with similar characteristics to facilitate efficient budget allocation in the future. The row data of the matrix to be clustered is determined as the data to be clustered. Row data refers to each row vector in the matrix to be clustered, and each row vector contains the feature information of the KV cache.

[0099] Each row of data indicates the cache of at least one target term in at least one target attention layer. Target terms refer to terms with high importance that are retained during the feature vector selection process. Target terms typically have high attention weights and citation counts, representing the most valuable portion of the KV cache. The target attention layer is a subset of layers selected from all attention layers in the predefined model during the construction of the clustering matrix. This target attention layer is selected based on importance vectors or other conditions to ensure that the clustering analysis focuses on key levels.

[0100] The data to be clustered is therefore a concretization of the row data, with each data point corresponding to a key-value cache feature of a target lexical at the target attention layer. The data captures multiple dimensions of the key-value cache, including importance, lexical identity, and inter-layer position, providing rich input for clustering. At least one data point is clustered using a pre-defined clustering algorithm to ultimately obtain the cache cluster.

[0101] During the clustering process, the algorithm calculates the Euclidean distance or similarity between the data to be clustered and groups them accordingly. Each cache cluster is a dataset containing a set of key-value (KV) cache samples with similar features. Specifically, each cache cluster contains KV caches of a target word on at least one target attention layer, indicating that the clustering result is based on words, but the cache attributes of the word on different target layers are comprehensively considered, thus forming cross-layer cache grouping.

[0102] The clustering process in this application reduces the dimensionality of the data while preserving the key features of the KV cache, making cache management more granular and efficient. This ensures that highly important KV caches are properly categorized, laying the foundation for subsequent budget allocation and ultimately improving cache space utilization and model inference performance.

[0103] In one possible implementation of this application embodiment, when allocating corresponding cache budget vectors for at least one cache cluster, the following methods can be used, but are not limited to: calculating the mean of the cluster importance vectors corresponding to each of the at least one cache cluster to obtain a first mean vector corresponding to each of the at least one cache cluster, wherein the cluster importance vector is the importance vector of the target word on at least one target attention layer; labeling the at least one cache cluster according to the magnitude of the first mean vector to obtain a category label corresponding to each of the at least one cache cluster; constructing cache budget features corresponding to each of the at least one cache cluster based on the category label, the cluster importance vector, and the cluster layer identifier corresponding to each of the at least one cache cluster, wherein the cluster layer identifier is the layer identifier corresponding to each of the at least one target attention layer; inputting the cache budget features into a preset cache budget model for cache allocation processing to obtain a cache budget vector corresponding to each of the at least one cache cluster.

[0104] In the embodiments of this application, cache clustering is a cache set obtained by grouping the data to be clustered using a preset clustering algorithm. Each cache cluster contains caches of target words with similar features on the target attention layer. The cluster importance vector refers to the set of importance vectors of all target words belonging to the same cache cluster on the target attention layer. The cluster importance vector is an index calculated by combining the attention vector and the citation count, reflecting the importance of the cache.

[0105] The first mean vector is obtained by averaging the importance vectors of all clusters within each cache cluster. It represents the central tendency of the overall importance level of that cluster. For example, calculating the arithmetic mean of the importance vectors within a cluster yields a general measure of importance. Calculating the first mean vector helps quantify the overall value of each cache cluster, providing a foundation for subsequent labeling and feature construction.

[0106] The cache clusters are then labeled according to the magnitude of the first mean vector, resulting in category labels. These category labels are symbols used to sort and identify the cache clusters based on the numerical value of the first mean vector. For example, the cache cluster with the largest first mean vector is labeled as category 1, the second largest as category 2, and so on, until all clusters are labeled. This labeling process ensures that the cache clusters are classified in order of importance, allowing highly important clusters to be processed first, thus providing a clear priority basis for budget allocation.

[0107] Subsequently, the clustering layer identifier is the layer number of the target attention layer corresponding to the cache cluster in the preset model, used to identify the model level to which the cache belongs, for example, the numbering from shallow to deep layers. The cache budget feature is a multi-dimensional data vector used to describe the statistical characteristics of the cache clusters, typically including class labels, the mean and standard deviation of the cluster importance vector, and the mean and standard deviation of the clustering layer identifier. For example, for each cache cluster, its cache budget feature can be represented as a vector containing elements such as the first mean vector, the standard deviation of the importance vector, and the mean and standard deviation of the layer identifier. The construction of the cache budget feature captures the importance distribution and inter-layer differences of the clusters, providing comprehensive input features for the budget model.

[0108] Finally, the cache budget features are input into a preset cache budget model for cache allocation, resulting in a cache budget vector. The preset cache budget model is a data-driven predictive model, such as a small neural network, which learns the mapping between cache budget features and the optimal budget through training. Cache allocation refers to the process by which the model infers based on the input cache budget features, outputting the number of budget shares allocated to each cache cluster. The cache budget vector is a numerical vector that defines the cache space size that each cache cluster can occupy, usually represented by the number of shares at the cache granularity level; for example, each element in the vector corresponds to the budget value of a cluster. Through model processing, budget allocation can adaptively match the importance level of clusters and inter-layer requirements, thereby optimizing the use of storage resources.

[0109] Specifically, the determination of the cache budget vector can also be achieved in the following ways, but are not limited to:

[0110] Cache budget feature The mean of the cluster importance vector of class cache clustering is The standard deviation is The average value of the layer identifier is The standard deviation is Then the first Cache budget features corresponding to class-based cache clustering Then the first Cache budget vector corresponding to class cache clustering ,in, This is the default cache budget model.

[0111] Assume the total available KV cache storage space is ,in, The granularity of the total key-value cache budget allocation, i.e., the key-value cache size generated by a token in one layer of the model. This represents the total number of distributable parts. If the input is categorized according to each category... The mean values, from highest to lowest, are referred to as category 1 to category 2. If the cache clusters are of different types, then the key-value cache budget vector for each cache cluster is: ,in, For the first The number of KV cache budgets allocated to class-based cache clustering.

[0112] This application integrates importance statistics and layer identification information to enable more refined and rational cache budget allocation. Category labels ensure priority management, cache budget features provide multi-dimensional descriptions, and preset models enable intelligent budget prediction. It can dynamically adjust cache space allocation while ensuring model performance, thereby improving cache reuse efficiency and inference speed.

[0113] In one possible implementation of this application embodiment, when constructing cache budget features corresponding to at least one cache cluster, the following methods can be used, but are not limited to: calculating the mean and standard deviation based on the target cluster importance vector corresponding to the target cache cluster to obtain the target mean vector and target standard deviation corresponding to the target cache cluster, wherein the target cache cluster is a cache cluster corresponding to any category label; calculating the mean and standard deviation based on the target cluster layer identifier corresponding to the target cache cluster to obtain the identifier mean and identifier standard deviation corresponding to the target cache cluster; constructing the target cache budget features corresponding to the target cache cluster based on the standard mean vector, target standard deviation, identifier mean, and identifier standard deviation, until at least one cache budget feature corresponding to each cache cluster is constructed.

[0114] In the embodiments of this application, the process of constructing cache budget features involves statistically analyzing the internal features of each cache cluster to generate a multidimensional descriptor that reflects its importance distribution and inter-layer differences. A target cache cluster refers to any cache cluster labeled according to its category, that is, a group of cache samples with similar features after being grouped by a clustering algorithm.

[0115] Each target cache cluster corresponds to a target cluster importance vector, which is the set of importance vectors of all KV cache samples belonging to that target cluster. The target cluster importance vector is processed by calculating its mean and standard deviation; that is, the arithmetic mean and standard deviation of the set of target cluster importance vectors are calculated to obtain the target mean vector and target standard deviation. The target mean vector represents the central tendency of the overall importance level of the target cache cluster, while the target standard deviation reflects the dispersion of importance vectors within the target cache cluster. A larger standard deviation indicates a more significant difference in cache importance. These statistics together characterize the importance distribution characteristics of the cluster.

[0116] Next, similar statistical analysis is performed on the target cluster layer identifiers corresponding to the target cache clusters. The target cluster layer identifier refers to the set of layer numbers of the attention layer to which all KV cache samples in the target cache cluster belong. The identifier mean and standard deviation are obtained through mean and standard deviation calculations. The identifier mean represents the average depth of the layer to which the cache belongs in the target cache cluster, such as shallow or deep layers, while the identifier standard deviation measures the range of fluctuation of the layer numbers. A smaller standard deviation indicates that the cache is mainly concentrated in adjacent layers, while a larger standard deviation indicates that the cache is widely distributed across layers. These layer identifier statistics capture the positional characteristics of the cache in the model structure.

[0117] Then, target cache budget features are constructed based on the target mean vector, target standard deviation, and label mean and standard deviation. The target cache budget feature is a structured vector; for example, the four statistics mentioned above are combined sequentially into a multi-dimensional feature vector to comprehensively describe the importance concentration, internal variability, inter-layer centroids, and layer distribution breadth of the target cache cluster. This ensures that the cache budget features not only cover importance information but also integrate the statistical characteristics of layer labels, providing rich input for subsequent budget allocation.

[0118] The above process is repeated until all cache clusters have completed feature construction, that is, the cache budget feature corresponding to each cluster is obtained. The cache budget feature will be used as the input of the preset cache budget model to generate the cache budget vector.

[0119] Specifically, the construction of cache budget features can be achieved in, but is not limited to, the following ways: Let the first... The mean of the cluster importance vector of the class data is The standard deviation is The average value of the layer identifier is The standard deviation is The cache budget characteristics of each cache cluster are then:

[0120] .

[0121] The cache budget feature construction in this application enables a refined description of cache clustering, allowing budget allocation to balance importance uniformity and inter-layer consistency. The target mean vector and label mean provide core metrics for clustering, while the target standard deviation and label standard deviation reveal internal diversity, ensuring the budget model can dynamically adapt to the needs of different clusters. This improves the accuracy and adaptability of cache space allocation, ultimately optimizing cache utilization efficiency in large model inference.

[0122] In one possible implementation of this application embodiment, when performing cache space compression allocation, the following methods can be used, but are not limited to: determining the cache budget corresponding to at least one cache cluster based on the cache budget vector; in response to the cache budget being able to allocate space for all target words in the cache cluster, performing cache space compression allocation for the cache of all target words in the target cache cluster on at least one target attention layer using the cache budget; in response to the cache budget being unable to allocate space for all target words in the cache cluster, sorting the target words in descending order according to the size of the corresponding cluster importance vector to obtain sorted target words; selecting the first preset number of target words as words to be allocated from the sorted target words, and performing cache space compression allocation for the cache of words to be allocated on at least one target attention layer using the cache budget.

[0123] In the embodiments of this application, during cache space compression allocation, it is necessary to determine the cache budget corresponding to each cache cluster based on the cache budget vector. Determining the cache budget ensures that each cache cluster is allocated a fixed space quota, providing a basis for subsequent compression allocation.

[0124] The system determines whether the cache budget is sufficient to allocate space for all target terms within their respective cache clusters. If so, it directly uses the cache budget to perform cache space compression allocation for all target terms in the target attention layer within the corresponding cache cluster. Cache space compression allocation refers to the process of selectively retaining key-value (KV) caches based on the budget size. For example, if the budget is sufficient, all KV caches within the cluster are retained to ensure the integrity of highly important KV caches; if compression is required, a portion of the KV caches is retained according to the budget value, while the remaining KV caches are discarded or swapped to secondary storage. This ensures that KV cache allocation matches the budget constraint, avoiding space overflow.

[0125] If the cache budget is insufficient to cover all target terms in the corresponding cache cluster, priority filtering is required. First, the target terms are sorted in descending order according to the magnitude of their corresponding cluster importance vectors. Descending order arranges the terms from highest to lowest importance vector, forming a sorted sequence of target terms. After sorting, the first preset number of target terms are selected from the sequence as terms to be allocated. This preset number is an integer determined by the cache budget; for example, if the cache budget is k, the first k target terms are selected. Finally, the cache space for these terms to be allocated is compressed and allocated on the target attention layer using the cache budget, retaining only the highest priority cache and evicting the rest.

[0126] This application's cache space compression allocation prioritizes the retention of highly important caches when budgets are limited, thereby maximizing cache value within a limited space. Through dynamic judgment and priority sorting, intelligent and refined cache allocation is achieved, which not only compresses storage usage but also minimizes performance loss, ultimately improving the efficiency and stability of large model inference services.

[0127] In one possible implementation of this application embodiment, before determining the cache budget vector, it is necessary to train a preset cache budget model. The training process of the preset cache budget model is a prerequisite for allocating appropriate cache budget vectors for cache clustering, aiming to achieve intelligent and adaptive optimization of cache budget allocation through data-driven modeling.

[0128] Specifically, the training process for the preset cache budget model can also employ, but is not limited to, the following methods: obtaining training cache budget features corresponding to multiple training cache clusters, wherein each training cache cluster includes multiple training clusters, and the training cache budget features include training features corresponding to multiple training clusters; performing gridded search processing on the training cache budget vectors based on the training cache budget features to obtain target training cache budget vectors corresponding to multiple training cache clusters; and performing cache budget model training processing based on multiple training cache clusters and target training cache budget vectors to obtain the preset cache budget model.

[0129] In the embodiments of this application, when constructing a preset cache budget model, it is first necessary to obtain the training cache budget features corresponding to each of multiple training cache clusters. The training cache cluster refers to a cache cluster set generated based on historical or simulated large model inference data. Each training cache cluster contains multiple training clusters. Each training cluster is the result of grouping KV cache samples in the training data by a clustering algorithm. Its structure is similar to that of cache clustering in actual applications.

[0130] Training cache budget features are statistical feature vectors describing the training clusters. They include elements such as the mean and standard deviation of the importance vector for each training cluster, and the mean and standard deviation of the layer labels. The calculation method for training cache budget features is consistent with that of actual cache budget features, and they are used to comprehensively characterize the attributes of the training clusters. By obtaining multiple sets of training cache budget features, rich input data can be provided for model training, ensuring that the model can learn the mapping relationship between different cluster features and the optimal budget.

[0131] A gridded search is performed on the training cache budget vector based on its training cache budget characteristics to determine the optimal budget allocation scheme. The training cache budget vector is the budget allocation vector to be optimized, where each element corresponds to the number of budget shares for a training cache cluster. Gridded search is a hyperparameter optimization method that iterates through a predefined space of budget vector combinations, evaluates the performance of each combination on the training data, and selects the optimal solution. Specifically, for each training cache cluster, a systematic search is performed within the possible budget vector range based on its training cache budget characteristics. For example, by changing the number of budget shares for each cluster and evaluating the throughput or latency metrics of model inference under each budget allocation, the budget vector with the best performance is finally selected as the target training cache budget vector. This target training cache budget vector represents the most reasonable budget allocation strategy under the current training data, providing labeled data for model training.

[0132] Finally, a cache budget model is trained based on multiple sets of training cache clusters and the target training cache budget vector to obtain a preset cache budget model. Cache budget model training refers to the process of building a predictive model using machine learning algorithms, typically employing a small neural network as the model architecture. During training, the training cache budget features are used as input, and the target training cache budget vector is used as output. The optimization objective is to minimize the mean squared error between the predicted and actual values, and the model parameters are adjusted using the backpropagation algorithm. The preset cache budget model obtained after training can automatically output the corresponding cache budget vector based on new cache budget features, enabling rapid inference for budget allocation.

[0133] Specifically, the training of the preset cache budget model can also be achieved in the following ways, but not limited to: based on a large-scale dataset, conduct the following experiments, i.e., generate data through data partitioning. Group training cache budget features For the first Group For the training cache budget vector Perform a grid search to select the target training cache budget vector with optimal performance. Thus obtain Group input / output data pairs .

[0134] Using a model based on a small neural network Features of training cache budget With target training cache budget vector Modeling the relationship, that is Neural networks The last layer is the L2Norm layer, that is, let the first layer be the L2Norm layer. Layer output is Using the second norm to... If the result of the layer is normalized to a vector with a magnitude of 1, then... It can be expressed by, but is not limited to, formula (4):

[0135] Formula (4)

[0136] use The model is trained using the minimum mean square error as the loss function. .

[0137] When new data to be processed is received, the KV cache budget characteristics of the current data to be processed are used. As input, utilize the model Output KV cache budget vector .

[0138] The pre-trained cache budget model in this application possesses generalization and adaptability, enabling dynamic optimization of cache space allocation in real-world inference scenarios. It ensures the rationality of budget allocation through gridded search and achieves intelligent prediction through model training, ultimately improving the efficiency of cache management and the overall performance of model inference.

[0139] In one possible implementation of this application, after obtaining at least one target feature vector and the target word corresponding to each target feature vector, it is necessary to process other words that are not selected into the target feature vector to ensure that all KV caches can be reasonably allocated, thereby improving the overall cache utilization efficiency.

[0140] Specifically, but not limited to the following methods can also be used: clustering the caches of other words on at least one attention layer into other word clusters, wherein other words are words other than the target word among at least one word; allocating a preset cache budget to other word clusters, and compressing and allocating cache space for the caches of other words on at least one attention layer based on the preset cache budget.

[0141] In the embodiments of this application, after obtaining the target feature vector and target words through descending sorting, the remaining words need to be processed separately to improve the cache allocation process. These remaining words refer to those words that were not selected as target words during the descending sorting process, i.e., words with lower importance vectors or those not given priority. These remaining words also correspond to KV caches at at least one attention layer, but because of their relatively low importance, they were not included in the target feature vector set during the initial sorting.

[0142] To ensure that other caches are not ignored and are incorporated into a unified cache management framework, the caches of other terms at the attention layer need to be clustered to form other term clusters. These other term clusters are collections specifically for low-importance caches. Simple clustering methods (such as grouping based on cache features or directly classifying them) are used to aggregate these caches, forming an independent processing unit. This ensures comprehensive cache management and avoids cache omissions caused by filtering.

[0143] Next, a preset cache budget is allocated to other term clusters. This preset cache budget is a pre-defined cache space quota, typically determined based on experience or system configuration, used to limit the storage usage of low-importance caches. This preset cache budget is relatively small to reflect the lower importance of other term caches, thus prioritizing space resources for high-importance caches. The purpose of budget allocation is to provide a fixed space limit for other term clusters under limited storage conditions, ensuring that the cache compression process is based on sound principles.

[0144] Based on a preset cache budget, cache space is compressed and allocated for each of the other lexical terms on the attention layer. This process involves selectively retaining or evicting caches based on the budget size. For example, if the preset cache budget is sufficient, all caches in other lexical term clusters are retained; if the budget is insufficient, further filtering is performed based on the importance of the caches or their recent usage, retaining only a portion of the caches, while the rest are discarded or swapped to secondary storage.

[0145] This application implements comprehensive cache management, prioritizing high-importance caches while also ensuring reasonable allocation of low-importance caches. This improves the granularity and fairness of cache management, avoids resource waste, and ultimately enhances the stability and efficiency of large-model inference services.

[0146] In one possible implementation of this application embodiment, when determining the importance vector of each of the at least one word element on at least one attention layer, it can be implemented in the following manner, but is not limited to: multiplying the first attention vector by the square root of the first citation number to obtain the first importance vector, wherein the first attention vector is the attention vector generated by the first word element on the first attention layer, the first citation number is the citation number corresponding to the first word element, the first word element is any word element among the at least one word element, and the first attention layer is any attention layer among the at least one attention layer; until the importance vector of each of the at least one word element on at least one attention layer is obtained.

[0147] In the embodiments of this application, the first word element refers to a word element arbitrarily selected from at least one word element, and the first attention layer refers to a layer arbitrarily selected from at least one attention layer of the preset model. The first word element generates a first attention vector on the first attention layer, which is a numerical vector generated during the attention calculation process. It represents the attention weight output of the first word element on the first attention layer and reflects the degree of attention of the first word element to other word elements.

[0148] Simultaneously, the first term has a first citation count, which is the number of times the key-value cache of this first term has been used by different users. The citation count records the shared value of the KV cache. In a radix tree-based cache management framework, a higher citation count indicates that the KV cache is reused by more requests, and its reuse potential is greater. The importance vector is calculated by multiplying the first attention vector by the square root of the first citation count.

[0149] The above process is repeated until the importance vectors of all terms have been calculated across all attention layers. That is, for each term, the same multiplication process is applied to each attention layer to obtain its respective importance vector. This ensures that the importance assessment of the entire cache set is consistent and comprehensive, laying the data foundation for subsequent cache clustering and budget allocation.

[0150] Specifically, the calculation of the importance vector can be achieved using, but is not limited to, formula (5):

[0151] Formula (5)

[0152] in, For token The first generation generated in large model inference operation The attention vector of the layer is , For token Key-value cache reference count, For token In the Importance vector of layer KV cache .

[0153] The importance vector determined in this application achieves multi-dimensional quantification of cache importance. The attention vector captures the intrinsic importance of the cache, while the square root of the reference count introduces the adjustment of reuse frequency, allowing frequently reused caches to obtain a high importance score even with moderate attention weights. This improves cache reuse rate and model inference efficiency while avoiding cache management bias caused by over-reliance on a single metric.

[0154] In one possible implementation of this application embodiment, the determination of the citation count can also be achieved by, but is not limited to, the following methods: inputting the data to be processed into a preset model for model inference processing to obtain at least one word element corresponding to the data to be processed, and the attention vector generated by each of the at least one word element on at least one attention layer; determining the historical citation count corresponding to each of the at least one word element in the preset model, and determining the citation count corresponding to each of the at least one word element based on the historical citation count.

[0155] In the embodiments of this application, the data to be processed is input into a preset model for model inference processing to generate necessary cache evaluation elements. The data to be processed typically refers to the prompt words or input text requested by the user, which is converted into a format acceptable to the model after preprocessing. The preset model is capable of deep processing of the input data. Model inference processing refers to the process by which the model performs forward computation on the input data, including steps such as lexicalization, embedding lookup, attention calculation, and output generation. Through this processing, the data to be processed is converted into a sequence of lexical units, where each lexical unit is the smallest unit of text. At the same time, the model generates an attention vector for each lexical unit at each attention layer. This vector is the output representation of the attention weights, reflecting the importance distribution of lexical units in a specific layer.

[0156] After the model inference process is complete, the citation count for each term needs to be determined. Historical citation count refers to the number of times a term has been used in the pre-defined model's cache management system. Historical citation counts are typically stored in the node information of a radix tree structure, reflecting the past sharing frequency of the term's key-value cache. The citation count is determined based on the historical citation count and updated in conjunction with the context of the current request. For example, if the current data to be processed uses an existing key-value cache for a term, the citation count is increased based on the historical citation count; if the key-value cache for a term is newly generated for the current data to be processed, the citation count is initialized to the baseline value. By tracking and updating the citation count, the system can dynamically capture cache reuse status, providing real-time data support for importance assessment.

[0157] Specifically, the calculation of citation count and importance vector can also be achieved through, but is not limited to, formulas (6) and (7):

[0158] Formula (6)

[0159] Formula (7)

[0160] in, For token The corresponding number of historical citations, For token The corresponding historical importance vector.

[0161] Furthermore, if the model inference process for the data to be processed terminates, then the reference count of all tokens involved will be... and existing KV importance vectors The update can be performed using, but is not limited to, formulas (8) and (9):

[0162] Formula (8)

[0163] Formula (9)

[0164] The model inference process in this application provides lexical and attention vectors, reflecting the internal characteristics of the cache; while the citation count determination introduces external reuse information, enhancing the comprehensiveness of the evaluation. This lays a reliable data foundation for subsequent importance vector calculation, feature construction, and cluster assignment, ultimately improving the accuracy and efficiency of cache management.

[0165] It should be noted that when the data to be processed arrives at the preset model for inference service, the lexical unit corresponding to the data to be processed is first matched with existing historical lexical units in the radix tree structure cache management framework of the preset model. If a historical lexical unit is matched, the historical citation count of the historical lexical unit is immediately increased as the citation count of the lexical unit corresponding to the data to be processed, indicating that this historical lexical unit is used by the current data to be processed. At the same time, the importance vector of the lexical unit on each attention layer is recalculated in real time based on the updated citation count. The recalculation of the importance vector adopts the same method as the initial calculation, that is, multiplying the attention vector with the square root of the updated citation count, thereby ensuring that the importance vector can reflect the latest reuse situation in a timely manner.

[0166] During user request processing, if a word in the vocabulary corresponding to the data to be processed cannot be matched with a historical word, the citation count for that word will be initialized, and an initial importance vector will be calculated based on the newly generated attention vector. The key-value cache for that word will be stored in the corresponding node of the radix tree and will begin participating in subsequent reuse statistics. This ensures that the key-value cache of a newly added word can immediately enter the importance evaluation, avoiding evaluation bias caused by data lag.

[0167] When the preset model processes the data to be processed, it matches the current word element corresponding to the data to be processed with the existing historical words elements in the radix tree structure of the preset model. If a historical word element is matched, the historical citation count of the historical word element is increased, and the increased historical citation count is used as the citation count of the current word element. At the same time, the importance vector of the current word element in each attention layer is calculated based on the citation count. If no historical word element is matched, the citation count of the current word element is initialized, and the importance vector is calculated based on the attention vector generated by the current word element in each attention layer. At the same time, the key-value cache of the current word element is stored in the corresponding node of the radix tree of the preset model.

[0168] In one possible implementation of this application, a key-value cache allocation system is also provided, such as... Figure 3 As shown, Figure 3 A schematic diagram of a key-value cache allocation system provided in this application. It includes:

[0169] 1. The KV (key-value) cache management module manages the KV cache based on a radix tree structure, continuously receives user requests, and performs calculations by the large model inference and calculation module.

[0170] 2. Using the KV cache importance assessment module, input the attention weights generated in the large model inference operation and the KV cache reference count of each token currently requested by the user in the KV cache management module, and output the importance of the KV cache for each token.

[0171] 3. Utilize KV cache multidimensional spectral clustering, input the importance of KV cache, and output the KV cache clustering results, that is, divide KV cache into multiple categories based on features such as importance.

[0172] 4. Using the KV cache budget allocation module, input the KV cache clustering results and allocate different KV cache budgets for different layers and tokens.

[0173] 5. Using the KV cache compression module, input the KV cache budget and output the KV cache compression command, that is, which KV caches to keep, and which KV caches to discard or exchange.

[0174] 6. Using the KV cache management module, input the KV cache compression command to compress the KV cache and update the KV cache management information.

[0175] 7. Repeat the above process to continuously process user requests.

[0176] In summary, this application can also achieve the following technical effects:

[0177] This application addresses the limitation of existing KV cache compression, which does not consider KV cache reusability, by using a KV cache importance metric based on radix tree request reference count. It proposes a KV cache management architecture based on radix tree, combining the classic AW metric with the KV cache reference count of each node in the radix tree to construct a new KV cache importance metric for subsequent retention, discarding, or swapping of KV caches.

[0178] This application addresses the limitations of existing KV cache budget allocation methods that only consider a single dimension such as the number of layers. It jointly considers three dimensions: the number of layers, the token index, and importance. First, it constructs an affinity metric based on similarity, then uses spectral clustering to classify KV caches to preserve the local proximity of language model data. Finally, it allocates different KV cache budgets to different layers and tokens through data modeling to match their importance.

[0179] This application addresses the issue of continuous updates to the KV cache cardinality tree during large model inference services by using reference number-aware KV cache compression. Based on importance metrics and KV cache budget allocation, it retains important KV caches for different levels and token classes, discards or swaps the rest of the KV caches, and designs an importance metric update and KV cache compression process that matches the KV cache cardinality tree management framework. This aims to significantly reduce the storage footprint of the KV cache, improve the KV cache reuse rate, and minimize performance loss.

[0180] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0181] Embodiments of this application also provide a key-value cache allocation device. Figure 4 A schematic diagram of a key-value cache allocation device provided in this application is shown below. Figure 4 As shown, it includes:

[0182] The determining unit 41 is used to determine the importance vector of at least one word element on at least one attention layer based on the attention vector generated by each of at least one word element on at least one attention layer of the preset model and the citation number of each of at least one word element in the preset model.

[0183] The first construction unit 42 is used to construct feature vectors for at least one word element on at least one attention layer based on the importance vector;

[0184] The second building unit 43 is used to construct an affinity matrix based on the eigenvectors and the reference count;

[0185] Clustering unit 44 is used to perform clustering processing on the cache of at least one word in at least one attention layer according to the affinity matrix, so as to obtain at least one cache cluster;

[0186] The first allocation unit 45 is used to allocate a corresponding cache budget vector to at least one cache cluster based on the importance vector and the layer identifier of at least one attention layer.

[0187] The second allocation unit 46 is used to perform cache space compression allocation for at least one terminology on at least one attention layer based on the cache budget vector.

[0188] In one embodiment of this application, the first building unit 42 is specifically used for:

[0189] In the preset model dictionary corresponding to the preset model, obtain at least one word index corresponding to each word element;

[0190] Based on the importance vector of at least one word element in at least one attention layer, the vocabulary index, and the layer identifier, construct the feature vector of at least one word element in at least one attention layer, where the layer identifier is the layer number of each of the at least one attention layer in the preset model.

[0191] In one embodiment of this application, the second building unit 43 is specifically used for:

[0192] The feature vectors are sorted in descending order of importance to obtain at least one target feature vector and the target word corresponding to each target feature vector.

[0193] Data calculations are performed based on the first target feature vector, the first target citation count, the second target feature vector, and the second target citation count to obtain matrix numerical elements. Here, the first target feature vector and the second target feature vector are any target feature vectors, the first target citation count is the citation count of the first target word corresponding to the first target feature vector, and the second target citation count is the citation count of the second target word corresponding to the second target feature vector.

[0194] Data calculations are performed on at least one target feature vector to obtain at least one matrix numerical element, and an affinity matrix is ​​constructed based on the at least one matrix numerical element.

[0195] In one embodiment of this application, the clustering unit 44 is specifically used for:

[0196] Calculate the diagonal matrix corresponding to the affinity matrix, and perform matrix construction based on the diagonal matrix and the affinity matrix to obtain the constructed matrix;

[0197] The construction matrix is ​​subjected to singular value decomposition by a preset decomposition algorithm to obtain at least one singular value vector in the construction matrix and the singular value corresponding to each singular value vector.

[0198] At least one singular value vector is sorted in descending order according to the size of the singular values ​​to obtain at least one target singular value vector, and the at least one target singular value vector is concatenated column by column to obtain a singular value matrix;

[0199] The singular value matrix is ​​standardized row by row to obtain the matrix to be clustered. The matrix to be clustered is then clustered using a preset clustering algorithm to obtain at least one cache cluster.

[0200] In one embodiment of this application, the clustering unit 44 is specifically used for:

[0201] The numerical elements of each row in the affinity matrix are summed to obtain the combined numerical elements corresponding to each row of the affinity matrix.

[0202] Construct the diagonal matrix corresponding to the affinity matrix based on the merged numerical elements.

[0203] In one embodiment of this application, the clustering unit 44 is specifically used for:

[0204] The row data of the matrix to be clustered is determined as the data to be clustered, wherein the row data is used to indicate the caching of at least one target word on at least one target attention layer, and the at least one target attention layer is the attention layer selected from at least one attention layer during the construction of the matrix to be clustered;

[0205] At least one set of data to be clustered is clustered using a pre-defined clustering algorithm to obtain at least one cached cluster, wherein each cached cluster contains a cache of a target word on at least one target attention layer.

[0206] In one embodiment of this application, the first allocation unit 45 is specifically used for:

[0207] Calculate the mean of the cluster importance vector corresponding to at least one cache cluster to obtain the first mean vector corresponding to at least one cache cluster, wherein the cluster importance vector is the importance vector of the target word on at least one target attention layer;

[0208] At least one cache cluster is labeled according to the size of the first mean vector to obtain the category label corresponding to each cache cluster;

[0209] Based on the category label, cluster importance vector, and cluster layer identifier corresponding to at least one cache cluster, construct cache budget features corresponding to at least one cache cluster, wherein the cluster layer identifier is the layer identifier corresponding to at least one target attention layer.

[0210] The cache budget features are input into a preset cache budget model for cache allocation processing, resulting in cache budget vectors corresponding to at least one cache cluster.

[0211] In one embodiment of this application, the first allocation unit 45 is specifically used for:

[0212] The mean and standard deviation of the target cache cluster are calculated based on the target cluster importance vector corresponding to the target cache cluster to obtain the target mean vector and target standard deviation of the target cache cluster. The target cache cluster is the cache cluster corresponding to any category label.

[0213] The mean and standard deviation of the target cache cluster are calculated based on the target cluster layer identifiers corresponding to the target cache clusters to obtain the identifier mean and identifier standard deviation of the target cache clusters.

[0214] Construct target cache budget features for target cache clusters based on the standard mean vector, target standard deviation, label mean, and label standard deviation, until at least one cache cluster has its own corresponding cache budget features.

[0215] In one embodiment of this application, the second allocation unit 46 is specifically used for:

[0216] Determine the cache budget corresponding to at least one cache cluster based on the cache budget vector;

[0217] In response to the cache budget, space allocation can be made for all target terms in the cache cluster, and cache space compression allocation can be made for all target terms in the target cache cluster at at least one target attention layer.

[0218] In response to the fact that the cache budget cannot allocate space for all target words in the cache cluster, the target words are sorted in descending order according to the size of the corresponding cluster importance vector to obtain the sorted target words;

[0219] Select the first preset number of target words as words to be assigned in the sorting target words, and use the cache budget to compress and allocate cache space for the words to be assigned in at least one target attention layer.

[0220] In one embodiment of this application, such as Figure 5 As shown, the key-value cache allocation device further includes a training unit 47, wherein the training unit 47 is used for:

[0221] Obtain the training cache budget features corresponding to each of the multiple training cache clusters. Each training cache cluster includes multiple training clusters, and the training cache budget features include the training features corresponding to each of the multiple training clusters.

[0222] Based on the training cache budget features, a grid search process is performed on the training cache budget vector to obtain the target training cache budget vector corresponding to each of the multiple training cache clusters.

[0223] The cache budget model is trained based on multiple training cache clusters and target training cache budget vectors to obtain the preset cache budget model.

[0224] In one embodiment of this application, the clustering unit 44 is further configured to cluster the caches of other lexical units on at least one attention layer into other lexical units clusters, wherein the other lexical units are lexical units other than the target lexical unit among at least one lexical unit.

[0225] The second allocation unit 46 is also used to allocate a preset cache budget for other word clusters, and to perform cache space compression allocation for each of the other word clusters on at least one attention layer based on the preset cache budget.

[0226] In one embodiment of this application, the determining unit 41 is specifically used for:

[0227] The first attention vector is multiplied by the square root of the first citation count to obtain the first importance vector. The first attention vector is the attention vector generated by the first word in the first attention layer, the first citation count is the citation count corresponding to the first word, the first word is any word among at least one word, and the first attention layer is any attention layer among at least one attention layer.

[0228] Continue until you obtain the importance vector of at least one word element on at least one attention layer.

[0229] In one embodiment of this application, such as Figure 5 As shown, the key-value cache allocation device further includes a processing unit 48, wherein the processing unit 48 is used for:

[0230] The data to be processed is input into a preset model for model inference processing to obtain at least one word element corresponding to the data to be processed, and the attention vector generated by each of the at least one word element on at least one attention layer.

[0231] Determine the historical citation count of at least one word element in the preset model, and determine the citation count of at least one word element based on the historical citation count.

[0232] For a description of the features in the embodiment corresponding to the key-value cache allocation device, please refer to the relevant description in the embodiment corresponding to the key-value cache allocation method, which will not be repeated here.

[0233] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any of the above-described key-value cache allocation method embodiments.

[0234] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described key-value cache allocation method embodiments when running.

[0235] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0236] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described key-value cache allocation method embodiments.

[0237] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described key-value cache allocation method embodiments.

[0238] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0239] The foregoing has provided a detailed description of the key-value cache allocation method, electronic device, and computer program provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A method for allocating key-value caches, characterized in that, include: Based on the attention vector generated by each of the at least one lexical units on at least one attention layer of the preset model, and the number of references of each of the at least one lexical units in the preset model, the importance vector of each of the at least one lexical units on the at least one attention layer is determined. Based on the importance vector, construct feature vectors for each of the at least one lexical unit on the at least one attention layer, and construct an affinity matrix based on the feature vectors and the citation count; Based on the affinity matrix, the caches of the at least one terminology on the at least one attention layer are clustered to obtain at least one cache cluster. Based on the importance vector and the layer identifier of the at least one attention layer, a corresponding cache budget vector is assigned to the at least one cache cluster. Based on the cache budget vector, the cache space of each of the at least one lexical unit on the at least one attention layer is compressed and allocated; The step of compressing and allocating cache space for each of the at least one lexical unit on the at least one attention layer based on the cache budget vector includes: The cache budget corresponding to each of the at least one cache cluster is determined based on the cache budget vector; In response to the cache budget being able to allocate space for all target lexical units in the cache cluster, the cache budget is used to perform cache space compression allocation for all target lexical units in the cache cluster on at least one target attention layer; In response to the cache budget being unable to allocate space for all the target words in the cache cluster, the target words are sorted in descending order according to the size of the corresponding cluster importance vector to obtain the sorted target words; The first preset number of target words are selected as words to be assigned in the sorted target words, and the cache space of the words to be assigned is compressed and allocated for the cache of the words to be assigned on the at least one target attention layer through the cache budget.

2. The key-value cache allocation method according to claim 1, characterized in that, The step of constructing the feature vector of each of the at least one word unit on the at least one attention layer based on the importance vector includes: In the preset model dictionary corresponding to the preset model, obtain the word index corresponding to each of the at least one word element; Based on the importance vector of each of the at least one word element on the at least one attention layer, the word index, and the layer identifier, the feature vector of each of the at least one word element on the at least one attention layer is constructed, wherein the layer identifier is the layer number of each of the at least one attention layer in the preset model.

3. The key-value cache allocation method according to claim 1, characterized in that, The construction of the affinity matrix based on the feature vector and the reference number includes: The feature vectors are sorted in descending order according to their importance to obtain at least one target feature vector and the target word corresponding to each of the at least one target feature vectors; Data calculations are performed based on the first target feature vector, the first target citation count, the second target feature vector, and the second target citation count to obtain matrix numerical elements. The first target feature vector and the second target feature vector are either of the target feature vectors, the first target citation count is the citation count of the first target word corresponding to the first target feature vector, and the second target citation count is the citation count of the second target word corresponding to the second target feature vector. The at least one target feature vector is processed to obtain at least one matrix numerical element, and the affinity matrix is ​​constructed based on the at least one matrix numerical element.

4. The key-value cache allocation method according to claim 3, characterized in that, The step of clustering the caches of the at least one lexical unit on the at least one attention layer according to the affinity matrix to obtain at least one cache cluster includes: Calculate the diagonal matrix corresponding to the affinity matrix, and perform matrix construction processing based on the diagonal matrix and the affinity matrix to obtain the constructed matrix; The construction matrix is ​​subjected to singular value decomposition by a preset decomposition algorithm to obtain at least one singular value vector in the construction matrix and the singular value corresponding to each of the at least one singular value vectors. The at least one singular value vector is sorted in descending order according to the size of the singular value to obtain at least one target singular value vector, and the at least one target singular value vector is concatenated column by column to obtain a singular value matrix; The singular value matrix is ​​standardized row by row to obtain a clustering matrix, and the clustering matrix is ​​then clustered using a preset clustering algorithm to obtain at least one cache cluster.

5. The key-value cache allocation method according to claim 4, characterized in that, The calculation of the diagonal matrix corresponding to the affinity matrix includes: The matrix numerical elements in each row of the affinity matrix are summed to obtain the merged numerical elements corresponding to each row of the affinity matrix. The diagonal matrix corresponding to the affinity matrix is ​​constructed based on the merged numerical elements.

6. The key-value cache allocation method according to claim 4, characterized in that, The step of performing clustering processing on the matrix to be clustered using a preset clustering algorithm to obtain the at least one cache cluster includes: The row data of the matrix to be clustered is determined as the data to be clustered, wherein the row data is used to indicate the caching of at least one of the target words on the at least one target attention layer, and the at least one target attention layer is the attention layer selected from the at least one attention layer during the construction of the matrix to be clustered; The at least one set of data to be clustered is clustered using the preset clustering algorithm to obtain at least one cached cluster, wherein each cached cluster contains a cache of the target word on the at least one target attention layer.

7. The key-value cache allocation method according to claim 6, characterized in that, The process of assigning corresponding cache budget vectors to the at least one cache cluster based on the importance vector and the layer identifier of each of the at least one attention layer includes: Calculate the mean of the cluster importance vector corresponding to each of the at least one cache cluster to obtain the first mean vector corresponding to each of the at least one cache cluster, wherein the cluster importance vector is the importance vector of the target word on the at least one target attention layer respectively; The at least one cache cluster is labeled according to the size of the first mean vector to obtain the category label corresponding to each of the at least one cache cluster; Based on the category label, the clustering importance vector, and the clustering layer identifier corresponding to each of the at least one cache cluster, construct the cache budget feature corresponding to each of the at least one cache cluster, wherein the clustering layer identifier is the layer identifier corresponding to each of the at least one target attention layer; The cache budget features are input into a preset cache budget model for cache allocation processing to obtain the cache budget vector corresponding to each of the at least one cache cluster.

8. The key-value cache allocation method according to claim 7, characterized in that, The step of constructing the cache budget feature corresponding to each of the at least one cache cluster based on the category label, the cluster importance vector, and the clustering layer identifier corresponding to each of the at least one cache cluster includes: The mean and standard deviation of the target cache cluster are calculated based on the target cluster importance vector corresponding to the target cache cluster to obtain the target mean vector and target standard deviation of the target cache cluster, wherein the target cache cluster is the cache cluster corresponding to any of the category labels; The mean and standard deviation of the target cache cluster are calculated based on the target cluster layer identifier corresponding to the target cache cluster to obtain the identifier mean and identifier standard deviation of the target cache cluster. Based on the standard mean vector, the target standard deviation, the identifier mean, and the identifier standard deviation, construct the target cache budget features corresponding to the target cache cluster, until the cache budget features corresponding to each of the at least one cache cluster are constructed.

9. The key-value cache allocation method according to claim 7, characterized in that, Before inputting the cache budget features into a preset cache budget model for cache allocation processing to obtain the cache budget vector corresponding to each of the at least one cache cluster, the method further includes: Obtain training cache budget features corresponding to each of multiple training cache clusters, wherein each training cache cluster includes multiple training clusters, and the training cache budget features include training features corresponding to each of the multiple training clusters. Based on the training cache budget features, the training cache budget vector is subjected to gridded search processing to obtain the target training cache budget vector corresponding to each of the multiple training cache clusters. The cache budget model is trained based on the multiple training cache clusters and the target training cache budget vector to obtain the preset cache budget model.

10. The key-value cache allocation method according to claim 3, characterized in that, After filtering the feature vectors in descending order of importance to obtain at least one target feature vector and the target word corresponding to each of the at least one target feature vectors, the method further includes: Other lexical units are each cached on the at least one attention layer and clustered into other lexical units clusters, wherein the other lexical units are lexical units other than the target lexical unit among the at least one lexical units; A preset cache budget is allocated for the clustering of the other lexical units, and the cache space of each of the other lexical units on the at least one attention layer is compressed and allocated based on the preset cache budget.

11. The key-value cache allocation method according to claim 1, characterized in that, Determining the importance vector of each of the at least one word element in the at least one attention layer includes: The first attention vector is multiplied by the square root of the first citation number to obtain the first importance vector, wherein the first attention vector is the attention vector generated by the first word in the first attention layer, the first citation number is the citation number corresponding to the first word, the first word is any word among the at least one word, and the first attention layer is any attention layer among the at least one attention layer. Until the importance vector of each of the at least one word element on the at least one attention layer is obtained.

12. The key-value cache allocation method according to claim 1, characterized in that, The method further includes: The data to be processed is input into the preset model for model inference processing to obtain at least one word element corresponding to the data to be processed, and the attention vector generated by each of the at least one word element on the at least one attention layer; Determine the historical citation count corresponding to each of the at least one word element in the preset model, and determine the citation count corresponding to each of the at least one word element based on the historical citation count.

13. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the allocation method of the key-value cache as described in any one of claims 1 to 12 when executing the computer program.

14. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the key-value cache allocation method as described in any one of claims 1 to 12.

Images