Model cache optimization method and device based on hierarchical quantification and dynamic priority management

Abstract

The invention discloses a hierarchical quantization and dynamic priority management-based model cache optimization method and device. The method comprises the following steps of: acquiring an importance score of each Token of a word sequence corresponding to a reasoning long text by utilizing a mask language model; constructing a Token importance scoring model, and dividing the cloud platform GPU into a head cache region, a middle cache region and a tail cache region; the service reasoning model stores the key value pairs of the first # imgabs0 # Token in a Token reasoning sequence corresponding to the reasoning long text in a head cache region in an FP16 format, stores the key value pairs of the first # imgabs1 # Token before the current processing Token in a tail cache region in the FP16 format according to an FIFO principle, and stores the key value pairs shifted out of the tail cache region in an intermediate cache region in an FP8 format; and after the intermediate cache region is full, updating the intermediate cache region by using the Token importance scoring model and the importance score. According to the method, the reasoning efficiency of the large language model in long text generation can be improved.

Description

Technical Field

[0001] The present invention belongs to the field of computer processing technology, and in particular relates to a model cache optimization method and device based on hierarchical quantization and dynamic priority management. Background Art

[0002] In the current development trend of artificial intelligence, large models such as the GPT series and BERT series for natural language processing have attracted widespread attention due to their excellent performance. These models usually require a lot of computing resources and memory, especially in providing real-time reasoning services to users as an AI service provider. In the reasoning process of large language models (LLMs), KV Cache (key-value cache) plays a vital role, especially when dealing with long text generation tasks. This cache mechanism is mainly used to store and quickly retrieve the key-value pairs of tokens, which are key data for self-attention calculations. In the self-attention mechanism, each input token is compared with other tokens to calculate the attention score, and keys (K) and values ​​(V) are generated in this process. The use of KV Cache can significantly reduce repeated calculations, especially when generating long texts. The key-value pairs of previously generated tokens can be reused by subsequent tokens, thereby improving the overall computing efficiency and response speed. For example, when users are doing document question and answer, novel writing and other scenarios that require long text input / output, the GPU needs to cache the KV matrices generated by a large number of tokens. Under the premise that the GPU video memory is limited, quantitative cache management of the KV matrix can effectively save video memory usage and improve the service throughput of the GPU.

[0003] The strategy of quantizing KV Cache further optimizes the use of storage and computing resources. By converting standard floating-point numbers (such as FP16) into smaller formats (such as FP8 or Int4), the required storage space can be greatly reduced while speeding up data processing. This optimization is particularly important in the scenario of long text generation, because the longer the generated text, the more tokens and corresponding key-value pairs are involved, and traditional storage and computing methods will quickly reach resource limitations. Quantization can also reduce bandwidth requirements when transmitting data, which is particularly critical for cloud-based language model services because data transmission is often a major source of latency. In addition, this quantization scheme has the following disadvantages:

[0004] 1. Large loss of precision, affecting the quality of inference: Traditional quantization algorithms usually adopt a unified quantization strategy, such as quantizing FP16 to INT8 or lower precision. This unified precision reduction method does not take into account the importance differences of data when processing all data, resulting in a large loss of precision during model inference. For some key information or important tokens, quantization may cause their information to be lost or inaccurate, which in turn affects the overall performance and generation quality of the model;

[0005] 2. Lack of dynamic adjustment and inability to adapt to the needs of different scenarios: Traditional quantization methods are mostly static. Once quantization is completed, the data accuracy will not be dynamically adjusted during the entire reasoning process. This means that when processing long text generation or other tasks, the model cannot adjust the importance and resource allocation of the previous and subsequent data processing according to actual needs. For long text generation tasks, some early tokens may become less important as the generation process progresses, but traditional quantization methods cannot further compress and optimize them;

[0006] 3. Global quantization strategy, it is difficult to perform differentiated processing for different types of data: Traditional quantization algorithms usually use a unified quantization method for the entire model or data set, and cannot flexibly distinguish the importance of tokens. For large language models, some tokens play a vital role in sequence generation, while other tokens may have little impact on the final result. However, traditional quantization cannot tailor appropriate quantization strategies for these different types of tokens, resulting in a waste of resources and degraded model performance;

[0007] 4. Inability to effectively balance storage and computing resource usage: When dealing with complex tasks such as long text generation, the limitation of storage and computing resources is a key issue. While traditional quantization schemes reduce computational complexity, they often fail to achieve the optimal balance of storage and computing resources because they use the same precision for all tokens. Especially when dealing with a large amount of irrelevant or redundant information, traditional quantization cannot effectively compress this data, increasing storage pressure. Summary of the invention

[0008] In view of the above problems, the present invention proposes a model cache optimization method and device based on hierarchical quantization and dynamic priority management, which can improve the reasoning efficiency of large language models in long text generation. To solve the above technical problems, the technical solutions adopted by the present invention are as follows:

[0009] A model cache optimization method based on hierarchical quantization and dynamic priority management includes the following steps:

[0010] S1, collects the inference long text received by the cloud platform, divides the inference long text into several word sequences of length N, and uses the masked language model to obtain the importance score of each token in the word sequence;

[0011] S2, using MLP multi-layer perceptron to build a token importance scoring model on the cloud platform, divide the GPU space of the cloud platform into a head buffer for storing the starting key-value pairs, a middle buffer for storing important key-value pairs, and a tail buffer for storing the key-value pairs associated with the current token, and initialize the text processing progress value

[0012] S3, input the Token inference sequence corresponding to the inference long text obtained in step S1 into the service inference model of the cloud platform, and the service inference model converts the first The key-value pairs corresponding to each Token are stored in the header buffer in FP16 format. The previous Token before the current one is processed is stored in the header buffer according to the FIFO principle. The key-value pairs corresponding to the tokens are stored in the tail buffer in FP16 format, and the key-value pairs removed from the tail buffer are stored in the middle buffer in FP8 format. When the middle buffer is full, the Token importance scoring model established in step S2 and the importance score of each token obtained in step S1 are used to update the middle buffer, where: Indicates the length of the header buffer and the tail buffer.

[0013] In step S1, the calculation formula of the importance score of the Token is:

[0014]

[0015] In the formula, Indicates the value of Token w l，n The mask word sequence obtained after masking, w l，n Represents the lth word sequence S l The nth Token in I(w l，n ) represents Token w l，n The importance score of Represents the masked language model for the masked word sequence The predicted probability of .

[0016] In step S2, the Token importance scoring model includes a first fully connected layer, a first activation function layer, a second fully connected layer, a second activation function layer, a third fully connected layer and a third activation function layer which are connected in sequence.

[0017] The step S3 comprises the following steps:

[0018] S3.1, use word embedding to convert the inference long text obtained in step S1 into the corresponding Token inference sequence, and input the Token inference sequence into the service inference model of the cloud platform;

[0019] S3.2, Execution The service inference model converts the first The key-value pairs corresponding to each token are stored in the header buffer in FP16 format;

[0020] S3.3, Judgment If yes, return to step S3.2, otherwise, execute step S3.4;

[0021] S3.4, Execution The service inference model converts the first The key-value pairs corresponding to each token are stored in the tail buffer in FP16 format;

[0022] S3.5, Judgment If yes, return to step S3.4, otherwise, execute step S3.6;

[0023] S3.6, Execution Update the tail buffer according to the FIFO principle based on the key-value pair corresponding to the mth token in the Token inference sequence;

[0024] S3.7, Judgment If yes, the key-value pairs removed from the tail buffer are quantized into FP8 format and stored in the middle buffer, and the process returns to step S3.6; otherwise, the process proceeds to step S3.8;

[0025] S3.8, using the Token importance scoring model established in step S2 and the importance score of each Token obtained in step S1, calculate the importance score of each Token corresponding to the intermediate cache, remove the key value corresponding to the Token with the lowest importance score from the intermediate cache to update the intermediate cache, quantize the key-value pairs removed from the intermediate cache into INT4 format and store them in other spaces of the GPU of the cloud platform, and quantize the key-value pairs removed from the tail cache into FP8 format and store them in the intermediate cache;

[0026] S3.9, Judgment If so, return to step S3.6, otherwise, the service reasoning model outputs the reasoning result based on all key-value pairs, where L represents the number of tokens in the Token reasoning sequence.

[0027] The method for quantizing a key-value pair in FP16 format to a key-value pair in FP8 format comprises the following steps:

[0028] i. Calculate the gradient of the key-value pairs in FP16 format according to the error function and weight matrix of the service inference model, and divide the key-value matrix corresponding to the key-value pairs into several key-value blocks of the same size;

[0029] ii. Construct an empty row element set corresponding to each key value block;

[0030] iii. Copy the first row elements of each key value block to the corresponding row element set, and update each row element set in the order of gradient from small to large;

[0031] iv. According to the first element in each row element set obtained in step iii, respectively confirm the position of the corresponding element in the corresponding key value block, quantize the elements according to the round-up and round-down decisions, calculate the quantization error of the key value matrix before and after quantization according to the Frobenius norm, determine the rounding decision with the goal of minimizing the quantization error, and update the key value block and key value matrix;

[0032] v, according to the quantization error obtained in step iv, correct the other elements in the first row of each key value block except the quantized elements in step iv one by one, and update the key value matrix synchronously;

[0033] vi. Operate the other elements in the row element set one by one according to the method of step iv and step v;

[0034] vii. Quantize and correct other row elements in each key-value block in turn using the methods of step iii, step iv, step v and step vi to generate key-value pairs in FP8 format.

[0035] In step iv, the expression of the objective function corresponding to the quantization error is:

[0036]

[0037] In the formula, represents the quantized key-value matrix, represents the key-value matrix before quantization, ‖·‖ F represents the Frobenius norm.

[0038] In step v, the formula for correcting the elements in the first row of each key value block except the quantized elements in step iv is:

[0039]

[0040] In the formula, Indicates the other elements in the key-value matrix except the quantized elements in the jth column. i，j represents the matrix error generated by quantizing the elements in the i-th row and j-th column of the key-value matrix. Represents all elements in the j-th row but not in the j-th column of the inverse of the Hessian matrix corresponding to the i-th row of the key-value matrix.

[0041] A model cache optimization device based on hierarchical quantization and dynamic priority management is applied to an AI service system consisting of a user terminal and a cloud platform, where the cloud platform is provided with:

[0042] Data collection module: used to receive the inference long text sent by the user terminal, and divide the inference long text into several word sequences of length N;

[0043] Importance score calculation module: used to obtain the importance score of each token in the word sequence generated by the data collection module using the masked language model;

[0044] Model storage module: used to store the Token importance scoring model and the service reasoning model. The Token importance scoring model is built based on the MLP multi-layer perceptron. The MLP multi-layer perceptron includes a first fully connected layer, a first activation function layer, a second fully connected layer, a second activation function layer, a third fully connected layer and a third activation function layer connected in sequence. The first activation function layer and the second activation function layer both use the ReLU activation function, and the third activation function layer uses the Sigmoid activation function. The service reasoning model is built based on the LLM model and is used to process the reasoning long text.

[0045] GPU space division module: used to divide the GPU space of the cloud platform into a head buffer area for storing the starting key-value pairs, a middle buffer area for storing important key-value pairs, and a tail buffer area for storing the key-value pairs associated with the current token;

[0046] Text encoding module: used to convert the inference long text obtained by the data collection module into a Token inference sequence, and send the Token inference sequence to the service inference model of the model storage module;

[0047] Key-value pair processing module: used to convert the first The key-value pairs corresponding to each Token are stored in the header buffer in FP16 format. The previous Token before the current one is processed is stored in the header buffer according to the FIFO principle. The key-value pairs corresponding to each Token are stored in the tail buffer in FP16 format, and the key-value pairs removed from the tail buffer are stored in the middle buffer in FP8 format. When the middle buffer is full, the importance score of each Token obtained by the Token importance scoring model of the model storage module and the importance score calculation module is used to update the middle buffer. Indicates the length of the header buffer and the tail buffer.

[0048] Beneficial effects of the present invention:

[0049] 1. Hierarchical quantization to improve the accuracy of key data retention: The quantization method proposed in this invention processes the Token KV cache in layers (innovatively divides the Token KV cache into high-precision and low-precision storage areas), with key Tokens stored in FP8 format and low-importance Tokens stored in INT4. This hierarchical quantization method can adjust the accuracy of Tokens of different importance so that the accuracy of key information can be retained, thereby reducing storage costs while ensuring the quality of model reasoning. This flexible precision management avoids the information loss caused by uniform precision reduction, significantly improves the reasoning effect, and greatly saves storage space;

[0050] 2. Dynamic priority management and real-time adjustment of KV cache: A predictor based on a multi-layer perceptron (MLP) is introduced (historical tokens are scored through self-supervised learning), which can dynamically evaluate the importance of each token and adjust the data accuracy in the KV cache in real time according to the needs of the current task. Whenever a new token is input, the system will re-score the existing key tokens and gradually downgrade the tokens with lower scores to INT4 format, thereby ensuring that only the most important data is retained in the cache. This dynamic management mechanism breaks through the static nature of traditional quantization schemes and has higher flexibility than traditional static quantization schemes (enabling the system to optimize storage allocation in real time based on the importance of the token, further improving reasoning efficiency and model performance), effectively adapting to the needs of different task scenarios;

[0051] 3. Self-supervised learning technology to accurately predict the importance of tokens: By using self-supervised learning technology to automatically annotate the importance of tokens, an MLP predictor that can score historical tokens is constructed. Unlike traditional quantization algorithms that rely on fixed quantization rules, the predictor of this invention can more accurately identify and distinguish the importance of tokens by learning the semantics and contextual relationships in the data;

[0052] 4. Globally tuned quantization to reduce precision loss: A globally tuned quantization technique is used, that is, the gradient information of the entire model is taken into account during the quantization process to ensure that the error between the quantized weight matrix and the original weight matrix is ​​minimized. Compared with the traditional quantization algorithm that directly performs simple rounding or truncation, the present invention further reduces the precision loss caused by quantization through global correction after quantization, so that the quality of model reasoning can be guaranteed even under low-precision storage. This global optimization strategy significantly improves the robustness of the model, especially in long text generation tasks, and can ensure the accuracy and consistency of the generated results. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0054] Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0055] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0056] A model cache optimization method based on hierarchical quantization and dynamic priority management, such as Figure 1 As shown, the following steps are included:

[0057] S1, collects the inference long text received by the cloud platform, divides the inference long text into several word sequences of length N, and uses the masked language model to obtain the importance score of each Token in the word sequence, including the following steps:

[0058] S1.1, collect long texts for AI service reasoning on the cloud platform, and divide the long texts into several word sequences of length N;

[0059] First, collect the long text data in the AI ​​service, and then divide the long text data into several word sequences, each of which contains N tokens. is a positive integer, and For example, it can be 320, 160. In addition, when segmenting long text data, When it is not an integer, represents the length of the inference long text, and Determine whether the length of the remaining text is greater than 32. If so, fill it with zeros to form the last word sequence. At this time, the total number of word sequences is Otherwise, the remaining text is discarded directly, that is, the total number of word sequences at this time is

[0060] Using S l represents the lth word sequence, S l ={w l，1 , w l，2 , wl，3 ,…,w l，n ,…,w l，N}, where w l，n Represents the word sequence S l The nth Token in , where N represents the number of Tokens in the word sequence.

[0061] S1.2, mask each token in the word sequence obtained in step S1.1 in turn and input the masked language model to obtain the importance score of each token;

[0062] In order to quantify the importance of a token, the key is how to measure the prediction accuracy of a token. If a token can be easily and accurately predicted based on the context after being masked, it means that the token contributes little to the overall meaning of the sentence and can be called a "low importance" token. On the contrary, if a token is difficult to predict after being masked, it means that the token carries key information and cannot be easily inferred from the context. It contributes a lot to the context and is a "high importance" token.

[0063] In order to evaluate the importance of each token, a masked language model (MLM) is used for each token to perform the prediction task. Self-supervision technology is used to automatically annotate the importance of tokens using the output difference of the masked language model prediction task. By masking a portion of the input tokens and letting the model predict the masked tokens, MLM can learn the dependencies between tokens in the context. That is, each token is masked in turn (for example, the nth token in a word sequence is replaced with [MASK]), and then the model is allowed to predict the masked token based on the contextual information. This prediction process reveals the importance of each token in preserving the contextual semantics.

[0064] In order to automatically annotate more accurately, the importance of the token is calculated using the predicted probability. For each masked token, the model generates a probability distribution, which indicates the probability that the model predicts that the token is a different word. If the probability of the correct prediction of the masked token is high, it means that the importance of the token is relatively low, because the model can still easily infer it based on the context in the absence of it. If the probability of the correct prediction is low, it means that the importance of the token in the context is high, because its existence is crucial to the completeness and semantic understanding of the sentence. Through this probability distribution, an importance score can be assigned to each token.

[0065] Specifically, after replacing each Token in the word sequence with [MASK], the masked word sequence is input into the masked language model for prediction. According to the predicted probability of each Token after masking by the masked language model, the importance score of each Token can be obtained. The corresponding calculation formula is:

[0066]

[0067] In the formula, Represents the word sequence S l w in l，n The masked word sequence obtained after Token masking, I(w l，n ) indicates Token l，n The importance score of Represents the masked language model for the masked word sequence The predicted probability of w is . Formula (1) shows that the lower the predicted probability, the higher the importance score, and vice versa. l，n The predicted probability after masking,

[0068] S2, using MLP multi-layer perceptron to build a token importance scoring model on the cloud platform, divide the GPU space of the cloud platform into a head buffer for storing the starting key-value pairs, a middle buffer for storing important key-value pairs, and a tail buffer for storing the key-value pairs associated with the current token, and initialize the text processing progress value m = 0;

[0069] In order to score each token of the input data, the embodiment of the present application uses a multi-layer perceptron (MLP) as a predictor of important tokens, and the predictor receives the embedding of N tokens as input. The dimension of each token embedding is consistent with the output of the pre-trained language model.

[0070] The token importance scoring model includes a first fully connected layer, a first activation function layer, a second fully connected layer, a second activation function layer, a third fully connected layer, and a third activation function layer connected in sequence. The first activation function layer and the second activation function layer both use the ReLU activation function, and the third activation function layer uses the Sigmoid activation function. The first fully connected layer and the first activation function layer serve as the input layer, the second fully connected layer and the second activation function layer serve as the intermediate layer, and the third fully connected layer and the third activation function layer serve as the output layer. The expressions corresponding to the input layer, the intermediate layer, and the output layer are:

[0071] h1=ReLU(W1·X+b1); (2)

[0072] h2=ReLU(W2·h1+b2); (3)

[0073] y=Sigmoid(W3·h2+b3); (4)

[0074] In the formula, X represents the input, that is, the mask word sequence, b1 represents the bias of the first fully connected layer, W1 represents the weight matrix of the first fully connected layer, corresponding to the projection dimension of the first fully connected layer, h1 represents the output of the input layer, b2 represents the bias of the second fully connected layer, W2 represents the weight matrix of the second fully connected layer, corresponding to the projection dimension of the second fully connected layer, h2 represents the output of the intermediate layer, y represents the output of the output layer, b3 represents the bias of the third fully connected layer, W3 represents the weight matrix of the third fully connected layer, corresponding to the projection dimension of the third fully connected layer.

[0075] In the process of optimizing the service inference model inference of long text generation, effectively managing the key-value cache (KV cache) is of great significance for improving model performance and reducing computing costs. Aiming at the characteristics of the attention mechanism (self-attention), that is, the attention tends to focus on the head and tail of the input sequence, while the attention to the tokens in the middle segment is relatively weak, this application proposes a hierarchical quantization strategy to achieve efficient use of resources.

[0076] This application divides KVcache into three parts: the head cache, the tail cache, and the middle cache. The head cache is used to store the start of the input word sequence. The key-value pair (also known as KV pair) corresponding to the token, and The tail buffer is used to retain the processed tokens before the current token in the word sequence. The key-value pairs of the tokens in the header buffer and the tail buffer are all in FP16 precision to ensure the calculation accuracy and speed of the service inference model for these key areas.

[0077] For the intermediate buffer area, the embodiment of the present application sets a fixed capacity, namely This area is used to store the KV pairs of important tokens identified by specific strategies so that these data can be quickly obtained during the calculation process. The intermediate cache area is stored in 8-bit floating point (FP8 precision) format to preserve data accuracy and importance.

[0078] When N=320, that is, a word sequence contains 320 tokens, Can be set to 32, then It can also be set to 16. In this case, When N=160, that is, a word sequence contains 160 tokens, Can be set to 16, then It can also be set to 8. In this case,

[0079] When N = 320 and , the input dimension of the Token importance scoring model is 320, and the output dimension is 256. Specifically, the embedding matrix of 320 Tokens is first input into the MLP. The first fully connected layer projects the input 320×768 matrix into a smaller space. Assuming that the output of the first fully connected layer is 512-dimensional, the embedding of each Token is compressed to 512 dimensions. After the nonlinear transformation of the first activation function layer, the embedding of each Token retains its important semantic features and compresses the information for subsequent processing. The second fully connected layer and the second activation function layer process the input again and project it into a 256-dimensional space. At this point, the input data is further simplified from the original embedding of 320 Tokens to a higher-dimensional representation representing its importance. After the activation function RELU of each layer, the nonlinear transformation further enhances the expression ability of the model, enabling it to recognize the complex relationship between the context and the target Token. The output layer is the final prediction layer, and its output dimension is 256, corresponding to the importance score of the middle 256 target Tokens. In order to ensure that the output value is between 0 and 1, this embodiment uses the Sigmoid activation function. In this way, each output value can be regarded as the importance score of the token in the entire sentence, 0 represents unimportant and 1 represents very important.

[0080] When N = 320 and When , the input dimension of the Token Importance Scoring Model is 320, and the output dimension is 288. When N = 160 and , the input dimension of the Token Importance Scoring Model is 160, and the output dimension is 128.

[0081] S3, input the Token inference sequence corresponding to the inference long text obtained in step S1 into the service inference model of the cloud platform, and the service inference model converts the first The key-value pairs corresponding to each Token are stored in the header buffer in FP16 format. The previous Token before the current one is processed is stored in the header buffer according to the FIFO principle. The key-value pairs corresponding to the tokens are stored in the tail buffer in FP16 format, and the key-value pairs removed from the tail buffer are stored in the middle buffer in FP8 format. When the middle buffer is full, the Token importance scoring model established in step S2 and the importance score of each token obtained in step S1 are used to update the middle buffer.

[0082] The service inference model refers to the large language model (LLM) used by the cloud platform that provides AI services to perform real-time inference services based on user needs. This is a prior art and the training details of the model will not be described in detail. When the cloud platform receives long text processing tasks such as document question answering and novel generation for users, the service inference model needs to quantify the KV cache to effectively save video memory usage and improve inference efficiency within an acceptable range of accuracy loss.

[0083] The step S3 comprises the following steps:

[0084] S3.1, use word embedding to convert the inference long text obtained in step S1 into the corresponding Token inference sequence, and input the Token inference sequence into the service inference model of the cloud platform;

[0085] S3.2, Execution The service reasoning model will reason about the first The key-value pairs corresponding to each token are stored in the header buffer in FP16 format;

[0086] S3.3, Judgment If yes, return to step S3.2, otherwise, execute step S3.4;

[0087] S3.4, Execution The service reasoning model will reason about the first The key-value pairs corresponding to each token are stored in the tail buffer in FP16 format;

[0088] S3.5, Judgment If yes, return to step S3.4, otherwise, execute step S3.6;

[0089] S3.6, Execution According to the inference of the long text The key-value pairs corresponding to each token are updated in the tail buffer according to the FIFO (First Input First Output) principle;

[0090] For the most recently generated tokens, this application adopts a progressive replacement strategy. Whenever a new token is generated, the KV pair of the earliest token is quantized from FP16 to FP8 and moved to the middle buffer to keep the tail buffer always at The size of the token.

[0091] S3.7, Judgment If yes, the key-value pairs removed from the tail buffer are quantized into FP8 format and stored in the middle buffer, and the process returns to step S3.6; otherwise, the process proceeds to step S3.8;

[0092] For the KV pairs in the intermediate cache, this application adopts a hierarchical quantization method. When the number of cached tokens in the intermediate cache is less than 256, the KV pairs of the newly generated tokens are quantized from FP16 format to FP8 format and added to the intermediate cache. The quantization operation in this process reduces the required storage space and computing resources by reducing the precision of the data, while trying to maintain the integrity of important information.

[0093] The method for quantizing the key-value pairs removed from the tail buffer area into the FP8 format includes the following steps:

[0094] i. Calculate the gradient of the key-value pair in FP16 format removed according to the error function and weight matrix of the service inference model, and divide the key-value matrix corresponding to the key-value pair into several key-value blocks of the same size;

[0095] The key-value matrix corresponding to the original FP16 format key-value pair, that is, the KV matrix, is 768 times 768. Therefore, the gradient matrix is ​​also 768 times 768. The KV matrix is ​​divided into 128*128 key-value blocks. Similarly, the gradient matrix can also be divided into 128*128 gradient blocks. Dividing the key-value blocks helps to make full use of hardware parallelism and speed up the quantization processing speed. The gradient is represented by G, and the corresponding calculation formula is L is the error function and W is the attention weight matrix. This gradient reflects the importance of each weight to the model output.

[0096] ii. Construct an empty row element set corresponding to each key value block;

[0097] iii. Copy the elements in the first row of each key-value block to the corresponding row element set, and update each row element set in the order of gradient from small to large;

[0098] iv. According to the first element in each row element set obtained in step iii, respectively confirm the position of the corresponding element in the corresponding key value block, quantize the elements according to the round-up and round-down decisions, calculate the quantization error of the key value matrix before and after quantization according to the Frobenius norm, determine the rounding decision with the goal of minimizing the quantization error, and update the key value block and key value matrix;

[0099] The expression of the objective function corresponding to the quantization error is:

[0100]

[0101] In the formula, represents the quantized key-value matrix, Represents the key-value matrix before quantization. ‖·‖ F Represents the Frobenius norm, which is used to measure the error between matrices before and after quantization. Gradient information is used to guide the rounding process, ensuring that the performance of the model does not drop significantly when using lower precision formats such as FP8 or INT4.

[0102] Explanation of the decision to round up or round down: For example, FP8 can accurately express the floating point number 1.75, while INT4 can only express 16 integers, 0, 1, 2, 3, ..., 15. When 1.75 in FP8 format is converted to data in INT4 format, you need to choose whether to round to 1 or 2. When rounded to 1, it is rounded down, and when rounded to 2, it is rounded up. Calculate the errors before and after the matrix under the two rounding decisions respectively, and select the rounding method with the smaller error to update the element and then update the key value matrix.

[0103] v, according to the quantization error obtained in step iv, correct the other elements in the first row of each key value block except the quantized elements in step iv one by one, and update the key value matrix synchronously. The corresponding correction formula is:

[0104]

[0105] In the formula, Indicates the other elements in the key-value matrix except the quantized elements in the jth column. i，j represents the matrix error generated by quantizing the elements in the i-th row and j-th column of the key-value matrix. Represents all elements in the j-th row but not in the j-th column of the inverse of the Hessian matrix corresponding to the i-th row of the key-value matrix.

[0106] Matrix error E i，j The calculation formula is:

[0107]

[0108] In the formula, represents the key value matrix before quantizing the elements in the i-th row and j-th column of the key value matrix, Q i，j It represents the quantization matrix after quantizing the elements in the i-th row and j-th column of the key value matrix, [H -1 ] jj Represents the j-th row and j-th column element of the inverse of the Hessian matrix corresponding to the i-th row of the key-value matrix.

[0109] Quantization matrix Q i，j The calculation formula is:

[0110]

[0111] Where quant(·) represents the quantization function.

[0112] vi. Operate the other elements in the row element set one by one according to the method of step iv and step v;

[0113] vii. Quantize and correct the elements of other rows in each key-value block in turn by using the methods of step iii, step iv, step v and step vi to obtain key-value pairs in FP8 format;

[0114] For each 128*128 key-value block, we adjust one row at a time. For each row, we quantize and adjust the gradient of the elements in the row from small to large. Each time an element is quantized, the values ​​of other elements in the row are corrected and adjusted to ensure that the overall error is minimized during the quantization process. At the same time, in order to avoid the accumulation of quantized errors in the entire model, this application adopts a global correction strategy, that is, after each quantization step, the global error is recalculated and the gradient information is used to correct the entire KV matrix. By gradually adjusting the weights in each block, ensure that the final quantized weight matrix is ​​as close to the original matrix as possible.

[0115] S3.8, using the Token importance scoring model established in step S2 and the importance score of each Token obtained in step S1, calculate the importance score of each Token corresponding to the intermediate cache, remove the key value corresponding to the Token with the lowest importance score from the intermediate cache to update the intermediate cache, quantize the key-value pairs removed from the intermediate cache into INT4 format and store them in other spaces of the GPU of the cloud platform, and quantize the key-value pairs removed from the tail cache into FP8 format and store them in the intermediate cache;

[0116] When the number of cache tokens in the intermediate cache reaches At the same time, we introduced a predictor, namely the Token Importance Scoring Model, to further manage these KV pairs. The main task of this predictor is to evaluate the importance of each token. According to the importance score provided by the predictor, we evict the KV pairs of the tokens with the lowest scores from the intermediate cache, and its data format is further quantized to INT4 format. This extremely low-precision format greatly saves storage space and is suitable for tokens that are considered to be the least important.

[0117] The method of quantizing the token with the lowest importance score into the INT4 format is the same as the method of quantizing the key-value pairs removed from the tail buffer into the FP8 format. The difference is that the method of quantizing into the FP8 format operates on the key-value pairs in the FP16 format, while the method of quantizing into the INT4 format operates on the key-value pairs in the FP8 format. The specific operation steps are not repeated here.

[0118] S3.9, Judgment If yes, return to step S3.6, otherwise, the service reasoning model outputs the reasoning result based on all key-value pairs, where L represents the number of tokens in the token reasoning sequence;

[0119] The above adopts global tuning quantization technology to quantize FP16 KV to FP8 and INT4, which not only reduces errors in the quantization process, but also optimizes rounding decisions through gradient information, which can further optimize storage efficiency. This global tuning strategy enables the model to retain important information even at low precision, ensuring that the quality of reasoning is not significantly affected by quantization accuracy. This differentiated processing of Token importance and accuracy requirements, as well as the dynamic management mechanism combined with intelligent predictors, is the core innovation of this technology, solving the key problem of balancing reasoning efficiency and storage resources in long text generation. In order to accurately manage the importance of Token KV, a predictor based on multi-layer perceptron (MLP) is constructed to score historical tokens. When the key Token KV cache is full of 256, each time a new Token is input, the system will call the predictor to score the existing key Token, downgrade the Token KV with the lowest score to INT4 format, and evict it from the key area to ensure efficient use of the cache area.

[0120] S4, the service reasoning model generates reasoning results by reasoning the long text according to the cached key-value pairs;

[0121] By adjusting the precision of the cached key-value pairs, the information loss caused by uniform precision reduction can be avoided, which can significantly improve the reasoning effect of the service reasoning model.

[0122] The embodiment of the present application also provides a model cache optimization device based on hierarchical quantization and dynamic priority management, which is mainly used in an AI service system composed of a user terminal and a cloud platform, wherein the cloud platform is provided with:

[0123] Data collection module: used to receive the inference long text sent by the user terminal, and divide the inference long text into several word sequences of length N;

[0124] Importance score calculation module: used to obtain the importance score of each token in the word sequence generated by the data collection module using the masked language model;

[0125] Model storage module: used to store the Token importance scoring model and the service reasoning model. The Token importance scoring model is built based on the MLP multi-layer perceptron. The MLP multi-layer perceptron includes a first fully connected layer, a first activation function layer, a second fully connected layer, a second activation function layer, a third fully connected layer and a third activation function layer connected in sequence. The first activation function layer and the second activation function layer both use the ReLU activation function, and the third activation function layer uses the Sigmoid activation function. The service reasoning model is built based on the LLM model and is used to process the reasoning long text.

[0126] GPU space division module: used to divide the GPU space of the cloud platform into a head buffer area for storing the starting key-value pairs, a middle buffer area for storing important key-value pairs, and a tail buffer area for storing the key-value pairs associated with the current token;

[0127] Text encoding module: used to convert the inference long text obtained by the data collection module into a Token inference sequence, and send the Token inference sequence to the service inference model of the model storage module;

[0128] Key-value pair processing module: used to convert the first The key-value pairs corresponding to each Token are stored in the header buffer in FP16 format. The previous Token before the current one is processed is stored in the header buffer according to the FIFO principle. The key-value pairs corresponding to each Token are stored in the tail buffer in FP16 format, and the key-value pairs removed from the tail buffer are stored in the middle buffer in FP8 format. When the middle buffer is full, the importance score of each Token obtained by the Token importance scoring model of the model storage module and the importance score calculation module is used to update the middle buffer.

[0129] An embodiment of the present application also provides an electronic device, which includes a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the steps of the above-mentioned model cache optimization method based on hierarchical quantization and dynamic priority management are performed.

[0130] The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above-mentioned model cache optimization method based on hierarchical quantization and dynamic priority management are executed. Specifically, the storage medium can be a general storage medium, such as a mobile disk, a hard disk, etc. When the computer program on the storage medium is run, the embodiment of the above-mentioned model cache optimization method based on hierarchical quantization and dynamic priority management can be executed.

[0131] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A model cache optimization method based on hierarchical quantization and dynamic priority management, characterized in that: The steps include: S1, collects the inference long text received by the cloud platform, divides the inference long text into several word sequences of length N, and uses the masked language model to obtain the importance score of each token in the word sequence; S2, using MLP multi-layer perceptron to build a token importance scoring model on the cloud platform, divide the GPU space of the cloud platform into a head buffer for storing the starting key-value pairs, a middle buffer for storing important key-value pairs, and a tail buffer for storing the key-value pairs associated with the current token, and initialize the text processing progress value S3, input the Token inference sequence corresponding to the inference long text obtained in step S1 into the service inference model of the cloud platform, and the service inference model converts the first The key-value pairs corresponding to each Token are stored in the header buffer in FP16 format. The previous Token before the current one is processed is stored in the header buffer according to the FIFO principle. The key-value pairs corresponding to the tokens are stored in the tail buffer in FP16 format, and the key-value pairs removed from the tail buffer are stored in the middle buffer in FP8 format. When the middle buffer is full, the Token importance scoring model established in step S2 and the importance score of each token obtained in step S1 are used to update the middle buffer, where: Indicates the length of the header buffer and the tail buffer.

2. The model cache optimization method based on hierarchical quantization and dynamic priority management according to claim 1 is characterized in that: In step S1, the calculation formula of the importance score of the Token is: In the formula, Indicates Tokenw l,n The mask word sequence obtained after masking, w l,n Represents the lth word sequence S l The nth Token in I(w l,n ) indicates Token l,n The importance score of Represents the masked language model for the masked word sequence The predicted probability of .

3. The model cache optimization method based on hierarchical quantization and dynamic priority management according to claim 1 is characterized in that: In step S2, the Token importance scoring model includes a first fully connected layer, a first activation function layer, a second fully connected layer, a second activation function layer, a third fully connected layer and a third activation function layer which are connected in sequence.

4. The model cache optimization method based on hierarchical quantization and dynamic priority management according to claim 1 is characterized in that: The step S3 comprises the following steps: S3.1, use word embedding to convert the inference long text obtained in step S1 into the corresponding Token inference sequence, and input the Token inference sequence into the service inference model of the cloud platform; S3.2, Execution The service inference model converts the first The key-value pairs corresponding to each token are stored in the header buffer in FP16 format; S3.3, Judgment If yes, return to step S3.2, otherwise, execute step S3.4; S3.4, Execution The service inference model converts the first The key-value pairs corresponding to each token are stored in the tail buffer in FP16 format; S3.5, Judgment If yes, return to step S3.4, otherwise, execute step S3.6; S3.6, Execution According to the Token inference sequence The key-value pairs corresponding to each token are updated in the tail buffer according to the FIFO principle; S3.7, Judgment If yes, the key-value pairs removed from the tail buffer are quantized into FP8 format and stored in the middle buffer, and the process returns to step S3.6; otherwise, the process proceeds to step S3.8; S3.8, using the Token importance scoring model established in step S2 and the importance score of each Token obtained in step S1, calculate the importance score of each Token corresponding to the intermediate cache, remove the key value corresponding to the Token with the lowest importance score from the intermediate cache to update the intermediate cache, quantize the key-value pairs removed from the intermediate cache into INT4 format and store them in other spaces of the cloud platform GPU, and quantize the key-value pairs removed from the tail cache into FP8 format and store them in the intermediate cache; S3.9, Judgment If so, return to step S3.6, otherwise, the service reasoning model outputs the reasoning result based on all key-value pairs, where L represents the number of tokens in the Token reasoning sequence.

5. The model cache optimization method based on hierarchical quantization and dynamic priority management according to claim 1 or 4, characterized in that: The method for quantizing a key-value pair in FP16 format to a key-value pair in FP8 format comprises the following steps: i. Calculate the gradient of the key-value pairs in FP16 format according to the error function and weight matrix of the service inference model, and divide the key-value matrix corresponding to the key-value pairs into several key-value blocks of the same size; ii. Construct an empty row element set corresponding to each key value block; iii. Copy the first row elements of each key value block to the corresponding row element set, and update each row element set in the order of gradient from small to large; iv. According to the first element in each row element set obtained in step iii, respectively confirm the position of the corresponding element in the corresponding key value block, quantize the elements according to the round-up and round-down decisions, calculate the quantization error of the key value matrix before and after quantization according to the Frobenius norm, determine the rounding decision with the goal of minimizing the quantization error, and update the key value block and key value matrix; v, according to the quantization error obtained in step iv, correct the other elements in the first row of each key value block except the quantized elements in step iv one by one, and update the key value matrix synchronously; vi. Operate the other elements in the row element set one by one according to the method of step iv and step v; vii. Quantize and correct other row elements in each key-value block in turn using the methods of step iii, step iv, step v and step vi to generate key-value pairs in FP8 format.

6. The model cache optimization method based on hierarchical quantization and dynamic priority management according to claim 5 is characterized in that: In step iv, the expression of the objective function corresponding to the quantization error is: In the formula, represents the quantized key-value matrix, represents the key-value matrix before quantization, ‖·‖ F represents the Frobenius norm.

7. The model cache optimization method based on hierarchical quantization and dynamic priority management according to claim 5 is characterized in that: In step v, the formula for correcting the elements in the first row of each key value block except the quantized elements in step iv is: In the formula, Indicates the other elements in the key-value matrix except the quantized elements in the jth column. i,j represents the matrix error generated by quantizing the elements in the i-th row and j-th column of the key-value matrix. Represents all elements in the j-th row but not in the j-th column of the inverse of the Hessian matrix corresponding to the i-th row of the key-value matrix.

8. A model cache optimization device based on hierarchical quantization and dynamic priority management, applied to an AI service system composed of a user terminal and a cloud platform, characterized in that: The cloud platform is provided with: Data collection module: used to receive the inference long text sent by the user terminal, and divide the inference long text into several word sequences of length N; Importance score calculation module: used to obtain the importance score of each token in the word sequence generated by the data collection module using the masked language model; Model storage module: used to store the Token importance scoring model and the service reasoning model. The Token importance scoring model is built based on the MLP multi-layer perceptron. The MLP multi-layer perceptron includes a first fully connected layer, a first activation function layer, a second fully connected layer, a second activation function layer, a third fully connected layer and a third activation function layer connected in sequence. The first activation function layer and the second activation function layer both use the ReLU activation function, and the third activation function layer uses the Sigmoid activation function. The service reasoning model is built based on the LLM model and is used to process the reasoning long text. GPU space division module: used to divide the GPU space of the cloud platform into a head buffer area for storing the starting key-value pairs, a middle buffer area for storing important key-value pairs, and a tail buffer area for storing the key-value pairs associated with the current token; Text encoding module: used to convert the inference long text obtained by the data collection module into a Token inference sequence, and send the Token inference sequence to the service inference model of the model storage module; Key-value pair processing module: used to convert the first The key-value pairs corresponding to each Token are stored in the header buffer in FP16 format. The previous Token before the current one is processed is stored in the header buffer according to the FIFO principle. The key-value pairs corresponding to each Token are stored in the tail buffer in FP16 format, and the key-value pairs removed from the tail buffer are stored in the middle buffer in FP8 format. When the middle buffer is full, the importance score of each Token obtained by the Token importance scoring model of the model storage module and the importance score calculation module is used to update the middle buffer. Indicates the length of the header buffer and the tail buffer.

Images