Large model key value cache quantization hyper-parameter selection method and system

Abstract

The invention discloses a large model key value cache quantization hyper-parameter selection method and system, and belongs to the technical field of large models. The method comprises the following steps: acquiring a memory bottleneck of a large model or a delay bottleneck of the large model; under a memory bottleneck or a delay bottleneck, calculating the maximum batch size which can be reached by each key value cache quantization hyper-parameter group; under the maximum batch size, a key value cache quantization hyper-parameter group is used for large model reasoning, and performance and precision points corresponding to the key value cache quantization hyper-parameter group are obtained; generating a Pareto curve corresponding to the large model based on all the performance and precision points; and selecting a key value cache quantization hyper-parameter group on a Pareto curve according to an application scene of the large model. According to the invention, the large model can reach higher throughput more easily.

Description

Technical Field

[0001] The present invention belongs to the technical field of large models, and in particular, relates to a large model key-value cache quantization hyperparameter selection method and system. Background Art

[0002] The emergence of large models has greatly improved the capabilities of artificial intelligence and has achieved remarkable results in speech recognition, image processing, natural language processing and other fields. For example, large language models are a class of advanced deep learning models in the field of natural language processing, designed to process and generate natural language text. Large language models are language models based on neural networks, usually with hundreds of millions to hundreds of billions of parameters. They are trained with large-scale text data and are able to understand and generate natural language text.

[0003] After the training of the large language model based on the Transformer architecture is completed, key-value cache will appear in the process of predicting new data or generating output. The key-value cache is used to store the intermediate results of the multi-head self-attention mechanism in the neural network model to accelerate calculations. Specifically, when processing long sequences or multiple iterations, the key-value cache allows the model to avoid repeated calculations and thus improve reasoning efficiency. By caching previously calculated keys and values, the model can directly access these cached results when generating subsequent outputs without recalculation, thus saving a lot of computing resources.

[0004] Generally, a key-value cache contains two main parts: key and value. Among them, the key cache is the key vector that stores the input sequence. These key vectors are used to perform dot product operations with the query vector to calculate the attention weights; the value cache is the value vector that stores the input sequence. These value vectors are used to generate the final attention output under the weight of the attention weights. The reason for choosing to cache the key-value vector but not the query vector is that in the inference stage, the Q matrix is ​​usually calculated from the model input, so each input is different, while the k and V matrices are calculated from the output of the encoder part and are usually relatively stable.

[0005] Quantization is the process of converting high-precision data (such as floating-point numbers) into low-precision data (such as fixed-point numbers). Quantization can significantly reduce the storage requirements and computational complexity of the model. The current quantization methods mainly include: static quantization, quantization-aware training, weight quantization, activation quantization, dynamic quantization, and post-training quantization. Among them, static quantization does not change the quantization parameters during the quantization process, and simplifies the reasoning process by determining the quantization range and scaling factor in advance. Quantization-aware training refers to simulating the impact of quantization during the neural network training process so that the model can adapt to low-precision operations during the training phase. Weight quantization refers to the quantization of weights in the neural network model. Weights are the weight coefficients of each node connection in the neural network layer. By quantizing them, the storage requirements and computational complexity of the model can be significantly reduced. Activation quantization refers to the quantization of the activation values ​​of nodes in the neural network model. The activation value is the result of the node output. After quantization, the memory usage and computational overhead of the model can be reduced while maintaining the prediction accuracy of the model. Dynamic quantization is to adjust the quantization parameters in real time during the inference process to adapt to the changes in input data, thereby improving the accuracy of the model. Post-training quantization refers to quantizing the model after model training is completed to reduce the storage and computing requirements of the model. Zero-shot quantization refers to quantizing the pre-trained model without any original training data.

[0006] For example, Chinese patent application CN117609444A discloses a search question-answering method based on a large model, which uses 8-bit 4 or 4-bit quantization to quantize the large model in feedback reinforcement training based on user satisfaction, which belongs to quantization-aware training; Chinese patent application CN117743548A discloses a local knowledge base intelligent question-answering method, system, device and readable storage medium based on a large model, in which in the large model quantization processing module, the quantization operation can use weight quantization, weight activation quantization or dynamic quantization; Chinese patent application CN118095371A discloses A large model quantization algorithm based on a low-rank dictionary, which compresses the large model using a post-training weight quantization algorithm; Chinese patent application CN118095396A discloses a bit-sensitive and flexible compression method for adaptive large model structure distribution, which performs bit-bit search on each layer of the large model to obtain a mixed precision quantization scheme suitable for each layer of the large model to be compressed; Chinese patent application CN118227766A discloses a tool-enhanced intelligent question-answering method in the financial field, which uses an activation-aware weight quantization method when using training corpus to fine-tune instructions for the basic open source large model.

[0007] Although these quantization methods can reduce the computing resources and storage space of the model to a certain extent, they have significant defects in dealing with key-value caches: 1) They do not consider the quantization problem of key-value caches; 2) They do not consider quantization schemes under resource-constrained situations; 3) They do not consider the generalization of their own quantization hyperparameters between different models. Summary of the invention

[0008] In view of the above problems, the present invention discloses a large model key-value cache quantization hyperparameter selection method and system, which can more easily enable large models to achieve higher throughput.

[0009] To achieve the above objectives, the technical solution of the present invention includes the following contents.

[0010] A method for selecting hyperparameters of a large model key-value cache quantization, the method comprising:

[0011] Get the memory bottleneck of large models or the latency bottleneck of large models;

[0012] Under the memory bottleneck or latency bottleneck, calculate the maximum batch_size that can be achieved by each key-value cache quantization hyperparameter group;

[0013] Use a key-value cache quantization hyperparameter group to perform large model inference under the maximum batch_size, and obtain the performance and accuracy points corresponding to the key-value cache quantization hyperparameter group;

[0014] Generate the Pareto curve corresponding to the large model based on all performance and accuracy points;

[0015] A key-value cache quantization hyperparameter group is selected on the Pareto curve according to the application scenario of the large model.

[0016] Furthermore, under the memory bottleneck or latency bottleneck, calculating the maximum batch_size that can be achieved by each key-value cache quantization hyperparameter group includes:

[0017] Get the initial batch_size;

[0018] If the memory or latency of the large model does not exceed the memory bottleneck or the latency bottleneck when batch_size is S1×n, and the memory or latency of the large model exceeds the memory bottleneck or the latency bottleneck when batch_size is S1×(n+1), then the batch_size is gradually reduced to S2 using the binary search method; where S1 is the initial batch_size, n is a natural number, and S2 is the set minimum batch_size;

[0019] When batch_size is S2×m, the memory or delay of the large model does not exceed the memory bottleneck or the delay bottleneck, and when batch_size is S2×(m+1), the memory or delay of the large model exceeds the memory bottleneck or the delay bottleneck, then the maximum batch_size is set to S2×m; wherein m is a positive integer.

[0020] Furthermore, the algorithm for generating the Pareto curve corresponding to the large model includes: a global traversal algorithm, an evolutionary algorithm or a genetic algorithm.

[0021] Furthermore, when the algorithm for generating the Pareto curve corresponding to the large model is a global traversal algorithm, the generating of the Pareto curve corresponding to the large model based on all performance and accuracy points includes:

[0022] Traverse all performance and precision points, and select all undominated points with precision as the horizontal axis and performance as the vertical axis; among which, the performance and precision points corresponding to the i-th key-value cache quantization hyperparameter group The value of each key-value cache quantization hyperparameter in the i-th key-value cache quantization hyperparameter group, represents the performance score of the large model attention layer during inference based on the i-th key-value cache quantization hyperparameter group, Represents the accuracy score of the large model based on the i-th key-value cache quantization hyperparameter group.

[0023] Further, the selecting all undominated points includes:

[0024] Based on performance score Sort the points P; if the performance is positive, sort them in descending order; if the performance is negative, sort them in ascending order;

[0025] The first point P after sorting is used as the initial point of the Pareto point set;

[0026] Traverse the remaining points P and sort them according to the accuracy score Decide whether to add the point P to the Pareto point set; wherein the accuracy score Deciding whether to add the point P to the Pareto point set includes:

[0027] When the accuracy is positive, if the accuracy score of point P is Not less than the accuracy score of the last point P added to the Pareto point set Then add the point P to the Pareto point set;

[0028] When the accuracy is negative, if the accuracy score of point P Not greater than the accuracy score of the last point P added to the Pareto point set Then add the point P to the Pareto point set;

[0029] The point P in the Pareto point set generated after traversal is regarded as an undominated point.

[0030] Furthermore, the step of selecting a key-value cache quantization hyperparameter group on the Pareto curve according to the application scenario of the large model includes:

[0031] When the application scenario of the large model is an intelligent customer service system or a voice assistant, a point close to high precision in the Pareto curve is selected to obtain a corresponding key-value cache quantization hyperparameter group;

[0032] When the application scenario of the large model is a real-time interactive system, NPC dialogue generation in a game, or summary generation in a search engine, a point close to high performance in the Pareto curve is selected to obtain a corresponding key-value cache quantization hyperparameter group;

[0033] When the application scenario of the large model is a customer service system, news summary generation, or an e-commerce recommendation system, a point in the middle area of ​​the Pareto curve is selected to obtain a corresponding key-value cache quantization hyperparameter group.

[0034] A large model key-value cache quantization hyperparameter selection system, the system comprising:

[0035] The bottleneck acquisition module is used to obtain the memory bottleneck or latency bottleneck of a large model.

[0036] The scale setting module is used to calculate the maximum batch_size that can be achieved by each key-value cache quantization hyperparameter group under memory bottleneck or latency bottleneck;

[0037] The model inference module is used to perform large model inference using a key-value cache quantization hyperparameter group under the maximum batch_size, and obtain the performance and accuracy points corresponding to the key-value cache quantization hyperparameter group;

[0038] A curve generation module is used to generate a Pareto curve corresponding to the large model based on all performance and accuracy points;

[0039] The parameter selection module is used to select a key-value cache quantization hyperparameter group on the Pareto curve according to the application scenario of the large model.

[0040] An electronic device comprises: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, the large model key-value cache quantization hyperparameter selection method as described in any one of the above items is implemented.

[0041] A computer-readable storage medium, characterized in that computer program instructions are stored on the computer-readable storage medium, and when the computer program instructions are executed by a processor, any of the large model key-value cache quantization hyperparameter selection methods described above is implemented.

[0042] Compared with the prior art, the present invention has at least the following technical effects.

[0043] 1) The present invention provides a theoretical analysis of the key-value cache quantization hyperparameter selection method. Compared with the original experience-based model weight and activation quantization, the present invention provides an analysis of the key-value cache quantization and uses a hyperparameter selection method. With the support of theoretical analysis, the selection can enable large models to achieve higher throughput.

[0044] 2) The present invention provides a Pareto curve of performance and accuracy of different key-value cache hyperparameter groups under memory-constrained conditions. Compared with the fixed environment quantization in the prior art, it is more conducive to the development of model quantization in general situations with memory constraints.

[0045] 3) The present invention has generalization of key-value cache quantization hyperparameters for different models. When there is an optimal key-value cache hyperparameter selection result for a model, the generalization can be used to directly bring it into a structurally similar model, so that the selection of key-value cache quantization hyperparameters converges faster. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 Flowchart of the hyperparameter selection method for large model key-value cache quantization.

[0047] Figure 2Flowchart for generalization of quantized hyperparameters for different model key-value caches. DETAILED DESCRIPTION

[0048] The present invention is further described in detail below in conjunction with specific embodiments. The given examples are only for illustrating the present invention, but not for limiting the scope of the present invention.

[0049] The large model key-value cache quantization hyperparameter selection method of the present invention is as follows: Figure 1 The method comprises the following steps.

[0050] Step 1: Get the memory bottleneck of large models or the latency bottleneck of large models.

[0051] The present invention can artificially set the memory bottleneck or the delay bottleneck of the large model. For the memory bottleneck, the memory threshold can be simply set to the upper limit of the memory of the GPU used, such as 40G, 80G, etc.; for the delay bottleneck, the delay threshold can be set to the common 50ms / token, etc. Afterwards, the memory bottleneck or delay bottleneck is input into the system so that the system can perform calculations in subsequent steps based on the memory bottleneck or delay bottleneck.

[0052] Step 2: Under the memory bottleneck or latency bottleneck, calculate the maximum batch_size that can be achieved by each key-value cache quantization hyperparameter group.

[0053] The maximum batch_size refers to the batch_size under which the large model will not exceed the corresponding bottleneck when running under each key-value cache quantization hyperparameter group, that is,

[0054] 1. Under memory constraints, for all optional key-value cache hyperparameter groups, the memory used by the large model runtime does not exceed the threshold.

[0055] 2. Under latency constraints, for all optional key-value cache hyperparameter groups, the average latency of generating each token when the large model is running does not exceed the threshold.

[0056] In one embodiment, a batch_size with a larger span is first set for model inference. For example, if the span step is 64, 64, 128, 196, etc. are set. If it is found that the threshold is not exceeded at 64n, but it is exceeded at 64(n+1), the step size is reduced by binary search until the step size is 4. The 4m obtained at this time is the maximum batch_size, where m means that the threshold is not exceeded at 4m, but it is exceeded at 4(m+1).

[0057] Step 3: Use a key-value cache quantization hyperparameter group to perform large model inference under the maximum batch_size, and obtain the performance and accuracy points corresponding to the key-value cache quantization hyperparameter group.

[0058] Model inference: refers to model inference with different key-value cache quantization hyperparameter groups under the corresponding key-value cache quantization framework. The framework can be KIVI, etc.

[0059] In one embodiment, the point where the corresponding performance and accuracy are obtained refers to the accuracy of the model inference result after setting the key-value cache quantization hyperparameter group under the maximum batch_size and the performance of the model Attention layer during the inference process. A point is a tuple with the shape of (p1, p2, ..., p n ,T P ,A S ), where (p1,p2,…,p n ) refers to the value of the hyperparameter, T P and A S These are the performance and accuracy obtained after selecting these hyperparameters. When memory is limited, performance refers to throughput (the average number of tokens output per second), and when latency is limited, performance refers to latency (the average time required to generate each token, in milliseconds). Accuracy refers to the accuracy score of the model output.

[0060] Step 4: Based on all quantization cache quantization hyperparameters, the corresponding performance and accuracy points are obtained, and the Pareto curve corresponding to the large model is generated.

[0061] The Pareto curve can be generated based on algorithms such as global traversal algorithm, evolutionary algorithm, genetic algorithm, etc. The following takes the global traversal algorithm as an example to explain in detail the generation process of the Pareto curve.

[0062] Traverse all quantization cache quantization hyperparameters to obtain the corresponding performance and accuracy points, with the horizontal axis as accuracy and the vertical axis as performance, select all the undominated points, and connect these points from left to right to obtain the Pareto curve.

[0063] How to select all the undominated points:

[0064] Suppose the point set of all points is The undominated point here means that if A is an undominated point, then there is no point B such that the performance and accuracy of B are better than those of A. "Better" here means that the performance and accuracy are greater in the positive direction. The specific steps are as follows:

[0065] 1. Sorting: According to Value Pair Sort in descending order if the positive direction is positive, otherwise in ascending order.

[0066] 2. Initialize the Pareto point set: take the first point after sorting as the initial point of the Pareto point set.

[0067] 3. Traverse and filter: traverse the remaining points, according to the accuracy of the point Decide whether to add it to the Pareto point set:

[0068] a) When the positive direction of the accuracy is positive, if the accuracy of the current point is If the precision is not less than the last point added to the Pareto point set, it will be added.

[0069] b) When the positive direction of the accuracy is negative, if the accuracy of the current point If the precision is not greater than the last point added to the Pareto point set, it will be added.

[0070] 4. Get the Pareto point set, which is the set of all undominated points.

[0071] Step 5: Select the key-value cache quantization hyperparameters in the Pareto curve based on the application scenario of the large model.

[0072] The application scenarios of this large model can be combined, for example:

[0073] 1. Scenarios that require medium to high precision.

[0074] These scenarios have certain requirements for the accuracy of the model, but do not pursue extreme accuracy and can accept a small amount of quantization error. For such scenarios, quantization still helps improve performance, especially in the face of large-scale applications.

[0075] ●Intelligent customer service system: Intelligent customer service needs to be able to handle users’ diverse questions and provide accurate and useful answers.

[0076] However, a slight reduction in precision will not affect the overall user experience.

[0077] ● Voice assistants: Although voice assistants require high comprehension and response accuracy, they do not require extremely precise answers. For problems such as weather forecasts and daily task scheduling, appropriately quantized models can still provide good enough performance.

[0078] In these scenarios, although a certain degree of accuracy is required, a full-precision model is not pursued, so points close to high accuracy in the Pareto curve can be selected, and the key-value cache quantization hyperparameters corresponding to these points can be used to run the model.

[0079] 2. Scenarios that require high performance (fast response):

[0080] These scenarios usually require the model's response time to be as short as possible and have relatively low accuracy requirements, so it is suitable to select performance-first hyperparameters.

[0081] ● Real-time interactive systems (such as chatbots): The user experience in real-time dialogue systems depends heavily on the response speed of the model, such as intelligent customer service or personal assistants. Fast replies are more important than extremely high language details, so you can choose the point on the Pareto curve that prioritizes performance.

[0082] ●NPC dialogue generation in games: Non-player character (NPC) dialogue in games usually requires fast generation of dialogue content, especially in multiplayer online games, where latency can greatly affect the player experience. In this scenario, accuracy is not the main concern, and fast response and smooth interaction are more important.

[0083] ●Summary generation in search engines: Search engines need to generate brief result summaries or direct answers for users in a very short time. This scenario has high requirements for speed, especially when accessed by a large number of users, and reasoning speed takes precedence over perfect language expression.

[0084] In these scenarios, it is appropriate to choose the high-performance side of the Pareto curve, sacrificing accuracy in exchange for faster inference speed.

[0085] 3. Balancing performance and accuracy

[0086] Some scenarios may require finding a balance between accuracy and speed to ensure output quality while maintaining a high response speed.

[0087] ●Customer service system: Some complex customer service systems require both timely response and accurate and professional answers, such as handling banking and insurance services. In this scenario, both accuracy and performance need to be balanced, and neither aspect can be completely sacrificed.

[0088] ● News summary generation: Automatically generating news summaries requires a certain speed to cope with breaking news scenarios, while also ensuring the accuracy and completeness of information to avoid misleading. Therefore, this type of scenario will take a middle value between speed and accuracy.

[0089] ●E-commerce recommendation system: The recommendation system in the e-commerce platform needs to generate personalized recommendations, which must ensure the relevance and accuracy of the recommended content and give recommendations in a timely manner while the user is browsing. Therefore, it must respond quickly and have a certain degree of accuracy.

[0090] In these scenarios, points in the middle area of ​​the Pareto curve can be chosen to maintain a balance between speed and accuracy.

[0091] In order to verify that the Pareto curve proposed in the present invention meets the actual needs and the generalization of the Pareto curve on different models, the present invention conducts

[0092] 1. Theoretical analysis of hyperparameter selection for key-value cache quantization.

[0093] (I) Quantitative performance analysis of key-value cache quantization parameter selection.

[0094] Taking the key-value cache quantization method of the present invention implemented under the KIVI framework as an example, in each step, assuming that the total number of current tokens is l, the window size of V (value vector) is R, and the group size of group quantization is g, then the window size of K (key vector) is r = l mod R. The data K in the window f ,V f Keep full precision (16bit), data outside the window K q , V q Perform group quantization, and set the quantized bit to bit (K-bit is equal to R-bit in this case). Note that when l≤R, in fact, not all K and V are in the window, so no data is quantized and no discussion is needed. Let's assume that l is always greater than R. At the same time, since KIVI currently only supports the multi-head attention mechanism, let d model =h×d, where d model is the model dimension, h is the number of attention heads, and d is the dimension of each head.

[0095] The main time spent in KIVI's Attention process is divided into two parts. One part is the time spent on quantizing the parts that need to be grouped and quantized, and then packing them into a 32-bit space. For example, if quantized to 2 bits, 16 data can be placed in an int32. The second step is the time spent on matrix multiplication calculation, including the full-precision matrix multiplication calculation Q●K f and QK●V f , and the calculation of the full-precision matrix multiplication of the quantized matrix Q●K q and QK●V q , where Q is the query vector. The following is an analysis of their computational complexity. Note that the following operations of data concatenation and data shaping are both O(1) and will be skipped.

[0096] 1. Quantization and packaging calculations.

[0097] 1) Minimum and maximum value calculation.

[0098] For each group, all elements need to be traversed to find the minimum and maximum values. The total number of calculations is:

[0099]

[0100] Among them, b is the quantization bit width. Then its time complexity is:

[0101]

[0102] 2) Calculate the quantization scaling factor and perform quantization.

[0103] Since Key is quantized per-channel and Value is quantized per-token, when grouping and quantizing, Key groups the token dimension and Value groups the channel dimension. Therefore, g needs to be divisible by R, otherwise Key cannot be grouped.

[0104] A. Calculate the quantization scaling factor.

[0105] The time complexity of calculating the quantization scaling factor for each group is:

[0106]

[0107] B. Quantification.

[0108] For each element, quantization is performed, that is, subtraction, division, and truncation, and the time complexity is:

[0109]

[0110] 3) Pack the data.

[0111] 32 / bit elements need to be packed into an integer, and the time complexity is:

[0112]

[0113] 4) The total time complexity of quantization and packaging calculations.

[0114] Based on the above analysis, the total time complexity of quantization and packaging calculation is:

[0115]

[0116] 2. Matrix multiplication calculation.

[0117] 1) Full precision matrix multiplication by quantized matrix.

[0118] In each thread block, actually loading data, dequantization, and multiplication, reduction and summation are all O(1). The time complexity mainly depends on the number of thread blocks and the number of threads in each block. Therefore, for each kernel, the total time complexity is:

[0119]

[0120] 2) Full-precision matrix multiplication by full-precision matrix.

[0121] The computational complexity of full-precision matrix multiplication is:

[0122]

[0123] Note that the full-precision QK here is QK q With QK f spliced ​​together. And the softmax and division are omitted This is an O(1) operation. Similarly, the output of the final attention layer is also composed of (QK)V q and (QK)V f Specifically,

[0124] QK=concat[QK q , QK f ] (8)

[0125]

[0126] 3) Analysis of the total complexity of matrix multiplication.

[0127] In each step, the total complexity of matrix multiplication is

[0128] TokenComputation=O(b●h●d●(l●R+l●r+2 lRr)) (10)

[0129] 3. The total computational complexity of Attention calculation.

[0130] Combining equation (5) and equation (10), the computational complexity of each step is:

[0131]

[0132] Therefore, the total computational complexity is:

[0133]

[0134] Among them, l0 is the length of the input prompt, max gen It is the maximum number of tokens generated, r = l mod R.

[0135] Simple calculation, take the expectation of r as Considering all Attention layers (let’s say N layers), the total computational complexity is

[0136]

[0137] Since l0 is very large (usually 4096), we only pay attention to the first half. Formula (13) assumes that the quantization bit width of each layer is consistent.

[0138] 4. Total computational complexity during model inference.

[0139] Furthermore, since the KV Cache parameters only affect the KV Cache, the computational complexity outside the Attention layer is independent of the KV Cache quantization parameters. Therefore, the entire model can be regarded as two parts, one part is related to the KV Cache quantization parameters, and the other part is independent of the KV Cache quantization parameters.

[0140] Assume that the computing power is P, and the computational complexity of the part that is not related to the quantization parameters of KV Cache is C - , the amount of data to be processed is D, then

[0141] The time T required to process data can be expressed as:

[0142]

[0143] ThroughputT P It can be expressed as:

[0144]

[0145] From formula (13), we can see that C Attention As R increases, g decreases, and bit increases. Combining formula (15) with Table 1, it can be concluded that the throughput (performance) decreases with R, increases with g, and decreases with bit.

[0146] (ii) Qualitative analysis of the accuracy of key-value cache quantization parameter selection.

[0147] Since the data inside the window size R is full precision and the data outside is quantized, the precision of Value increases with R. For Key, the value of its window size r changes with the number of tokens during the inference process, and its mean is R / 2, so the precision of Key also increases with R.

[0148] It can be seen from formula (9) that the accuracy of the model reasoning results increases with R.

[0149] It is also easy to see that the accuracy of the model reasoning results increases with the bit.

[0150] When reasoning with a large model, the impact of KV Cache quantization on the reasoning results will vary depending on the change in group size $g$. Specifically:

[0151] ● g increases: If the group size increases, the number of elements in each group increases, which means that more elements are considered during quantization, which may lead to a coarser quantized representation. In general, a larger group size may reduce the fineness of quantization, resulting in a decrease in the inference accuracy of the model. Therefore, the score of the inference result may decrease.

[0152] ●g decreases: If the group size decreases, the number of elements in each group decreases, which means that the elements in each group are more concentrated during quantization and the fineness of quantization is improved. This usually maintains higher inference accuracy and the score of the inference result may increase incrementally.

[0153] In summary, the accuracy of the model inference results decreases with g, and the lower the number of quantization bits, the worse the accuracy. Therefore, the accuracy increases with R, decreases with g, and increases with bits. This is exactly the opposite of performance.

[0154] 2. Pareto curves of performance and accuracy of different key-value cache hyperparameter groups under memory bottleneck and latency bottleneck.

[0155] 1. Performance and accuracy experiments of key-value cache quantization parameters under memory bottleneck.

[0156] Experimental scenario:

[0157] ●dataset: THUDM / LongBench.

[0158] ●Task: trivialqa, trec, qasper, qmsum, gov_report, corresponding maxgen is 32, 64, 128, 512, 512, corresponding accuracy scores are qa_f1_score, classification_score, qa_f1_score, rouge_score, rouge_score.

[0159] ●Environment: Ampere, A100_80G, NVBridge, Rocky8.

[0160] ●Model: Llama-2-7B, Llama-2-13B.

[0161] Key-value cache quantization hyperparameters:

[0162] ■bit: 2, 4;

[0163] ■g: 32, 64, 128;

[0164] ■R: 32, 64, 96, 128 (Note: g needs to be divisible by R).

[0165] Using the above-mentioned large-model key-value cache quantization hyperparameter selection method, this embodiment constructs a Pareto curve. Further analysis of the Pareto curve shows that the Pareto curve conforms to the theoretical analysis results of the above-mentioned key-value cache quantization hyperparameter selection, that is, when bits and g are fixed, performance decreases with R, and precision increases with R; when bits and R are fixed, performance increases with g, and precision decreases with g; when R and g are fixed, performance decreases with bits, and precision increases with bits.

[0166] 2. Performance and accuracy experiments of key-value cache quantization parameters under latency bottleneck.

[0167] Experimental scenario:

[0168] ●dataset: THUDM / LongBench.

[0169] ●Task: trivialqa, trec, qasper, qmsum, gov_report, corresponding to max gen The values ​​are 32, 64, 128, 512, 512, and the corresponding accuracy scores are qa_f1_score, classification_score, qa_f1_score, rouge_score, rouge_score.

[0170] ●Environment: Ampere, A100_80G, NVBridge, Rocky8.

[0171] ●Model: Llama-2-7B, Llama-2-13B.

[0172] Key-value cache quantization hyperparameters:

[0173] ■bit: 2, 4;

[0174] ■g: 32, 64, 128;

[0175] ■R: 32, 64, 96, 128 (Note: g needs to be divisible by R).

[0176] Using the above-mentioned large-model key-value cache quantization hyperparameter selection method, this embodiment constructs another Pareto curve. Further analysis of the Pareto curve shows that the Pareto curve also conforms to the theoretical analysis results of the above-mentioned key-value cache quantization hyperparameter selection, that is, when bits and g are fixed, performance decreases with R, and precision increases with R; when bits and R are fixed, performance increases with g, and precision decreases with g; when R and g are fixed, performance decreases with bits, and precision increases with bits.

[0177] 3. Generalization of quantified hyperparameters for different model key-value caches.

[0178] 1. Theoretical analysis.

[0179] Consider the structural similarity model shown in Table 1

[0180] model Attention head type hd N Llama-2-7b Multi-head attention 321 28 32 Llama-2-1 3b Multi-head attention 40128 40 OPT-6.7b Multi-head attention 32128 32

[0181] Table 1: Attention head types and configurations of different models

[0182] Suppose we obtain the Pareto Curve of Llama-2-7b under Momoey Bound (or Latency Bound), and want to generalize it to the OPT-125m model.

[0183] Since the Attention layer structure of this OPT model is similar to that of Llama-2-7b (both are Multi-HeadAttention and both use the L1amaSdpaAttention class), it can be proved that its generalization is feasible.

[0184]

[0185] Where T P , T′ P They are the throughput of llama-2-7b and opt-125m, respectively, and the computing power P is independent of the amount of data D to be processed and the model. For a certain and identical data set, C - and C′ - It has nothing to do with the KV Cache quantization parameter and is a constant. Substituting it into formula (13), we get

[0186]

[0187] Where N and N′ are the number of attention layers of llama-2-7b and OPT-125m respectively. Equation (17) proves the possibility of generalization of R and g. Due to the different number of layers, the number of quantization bits of each layer does not support generalization, but if each layer takes the same bit, it can be generalized.

[0188] In addition, if it is assumed that all Attention layers have the same quantization bit width b, mixed precision quantization is actually supported. Formula (17) can be changed to Formula (18):

[0189]

[0190] where b i and b i ′ are the quantization bit widths of each layer of the first model and the second model respectively.

[0191] This is still independent of R and g, proving that R and g meet the generalization property.

[0192] 2. Experimental proof.

[0193] Assume that we have the Pareto curve A obtained by llama-2-7b in 3.2.2, and want to verify its generalization on the opt-125m model, such as Figure 2 As shown, the verification steps include:

[0194] Step 1: Record the key-value cache hyperparameter group corresponding to each point on A, as a series of points {A i}, A i =(bit i , R i , g i ).

[0195] Step 2: List the points {A i}Substitute it into the opt-125m model and infer the performance and accuracy results P A .

[0196] Step 3: Randomly generate a series of points {B j}, B j =(bit j ′,R j ′,g j ′).

[0197] Step 4: Arrange the points {B j}Substitute it into the opt-125m model and infer the performance and accuracy results P B .

[0198] Step 5: Experimental results show that P A Effect ratio B Well, it is proved that the Pareto curve proposed in the present invention is generalizable.

[0199] The above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable people familiar with the technology in this field to understand the content of the present invention and implement it, and they cannot be used to limit the protection scope of the present invention. Any equivalent changes or modifications made according to the spirit of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for selecting hyperparameters for large model key-value cache quantization, characterized in that: The method comprises: Get the memory bottleneck of large models or the latency bottleneck of large models; Under the memory bottleneck or latency bottleneck, calculate the maximum batch_size that can be achieved by each key-value cache quantization hyperparameter group; Use a key-value cache quantization hyperparameter group to perform large model inference under the maximum batch_size, and obtain the performance and accuracy points corresponding to the key-value cache quantization hyperparameter group; Generate the Pareto curve corresponding to the large model based on all performance and accuracy points; A key-value cache quantization hyperparameter group is selected on the Pareto curve according to the application scenario of the large model.

2. The method according to claim 1, characterized in that The maximum batch_size that can be achieved by each key-value cache quantization hyperparameter group under the memory bottleneck or latency bottleneck is calculated, including: Get the initial batch_size; If the memory or latency of the large model does not exceed the memory bottleneck or the latency bottleneck when batch_size is S1×n, and the memory or latency of the large model exceeds the memory bottleneck or the latency bottleneck when batch_size is S1×(n+1), then the batch_size is gradually reduced to S2 using the binary search method; where S1 is the initial batch_size, n is a natural number, and S2 is the set minimum batch_size; When batch_size is S2×m, the memory or delay of the large model does not exceed the memory bottleneck or the delay bottleneck, and when batch_size is S2×(m+1), the memory or delay of the large model exceeds the memory bottleneck or the delay bottleneck, then the maximum batch_size is set to S2×m; wherein m is a positive integer.

3. The method according to claim 1, characterized in that The algorithm for generating the Pareto curve corresponding to the large model includes: a global traversal algorithm, an evolutionary algorithm or a genetic algorithm.

4. The method according to claim 3, characterized in that In the case where the algorithm for generating the Pareto curve corresponding to the large model is a global traversal algorithm, the generating of the Pareto curve corresponding to the large model based on all performance and accuracy points includes: Traverse all performance and precision points, and select all undominated points with precision as the horizontal axis and performance as the vertical axis; among which, the performance and precision points corresponding to the i-th key-value cache quantization hyperparameter group The value of each key-value cache quantization hyperparameter in the i-th key-value cache quantization hyperparameter group, represents the performance score of the large model attention layer during inference based on the i-th key-value cache quantization hyperparameter group, Represents the accuracy score of the large model based on the i-th key-value cache quantization hyperparameter group.

5. The method according to claim 4, characterized in that The method of selecting all undominated points includes: Based on performance score Sort the points P; if the performance is positive, sort them in descending order; if the performance is negative, sort them in ascending order; The first point P after sorting is used as the initial point of the Pareto point set; Traverse the remaining points P and sort them according to the accuracy score Decide whether to add the point P to the Pareto point set; wherein the accuracy score Deciding whether to add the point P to the Pareto point set includes: When the accuracy is positive, if the accuracy score of point P is Not less than the accuracy score of the last point P added to the Pareto point set Then add the point P to the Pareto point set; When the accuracy is negative, if the accuracy score of point P Not greater than the accuracy score of the last point P added to the Pareto point set Then add the point P to the Pareto point set; The point P in the Pareto point set generated after traversal is regarded as an undominated point.

6. The method according to claim 1, characterized in that The step of selecting a key-value cache quantization hyperparameter group on the Pareto curve according to the application scenario of the large model includes: When the application scenario of the large model is an intelligent customer service system or a voice assistant, a point close to high precision in the Pareto curve is selected to obtain a corresponding key-value cache quantization hyperparameter group; When the application scenario of the large model is a real-time interactive system, NPC dialogue generation in a game, or summary generation in a search engine, a point close to high performance in the Pareto curve is selected to obtain a corresponding key-value cache quantization hyperparameter group; When the application scenario of the large model is a customer service system, news summary generation, or an e-commerce recommendation system, a point in the middle area of ​​the Pareto curve is selected to obtain a corresponding key-value cache quantization hyperparameter group.

7. A large model key-value cache quantization hyperparameter selection system, characterized in that: The system comprises: The bottleneck acquisition module is used to obtain the memory bottleneck or latency bottleneck of a large model. The scale setting module is used to calculate the maximum batch_size that can be achieved by each key-value cache quantization hyperparameter group under memory bottleneck or latency bottleneck; The model inference module is used to perform large model inference using a key-value cache quantization hyperparameter group under the maximum batch_size, and obtain the performance and accuracy points corresponding to the key-value cache quantization hyperparameter group; A curve generation module is used to generate a Pareto curve corresponding to the large model based on all performance and accuracy points; The parameter selection module is used to select a key-value cache quantization hyperparameter group on the Pareto curve according to the application scenario of the large model.

8. An electronic device, characterized in that: The electronic device comprises: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, the large model key-value cache quantization hyperparameter selection method as described in any one of claims 1-6 is implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by the processor, the large model key-value cache quantization hyperparameter selection method according to any one of claims 1 to 6 is implemented.

Images