Quantization method and system of large language model and electronic device

Abstract

The application provides a quantization method of a large language model. For each linear layer: a weight quantization sensitivity of the linear layer is calculated; a high-precision bit width ratio of each linear layer is determined according to the weight quantization sensitivity of each linear layer and a target mixed precision ratio; for multiple channels of each linear layer, a channel quantization sensitivity of each channel in the linear layer is obtained based on input activation of the linear layer; the multiple channels in the linear layer are sorted according to the channel quantization sensitivity; the sorted multiple channels are clustered in combination with the high-precision bit width ratio of the linear layer; different-precision quantization bit widths are assigned to weight parameters corresponding to channels of different clusters; for multiple channels of the same cluster, the multiple channels are rearranged according to a distribution similarity index; the rearranged channels are grouped; weight parameters of each group of channels are quantized synchronously according to the assigned quantization bit width; and a quantized large language model is obtained.

Description

Technical Field

[0001] This application relates to the field of large language model technology, and in particular to a method, system and electronic device for quantizing large language models. Background Technology

[0002] Large Language Models (LLMs) have made significant progress in many fields, including natural language understanding, dialogue systems, code generation, and information retrieval. However, the great success of LLMs relies on a massive number of parameters and enormous computational demands, posing a significant challenge to deployment in resource-constrained environments. To improve the inference speed of LLMs, a model quantization scheme is adopted, quantizing the weight parameters of the LLM into low-precision values. This reduces the memory footprint of the LLM and accelerates model inference through faster low-precision computation.

[0003] If the model is quantized by sharing the same quantization parameter (such as a scaling factor) for all weights, the model's performance will degrade due to outliers (such as large weight values). Summary of the Invention

[0004] This application provides a quantization method for large language models, which can solve the problem of performance degradation caused by quantization errors due to outlier weight data when the model weights are quantized using an overall quantization scheme.

[0005] Firstly, this application provides a quantization method for a large language model, which is trained using sample texts. This method includes:

[0006] For each linear layer of the large language model:

[0007] Calculate the weight quantization sensitivity of the linear layer, and determine the high-precision bit width ratio of each linear layer based on the weight quantization sensitivity of each linear layer and the target mixed precision ratio;

[0008] For each linear layer with multiple channels, based on the input activation of that linear layer, the channel quantization sensitivity of each channel in that linear layer is obtained;

[0009] The multiple channels in the linear layer are sorted according to their channel quantization sensitivity. Combined with the high-precision bit width ratio of the linear layer, the sorted channels are clustered, and different precision quantization bit widths are assigned to the weight parameters corresponding to the channels in different clusters.

[0010] For multiple channels in the same cluster, they are rearranged according to the distribution similarity index. The rearranged channels are then grouped, and the weight parameters of each group of channels are quantized synchronously according to the allocated quantization bit width to obtain the quantized large language model.

[0011] Optionally, multiple channels in the linear layer are sorted according to their channel quantization sensitivity, and then clustered based on the high-precision bit-width ratio of the linear layer, including:

[0012] According to the corresponding channel quantization sensitivity, multiple channels are sorted in descending order. Combined with the high-precision bit width ratio of the linear layer, multiple channels are divided into high-sensitivity clusters with high channel quantization sensitivity and low-sensitivity clusters with low channel quantization sensitivity.

[0013] For multiple channels within the same cluster, they are rearranged according to a distribution similarity index. The rearranged channels are then grouped, and the weight parameters of each group are quantized synchronously according to the allocated quantization bit width, including:

[0014] For multiple channels in a high-sensitivity cluster, the product of the maximum absolute value of the weight parameter in each channel and the maximum value of the input activation is obtained as a distribution similarity index. The multiple channels in the high-sensitivity cluster are rearranged according to the distribution similarity index, and each group of 128 adjacent channels is divided into a group. The weight parameters of each group of channels are quantized synchronously according to the high-precision bit width.

[0015] For multiple channels in a low-sensitivity cluster, the product of the maximum absolute value of the weight parameter in each channel and the maximum value of the input activation is obtained as a distribution similarity index. The multiple channels in the low-sensitivity cluster are rearranged according to the distribution similarity index, and each group of 128 adjacent channels is divided into groups. The weight parameters of each group of channels are quantized synchronously according to the low-precision bit width.

[0016] Optionally, for each linear layer of the large language model: calculate the weight quantization sensitivity of that linear layer, including:

[0017] A small amount of calibration data is input into the large language model, and forward and backward propagation are performed to generate gradient vectors. The small amount of calibration data is obtained by sampling the sample text.

[0018] Based on the gradient vector and the low-precision quantization error, the weight quantization sensitivity of the linear layer is generated.

[0019] Optionally, based on the weighted quantization sensitivity and target mixed precision ratio of each linear layer, the high-precision bit width ratio of each linear layer is determined, including:

[0020] The basic mixed sperm ratio of a large language model was determined through ablation experiments;

[0021] Based on the memory usage of the original floating-point weights of the large language model, set the target mixing ratio of the large language model;

[0022] For the i-th linear layer, the high-precision bit width ratio of the i-th linear layer is calculated based on the weight quantization sensitivity of the i-th linear layer, the weight parameter quantity of the i-th linear layer, the basic mixing ratio, and the target mixing ratio.

[0023] Optionally, for each linear layer of the large language model: calculate the weight quantization sensitivity of that linear layer, including:

[0024] The weight quantization sensitivity of each linear layer is calculated using the following formula:

[0025]

[0026] Among them, b i δw represents the quantization sensitivity of the weights of the i-th linear layer, δw represents the quantization error generated by quantizing the i-th linear layer with low precision bit width, and g represents the gradient vector generated by forward and backward propagation based on a small amount of calibration data.

[0027] Optionally, for each linear layer of the large language model: calculate the weight quantization sensitivity of that linear layer, including:

[0028] The high-precision bit-width ratio of each linear layer is determined based on the following formula:

[0029]

[0030] Among them, b i p represents the weight quantization sensitivity of the i-th linear layer. i The weight parameters s represent the weights of the i-th linear layer. T Characterizing the target mixed sperm ratio, s B Characterizes the basic mixed sperm ratio.

[0031] Optionally, based on the input activation of the linear layer, the channel quantization sensitivity of each channel in the linear layer is obtained, including:

[0032] The Levenberg-Marquadt algorithm is used to process the input activation of the linear layer. The Hessian matrix is ​​approximated by the Levenberg-Marquadt algorithm, and the inverse of the Hessian matrix is ​​obtained. For each channel, the original floating-point weights of the channel are obtained, and the original floating-point weights of the channel are quantized using low-precision bit width to obtain the quantized weights. The difference between the original floating-point weights and the quantized weights is used as the low-precision quantization error of the channel. The channel quantization sensitivity of the channel is determined based on the low-precision quantization error and the inverse of the Hessian matrix.

[0033] Optionally, based on the input activation of the linear layer, the channel quantization sensitivity of each channel in the linear layer is obtained, including:

[0034] The quantization sensitivity of any channel in the linear layer is obtained using the following formula:

[0035] [(wQ(w)) 2 / 2(XX T +λmean(diag(XX T )) -1 ];

[0036] Where X represents the input activation of the linear layer, X T Let w represent the transpose of the input activation vector of the linear layer, Q(w) represent the original floating-point weights of the channel, Q(w) represent the weights of the channel after low-precision bit-width quantization of the original floating-point weights, and λ represent the damping factor. mean(diag(XX) T )) indicates taking XX T The mean of the diagonal elements of a matrix.

[0037] Optionally, the weight parameters of the linear layers of the large language model include at least one of the following:

[0038] Query weight, key-value weight, value-value weight, dimensionality-upgrading weight, gating weight, and dimensionality-reducing weight;

[0039] The activation values ​​of the large language model are quantized using high-precision bit-width quantization.

[0040] Secondly, embodiments of this application provide an electronic device, including:

[0041] processor;

[0042] The memory stores computer-readable instructions, which, when executed by the processor, implement the quantization method of the large language model described above.

[0043] Thirdly, embodiments of this application provide a large language model quantization system, including a server, a client, and a display:

[0044] A server is used to quantize large language models using the methods described above.

[0045] The client is used to input the problem into the vectorized large language model to obtain the inference results;

[0046] A display used to show the inference results.

[0047] This application achieves mixed-precision quantization between linear layers of a large language model by determining the weight quantization sensitivity of each linear layer and then determining the high-precision bit width ratio of each linear layer. Channels are sorted according to their channel quantization sensitivity, and then clustered based on the high-precision bit width ratio. Different precision quantization bit widths are assigned to the weight parameters corresponding to channels in different clusters, achieving mixed-precision quantization within each linear layer of the large language model. For multiple channels in the same cluster, the channels are rearranged according to a distribution similarity index, and the rearranged channels are grouped and quantized according to the assigned quantization bit width. This reduces quantization errors caused by data differences between different channels within the same group, minimizes static quantization errors, and achieves near-lossless compression of the large language model. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 This is a first flowchart of a quantization method for a large language model provided in an embodiment of this application;

[0050] Figure 2 This is a schematic diagram of the network structure of a large language model provided in an embodiment of this application;

[0051] Figure 3 This is a second flowchart of a quantization method for a large language model provided in an embodiment of this application;

[0052] Figure 4 This is a third flowchart of a quantization method for a large language model provided in an embodiment of this application;

[0053] Figure 5 This is the fourth flowchart of a quantization method for a large language model provided in an embodiment of this application;

[0054] Figure 6 This is a schematic diagram illustrating the application scenario of the quantized large language model provided in the embodiments of this application;

[0055] Figure 7 This is a schematic diagram of activation quantization provided in an embodiment of this application;

[0056] Figure 8 This is a schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0057] The present application will be described in detail below with reference to the specific embodiments shown in the accompanying drawings. However, these embodiments do not limit the present application. Any structural, methodological, or functional modifications made by those skilled in the art based on these embodiments are included within the protection scope of the present application.

[0058] Please refer to Figure 1 This application provides a method for quantizing a large language model, which includes steps S101-S104.

[0059] S101, for each linear layer of the large language model: calculate the weight quantization sensitivity of the linear layer, and determine the high-precision bit width ratio of each linear layer based on the weight quantization sensitivity of each linear layer and the target mixed precision ratio.

[0060] Quantization refers to converting high-precision data into low-precision data. For example, mapping BF16 (Brain Floating Point) to an 8-bit integer (INT8) tensor. Quantization is equivalent to discretizing the data, reducing computational workload by decreasing data precision, but it introduces information loss due to precision errors. In other words, reducing the bit width of weights and activation values ​​in a neural network can reduce model size and inference latency.

[0061] In this application's embodiments, the large language model is based on the Transformer architecture, such as... Figure 2 As shown, the large language model comprises multiple Transformer networks, each including a multi-head attention module and a feedforward layer. The linear layers within the multi-head attention module include structures such as query (Q), key (K), value (V), and output. The linear layers within the feedforward layer include structures such as gate, up, and down. Each linear layer has raw floating-point weights that need to be quantized.

[0062] The aforementioned large language model can be pre-trained using sample text. In this case, the weights of the large language model are called raw floating-point weights. When quantizing this large language model, the raw floating-point weights corresponding to each linear layer are obtained, and the raw floating-point weights of all or some of the linear layers in the large language model are quantized. The raw floating-point weights of each linear layer of the large language model can be stored in memory. The type of the raw floating-point weights is floating-point numbers; for example, the raw floating-point weights are BF16. For example, the raw weights of BF16 can be quantized into 8-bit fixed-point data (int8), or they can be quantized into other low-precision data.

[0063] Based on the original floating-point weights of each linear layer in the large language model, the quantization sensitivity of each linear layer is calculated. Then, based on the quantization sensitivity and the target mixed-precision quantization ratio, the high-precision bit-width ratio of each linear layer is determined. The high-precision bit-width ratio can be understood as the ratio of channels using high-precision bit-width quantization to the total number of channels in that linear layer. Because the sensitivity to quantization errors caused by quantization in different linear layers of the large language model can vary, the number of quantization bits in the large language model can be mixed; that is, the number of quantization bits in different linear layers can be different. For example, the weights of layers with higher quantization error sensitivity can be quantized using high-precision bit-width quantization with a larger number of quantization bits, while the weights of layers with lower quantization error sensitivity can be quantized using low-precision bit-width quantization with a smaller number of quantization bits, in order to minimize the precision loss of the large language model caused by quantization. The quantization sensitivity of each linear layer is converted into the corresponding high-precision bit-width ratio of the linear layer. Then, based on the high-precision bit-width ratio of each linear layer, corresponding quantization bit widths are allocated to different levels of linear layers, realizing mixed-precision quantization between linear layers in the large language model.

[0064] S102, for multiple channels of each linear layer, based on the input activation of the linear layer, obtain the channel quantization sensitivity of each channel in the linear layer.

[0065] In large language models, outliers often appear in certain input channels, which are typically randomly distributed. If all channels are quantized with the same bit width, the channels containing outliers will introduce larger errors. Therefore, different bit widths need to be assigned to different channels. However, if quantization is performed directly by channel grouping, the data distribution of different input channels within the same group may vary significantly, leading to large quantization errors.

[0066] For example, the channel can be an input channel.

[0067] S103, sort the multiple channels in the linear layer according to the channel quantization sensitivity, and combine the high-precision bit width ratio of the linear layer to cluster the sorted multiple channels, and assign different precision quantization bit widths to the weight parameters corresponding to the channels in different clusters.

[0068] For each linear layer, the channel quantization sensitivity of each channel in that layer is obtained. Based on the channel quantization sensitivity, the channels are sorted from largest to smallest. According to the high-precision bit width ratio of the linear layer, the sorted channels are clustered, and different precision quantization bit widths are assigned to the weight parameters corresponding to the channels in different clusters. For example, it can be determined which channels use high-precision bit widths for their weight parameters and which use low-precision bit widths, thus determining the allocated bit width for the weight parameters of each input channel.

[0069] S104: For multiple channels in the same cluster, rearrange them according to the distribution similarity index. Group the rearranged channels and quantize the weight parameters of each group of channels synchronously according to the allocated quantization bit width to obtain the quantized large language model.

[0070] For multiple channels within the same cluster, the channels are rearranged according to a distribution similarity index. The rearranged channels are then grouped, and the weight parameters of each group are quantized synchronously according to the allocated quantization bit width, resulting in a quantized large language model. Grouping channels with similar data distributions together reduces the data distribution gap between adjacent channels. Therefore, during channel grouping and quantization, channels with similar data distributions can have the same quantization parameter, while channels with large data distribution differences can have different quantization parameters. This reduces the quantization error caused by data differences between different channels within the same weight matrix, minimizing static quantization error.

[0071] In this application example, by determining the weight quantization sensitivity of each linear layer in a large language model, and then determining the high-precision bit width ratio of each linear layer, the inter-layer mixed precision quantization of each linear layer in the large language model is achieved. Channels are sorted according to the channel quantization sensitivity of each channel in the linear layer, and the sorted channels are clustered based on the high-precision bit width ratio. Different precision quantization bit widths are assigned to the weight parameters corresponding to channels in different clusters. This allows it to determine which input channels use high-precision bit widths for their weight parameters and which use low-precision bit widths, achieving intra-layer mixed precision quantization of each linear layer in the large language model. For multiple channels within the same cluster, the channels are rearranged according to a distribution similarity index, such as... Figure 2 As shown, the weight parameters of the rearranged channels are quantized according to the allocated quantization bit width, thereby reducing the quantization error caused by the data differences between different channels within the same group of the weight matrix, minimizing the static quantization error, and achieving near-lossless compression of large language models.

[0072] One embodiment of this application, such as Figure 3 As shown, the method includes steps S301-S303.

[0073] S301, according to the corresponding channel quantization sensitivity, sort multiple channels in descending order, and combined with the high-precision bit width ratio of the linear layer, divide multiple channels into high-sensitivity clusters with high channel quantization sensitivity and low-sensitivity clusters with low channel quantization sensitivity;

[0074] S302: For multiple channels in a high-sensitivity cluster, obtain the product of the maximum absolute value of the weight parameter in each channel and the maximum value of the input activation, and use it as a distribution similarity index. Rearrange the multiple channels in the high-sensitivity cluster according to the distribution similarity index, group multiple adjacent channels into a group, and quantize the weight parameters of each group of channels synchronously according to the high-precision bit width.

[0075] S303: For multiple channels in a low-sensitivity cluster, obtain the product of the maximum absolute value of the weight parameter in each channel and the maximum value of the input activation, and use it as a distribution similarity index. Rearrange the multiple channels in the low-sensitivity cluster according to the distribution similarity index, group multiple adjacent channels into a group, and quantize the weight parameters of each group of channels synchronously according to the low-precision bit width.

[0076] For each linear layer, the channel quantization sensitivity of each channel in that layer is obtained. Based on the channel quantization sensitivity of each channel, the corresponding channels are sorted in descending order. According to the high-precision bit-width ratio of the linear layer and the total number of channels, the number of channels using high-precision bit-width and the number of channels using low-precision bit-width are determined. Starting from the channel corresponding to the maximum value of the channel quantization sensitivity, channels corresponding to the number of channels using high-precision bit-width are selected sequentially. These selected channels are assigned to a high-sensitivity cluster, and the remaining channels are assigned to a low-sensitivity cluster. Channels within the high-sensitivity cluster use high-precision bit-width quantization, and channels within the low-sensitivity cluster use low-precision bit-width quantization.

[0077] For example, assuming the linear layer has 1024 input channels and the high-precision bit width ratio is 20%, based on the channel quantization sensitivity of the 1024 input channels, the channels are sorted from largest to smallest, and the selection starts from the maximum value. The first 1024 * 10% = 205 channels are the high-sensitivity cluster, and the remaining channels are the low-sensitivity cluster.

[0078] For each channel in the high-sensitivity cluster, the maximum absolute value of the weight parameter and the maximum activation value among the input activation values ​​are calculated, and their product is used as the distribution similarity index for that channel. This allows us to obtain the distribution similarity index for each channel. The channels in the high-sensitivity cluster are then sorted in ascending order of distribution similarity index, ensuring that channels with similar data distributions are grouped together, resulting in an ordered data distribution across all channels. For each sorted channel, every 128 adjacent channels are grouped together, and the weight parameters of each group are simultaneously quantized according to a high-precision bit width.

[0079] For example, in a high-sensitivity cluster of a linear layer, there are 8 input channels C1-C8. The weight matrix represents the 8 weight values ​​of the first input channel as w1 to w8, and the maximum absolute value of the weight of this input channel is |w4|. The input activation matrix represents the 8 activation values ​​of the first input channel as x1 to x8, and the maximum absolute value of the activation of this input channel is |x8|. Then, the distribution similarity index corresponding to this input channel is |w4|×|x8|. By analogy, the distribution similarity indexes of the 8 input channels can be obtained. The channels are sorted from largest to smallest according to the size of the distribution similarity index. If the input channels of the high-sensitivity cluster were sorted as "C1, C2, C3, C4, C5, C6, C7, C8" before sorting, the input channels of the linear layer will be sorted as "C2, C4, C6, C3, C1, C5, C7, C8" after resorting in the above manner.

[0080] For each channel in the low-sensitivity cluster, the maximum absolute value of the weight parameter and the maximum activation value among the input activation values ​​are calculated, and their product is used as the distribution similarity index for that channel. This allows us to obtain the distribution similarity index for each channel. The channels in the low-sensitivity cluster are then sorted in ascending order of distribution similarity index, ensuring that channels with similar data distributions are grouped together, resulting in an ordered data distribution across all channels. For each sorted channel, every 128 adjacent channels are grouped together, and the weight parameters of each group are simultaneously quantized using a low-precision bit width.

[0081] In this embodiment, based on the channel quantization sensitivity and high-precision bit width ratio corresponding to each input channel, it is determined which input channels use high-precision bit width for their weight parameters and which input channels use low-precision bit width for their weight parameters. The input channels using high-precision bit width are then rearranged based on the distribution similarity index, and the input channels using low-precision bit width are also rearranged based on the distribution similarity index. This is to ensure that channels with similar data distributions within the same weight matrix are grouped together, thereby further reducing quantization errors.

[0082] One embodiment of this application, such as Figure 4 As shown, for each linear layer of the large language model: calculate the weight quantization sensitivity of the linear layer, including steps S401-S403.

[0083] S401, a small amount of calibration data is input into the large language model, and forward and backward propagation are performed to generate gradient vectors; wherein, the small amount of calibration data is a small amount of data obtained by sampling the sample text.

[0084] S402 uses low-precision bit-width quantization of the weights of the linear layer to obtain low-precision quantization error;

[0085] S403 generates the weight quantization sensitivity of the linear layer based on the gradient vector and low-precision quantization error.

[0086] For each linear layer, the raw floating-point weights are obtained. Based on a small amount of calibration data, forward propagation is performed using the raw floating-point weights, and the gradient of the weight parameters is obtained through backpropagation. The weights of the linear layer are quantized using a low-precision bit width, and the low-precision quantization error is determined based on the raw floating-point weights and the quantized weights. Based on the gradient vector and the low-precision quantization error, the weight quantization sensitivity of the linear layer is generated. Based on the weight quantization sensitivity of each linear layer, the quantization bit width of each linear layer can be determined; for example, linear layers with high weight quantization sensitivity use a high-precision quantization bit width, and linear layers with low weight quantization sensitivity use a low-precision quantization bit width.

[0087] Quantization converts the weights of a large language model from floating-point numbers to integers. This conversion inevitably introduces errors into the model parameters, known as "quantization perturbations." This application's embodiments utilize the Hessian matrix to evaluate these perturbations. The Hessian matrix is ​​a matrix containing second-order partial derivatives, and quantization perturbations provide detailed information about the impact of small changes in model parameters on the output. Therefore, based on the Hessian matrix and the weight quantization error, a target quantization perturbation function can be constructed for the model. This function represents the perturbation introduced by quantizing the weights into the loss function, with the goal of finding the optimal quantization bit width allocation strategy to minimize the resulting perturbation. Based on this target quantization perturbation function, the calculation methods for the weight quantization sensitivity of linear layers and the channel quantization sensitivity are derived.

[0088] The target quantization perturbation function can be expressed as:

[0089]

[0090] Where w represents the original floating-point weight, Q(w) represents the target quantized weight, H represents the Hessian matrix, wQ(w) represents the weight quantization error, and L represents the loss function.

[0091] Calculating the Hessian matrix in large language models with a huge number of parameters is quite difficult; therefore, the Fisher information matrix is ​​used as an approximation of the Hessian matrix. When calculating the Fisher information matrix of the weight parameters, a small amount of calibration data is used for forward propagation, and the gradients of the weight parameters are obtained through backpropagation. The Fisher information matrix is ​​then calculated based on these gradients. Therefore, the Fisher information matrix can be calculated based on the gradients of the weight parameters. However, explicitly calculating the Fisher information matrix requires a significant amount of memory. Through formula derivation, the quantization sensitivity of the linear layer weights can be directly calculated from the gradients and quantization errors, i.e., the Fisher information matrix is ​​implicitly calculated.

[0092] Therefore, based on the aforementioned target quantization perturbation function, the Fisher information matrix is ​​used as an approximation of the Hessian matrix. The Fisher information matrix can be calculated based on the gradient of the weight parameters. Therefore, the weight quantization sensitivity of the linear layer can be determined based on the quantization error and the gradient, and thus the calculation formula for the weight quantization sensitivity of each linear layer can be derived:

[0093] H≈F=ggT;

[0094]

[0095] δw=wQ(w);

[0096] Where F represents the Fisher information matrix, b i The quantization sensitivity of the weights of the i-th linear layer is represented by δw, which represents the quantization error generated by quantizing the i-th linear layer with low precision bit width. g represents the gradient vector generated by forward and backward propagation based on a small amount of calibration data. w represents the original floating-point weights, and Q(w) represents the weights quantized with low precision bit width. For example, when the model uses a combination of mixed quantization precision of {8-bit, 4-bit}, Q(w) is the weight after quantization with int4 bit width.

[0097] In this embodiment, the gradient of the weight parameters is used to calculate the Fisher information matrix. The quantization error and gradient are used to determine the quantization sensitivity of the linear layer weights, which reduces the complexity of model calculation and memory usage. It can capture and evaluate the potential impact of quantization operations on model performance, so that the quantized large language model can maintain a small size and improve computational efficiency while maintaining the accuracy of the original model as much as possible.

[0098] One embodiment of this application, such as Figure 5 As shown, the high-precision bit width ratio of each linear layer is determined based on the weight quantization sensitivity and target mixed precision ratio of each linear layer, including steps S501-S503.

[0099] S501, the basic mixed sperm ratio of a large language model was determined through ablation experiments;

[0100] S502, combined with the memory usage of the original floating-point weights of the large language model, sets the target mixed precision ratio of the large language model;

[0101] S503, for the i-th linear layer, calculate the high-precision bit width ratio of the i-th linear layer based on the weight quantization sensitivity of the i-th linear layer, the weight parameter quantity of the i-th linear layer, the basic mixing ratio, and the target mixing ratio.

[0102] The basic mixed sperm ratio of a large language model is a hyperparameter, and its optimal value can be determined through ablation experiments.

[0103] Based on the multiple different quantization precipitates that the target hardware can support, and the total number of weight parameters of the large language model, determine the memory required for each quantization precipitate, and determine the target mixed precision ratio based on the memory allocated to the total number of weight parameters.

[0104] For example, assuming the hardware supports quantization precision of {8-bit, 4-bit}, INT8 occupies 1 byte, and INT4 occupies 0.5 bytes. The total number of weight parameters in the large language model is 7 bytes (7 billion parameters), and the memory allocated for the total number of weight parameters is 4 GB. Calculate the target mixed precision ratio r of INT4 and INT8.

[0105] If the model uses INT8 throughout, its memory usage is:

[0106] The memory required for a 7B (7 billion parameters) INT8 model is approximately 7e9 × 1 Byte, which is approximately 6.53 GB.

[0107] If the model uses INT4 throughout, its memory usage is:

[0108] The memory requirement for a 7B (7 billion parameters) INT4 model is approximately 7e9 × 0.5 Bytes, which is approximately 3.26 GB.

[0109] Assume the target mixed sperm ratio of INT8 is s T Then the proportion of INT4 is (1-s T From this, we can obtain the following equation:

[0110] 6.53*s T +3.26*(1-s T = 4GB;

[0111] Solving for s, we get s T It is 22.63%.

[0112] For example, the high-precision bit-width ratio of each linear layer is determined based on the following formula:

[0113]

[0114] Among them, b i p represents the weight quantization sensitivity of the i-th linear layer. i The weight parameters s represent the weights of the i-th linear layer. T Characterizing the target mixed sperm ratio, s B Characterizing the basic mixed semen ratio, satisfying S B Less than S T .

[0115] In this embodiment, the high-precision bit-width ratio of each linear layer is determined based on the weight quantization sensitivity and the target mixed-precision ratio. For example, linear layers with high weight quantization sensitivity will have a higher high-precision bit-width ratio, while linear layers with low weight quantization sensitivity will have a lower high-precision bit-width ratio. This mixed-precision quantization allows for more effective error control during quantization operations, while also optimizing the deployment and operational efficiency of large language models in resource-constrained environments.

[0116] In one specific embodiment of this application, the channel quantization sensitivity of each channel in the linear layer is obtained based on the input activation of the linear layer, including: processing the input activation of the linear layer using the Levenberg-Marquadt algorithm, approximating the corresponding Hessian matrix with the Levenberg-Marquadt algorithm, and obtaining the inverse of the Hessian matrix; for each channel, obtaining the original floating-point weights of the channel, quantizing the original floating-point weights using low-precision bit width to obtain quantized weights, and using the difference between the original floating-point weights and the quantized weights as the low-precision quantization error of the channel; and determining the channel quantization sensitivity of the channel based on the low-precision quantization error and the inverse of the Hessian matrix.

[0117] Since the Hessian matrix is ​​difficult to solve, the Levenberg-Marquadt (LM) algorithm is used to approximate the Hessian matrix. The Hessian matrix H is expressed as:

[0118] H = XX T ;

[0119] To address the issue of the Hessian matrix being non-invertible, a damping term λmean(diag(XX) is introduced. T Therefore, the Hessian matrix can be approximately expressed as:

[0120] H≈XX T +λmean(diag(XX T );

[0121] Where λ represents the damping factor, mean(diag(XX) T )) indicates taking XX T The mean of the diagonal elements of a matrix.

[0122] The calculation method for the channel quantization sensitivity is derived based on the above target quantization perturbation function, as follows:

[0123]

[0124] Assuming the q-th weight parameter is quantized, the above equation is transformed into a constrained optimization problem:

[0125]

[0126] stδw q +w q =W(w q );

[0127] The constrained optimization problem is transformed into an unconstrained optimization problem using the Lagrange multiplier method, as follows:

[0128]

[0129] make

[0130] We can obtain:

[0131] δw=-βH -1 e q T ;

[0132] According to δw q +w q =Q(w q ),get:

[0133]

[0134] The above derivation yields the formula for calculating the quantization sensitivity of any channel in a linear layer:

[0135]

[0136] Where X represents the input activation of the linear layer, X T Let represent the transpose of the input activation vector of the linear layer, w represent the original floating-point weights of the channel, Q(w) represent the weights after low-precision bit-width quantization of the original floating-point weights of the channel, and λ represent the damping factor, typically taken as 0.1. mean(diag(XX) T )) indicates taking XX T The mean of the diagonal elements of the matrix. Based on this formula, the channel quantization sensitivity of each channel can be calculated.

[0137] In a specific embodiment of this application, the parameters pre-set for the large language model are as follows: the hardware supports quantization bit widths of int8 and int4. The target mixing ratio S is set. T The value is 20%, meaning that 20% of the linear layers in the model use int8 quantization and 80% use int4 quantization. The base blending ratio for each linear layer is set to S. B Satisfying S B Less than S TThe linear layers in the large language model include query, key-value, value, and output layers, as well as dimensionality up, gating, and dimensionality down layers. Based on the weight quantization sensitivity calculations for the linear layers, the weight quantization sensitivities for each of the above linear layers are, in order: 5.2891e-06, 1.0855e-05, 2.0029e-05, 8.9772e-06, 5.7826e-06, 4.6932e-06, and 1.2573e-05. Based on the calculation formula for the high-precision bit-width ratio of the linear layers, the high-precision bit-width (int8) ratios for each of the above linear layers are, in order: 14%, 27%, 48%, 23%, 15%, 13%, and 31%. Taking a linear layer as an example, assuming the layer has 4096 input channels and a high-precision bit width ratio of 14%, that is, 4096 * 14% of the channels use int8 quantization, and 4096 * (1 - 14%) of the channels use int4 quantization. The formula for calculating channel quantization sensitivity determines which input channels use int8 quantization and which use int4 quantization. The input channels using int8 quantization are then rearranged based on the product of the maximum absolute value of the weight parameter and the maximum value of the input activation, ensuring that channels with similar data distributions are grouped together. Similarly, the input channels using int4 quantization are rearranged again, again based on the product of the maximum absolute value of the weight parameter and the maximum value of the input activation, further reducing quantization error.

[0138] As shown in Figure 6, this is a schematic diagram of the application scenario of the quantized large language model. After tokenizing the user input, the token is input into the embedding layer for encoding to obtain the corresponding embedding vector. The embedding vector is then input into multiple cascaded Transformer layers for forward encoding. After normalization in the normalization layer, the input is fed into a multi-head self-attention (MOA) layer. The linear layers in the MOA layer include query, key, value, and output. After MOA, the activations are input into a feedforward neural network. The linear layers in the feedforward neural network include gate, up, and down layers. During the computation of each linear layer, the quantized linear layer weights are first obtained, then the input activations are quantized, the matrix multiplication of the quantized input activations and weights is calculated, and then dequantization is performed to obtain the floating-point output. After the user input text is processed by multiple cascaded Transformer layers, the final inference result is obtained.

[0139] In one embodiment of this application, the weight parameters of the linear layer of the large language model include at least one of the following: query weight, key-value weight, value-value weight, dimensionality-up weight, gating weight, and dimensionality-down weight; the activation values ​​of the large language model are quantized using high-precision bit-width quantization.

[0140] The weight quantization of the large language model is completed before inference, and the activation of the input is quantized during the inference process. Therefore, after the weight parameters of the large language model are quantized according to the above implementation method, the input activation in the model is then quantized. Specifically, for each linear layer, an activation quantization node is inserted; using a small amount of calibration data, forward propagation is performed in the large language model after the activation quantization node is inserted, so that the activation quantization node obtains the quantization weight parameters required for activation quantization and stores these activation quantization weight parameters; during the inference process of the large language model, for each linear layer, the corresponding activation quantization node quantizes the input activation of that linear layer, where a high-precision bit width is used for quantization of the input activation; integer matrix operations are performed on the quantized input activation and the stored quantization weight parameters to obtain intermediate calculation results, and the intermediate calculation results are dequantized to obtain the calculation result of the current linear layer output. Figure 7 The diagram illustrates activation quantization. At any activation quantization node, the input activation (FP16) is quantized using a high-precision bit width to obtain the quantized input activation. The quantized input activation is then subjected to integer matrix operations (INT8 general matrix multiplication weights) with the quantization weight parameters (INT8 weights or mixed INT8 and INT4 weights) stored at that activation quantization node. This is then processed by an INT8 accumulator, performing matrix addition on the INT8 data type. In some biased linear layers, an INT8 bias term is required as input. The calculation result is then dequantized to obtain the calculation result of the current linear layer output (FP16 output activation).

[0141] One embodiment of this application provides a large language model quantization system, including a server, a client, and a display. The server quantizes the large language model using the method described above. The client inputs a question into the vectorized large language model to obtain inference results. The display shows the inference results.

[0142] Figure 8 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Figure 8 The illustrated electronic device includes: a processor 801, a communication interface 802, a memory 803, and a communication bus 804. The processor 801, communication interface 802, and memory 803 communicate with each other via the communication bus 804. Figure 8The connection method between the processor 801, communication interface 802, and memory 803 shown is merely exemplary. In the implementation process, the processor 801, communication interface 802, and memory 803 may also communicate with each other using other connection methods besides the communication bus 804.

[0143] The memory 803 can be used to store computer programs, which may include instructions and data to implement the steps of any of the large language model quantization methods described above. In this embodiment, the memory 803 can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, optical memory, and registers. The memory 803 may include a hard disk and / or RAM.

[0144] Processor 801 can be a general-purpose processor, which can be a processor that performs specific steps and / or operations by reading and executing a computer program (e.g., a computer program) stored in memory (e.g., memory 803). The general-purpose processor may use data stored in memory (e.g., memory 803) during the execution of said steps and / or operations. The general-purpose processor can be, for example, but not limited to, a central processing unit (CPU). Furthermore, processor 801 can also be a special-purpose processor, which can be a processor specifically designed to perform specific steps and / or operations. Special-purpose processors can be, for example, but not limited to, ASICs and FPGAs. Additionally, processor 801 can also be a combination of multiple processors, such as a multi-core processor.

[0145] Communication interface 802 may include input / output (I / O) interfaces, physical interfaces, and logical interfaces for interconnecting devices within the network device, as well as interfaces for interconnecting the network device with other devices (e.g., network devices). The communication network may be Ethernet, a radio access network (RAN), a wireless local area network (WLAN), etc. Communication interface 802 may be a module, circuit, transceiver, or any device capable of enabling communication.

[0146] In implementation, each step of the above method can be completed by the integrated logic circuits in the hardware of the processor 801 or by instructions in software. The method disclosed in the embodiments of this application can be directly implemented by the hardware processor, or by a combination of hardware and software modules in the processor. The software modules can reside in readily available storage media in the art, such as random access memory (RAM), read-only memory (ROM), programmable read-only memory (ROM), electrically erasable programmable memory (EPR), or registers. This storage medium is located in memory 803, and the processor 801 reads the information in memory 803 and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, detailed descriptions are omitted here.

[0147] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0148] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0149] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0150] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0151] The above are merely preferred embodiments of this application and are not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A quantization method for a large language model, characterized in that, The large language model is trained using sample text, and the method includes: For each linear layer of the large language model: Calculate the weight quantization sensitivity of the linear layer, and determine the high-precision bit width ratio of each linear layer based on the weight quantization sensitivity of each linear layer and the target mixed precision ratio; For each linear layer with multiple channels, based on the input activation of that linear layer, the channel quantization sensitivity of each channel in that linear layer is obtained; According to the channel quantization sensitivity, multiple channels in the linear layer are sorted, and combined with the high-precision bit width ratio of the linear layer, the sorted channels are clustered, and different precision quantization bit widths are assigned to the weight parameters corresponding to the channels in different clusters. For multiple channels in the same cluster, they are rearranged according to the distribution similarity index. The rearranged channels are grouped, and the weight parameters of each group of channels are quantized synchronously according to the allocated quantization bit width to obtain a quantized large language model. A question is input into the quantized large language model to obtain the inference result.

2. The method as described in claim 1, characterized in that, The process of sorting multiple channels in the linear layer according to the channel quantization sensitivity, and then clustering the sorted channels based on the high-precision bit-width ratio of the linear layer, includes: According to the corresponding channel quantization sensitivity, multiple channels are sorted in descending order. Combined with the high-precision bit width ratio of the linear layer, the multiple channels are divided into a high-sensitivity cluster with high channel quantization sensitivity and a low-sensitivity cluster with low channel quantization sensitivity. The process involves rearranging multiple channels within the same cluster according to a distribution similarity index, grouping the rearranged channels, and synchronously quantizing the weight parameters of each group of channels according to the allocated quantization bit width, including: For multiple channels in a high-sensitivity cluster, the product of the maximum absolute value of the weight parameter in each channel and the maximum value of the input activation is obtained as a distribution similarity index. The multiple channels in the high-sensitivity cluster are rearranged according to the distribution similarity index, and each group of 128 adjacent channels is divided into groups. The weight parameters of each group of channels are quantized synchronously according to the high-precision bit width. For multiple channels in a low-sensitivity cluster, the product of the maximum absolute value of the weight parameter in each channel and the maximum value of the input activation is obtained as a distribution similarity index. The multiple channels in the low-sensitivity cluster are rearranged according to the distribution similarity index, and every 128 adjacent channels are grouped together. The weight parameters of each group of channels are quantized synchronously according to the low precision bit width.

3. The method as described in claim 1, characterized in that, For each linear layer of the large language model: calculate the weight quantization sensitivity of that linear layer, including: A small amount of calibration data is input into the large language model, and forward and backward propagation are performed to generate gradient vectors. The small amount of calibration data is obtained by sampling the sample text. The weights of the linear layer are quantized using low-precision bit width to obtain low-precision quantization error; Based on the gradient vector and the low-precision quantization error, the weight quantization sensitivity of the linear layer is generated.

4. The method as described in claim 1, characterized in that, Based on the weighted quantization sensitivity and target mixed precision ratio of each linear layer, the high-precision bit width ratio of each linear layer is determined, including: The basic mixed sperm ratio of the large language model was determined through ablation experiments; Based on the memory usage of the original floating-point weights of the large language model, the target mixed precision ratio of the large language model is set. For the i-th linear layer, the high-precision bit width ratio of the i-th linear layer is calculated based on the weight quantization sensitivity of the i-th linear layer, the weight parameter quantity of the i-th linear layer, the basic mixed precision ratio, and the target mixed precision ratio.

5. The method as described in claim 3, characterized in that, For each linear layer of the large language model: calculate the weight quantization sensitivity of that linear layer, including: The weight quantization sensitivity of each linear layer is calculated using the following formula: ； in, Characterizing the weight quantization sensitivity of the i-th linear layer, Characterizes the quantization error generated by quantizing the i-th linear layer using low-precision bit width. Characterizes the gradient vector generated by forward and backward propagation based on a small amount of calibration data.

6. The method as described in claim 5, characterized in that, For each linear layer of the large language model: calculate the weight quantization sensitivity of that linear layer, including: The high-precision bit-width ratio of each linear layer is determined based on the following formula: ； in, Characterizing the weight quantization sensitivity of the i-th linear layer, The weight parameters representing the i-th linear layer Characterizing the target mixed semen ratio, Characterizes the basic mixed sperm ratio.

7. The method as described in claim 1, characterized in that, Based on the input activation of this linear layer, the channel quantization sensitivity of each channel in this linear layer is obtained, including: The Levenberg-Marquadt algorithm is used to process the input activation of the linear layer. The Hessian matrix is ​​approximated by the Levenberg-Marquadt algorithm, and the inverse of the Hessian matrix is ​​obtained. For each channel, the original floating-point weights of the channel are obtained, and the original floating-point weights of the channel are quantized using low-precision bit width to obtain the quantized weights. The difference between the original floating-point weights and the quantized weights is used as the low-precision quantization error of the channel. The channel quantization sensitivity of the channel is determined based on the low-precision quantization error and the inverse of the Hessian matrix.

8. The method as described in claim 1, characterized in that, Based on the input activation of this linear layer, the channel quantization sensitivity of each channel in this linear layer is obtained, including: The quantization sensitivity of any channel in the linear layer is obtained using the following formula: ； in, This indicates the input activation of the linear layer. This represents the transpose of the input activation of the linear layer. Characterizes the original floating-point weights of this channel. This represents the weights after low-precision bit-width quantization of the original floating-point weights for this channel. Indicates the damping factor. Indicates taking The mean of the diagonal elements of a matrix.

9. The method as described in claim 1, characterized in that, The weight parameters of the linear layer of the large language model include at least one of the following: Query weight, key-value weight, value-value weight, dimensionality-upgrading weight, gating weight, and dimensionality-reducing weight; The activation values ​​of the large language model are quantized using high-precision bit-width quantization.

10. An electronic device, characterized in that, include: processor; A memory storing computer-readable instructions that, when executed by the processor, implement the method as described in any one of claims 1 to 8.

11. A large language model quantization system, characterized in that, Includes server, client, and display: The server is configured to quantize a large language model using the method described in any one of claims 1-8; The client is used to input a question into the quantized large language model to obtain inference results; The display is used to show the reasoning results.

Images