Large model quantization algorithm based on outlier suppression

Abstract

The invention discloses a large language model quantification algorithm based on outlier suppression, and aims to solve the problem of performance reduction caused by outliers in the existing quantification technology. According to the algorithm, high-precision quantization in a large model reasoning process is realized through suppression of weights, activation values and self-attention outliers, and activation and weight reforming type 4-bit quantization operation is supported. Particular techniques include random Hadamard transforms to reduce weight outliers, scaling factor adjustments to process activation outliers, and self-attention migration techniques to correct attention outliers. According to the method, the numerical stability and precision of model reasoning are comprehensively improved. The quantized model significantly reduces the memory and computing power requirements, is suitable for resource-limited environments such as edge device deployment, and improves the reasoning efficiency at the same time. The method provides powerful support for low-cost and high-efficiency deployment of the large language model, and has important application value in a plurality of actual scenes.

Description

Technical Field

[0001] The present invention belongs to the field of large language models and deep learning, and in particular relates to a large model quantization algorithm based on outlier suppression. Background Art

[0002] With the development of society, large language models have demonstrated excellent performance in many fields. They are widely used in chat assistants, medical diagnosis, game content generation, programming assistance, embodied intelligence and other scenarios. These models are trained with massive data and can handle complex tasks, but this is accompanied by a huge number of parameters and high inference costs. With the in-depth application of large models in various fields, inference costs have become an urgent problem to be solved. In order to reduce costs and improve efficiency, post-training quantization technology has emerged. It quantizes the weights or activations of the model to a lower precision, effectively reducing the model's memory usage and may significantly reduce latency. This is critical for server-side inference as well as inference on edge devices.

[0003] Post-training quantization technology enables models to be deployed in resource-constrained environments, such as mobile and embedded devices, by reducing model size and improving inference speed. However, the handling of outliers remains a challenge during the quantization process. Outliers will expand the quantization range, resulting in a reduction in the effective bits of most values, affecting the performance of the quantized model. Outliers are divided into weight outliers, activation outliers, and self-attention outliers. Existing methods often deal with one or two outliers, and the suppression of outliers is incomplete, resulting in the inability to achieve good results at lower quantization bits. Summary of the invention

[0004] The present invention relates to a large language model quantization algorithm based on outlier suppression, which specifically includes the elimination of weight outliers, activation outliers, and self-attention outliers, and finally realizes the quantization of integer 4-bit data types of weights and activation values; it is used for lightweight deployment of large models, and suppresses weight outliers, activation outliers, and self-attention outliers in the reasoning process of the large model respectively.

[0005] The present invention comprises the following steps:

[0006] Step 1: Process the outliers in the weight parameters of the large model, perform random Hadamard transformation on the weight parameter matrix, and eliminate the outliers in the weight matrix;

[0007] Step 2: Process the outliers of activation values ​​during the large model inference process. During inference, record the maximum value of each channel in the activation value, and determine whether it is an abnormal channel based on the maximum value. If so, multiply a scaling factor in the abnormal activation value channel to reduce the abnormal activation value, and divide the corresponding weight parameter channel by the scaling factor to ensure the invariance of the calculation;

[0008] Step 3: Process the outliers in the self-attention weights during the large model reasoning process. By self-attention migration, the attention weights of the positions where attention convergence occurs are assigned to other positions to eliminate autonomous outliers. This completes the suppression of three types of outliers during the large model reasoning process and realizes the quantization of the 4-bit integer data type of the weight activation value;

[0009] The method of the present invention comprehensively improves the numerical stability, accuracy and performance in the reasoning process of large language models by combining technical means such as random Hadamard transform, activation value channel adjustment, and self-attention migration, and solves the problems of decreased reasoning accuracy and quantization difficulties caused by outliers in the prior art.

[0010] The technical solution of the present invention is as follows:

[0011] A large model quantization algorithm based on outlier suppression, comprising:

[0012] Weight outlier suppression, activation outlier suppression, self-attention outlier suppression, model integer 4-bit quantization;

[0013] Among them, weight outliers are usually manifested as extremely large or extremely small abnormal values ​​in the weight matrix. These abnormal values ​​will introduce significant numerical errors in reasoning. To this end, the present invention uses random Hadamard transform to process the weight matrix. By increasing the irrelevance of each vector in the weight matrix, the quantization error of each rounding operation is reduced during the quantization process. The specific steps are as follows: weight parameter acquisition, random Hadamard matrix generation, and weight matrix Hadamard transform. First, the weight parameters are acquired. The weight parameters in the large model are mainly concentrated in the self-attention module and the feedforward neural network module of each Transformer block. The parameter matrix in the self-attention module is W o , W k , W v , W o , in the feedforward neural network module it is W up , W gate , W down . Then, random Hadamard matrix is ​​generated. Hadamard matrix is ​​a special matrix composed of +1 and -1 elements and satisfies orthogonality. First, according to the recursive formula of Hadamard matrix.

[0014]

[0015] After obtaining the Hadamard matrix consistent with the model feature dimension, since the Hadamard matrix is ​​still a Hadamard matrix after exchanging rows or columns, a random row and column transformation is used here to obtain a random Hadamard matrix. A total of two random Hadamard matrices need to be generated, denoted as R 1 , R 2 . Then apply the Hadamard transform to the parameter matrix, that is After the above processing, the outliers in the weight parameters are basically suppressed.

[0016] Among them, activation outliers mainly occur during the inference process, which is manifested as the values ​​of some channels in the activation value are much higher than those of other channels. Such outliers will significantly affect the stability and quantization accuracy of the inference. The present invention proposes an activation outlier processing method based on scaling factor adjustment, and its specific steps are as follows: activation value recording, abnormal channel detection, scaling factor selection, activation value adjustment, and weight parameter adjustment. First, record the maximum value in each activation value channel, that is, s j =max(|X j |),j=1,2,...,C i If the activation value in a channel is greater than 10 times the average channel activation value, the activation value channel is judged to be an abnormal activation value channel, that is, Where n is the number of channels.

[0017] Then select a suitable scaling factor, usually 0.5, and multiply the value in the abnormal activation value channel by the activation factor, that is, At the same time, the weight value in the corresponding weight channel is multiplied by the activation factor to maintain the calculation invariance. After the above processing, the outliers in the activation value are basically suppressed.

[0018] The self-attention outlier is manifested as an extreme concentration phenomenon of attention weights, that is, the attention weights are highly concentrated in certain positions, while the weights in other positions are close to zero. To solve this problem, the present invention adopts a self-attention migration technology, and its specific steps are as follows: dormant attention head detection, self-attention weight migration, and attention head weight adjustment. First, traverse each attention head. If the attention weight of the attention head shows an attention concentration phenomenon, that is, a self-attention outlier appears, then the attention head is considered to be in a dormant state. The self-attention outlier is defined by the following formula: in is the cumulative attention contribution of Token j, μ A is the mean of the cumulative attention contribution, Then, according to the attention weight distribution of the attention head in the non-dormant state, i.e., the activated state, the self-attention migration is performed on the dormant state attention head. The token where the self-attention outlier is located is called the sink token. First, the sink token position in the attention head is determined. Where T is the number of tokens, represents the sinktoken in the h-th attention head of the l-th layer, represents the average attention of the tth token of the hth attention head in the lth layer, and then reduces the weight of the sink token’s attention. Then distribute the reduced attention weight to other tokens, that is

[0019]

[0020] Then, the attention weight of the dormant attention head is multiplied by a scaling factor, which is 0.8 in this invention, to reduce the importance of the dormant attention head. After the above processing, the outliers in the self-attention values ​​are basically suppressed.

[0021] The model is quantized to 4 bits, and an asymmetric quantization strategy is adopted. Per-Token quantization is used for activation values, and Per-Channel quantization is used for weight values. The quantization formula is as follows: q = [(xz) / s], where [] represents a rounding operation, s = (max(x)-min(x)) / (2 n-1 -1), z = [-min(x) / s]. The model quantization process is performed in the first step of weight outlier suppression, and activation outlier suppression and self-attention outlier suppression are online operations in the reasoning process.

[0022] The beneficial effects of the present invention are as follows:

[0023] The present invention provides a large model quantization algorithm based on outlier suppression, which can effectively suppress outliers in weights, activation values, and self-attention weights, and significantly reduce the model quantization error. In the integer 4-bit quantization of weights and activation values, the present invention reduces the impact of outliers on quantization accuracy and achieves a significant improvement in the performance of the quantized model. The quantized model reduces the computing power requirements for hardware and improves the inference efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 This is a diagram for briefly introducing the algorithm architecture in the present invention;

[0025] Figure 2 This is a detailed architecture diagram of the quantization model in the present invention;

[0026] Figure 3 This is the weight outlier suppression process in the present invention;

[0027] Figure 4 Activate the outlier suppression process in the present invention;

[0028] Figure 5 This is the self-attention outlier suppression process in the present invention. Specific implementation plan

[0029] The present invention will be further described below in conjunction with the accompanying drawings.

[0030] like Figure 1 As shown in the figure, this algorithm is mainly divided into two stages, namely the offline stage and the online stage. In the offline stage, the weight outliers of the large model are suppressed by random Hadamard transform, and then the integer 4-bit quantization is performed to obtain the quantized large model. In the online inference stage, activation outlier suppression and self-attention outlier suppression are performed in each Transformer block.

[0031] exist Figure 2 The specific operations in each Transformer block are shown in detail in Figure 1. When the word vector enters each Transformer block, it is first subjected to RMS normalization. The process is as follows: γ is a learnable parameter, σ is the standard deviation. The normalized result X is multiplied by R on the right 1 It is the activation value of the self-attention module. If there are outliers in the activation value, the activation outlier suppression operation is performed on it. 1 Added to the output position of the previous Transformer block to offset R 1 The effect on matrix multiplication requires the parameter matrix W q , W k , W v Left multiplication XW q and XW k The results are encoded by rotation positions to obtain q and k in self-attention. After multiplication, the two are passed through the Softmax function. The process is as follows: where x i is the i-th element of the input vector, and finally the self-attention weight is obtained. If outliers appear in the self-attention weight, the self-attention outliers are calibrated online through self-attention migration. The calibrated self-attention weight is compared with the rotated W v Matrix multiplication, where W V R 2 The influence of W o Matrix left multiplication The output value after a residual link is used as the activation value of the feedforward neural network module. If there are outliers in the activation value, an activation outlier suppression operation is performed on it. o R in the matrix right multiplication 1 The matrix is ​​W up , W gats Left multiplication Offset. XW up The results obtained are consistent with XW gate The result obtained after SiLU is ANDed and then multiplied by W down , where W down R 1 Then the next Transformer block W q , W k , W v Left multiplication After 32 such Transformer blocks, a linear layer and a Softmax function are finally passed to obtain the final output result.

[0032] like Figure 3 As shown in Figure 1, this is the step of weight outlier suppression. First, the weight parameters are obtained, that is, W in the 32 Transformer blocks. q , W k , Wv ， W o , W up , W gate , W down Then, a random Hadamard matrix is ​​generated. For example, in the most widely used open source large model Llama, the feature dimension is 4096. First, a 4096-dimensional Hadamard matrix is ​​generated through a recursive formula. The 4×4 submatrix in the upper left corner is as follows:

[0033]

[0034] Then, the 4096-dimensional Hadamard matrix generated by the recursive formula is subjected to n random row and column transformations, n∈(100, 200), to obtain a random Hadamard matrix. Finally, the obtained matrix is ​​converted into Figure 2 Multiplying with the corresponding weight matrix, due to the orthogonality of the Hadamard matrix, the outliers in the weight matrix are reduced without introducing additional calculations while keeping the inference results unchanged.

[0035] like Figure 4As shown, it is the step of suppressing the activation outliers. First, the activation values ​​are recorded to obtain the activation value channel, that is, the maximum value of each column of the activation value. Then, abnormal channel detection is performed. If the maximum value of the channel significantly exceeds the average value of the remaining channels, 10 times is selected as the threshold in the present invention, and the channel is considered to be an abnormal channel, and outlier suppression is required. Then the scaling factor is selected, and 0.5 is selected as the scaling factor in the present invention. The abnormal channel is multiplied by the scaling factor, and outlier suppression is performed. At the same time, in order to offset the influence of the scaling factor on the matrix multiplication, it is also necessary to divide the corresponding channel by the scaling factor in the weight value, and finally the outliers in the activation are suppressed, and a large amount of calculation is not introduced.

[0036] like Figure 5 As shown in FIG. 1 , it is a step to suppress outliers of self-attention. In the present invention, it is considered that the reason for the generation of outliers of self-attention is that in the multi-head attention mechanism, when facing problems in different fields, some attention heads are in a dormant state and some attention heads are in an active state. The dormant attention heads will focus the extra attention on <s>, / n and other tokens with very little semantic information. Therefore, in order to suppress self-attention outliers in the present invention, it is first necessary to determine the status of the attention heads. In the present invention, it is considered that the attention heads with attention outliers are in a dormant state. After finding these attention heads, according to the attention distribution of the activated attention heads, the excess attention is partially transferred from the tokens with very little semantic information to the tokens with strong semantic information, which is self-attention migration. Then, the weights of these dormant attention heads are reduced, and finally the outliers in the self-attention are suppressed without introducing a large amount of computation.

[0037] The above contents are further detailed descriptions of the present invention in combination with specific / preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, they can also make several substitutions or modifications to these described embodiments without departing from the concept of the present invention, and these substitutions or modifications should be regarded as belonging to the protection scope of the present invention.

[0038] Parts of the present invention that are not described in detail belong to the well-known technologies of those skilled in the art.

[0039] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0040] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.< / s>

Claims

1. A large model quantization algorithm based on outlier suppression, characterized in that: It is used for lightweight deployment of large models and suppresses weight outliers, activation outliers, and self-attention outliers in the reasoning process of large models.

2. A large model quantization algorithm based on outlier suppression as claimed in claim 1, characterized in that: The following steps are involved: Step 1: Process the outliers in the weight parameters of the large model, perform random Hadamard transformation on the weight parameter matrix, and eliminate the outliers in the weight matrix; Step 2: Process the outliers of activation values ​​during the large model inference process. During inference, record the maximum value of each channel in the activation value, and determine whether it is an abnormal channel based on the maximum value. If so, multiply a scaling factor in the abnormal activation value channel to reduce the abnormal activation value, and divide the corresponding weight parameter channel by the scaling factor to ensure the invariance of the calculation; Step 3: Process the outliers in the self-attention weights during the large model reasoning process. Through self-attention migration, the attention weights of the positions where attention convergence occurs are allocated to other positions to eliminate autonomous outliers. This completes the suppression of three types of outliers during the large model reasoning process and realizes the quantization of the 4-bit integer data type of the weight activation value.

Images