Quantization method of large language model, related equipment and computer program product

Abstract

The invention provides a large language model quantification method, related equipment and a computer program product, and the method comprises the steps: carrying out the reasoning of a to-be-quantized large language model through calibration data, obtaining the activation of the large language model, and carrying out the statistics of the activation distribution of a target layer according to channels; calculating a smoothing factor of each channel according to the activation distribution; compensating the weight of the target layer channel by channel according to the smoothing factor to obtain a compensated weight; performing 4-bit quantization on the compensated weight; in the model reasoning process, activation of a target layer is smoothed, and 16-bit quantization is carried out on the smoothed activation. According to the method, a W4A16 quantification scheme is adopted, compared with W8A8, the weight storage amount is compressed by half, meanwhile, the precision loss is controlled to be smaller than 3%, and deployment of a large language model on edge equipment is facilitated.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of deep learning model compression, and in particular to a large language model quantization method, related equipment and computer program product. BACKGROUND

[0002] In the rapid development process of the field of artificial intelligence, large language models (LLM) have shown excellent application value in many fields due to their powerful language understanding and generation capabilities. However, their deployment faces severe challenges in terms of storage bandwidth and computing resources, which greatly limits the application and promotion of the model in resource-constrained scenarios such as edge devices.

[0003] Considering the limited performance of edge devices, it is generally necessary to quantize the large model before deploying it to the edge device to reduce storage occupancy and computing overhead. Current mainstream quantization techniques include: 1) Static quantization technology: such as the AWQ algorithm, which only quantizes the model weights, converting the model weight data from a high-precision data format to a 4-bit data format, greatly reducing the storage space requirement of the model weights at the storage level, but it only optimizes in the storage bandwidth dimension, and in the actual calculation process, it still uses the traditional floating-point calculation method, which does not substantially improve the calculation overhead of the model in the inference process, making it difficult to meet the efficient operation needs of low-power devices. For example, the SmoothQuant algorithm quantizes both the model weights and the activations, and by migrating the quantization difficulty of the activations to the weights, it makes the activations easy to quantize, and quantizes both the weights and the activations to 8 bits, but if the 4-bit weight quantization is performed directly, there will be a significant loss of precision.

[0004] 2) Dynamic quantization technology: adjusts the quantization factor of the activation data in real time according to the dynamic data distribution of the model during runtime, but this increases the computational burden on edge devices with limited computing resources.

[0005] As mentioned above, the existing technology faces the dilemma of being unable to simultaneously consider storage bandwidth optimization, computing efficiency improvement, and model accuracy when pursuing lightweight deployment of the model. SUMMARY

[0006] One of the purposes of the present application is to overcome at least one of the deficiencies in the prior art and provide a large language model quantization method, related equipment and computer program product.

[0007] The technical solutions provided by the present application are as follows: A large language model quantization method, comprising: using calibration data to perform inference on a large language model to be quantized, obtaining activations of the large language model, and counting activation distribution by channel; the activation distribution includes the maximum and minimum values of the activations; The smoothing factor of each channel of the target layer is calculated according to the activation distribution of the corresponding channel; The weight of the target layer is compensated channel by channel according to the smoothing factor, to obtain the compensated weight; The compensated weight is quantized by 4 bits; In the model inference process, the activation of the target layer is smoothed channel by channel according to the smoothing factor, and the smoothed activation is quantized by 16 bits.

[0008] In some embodiments, the calculation of the smoothing factor of the target layer channel j includes: According to the maximum and minimum values of the activation of the target layer channel j, the maximum value of the absolute value of the activation of the target layer channel j is obtained; According to the maximum and minimum values of the weight of the target layer channel j, the maximum value of the absolute value of the weight of the target layer channel j is obtained; The smoothing factor of the target layer channel j is calculated according to the following formula : ； Wherein, represents the activation of the target layer channel j, represents the weight of the target layer channel j, and a represents a migration factor.

[0009] In some embodiments, the weight of the target layer is compensated channel by channel, including: The weight of each channel is multiplied by the smoothing factor of the corresponding channel to obtain the compensated weight.

[0010] In some embodiments, before calculating the smoothing factor of each channel of the target layer according to the activation distribution of the corresponding channel, it includes: The target migration factor is determined according to the activation distribution and the weight distribution of the target layer in the calibration stage, specifically including: The migration factor is initialized; The output of the unquantized target layer is calculated by inputting the calibration data; The output of the quantized target layer is calculated according to the migration factor by inputting the calibration data, including: The smoothing factor of the target layer is obtained according to the migration factor, the activation of the target layer is smoothed and the weight is compensated according to the smoothing factor, and then quantized, and the output of the quantized target layer is obtained according to the quantized weight and activation; The loss caused by quantization is calculated by comparing the output of the quantized target layer with the output of the unquantized target layer; The target migration factor that makes the quantization loss converge is found by using gradient descent method.

[0011] In some embodiments, the migration factor is equal to sigmoid(θ), sigmoid() is a sigmoid activation function; the determination of the target migration factor comprises: initializing a parameter θ; inputting calibration data, calculating the output of the corresponding quantized target layer according to the parameter θ, comprising: obtaining the corresponding migration factor according to the parameter θ; obtaining the smoothing factor of the target layer according to the migration factor, respectively smoothing the activation of the target layer and compensating the weight according to the smoothing factor, then quantizing respectively, and obtaining the output of the quantized target layer according to the quantized weight and activation; comparing the output of the quantized target layer with the output of the unquantized target layer, calculating the loss caused by quantization; using gradient descent method to find the target parameter θ value that makes the quantization loss converge, and obtaining the target migration factor according to the target parameter θ value.

[0012] In some embodiments, the compensated weight is quantized to 4 bits, comprising: grouping the compensated weight, performing quantization by group, and using the following formula to quantize each group of compensated weight ; ; ； wherein, is the quantized value of , n and m are the minimum and maximum values of the integer data respectively, clip() is a clipping function, indicates that the value of y is limited between n and m, and are the quantization factors of the grouped and compensated weight, is a quantization threshold parameter; updating the quantization threshold parameter using gradient descent method , and the optimization target is ; wherein, is the regularization strength, indicates the Lp norm distance between the weight before quantization and the weight after quantization, p≧1, indicates the L2 norm of β, indicates the L2 norm of .

[0013] In some embodiments, the quantization threshold parameter is updated using gradient descent method , and the optimization target is ​ ; wherein, is the quantization sensitivity of the kth target layer, used to measure the degree of influence of the weight on the loss of the output of the kth target layer in the quantization process; The calculation is as follows: For the kth target layer: For the same input data, the L2 norm distance between the output of the unquantized target layer and the output of the target layer whose weight is only quantized is calculated, denoted as the quantization loss of the input data; The sum of the quantization losses of all calibration data in the calibration data set is counted, and divided by the sample number of the calibration data set to obtain the average quantization loss of the kth target layer; is equal to the average quantization loss.

[0014] The present application also provides an electronic device comprising: a memory for storing a computer program; a processor for implementing the quantization method of the large language model of any of the preceding embodiments when running the computer program.

[0015] The present application also provides a readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the quantization method of the large language model of any of the preceding embodiments.

[0016] The present application also provides a computer program product comprising a computer program, wherein the computer program, when executed by a processor, implements the steps of the quantization method of the large language model of any of the preceding embodiments.

[0017] The large language model quantization method, related device and computer program product provided by the present application can at least bring the following beneficial effects: 1、The present application adopts the quantization scheme of W4A16 (4-bit weight + 16-bit activation), which compresses the weight storage by half compared with W8A8, while controlling the precision loss; by performing weight quantization and activation factor calculation of the model offline in the calibration stage, the overhead of the model inference stage is reduced, thereby reducing the requirement for the operation capacity of the device, which is conducive to the deployment of large language models on edge devices.

[0018] 2、The present application uses an algorithm to quantize the weight to 4 bits, further controlling the precision loss of the model to be less than 3%.

[0019] 3. By selecting appropriate transfer factors based on the activation and weight distributions during the calibration phase, this invention can help to more accurately map the weights and activation information of the original model to a low-bit representation, reduce information loss, improve the accuracy of the quantization model, enhance the model's robustness to quantization noise, and improve the stability of quantization.

[0020] 4. By introducing a quantization sensitivity driving mechanism into the weight quantization, this invention can make minimizing the quantization error of the high-sensitivity layer a higher priority, making the overall weight after quantization closer to the weight before quantization, further reducing the total loss of quantization and improving quantization accuracy. Attached Figure Description

[0021] The preferred embodiments will be described below in a clear and easy-to-understand manner, with reference to the accompanying drawings, to further explain the above-mentioned characteristics, technical features, advantages, and implementation methods of a large language model quantization method, related equipment, and computer program products.

[0022] Figure 1 This is a flowchart of an embodiment of a quantization method for a large language model according to the present invention; Figure 2 This is a schematic diagram of the structure of an embodiment of an electronic device according to the present invention. Detailed Implementation

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the specific implementation methods of the present invention will be described below with reference to the accompanying drawings. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings and other implementation methods can be obtained based on these drawings without any creative effort.

[0024] To keep the drawings concise, only the parts relevant to the invention are shown schematically in each figure, and they do not represent the actual structure of the product. Furthermore, for ease of understanding, in some figures, components with the same structure or function are shown only schematically, or only one is labeled. In this document, "a" can mean not only "only one" but also "more than one".

[0025] One embodiment of the present invention, such as Figure 1 As shown, a quantization method for a large language model includes: Step S100 uses calibration data to perform inference on the large language model to be quantized, obtains the activation of the large language model, and statistically analyzes the activation distribution and weight distribution of the target layer by channel.

[0026] Specifically, large language models consist of several neural network layers. Generally, it is necessary to quantize the weights and activations of some of these neural network layers (such as attention layers and multilayer perceptron (MLP) layers) to reduce the resource consumption of deploying large language models. The neural network layer that needs to be quantized is called the target layer.

[0027] This invention employs a post-training quantization method to quantize large language models. During the calibration phase, the calibration dataset is... Input a large language model to be quantized (its target layer weight tensor is...) The activation of the model is obtained through forward reasoning. The activation of the target layer is collected along the channel dimension. Distribution and weight tensor The distribution of activations, by default, refers to the inputs of the neural network layers. The activation distribution includes the maximum and minimum values ​​of the activations, while the weight distribution includes the maximum and minimum values ​​of the weights.

[0028] Step S200 calculates the smoothing factor for each channel based on the activation distribution of each channel in the target layer.

[0029] Typically, the weights of a model are relatively evenly distributed and easy to quantize, while outliers exist in certain channels of the activation values, making activation more difficult to quantize. To reduce the difficulty of activation quantization, the idea of ​​the SmoothQuant algorithm is borrowed to calculate a smoothing factor. Based on the smoothing factor, the activation is smoothed and the weights are compensated, thereby transferring part of the difficulty in activation quantization to the quantization of the weights.

[0030] Taking channel j as an example, calculate the smoothing factor of the channel: Calculate the maximum absolute value of all activations in target layer channel j. ,for example, Calculate the maximum absolute value of all weights for channel j in the target layer. ,for example, Then, based on the maximum absolute value of the activation of channel j, the maximum absolute value of the weight, and the migration factor, the smoothing factor of channel j is calculated. Specifically, the smoothing factor of channel j can be obtained using the following formula. :

[0031] in, This indicates the activation of target layer channel j. α represents the weight of target layer channel j, and α represents the migration factor.

[0032] The migration factor α ranges between 0.0 and 1.0, and can be fixed at 0.5 or adjusted according to the actual activation and weight distributions. The migration factor α characterizes the migration strength; its value reflects the degree to which the quantization difficulty is transferred from the activation values ​​to the weights. Setting an appropriate α ensures that the quantization difficulty of the smoothed activations and compensated weights is balanced.

[0033] Step S300 compensates the weights of the target layer channel by channel according to the smoothing factor to obtain the compensated weights; Step S400 performs 4-bit quantization on the compensated weights.

[0034] Specifically, smoothing activations and weights channel by channel includes: smoothing activations channel by channel using a smoothing factor for each channel using the following formula: The following formula can be used to compensate for the weights on a channel-by-channel basis using the smoothing factor for each channel: .in, Activation of the smoothed channel j The weights are those of channel j after compensation. Smoothing transfers the error of the quantized activation values ​​to the weights, making the activation distribution of each channel more uniform and smooth.

[0035] The 4-bit standard quantization technique can be used to quantize the weights of the target layer after compensation.

[0036] Since the weights are fixed at the end of model training, while the activation is a value that can only be determined during online inference, the compensation and quantization of the weights can be processed offline during the calibration phase, while the smoothing of the activation and the quantization of the smoothed activation are performed during the model inference process with the input actual data.

[0037] In step S500, the activation of the target layer is smoothed channel by channel according to the smoothing factor during the model inference process, and the smoothed activation is quantized in 16 bits.

[0038] During the inference phase, as mentioned earlier, the activation of the target layer is smoothed channel by channel according to the smoothing factor of each channel to obtain the smoothed activation.

[0039] Next, perform 16-bit quantization on the smoothed activation. A 16-bit standard quantization technique can be used, as shown in the following formula: ; in, The activation value to be quantized. Let X be the quantized value. This indicates the rounding operation. , The quantization factor for activation.

[0040] If asymmetric quantization is used, the activated quantization factor , for: .

[0041] If symmetric quantization is used, then the activated quantization factor , for: .

[0042] Since the weight quantization values ​​of the large language model have been obtained offline during the calibration phase, we replace the weights of the large language model with the corresponding quantization values, and at the same time, we perform activation quantization in the inference process of the actual data. The resulting model is called the quantized model of the large language model.

[0043] The inference process of the quantization model is as follows: First, the actual data is input and processed through several layers of neural network operations (e.g., using floating-point operations) to the target layer. The input activation values ​​of the target layer are smoothed channel by channel to eliminate outliers (i.e., smoothing the activations using a pre-calculated smoothing factor). Then, the smoothed activation values ​​are quantized into 16-bit integers. Next, using a dedicated hardware accelerator, the 16-bit quantized values ​​of the smoothed activations are multiplied and added to the 4-bit quantized values ​​of the compensated weights. Finally, the calculation results are restored to their numerical range using the smoothing factor.

[0044] If the next network layer is a quantized layer (i.e., the weights of this layer use integer quantization values, and the operations of this layer use integer operations), then repeat the above steps. If the next network layer is a floating-point layer (i.e., the weights of this layer use floating-point numbers, and the operations of this layer use floating-point operations), then dequantize the calculation result to a floating-point number, such as FP16 format, and continue the calculation.

[0045] In one embodiment, the activation quantization factor is calculated based on the activation distribution of the target layer obtained during the calibration phase.

[0046] This allows for offline calculation of the activated quantization factor during the calibration phase, and the pre-calculated activated quantization factor can be directly used during the inference phase, further reducing the processing overhead during the model inference phase and matching the computing power of edge devices.

[0047] In this embodiment, by performing weight quantization and activation quantization factor calculation offline during the calibration phase, the overhead of the model inference phase can be reduced; by implementing 4-bit quantization on the weights, the storage bandwidth of the model is reduced, and by implementing 16-bit quantization on the activations, the typical multiply-accumulate-integer operations in neural networks are completed using a dedicated hardware accelerator during the inference process, which ensures the accuracy of the model while matching the computing power of edge devices.

[0048] In one embodiment, the AutoRound algorithm is used to perform 4-bit quantization on the compensated weights of the target layer, specifically including: For the compensated weights Grouping, quantization is performed on each group, and the compensated weights for each group are calculated using the following formula. Quantification: ; ; ； in, for The quantized value, clip() is the clipping function, and n and m are the minimum and maximum values ​​of the integer data, respectively. This means that the value of y is restricted to between n and m. and These are the quantization factors of the weights after group compensation. Here, the quantization threshold parameter is a learnable parameter. Used to control the value of floating-point integers rounded, typically between -0.5 and 0.5; Quantization factor used to control weights It is generally between 0.5 and 1.0.

[0049] Next, the gradient descent method is used to update the quantization threshold parameter. The optimization objective is: .

[0050] in, For regularization strength, Represents the weights before quantization With quantized weights The Lp norm distance between them, p≧1, for example, p=2. Denotes the L2 norm of β. express The L2 norm. The optimizer uses Signed Stochastic Gradient Descent (SignSGD) to quickly explore the entire space for each parameter until the final iteration converges.

[0051] In this embodiment, the AutoRound algorithm is used to quantize the weights to 4 bits, which further optimizes the loss caused by the 4-bit integer quantization of the weights and improves the accuracy of the model.

[0052] Performance comparison of the quantization model provided in this embodiment with existing technologies:

[0053] As shown in the table above, this invention achieves weight compression to 4 bits while maintaining model accuracy loss of <3% through offline joint optimization of SmoothQuant and AutoRound. This reduces bandwidth by 75% compared to FP16 and by another 50% compared to W8A8. The calculation is converted to hardware-accelerated INT16 integer operations, avoiding floating-point calculations in the W4 scheme. The offline pre-calculation of weight quantization and activation quantization factors avoids the real-time burden caused by dynamic quantization, provides controllable accuracy assurance, and forms an error cancellation closed loop of activation outlier migration and weight grouping optimization. This meets the stringent requirements of edge devices for storage bandwidth, computing efficiency, and model performance.

[0054] In one embodiment, the gradient descent method is used to update the quantization threshold parameter. When, the optimization objective is defined as: ; in denoted as the quantization sensitivity of the k-th target layer.

[0055] For the k-th target layer: For the same input data, calculate the L2 norm distance between the output of the unquantized target layer and the output of the target layer with only quantized weights, and denote it as the quantization loss of the input data; sum the quantization losses of all calibration data in the calibration dataset, divide it by the number of samples in the calibration dataset, and obtain the average quantization loss; It equals the average quantization loss.

[0056] This metric measures the impact of weight quantization on the loss of the k-th network output (i.e., the output of the k-th target layer). If quantization of a weight layer causes a significant change in the network output, then... The larger the value, the higher the sensitivity; conversely, if it causes a small change in the network output, then... The smaller the value, the lower the sensitivity.

[0057] Since the quantization loss of the sensitive layer accounts for a larger proportion of the total loss, more attention is paid to optimizing the sensitive layer, and this is incorporated into the optimization objective. This makes minimizing the quantization error of the highly sensitive layer a higher priority, making the quantized weights closer to the unquantized weights overall, and reducing the total loss of quantization.

[0058] The above process can be based on calibration data, so it can be performed offline during the calibration phase.

[0059] In one embodiment, the migration factor α is not set to a fixed empirical value, but rather a better α value is determined based on the activation distribution and weight distribution of the target layer during the calibration phase, so as to achieve a balance between the difficulty of activation quantization and weight quantization.

[0060] The optimization of the migration factor specifically includes the following steps: 1) Initialize the migration factor α, for example, to 0.5.

[0061] 2) Input calibration data and calculate the output of the unquantized target layer.

[0062] 3) Input calibration data and calculate the corresponding quantized target layer output based on the migration factor α, including: 3.1) Obtain the smoothing factor of the target layer based on the migration factor α. Smooth the activation of the target layer and compensate the weights based on the smoothing factor. Then quantize them separately (e.g., 4-bit quantization for weights and 16-bit quantization for activations, which must be consistent with the subsequent quantization dimensions). Obtain the output of the quantized target layer based on the quantized weights and activations.

[0063] 4) Compare the output of the quantized target layer with the output of the unquantized target layer, and calculate the loss caused by quantization. The mean squared error loss (MSE Loss) can be used to represent this.

[0064] 5) Use gradient descent to find the target migration factor α that makes the quantization loss converge (i.e., the loss value no longer decreases significantly).

[0065] The above process can be performed offline during the calibration phase. The appropriate transfer factor for each layer of the neural network to be quantized can be obtained through this process. Alternatively, several adjacent layers to be quantized can be grouped together to form a block, and the appropriate transfer factor can be obtained through the above process on a block-by-block basis. This can accelerate the optimization of the transfer factor.

[0066] For example, the attention layer and the MLP layer can be combined into a single block, and the transfer factor can be optimized on a block-by-block basis.

[0067] Considering that the migration factor α needs to be between 0.0 and 1.0, but the above process cannot guarantee that the optimized value will fall within the range of 0 to 1, it may appear outside the range of 0 to 1. In order to ensure that the optimized value of the migration factor α falls within the range of 0 to 1, in one embodiment, the optimization of the α value is improved as follows: Let the migration factor α equal to Here, sigmoid() is a sigmoid activation function, whose output ranges from 0 to 1. The parameter θ is used as a learning parameter, and optimization is performed using gradient descent. The optimizer can be the Adam optimizer, an adaptive moment estimator.

[0068] The optimization process for parameter θ is as follows: 1) Initialize the parameter θ, for example, set it to 0.

[0069] 2) Input calibration data and calculate the output of the unquantized target layer.

[0070] 3) Input calibration data and calculate the corresponding quantized target layer output based on parameter θ, including: 3.1) Obtain the corresponding migration factor α based on the parameter θ; 3.2) Obtain the smoothing factor of the target layer based on the migration factor α, smooth the activation of the target layer and compensate the weights based on the smoothing factor, then quantize them respectively, and obtain the output of the quantized target layer based on the quantized weights and activations.

[0071] 4) Compare the output of the quantized target layer with the output of the unquantized target layer, and calculate the loss caused by quantization.

[0072] 5) Use gradient descent to find the target parameter θ that makes the quantization loss converge (i.e. the loss value no longer decreases significantly), and obtain the target migration factor α based on the target parameter θ.

[0073] Similarly, the target parameter θ value can be obtained for each neural network layer to be quantized or for each block, and then the target transfer factor α can be obtained based on the target parameter θ value.

[0074] One embodiment of the present invention, such as Figure 2 As shown, an electronic device 100 includes a memory 110 and a processor 130; Memory 110 is used to store computer program 120; The processor 130 is used to implement the quantization method of the large language model as described in any of the foregoing embodiments when running the computer program 120.

[0075] Memory can be any internal storage unit and / or external storage device capable of storing data and programs. For example, memory can be a plug-in hard drive, a smart memory card (SMC), a flash memory card, an SD card, etc.

[0076] Depending on the requirements, the processor can be a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a general-purpose processor, or other logic devices.

[0077] This application also provides a computer-readable storage medium carrying a computer program. When the computer program is executed by an electronic device, it enables the electronic device to implement any of the large language model quantization methods provided in this application.

[0078] This application also provides a computer program product, which includes a computer program that, when run on an electronic device, enables the electronic device to implement any of the large language model quantization methods provided in this application.

[0079] It should be noted that the above embodiments can be freely combined as needed. The above description is only a preferred embodiment of the present invention. It should be pointed out that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A quantization method for a large language model, characterized in that, include: The calibration data is used to perform inference on the large language model to be quantized, the activation of the large language model is obtained, and the activation distribution of the target layer is statistically analyzed by channel. The activation distribution includes the maximum and minimum activation values; The smoothing factor for each channel is calculated based on the activation distribution of each channel in the target layer. The weights of the target layer are compensated channel by channel according to the smoothing factor to obtain the compensated weights; The compensated weights are quantized using 4 bits. During model inference, the activation of the target layer is smoothed channel by channel according to the smoothing factor, and the smoothed activation is quantized in 16 bits.

2. The quantization method for large language models according to claim 1, characterized in that, The calculation of the smoothing factor for target layer channel j includes: Based on the maximum and minimum activation values ​​of the target layer channel j, the maximum absolute value of the activation of the target layer channel j is obtained; Based on the maximum and minimum values ​​of the weights of the target layer channel j, the maximum absolute value of the weights of the target layer channel j is obtained; The smoothing factor of the target layer channel j is calculated according to the following formula. : ； in, This indicates the activation of target layer channel j. α represents the weight of target layer channel j, and α represents the migration factor.

3. The quantization method for large language models according to claim 2, characterized in that, Compensation is performed on the weights of the target layer channel by channel, including: The weight of each channel is multiplied by the corresponding channel's smoothing factor to obtain the compensated weight.

4. The quantization method for large language models according to claim 2, characterized in that, Before calculating the smoothing factor for each channel based on the activation distribution of each channel in the target layer, the process includes: The target migration factor is determined based on the activation distribution and weight distribution of the target layer during the calibration phase, specifically including: Initialize the migration factor; Input the calibration data and calculate the output of the unquantized target layer; Input the calibration data, and calculate the corresponding quantized target layer output based on the migration factor, including: The smoothing factor of the target layer is obtained based on the migration factor. The activation of the target layer is smoothed and the weights are compensated based on the smoothing factor. Then, they are quantized. The output of the target layer is obtained based on the quantized weights and activations. Compare the output of the quantized target layer with the output of the unquantized target layer, and calculate the loss caused by quantization; The gradient descent method is used to find the target migration factor that makes the quantization loss converge.

5. The quantization method for large language models according to claim 4, characterized in that, Let the migration factor be equal to sigmoid(θ), where sigmoid() is a sigmoid activation function; The determination of the target migration factor includes: Initialize parameter θ; Input the calibration data, and calculate the corresponding quantized target layer output based on the parameter θ, including: The corresponding migration factor is obtained based on the parameter θ. The smoothing factor of the target layer is obtained based on the migration factor. The activation of the target layer is smoothed and the weights are compensated based on the smoothing factor. Then, they are quantized. The output of the quantized target layer is obtained based on the quantized weights and activations. Compare the quantized output of the target layer with the unquantized output of the target layer, and calculate the loss caused by quantization; The gradient descent method is used to find the target parameter θ value that makes the quantization loss converge, and the target migration factor is obtained based on the target parameter θ value.

6. The quantization method for large language models according to claim 2, characterized in that, The compensated weights are quantized to 4 bits, including: The compensated weights are grouped, and quantization is performed on each group. The following formula is used to calculate the compensated weights for each group. Quantification: ； ； ； in, for The quantized value, n and m are the minimum and maximum values ​​of the integer data, respectively, and clip() is the clipping function. This means that the value of y is restricted to between n and m. and This is the quantization factor for the weights after group compensation. This is the quantization threshold parameter; Update the quantization threshold parameter using gradient descent. The optimization objective is ； in, For regularization strength, This represents the Lp norm distance between the weights before and after quantization, where p ≥ 1. Denotes the L2 norm of β. express The L2 norm.

7. The quantization method for a large language model according to claim 6, characterized in that, Update the quantization threshold parameter using gradient descent. The optimization objective is ； in, is the quantization sensitivity of the k-th target layer, used to measure the degree of influence of the weights on the loss of the k-th network output during the quantization process; The calculation is as follows: For the k-th target layer: For the same input data, calculate the L2 norm distance between the output of the unquantized target layer and the output of the target layer with only quantized weights, and denote it as the quantization loss of the input data; The average quantization loss is obtained by summing the quantization losses of all calibration data in the statistical calibration dataset and dividing it by the number of samples in the calibration dataset. It equals the average quantization loss.

8. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for implementing the quantization method of a large language model as described in any one of claims 1 to 7 when running a computer program.

9. A readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the quantization method for a large language model as described in any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the quantization method for a large language model as described in any one of claims 1 to 7.

Images