Non-linear function quantization method in transformer compiler

Abstract

The application discloses a nonlinear function quantization method in a Transformer compiler. The application adopts a quantization method to quantize and compress a transformer and a derivative model thereof, and obtains quantized weights and quantized parameters through a quantization-aware training (QAT) method. In the quantization process, a segmented linear approximation method is used to calculate a nonlinear function. In the determination of the segmented boundary, the QAT method is also used, and the linear approximation function of the nonlinear function is expressed as a differentiable function, so that it participates in the back propagation of the QAT, thereby obtaining the optimal segmented boundary combination. The application can be applied to the multi-platform deployment of a BERT model, and is especially suitable for a deployment platform such as FPGA which is suitable for rule calculation. In the quantization process, the method of the application can improve the hardware efficiency and maximize the accuracy.

Description

Technical Field

[0001] This invention belongs to the field of signal processing technology, specifically, it relates to a nonlinear function quantization method in a Transformer compiler. Background Technology

[0002] Neural networks are a method for transforming complex real-world problems into computable ones. Many real-world problems, such as image recognition and natural language processing, cannot be precisely calculated and analyzed using interpretable theories, but their computational processes can be simulated using a large number of parameters, resulting in non-interpretable computational models—that is, neural networks. In recent years, with the rapid development of computer equipment, the massive computational demands of neural networks have been met, ushering in a golden age of rapid development for neural networks. In 2017, the invention of the Transformer propelled the development of large-scale neural networks to a new peak. Models based on Transformers, such as GPT, BERT, and GPT-2, have brought the technological level of the field of natural language processing to unprecedented heights.

[0003] However, in many specific application scenarios, such as mobile devices, edge devices, and the Internet of Things (IoT), computing and storage resources are limited, while low power consumption and high inference speed are required. Large-scale neural networks are difficult to apply directly to these scenarios. Therefore, model compression methods are needed to reduce the model size, computational load, and storage resource consumption while maintaining model accuracy, thereby reducing power consumption and inference latency. Quantization is one method of model compression. In conventional neural network computation, data (including weights, biases, intermediate results, etc.) is often stored and computed using FP32 (full precision) or FP16 (half precision) data types. However, using quantization methods, this data can be converted into 8-bit integers. This method reduces the storage space and computing resource consumption required by the model, enabling the model to meet the requirements of running on edge devices.

[0004] Model quantization has two main impacts: it reduces model size but increases accuracy loss. Direct quantization without modification often has a devastating effect on model accuracy. Therefore, a method is needed to maintain model accuracy during quantization. In the inference process of quantized models, the calculation results of nonlinear functions often have a more significant impact on the final result, but currently, there is a lack of efficient, convenient, and universal nonlinear function quantization methods. The paper "I-BERT: Integer-only BERT quantization" published by Sehoon Kim et al. in the 2021 Proceedings of Machine Learning Research (PMLR) uses quantization to compress the Transformer-derived model BERT. The authors employ a full quantization scheme. Specifically, for linear layers in the model, weights and activations are compressed to 8 bits. For nonlinear layers, the authors use approximation functions to replace the nonlinear functions in the model, achieving performance improvements. However, the approximation function logic used in this work is relatively complex. When deployed in terminal devices such as FPGAs, each different nonlinear function needs to be configured and processed separately, thus lacking universality.

[0005] The paper "NPE: An FPGA-based Overlay Processor for Natural Language," published by Hamza Khan et al. at the 2021 International Symposium on Field-Programmable Gate Arrays (FPGA), uses linear fitting to compute nonlinear functions in transformers and their derived models. However, while approximating the nonlinear functions piecewise, the paper does not specify the specific piecewise strategy, which may lead to unpredictable losses in the final results. Summary of the Invention

[0006] To address the technical problems existing in the prior art, the present invention aims to provide a novel method for quantizing nonlinear functions in a Transformer compiler for FPGAs (Field Programmable Gate Arrays) to quantize and compress Transformer and its derived models. This invention obtains quantization weights and parameters through quantization-aware training (QAT). During the quantization process, a piecewise linear approximation method is used to calculate the nonlinear function. The determination of the piecewise boundaries also employs QAT, representing the linear approximation function of the nonlinear function as a differentiable function, which participates in the backpropagation of QAT. By optimizing the differentiable function, the optimal combination of piecewise boundaries is obtained. This method is universal and can uniformly configure nonlinear functions in the Transformer compiler. Furthermore, the piecewise linear approximation method results in more accurate quantization results.

[0007] The technical solution of the present invention is described in detail below.

[0008] A nonlinear function quantization method for a Transformer compiler for FPGAs is proposed. Since BERT model training consists of two steps—pre-training and fine-tuning—and this method quantizes the model during the second fine-tuning step, it assumes that a pre-trained BERT model has already been obtained. The method includes the following steps:

[0009] 1) Deploy the BERT pre-trained model on the FPGA platform, connect different classifier layers to the pre-trained model according to the different tasks to be fine-tuned, and initialize the parameters of the classifier;

[0010] 2) Quantization model: Quantize the weights, biases and activations of the model. The specific steps are: calculate the mean square error between the quantized parameters and the original parameters under different quantization coefficients based on the parameter values, and take the quantization coefficient with the minimum mean square error to obtain the corresponding quantized parameters.

[0011] 3) Forward Propagation: Substitute the sample input corresponding to the fine-tuning task, and the quantization parameters obtained in the previous step, calculate the cross-entropy between the final result and the sample label, and use it as the loss function. The nonlinear functions involved in the model are replaced by linear approximations. The method for making linear approximations of the nonlinear functions in the model is as follows:

[0012] Divide the domain of the nonlinear function into equally spaced parts, and at the division points (x... i ,y i ), (x i+1 ,y i+1 Between ), the linear approximation expression is:

[0013]

[0014] 4) Backpropagation: Calculate the gradient between the loss function and the model parameters, and use gradient descent to compute the new parameters. For nonlinear functions, use a differentiable linear approximation function to calculate the gradient of the loss function with respect to the split boundary and the input parameters. (Interval point x) i and x i+1 The differentiable linear approximation function between them is expressed as:

[0015]

[0016] Where α represents the similarity between the differentiable linear function and the original linear function, the larger α is, the higher the similarity, and kx+b is the analytical expression of the linear function;

[0017] Find the optimal split boundary based on the gradient;

[0018] 5) After a finite number of iterations, the convergent quantization model parameters and the corresponding nonlinear segmentation boundary are obtained.

[0019] In this invention, in step 2), the nonlinear function includes the exponential function, the arithmetic square root function, and the activation function GELU.

[0020] In this invention, in step 2), the initial linear approximate boundary value of each nonlinear function is obtained by uniform segmentation.

[0021] In this invention, during the inference and deployment process, the quantization parameters are deployed on the corresponding platform. On the FPGA, the nonlinear function is deployed in the form of a read-only memory (ROM), and the optimized segmentation boundary is stored in the ROM.

[0022] Compared with existing technologies, the advantages of this invention are as follows: This invention can be applied to multi-platform deployment of BERT models, especially suitable for FPGAs, which are suitable for rule-based computation. Because it uses quantization to compress the model and linear approximation to calculate nonlinear functions, it is highly beneficial for FPGA deployment. Since this invention obtains QAT-optimized quantization parameters and segmentation boundaries, and uses the method of this invention during the quantization process, it can improve hardware efficiency while preserving accuracy to the greatest extent. Attached Figure Description

[0023] Figure 1 It is the graph of an exponential function.

[0024] Figure 2 It is a linear approximation of the image.

[0025] Figure 3 It is the graph of a differentiable linear function.

[0026] Figure 4 It compares a combination of differentiable linear functions with an exponential function. Detailed Implementation

[0027] The technical solution of the present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0028] The overall flow of the quantization method proposed in this invention is as follows:

[0029] 1. Obtain the pre-trained BERT model;

[0030] 2. There are three types of nonlinear functions in BERT: exponential function, arithmetic square root function and activation function GELU. The second step is to divide the initial value of the linear approximation boundary of each nonlinear function. In this invention, this step usually adopts a uniform division strategy.

[0031] 3. Use QAT to fine-tune the model. During the fine-tuning process, the nonlinear function is forward-propagated with the result of a linear approximation. When calculating the gradient, the nonlinear function is represented by a differentiable linear approximation. The gradient of the loss function with respect to the linear fitting boundary is calculated, and then the gradient descent method is used to backpropagate it to obtain the optimal combination of linear approximation boundaries.

[0032] The core of this invention lies in the linear approximation of nonlinear functions; in this embodiment, an exponential function is used as an example. Assuming the range of the exponential function in the calculation is -1 to 1, the exponential function is as follows: Figure 1 As shown.

[0033] During the forward propagation, the exponential function is calculated using a linear approximation method. Taking a linear approximation interval of 0.5 as an example, the corresponding image is as follows: Figure 2 As shown.

[0034] During the backpropagation process, the interval point x i and x i+1 The analytical expression of the differentiable linear function between them is:

[0035]

[0036] Here, α determines the similarity between the differentiable linear function and the original linear function; the larger α is, the higher the similarity. kx+b is the analytical expression of the linear function. The graph of the above differentiable linear function is shown below. Figure 3 As shown. A combination of all differentiable linear functions, such as Figure 4 As shown. In differentiable linear functions, we can calculate the result with respect to the interval point x. i and x i+1 The gradient can then be obtained by optimizing the interval point x. i and x i+1 The numerical value is used to maximize the accuracy of linear fitting.

Claims

1. A nonlinear function quantization method in a Transformer compiler for FPGAs, characterized in that, Includes the following steps: 1) Deploy the BERT pre-trained model on the FPGA platform, connect different classifier layers to the pre-trained model according to the different tasks to be fine-tuned, and initialize the parameters of the classifier; 2) Quantization model: Quantize the weights, biases and activations of the model. The specific steps are: calculate the mean square error between the quantized parameters and the original parameters under different quantization coefficients based on the parameter values, and take the quantization coefficient with the minimum mean square error to obtain the corresponding quantized parameters. 3) Forward propagation: Substitute the sample input corresponding to the fine-tuning task and the quantization parameters obtained in the previous step to calculate the cross-entropy between the final result and the sample label, which is used as the loss function; The nonlinear functions involved in the model are replaced by linear approximations; 4) Backpropagation: Calculate the gradient between the loss function and the model parameters, and use the gradient descent method to calculate the new parameters; for nonlinear functions, use a differentiable linear approximation function, calculate the gradient of the loss function with respect to the segmentation boundary and the input parameters, and find the optimal segmentation boundary based on the gradient. 5) After a finite number of iterations, the convergent quantization model parameters and the corresponding nonlinear segmentation boundary are obtained.

2. The nonlinear function quantization method according to claim 1, characterized in that, In step 3), the nonlinear functions include the exponential function, the arithmetic square root function, and the activation function GELU.

3. The nonlinear function quantization method according to claim 1, characterized in that, In step 3), the method for linear approximating the nonlinear function is as follows: Divide the domain of the nonlinear function into equally spaced parts, and at the division points (x... i ,y i ), (x i+1 ,y i+1 Between ), the linear approximation expression is:

4. The nonlinear function quantization method according to claim 1, characterized in that, In step 4), the interval point x i and x i+1 The analytical expression of the differentiable linear function between them is: Where α represents the similarity between the differentiable linear function and the original linear function, the larger α is, the higher the similarity, and kx+b is the analytical expression of the linear function.

Images