An FPGA acceleration method for normalized nonlinear functions in BERT layers
By applying asymmetric and symmetric quantization formulas in the BERT model, the floating-point operations of the layer normalization module are transformed into integer operations and shift operations. The algorithm is then deployed on an FPGA, which solves the problems of high latency and inaccurate quantization strategy of the BERT model on edge devices, thereby accelerating the model and optimizing resources.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-09-19
- Publication Date
- 2026-04-03
AI Technical Summary
In existing technologies, the deployment of BERT models on edge devices is limited by the high latency of the layer normalization module and the lack of accuracy and low computational efficiency of the quantization strategy, resulting in excessive computational complexity and resource requirements.
Asymmetric and symmetric quantization formulas are used to process the layer normalization module, converting floating-point operations into integer operations and shift operations. The algorithm is deployed on the FPGA, and the calculation of the layer normalization module is optimized by quantization formulas to improve the computational efficiency and accuracy.
While ensuring acceptable accuracy loss, it significantly improves the running speed and hardware resource utilization efficiency of the BERT model, and reduces hardware resource requirements.
Smart Images
Figure CN115965062B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer algorithm technology, and in particular relates to an FPGA acceleration method for the normalized nonlinear function in the middle layer of BERT. Background Technology
[0002] BERT (Bidirectional Encoder Representations from Transformers) is a pre-trained model that combines the advantages of Transformers to generate deep bidirectional language representation models. After fine-tuning by adding an extra layer, it can achieve excellent results in multiple Natural Language Processing (NLP) tasks.
[0003] BERT's high performance comes at the cost of high computational complexity and a large number of parameters, making it extremely difficult to deploy BERT on edge devices with limited computing and storage resources while meeting real-time requirements. The Layer Normalization (LN) module, a crucial component of BERT, accounts for a significant portion of the overall BERT latency due to its non-linear nature and its frequent occurrence within the module. Therefore, accelerating the Layer Normalization module in BERT is essential.
[0004] Quantization is a common method in model compression and acceleration schemes. Quantization involves converting floating-point numbers into integers (int) through a functional relationship, and then using these integers in the model for computation. Deploying the algorithm on a Field Programmable Gate Array (FPGA) can accelerate the FPGA. Currently, a common quantization method is to convert 32-bit floating-point numbers into int8 integers (i.e., 8-bit signed integers). This reduces the bit width to one-quarter of the original 32-bit floating-point number, thus significantly compressing the model size and accelerating the model.
[0005] In existing technologies, although quantization methods have accelerated LN, their scaling factors are usually set to constants, resulting in a lack of precision and low computational efficiency in their quantization strategies. Summary of the Invention
[0006] To address the problems of existing technologies, this invention provides an FPGA acceleration method for normalized nonlinear functions in BERT layers, which has the advantages of high accuracy and high computational efficiency, and solves the problems of lack of accuracy and low computational efficiency of existing normalization strategies.
[0007] This invention is implemented as follows: an FPGA acceleration method for normalized nonlinear functions in BERT layers, comprising the following steps:
[0008] Asymmetric and symmetric quantization formulas are obtained through quantization formulas;
[0009] Using the asymmetric quantization formula and the symmetric quantization formula, the layer normalization operation module is processed by the quantization method to obtain an algorithm that converts floating-point operations into integer operations and shift operations.
[0010] The algorithm is deployed on a field-programmable gate array (FPGA). While ensuring acceptable accuracy loss, the layer normalization module is compressed and accelerated, ultimately improving the running speed of the entire BERT model.
[0011] As a preferred embodiment of the present invention, the method for obtaining asymmetric and symmetric quantization formulas through quantization formulas includes:
[0012] The quantization formula is set as: r = S(qZ);
[0013] Where r is a floating-point number, q is the integer corresponding to the floating-point number, S represents the ratio between the floating-point number and the integer, and Z represents the integer corresponding to 0 in the real number after quantization.
[0014] Using r = S(qZ) as the asymmetric quantization formula, we obtain the asymmetric quantization formula r = S(qZ);
[0015] The formula for integers corresponding to floating-point numbers is: Among them, round(·) is the rounding operation;
[0016] Set the formulas for S and Z:
[0017]
[0018] Symmetric quantization is used, i.e., r max =-r min ,q max =2 B-1 -1,q min =-2 B Where B is the number of bits in the quantized integer, substituting it into the formula... We can obtain Z = 0. Substituting Z = 0 into the asymmetric quantization formula r = S(qZ), we obtain the symmetric quantization formula.
[0019] r = Sq.
[0020] As a preferred embodiment of the present invention, for the integer values obtained after quantization, a clipping operation clip(x, min, max) is also performed to limit the integer corresponding to the floating-point number to the interval (min, max). If x is greater than max, then x is set to max; if x is less than min, then x is set to min; thus obtaining...
[0021]
[0022] As a preferred embodiment of the present invention, the method of using the asymmetric quantization formula and the symmetric quantization formula to process the layer normalization operation module with a quantization method to convert floating-point operations into integer operations and shift operations includes:
[0023] The layer normalization formula is obtained as follows:
[0024]
[0025] Where, x i Let be a value in the input vector X, where the subscript i indicates the index; μ is the average value of the input vector X; σ 2 is the variance of the input vector X; ∈ is a very small preset value used to prevent σ 2 Too small, resulting in a denominator of the fraction being 0; in this paper, it is set to 0.01. For the normalized result vector One of the values, where the subscript i represents the sequence number; γ i β is a value in the gain vector γ, where the subscript i indicates the index; i And a value in the offset vector β, where the subscript i represents the index; y i It is a value in the result vector Y, and the subscript i represents the index.
[0026] Symmetric quantization is performed on parameters γ and β, and asymmetric quantization is performed on input X and output Y;
[0027] Substituting the asymmetric quantization formula r = S(qZ) and the symmetric quantization formula r = Sq into... β i We can obtain:
[0028]
[0029] in,
[0030] From the formula It can be seen that the zeros of the input matrix have no effect on the operation; after the transformation,
[0031] It can be transformed into:
[0032]
[0033] Where, q γ , q μ q σ 2 q β Z Y All are integers. The calculation using Newton's iteration method also yields an integer; therefore... Only in China and Both items are floating-point numbers;
[0034] Convert floating-point arithmetic into integer arithmetic and perform bit shift operations;
[0035] The ratio of the two floating-point scaling factors and The following approximation is made to improve computational efficiency while keeping the error acceptable:
[0036]
[0037] Where M0 and M1 are both integers, and This can be achieved by right shifting by n0 and n1 bits;
[0038] Will and Substitution
[0039] In the middle, we get:
[0040]
[0041] As a preferred embodiment of the present invention, under the premise that the error is acceptable, in order to improve the computational efficiency, let n0 = n1 = n, then
[0042] It can be rewritten as:
[0043]
[0044] As a preferred embodiment of the present invention, the method for deploying the algorithm on an FPGA includes:
[0045] Deploy the algorithm onto an FPGA;
[0046] Enables the algorithm to run on an FPGA;
[0047] Verify the algorithm's running status on the FPGA.
[0048] As a preferred embodiment of the present invention, the method for deploying the algorithm on an FPGA includes the following steps:
[0049] The algorithm is programmed to obtain the corresponding programming information;
[0050] Check and modify the programming information corresponding to the algorithm;
[0051] Check the working status of the FPGA. If the FPGA can work normally, deploy the programming information corresponding to the algorithm onto the FPGA.
[0052] Verify that the programming information can run normally in the FPGA; if it cannot run normally, return to the step of checking and modifying the programming information and repeat until the programming information can run normally in the FPGA.
[0053] Test the running speed of BERT.
[0054] As a preferred embodiment of the present invention, the method for programming the algorithm to obtain the corresponding programming information includes the following steps:
[0055] The algorithm is programmed into pseudocode corresponding to the algorithm.
[0056] Based on the pseudocode corresponding to the algorithm and the requirements of the programming language, the pseudocode corresponding to the algorithm is compiled into the formal code corresponding to the algorithm, and the formal code corresponding to the algorithm is used as the programming information corresponding to the algorithm.
[0057] As a preferred embodiment of the present invention, the hardware resources are gradually reduced multiple times; the running speed of BERT is detected; and a balance range between hardware resources and BERT running speed is found.
[0058] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0059] This invention uses a quantization method to process the layer normalization module and deploys it on an FPGA. While ensuring acceptable accuracy loss, the layer normalization module is compressed and accelerated, ultimately improving the running speed of the entire BERT model. Attached Figure Description
[0060] Figure 1 This is a flowchart illustrating an FPGA acceleration method for a BERT layer normalized nonlinear function provided in an embodiment of the present invention.
[0061] Figure 2 This is a flowchart illustrating the method for obtaining asymmetric and symmetric quantization formulas through quantization formulas according to an embodiment of the present invention.
[0062] Figure 3This is a flowchart illustrating a method provided in this embodiment of the invention that uses asymmetric and symmetric quantization formulas to process the layer normalization operation module using quantization methods, converting floating-point operations into integer operations and shift operations.
[0063] Figure 4 This is a flowchart illustrating the method for deploying an algorithm onto an FPGA according to an embodiment of the present invention.
[0064] Figure 5 This is a flowchart illustrating the method for deploying an algorithm onto an FPGA according to an embodiment of the present invention.
[0065] Figure 6 This is a flowchart illustrating a method for programming an algorithm to obtain corresponding programming information, as provided in an embodiment of the present invention.
[0066] Figure 7 This is a schematic diagram of the pseudocode of the algorithm provided in the embodiments of the present invention.
[0067] Figure 8 This is a schematic diagram of the process for finding a balance between hardware resources and BERT running speed provided in an embodiment of the present invention. Detailed Implementation
[0068] To further understand the invention's content, features, and effects, the following embodiments are provided, and detailed descriptions are given in conjunction with the accompanying drawings.
[0069] In existing technologies, the literature (Li J, Alvarez R. On the quantization of recurrent neural networks[J]. arXiv preprint arXiv:2101.05453,2021.) quantizes LN. However, the literature directly sets the scaling factor of the input X to a constant 2. -10 And directly set the scaling factor S of the bias coefficient β β =2 -10 ·S γ S γ The scaling factor for the weighting coefficient γ is used; furthermore, the paper does not handle the nonlinear operation of the square root. These two shortcomings result in the method being suboptimal in both accuracy and computational efficiency.
[0070] The paper (Kim S, Gholami A, Yao Z, et al. I-bert: Integer-only bert quantization[C]. International conference on machine learning. PMLR, 2021:5506-5518.) proposes to approximate the square root operation using Newton's iteration method and return an integer, thus making all calculations of the LN function integer operations. However, this paper only discusses the approximate calculation of the square root operation in LN, and does not provide detailed quantization strategies for other parts of LN.
[0071] The structure of the present invention will now be described in detail with reference to the accompanying drawings.
[0072] Please see Figure 1 An FPGA acceleration method for the normalized nonlinear function in BERT provided by this invention includes the following steps:
[0073] Step S1: Obtain the asymmetric quantization formula and the symmetric quantization formula through the quantization formula;
[0074] Step S2: Using the asymmetric quantization formula and the symmetric quantization formula, the layer normalization operation module is processed by the quantization method to obtain an algorithm that converts floating-point operations into integer operations and shift operations.
[0075] Step S3: Deploy the algorithm onto the FPGA.
[0076] In this embodiment, by using asymmetric and symmetric quantization formulas, the layer normalization operation module is processed using quantization methods to obtain an algorithm that converts floating-point operations into integer operations and shift operations.
[0077] At the hardware level, FPGAs offer lower latency and are better suited for computationally intensive tasks compared to CPUs; they also boast better energy efficiency and lower power consumption compared to GPUs; and they have lower initial investment costs compared to ASICs. Furthermore, FPGAs are not based on the von Neumann architecture and are programmable, allowing for the design of specific acceleration architectures. Therefore, on hardware computing platforms, the parallelism, low power consumption, low latency, and programmability of FPGAs make them an excellent choice for accelerating deep learning models.
[0078] For further details, please refer to Figure 2 In step S1, the method for obtaining the asymmetric quantization formula and the symmetric quantization formula through the quantization formula includes:
[0079] Step S11: Set the quantization formula as: r = S(qZ);
[0080] Where r is a floating-point number, q is the integer corresponding to the floating-point number, S represents the ratio between the floating-point number and the integer, and Z represents the integer corresponding to 0 in the real number after quantization.
[0081] Step S12: Using r = S(qZ) as the asymmetric quantization formula, we obtain the asymmetric quantization formula r = S(qZ);
[0082] Step S13, the integer formula corresponding to the floating-point number is:
[0083] Among them, round(·) is the rounding operation;
[0084] Step S14: Set the formulas for S and Z:
[0085]
[0086] Step S15: Perform symmetric quantization, i.e., r max =-r min ,q max =2 B-1 -1,q min =-2 B Where B is the number of bits in the quantized integer, substituting it into the formula... We can obtain Z = 0. Substituting Z = 0 into the asymmetric quantization formula r = S(qZ), we get the symmetric quantization formula r = Sq.
[0087] In the above process, the integer formula corresponding to the floating-point number is used. Integers can be obtained, but the range is relatively large. Therefore, for the integer values obtained after quantization, a clipping operation (clip(x, min, max)) is used to limit the integer corresponding to the floating-point number to the interval (min, max). If x is greater than max, then x is set to max; if x is less than min, then x is set to min.
[0088]
[0089] For further details, please refer to Figure 3 In step S2, the layer normalization operation module is processed using asymmetric and symmetric quantization formulas and quantization methods to convert floating-point operations into integer operations and shift operations. The methods include:
[0090] Step S21, the layer normalization formula is obtained as follows:
[0091]
[0092] Where ∈ is a very small number; to prevent the denominator of the division from being 0, ∈ is set to 0.01 in this embodiment; γ and β are the gain and offset parameters, respectively, and are vectors; the subscript i indicates one of the data in the word vector; μ is the average value of the word vector, and σ 2 It is the variance of the word vectors;
[0093] Step S22: Perform symmetric quantization on parameters γ and β, and perform asymmetric quantization on input X and output Y;
[0094] Step S221: Substitute the asymmetric quantization formula r = S(qZ) and the symmetric quantization formula r = Sq into... We can obtain:
[0095]
[0096] in,
[0097] Step S23, by It can be seen that the zeros of the input matrix have no effect on the operation; after the transformation,
[0098] Transformed into: Z Y ;
[0099] Where, q γ , q μ q σ 2 q β Z Y All are integers. The calculation using Newton's iteration method also yields an integer; therefore... Only in China and Both items are floating-point numbers;
[0100] Step S24: Convert floating-point operations into integer operations and shift operations;
[0101] Step S241, for and The following steps were taken:
[0102]
[0103] Where M0 and M1 are both integers, and This can be achieved by right shifting by n0 and n1 bits.
[0104] Step S242, will and Substitution
[0105] In the middle, we get:
[0106]
[0107] Furthermore, let n0 = n1 = n, then
[0108] It can be rewritten as:
[0109]
[0110] Please see Figure 4 In step S3, the method for deploying the algorithm on the FPGA includes:
[0111] Step S31: Deploy the algorithm onto the FPGA;
[0112] Step S32: Run the algorithm on the FPGA;
[0113] Step S33: Verify the running status of the algorithm on the FPGA.
[0114] This method can be used to check whether the algorithm is correct and whether it can run normally.
[0115] For further details, please refer to Figure 5 In step S31, the method for deploying the algorithm on the FPGA includes the following steps:
[0116] Step S311: Program the algorithm to obtain the corresponding programming information;
[0117] Step S312: Check and modify the programming information corresponding to the algorithm;
[0118] Step S313: Check the working status of the FPGA. If the FPGA can work normally, deploy the programming information corresponding to the algorithm onto the FPGA.
[0119] Step S314: Verify whether the programming information can run normally in the FPGA; if it cannot run normally, return to the step of checking and modifying the programming information and repeat until the programming information can run normally in the FPGA.
[0120] Step S315: Detect the running speed of BERT.
[0121] For further details, please refer to Figure 6 and Figure 7 Step S311, the method for programming the algorithm and obtaining the corresponding programming information includes the following steps:
[0122] Step S3111: Program the algorithm into pseudocode corresponding to the algorithm;
[0123] Step S3112: Based on the pseudocode corresponding to the algorithm and the requirements of the programming language, compile the pseudocode corresponding to the algorithm into the formal code corresponding to the algorithm, and use the formal code corresponding to the algorithm as the programming information corresponding to the algorithm.
[0124] Pseudocode is an informal, English-structured language used to describe module structure diagrams. When implementing the same algorithm in different programming languages, it's recognized that their implementations can differ significantly. This is especially true for programmers proficient in different languages, who may find it difficult to understand the functionality of a program (written in other languages) because the formalities of the programming language limit their understanding of key parts. Pseudocode was developed to address this need. It provides more design information; the description of each module must correspond to the design structure. Figure 1 It appeared.
[0125] The purpose of using pseudocode is to make the described algorithm easily implemented in any programming language (Pascal, C, Java, etc.). Therefore, pseudocode must be clearly structured, simple, highly readable, and resemble natural language. Pseudocode lies between natural language and programming languages, specifying the algorithm's function in a programming language style. Using pseudocode, one doesn't need to be bound by a specific implementation. Compared to programming languages (such as Java, C++, C, Delphi, etc.), it is more similar to natural language. It is a semi-scripted, non-standard language that can describe the entire algorithm's structure in a form close to natural language (any familiar language can be used, the key is to express the program's meaning).
[0126] In this embodiment, the pseudocode of the algorithm is as follows: Figure 7 As shown, pseudocode can be flexibly converted to various programming languages, such as Java, C++, C, and Delphi. This setup makes it easier for programmers of different programming languages to understand, increasing its accessibility and making the conversion of pseudocode to different programming languages more flexible.
[0127] Please see Figure 8 Furthermore, after the algorithm is deployed to BERT, the following operations are performed:
[0128] Step S316: Gradually reduce hardware resources multiple times;
[0129] Step S317: Detect the running speed of BERT;
[0130] Step S318: Find the balance range between hardware resources and BERT running speed.
[0131] After deployment on an FPGA, compared to the results of 32-bit floating-point operations, this invention significantly reduces all hardware resources while maintaining acceptable precision loss. By finding a balance between hardware resources and BERT's operating speed, it is easy to reduce hardware resource investment according to requirements, thereby saving hardware resources while ensuring BERT's operating speed.
[0132] Furthermore, quantification also includes: through... and Approximation processing Improve computational efficiency while ensuring acceptable error.
[0133] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.
[0134] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.
Claims
1. An FPGA acceleration method for normalized nonlinear functions in BERT layers, characterized in that, Includes the following steps: Asymmetric and symmetric quantization formulas are obtained through quantization formulas; Using the asymmetric quantization formula and the symmetric quantization formula, the layer normalization operation module is processed by the quantization method to obtain an algorithm that converts floating-point operations into integer operations and shift operations. The algorithm is deployed on an FPGA. While ensuring acceptable accuracy loss, the layer normalization module is compressed and accelerated, ultimately improving the overall running speed of the BERT model. Specifically: Methods for obtaining asymmetric and symmetric quantization formulas through quantization formulas include: The quantification formula is set as follows: ； （1） in, It is a floating-point number. It is the integer corresponding to the floating-point number. It represents the proportional relationship between floating-point numbers and integers. This represents the integer corresponding to 0 in a real number after quantization. Will As an asymmetric quantization formula; The formula for integers corresponding to floating-point numbers is: ； （2） in, It is a rounding operation; set up and The formula: (3) ； （4） Symmetric quantization is used, that is , , , in Substitute the number of digits of the quantized integer into the formula. Zhongde ,Will Substituting these values into the asymmetric quantization formula, we obtain the symmetric quantization formula: ； （5） Using the aforementioned asymmetric quantization formula and the aforementioned symmetric quantization formula, the layer normalization operation module is processed by a quantization method to convert floating-point operations into integer operations and shift operations, including the following methods: The layer normalization formula is: 、 ； （7） in, For input vectors One of the values, index Indicates the serial number; For input vectors The average value; It is the input vector The variance; It is a very small preset value used to prevent Too small, resulting in a denominator of the fraction being 0; in this paper, it is set to 0.
01. For the normalized result vector One of the values, index Indicates the serial number; Gain vector One of the values, index Indicates the serial number; and offset vector One of the values, index Indicates the serial number; It is the result vector One of the values, index Indicates the serial number; For parameter vectors and parameter vector Perform symmetric quantization on the input vector and output vector Perform asymmetric quantization; Substituting the asymmetric and symmetric quantization formulas into the layer normalization formula, we obtain: ； （8） in, , ; The zeros of the input matrix have no effect on the operation. After transformation, equation (8) becomes: ； （9） in, , , , , , All are integers. After calculating using Newton's iteration method, an integer is obtained. Therefore, only one integer is found in equation (9). and Both items are floating-point numbers, representing the ratio of two floating-point scaling factors; Convert floating-point arithmetic into integer arithmetic and perform bit shift operations; right and Both items are approximated as shown in equation (10) to improve computational efficiency while keeping the error acceptable: ； ； （10） in, , , , All are integers. and By moving right and Bit implementation; Substituting equation (10) into equation (9), we obtain equation (11): 。（11） 2. The FPGA acceleration method for the normalized nonlinear function in BERT as described in claim 1, characterized in that, For the integer values obtained after quantization, a truncation operation is also performed. This restricts the integer corresponding to the floating-point number to a range. inside, if Greater than , then let for ;like Less than , then let for ;get 。 （6） 3. The FPGA acceleration method for the normalized nonlinear function in BERT as described in claim 1, characterized in that: Further approximation of equation (11): make Under the premise that the error is acceptable, the calculation formula is simplified and the amount of calculation is reduced. Equation (11) is rewritten as: 。 （12） 4. The FPGA acceleration method for the normalized nonlinear function in BERT as described in claim 1, characterized in that: Methods for deploying algorithms on FPGAs include: Deploy the algorithm onto an FPGA; Enables the algorithm to run on an FPGA; Verify the algorithm's running status on the FPGA.
5. The FPGA acceleration method for the normalized nonlinear function in BERT as described in claim 4, characterized in that: Deploying algorithms onto FPGAs involves the following steps: The algorithm is programmed to obtain the corresponding programming information; Check and modify the programming information corresponding to the algorithm; Check the working status of the FPGA. If the FPGA can work normally, deploy the programming information corresponding to the algorithm onto the FPGA. Verify that the programming information can run normally in the FPGA; if it cannot run normally, return to the step of checking and modifying the programming information and repeat until the programming information can run normally in the FPGA. Test the running speed of BERT.
6. The FPGA acceleration method for the normalized nonlinear function in BERT as described in claim 5, characterized in that: The method for programming an algorithm and obtaining the corresponding programming information includes the following steps: The algorithm is programmed into pseudocode corresponding to the algorithm. Based on the pseudocode corresponding to the algorithm and the requirements of the programming language, the pseudocode corresponding to the algorithm is compiled into the formal code corresponding to the algorithm, and the formal code corresponding to the algorithm is used as the programming information corresponding to the algorithm.
7. The FPGA acceleration method for the normalized nonlinear function in BERT as described in claim 5, characterized in that: After the algorithm is deployed on the FPGA, the following operations are performed: Hardware resources were gradually reduced in multiple stages; Detect the running speed of BERT; Find a balance between hardware resources and BERT's operating speed.