Approximate computing circuit for gelu non-linear activation function in visual transformer

Abstract

The application discloses a GELU nonlinear activation function approximate calculation circuit in a visual transformer, belongs to the technical field of calculation, estimation or counting, and has the function of hardware-friendly optimization of a Gaussian error linear unit nonlinear activation function of a visual transformer model. The method performs GELU calculation on an activation value with hardware friendliness as the target, uses a linear function or a piecewise linear function approximation for small and extremely large activation values by detecting the size of the activation value, and applies a more accurate polynomial approximation function to other values to ensure model accuracy. With hardware friendliness as the principle, the whole calculation process is performed in a full integer calculation mode, high hardware benefits are achieved at the cost of extremely low precision loss under the condition of improving energy efficiency and area efficiency, and the method is suitable for deployment in a hardware environment.

Description

Technical Field

[0001] This invention relates to artificial intelligence computer vision technology, and discloses an approximate calculation circuit for the GELU nonlinear activation function in a vision transformer, belonging to the technical field of calculation, estimation or counting. Background Technology

[0002] In the field of deep learning, Transformer-based models have achieved unprecedented performance in Natural Language Processing (NLP) tasks due to their self-attention mechanism. Meanwhile, Vision Transformer (ViT) models have outperformed traditional Convolutional Neural Networks (CNNs) in object detection and image classification tasks. However, ViT models introduce more complex nonlinear functions and higher computational costs than previous CNNs, posing challenges for hardware deployment. Current research on ViT has not thoroughly discussed the deployment of nonlinear functions, particularly the optimization of the Gaussian Error Linear Element (GELU) nonlinear activation function. Researchers have attempted various optimization methods for the GELU function, including algorithmic approximation, hardware acceleration, and weight quantization. However, these methods often sacrifice model accuracy while reducing computational complexity. Meanwhile, the hardware design of nonlinear activation functions is optimized for activation functions such as ReLU, Sigmoid, and Tanh in traditional neural networks. It is not suitable for optimizing the computationally complex GELU nonlinear activation function, and therefore is not suitable for the efficient deployment of visual transformer models on hardware.

[0003] In response to the above-mentioned situation, this application proposes a circuit design for approximate calculation of the GELU activation function in a visual converter. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of the aforementioned background technology by providing an approximate calculation circuit for the GELU nonlinear activation function in a visual transformer. This circuit solves the technical problems of high computational complexity and high resource consumption in the hardware implementation of the GELU nonlinear activation function in traditional visual transformer models, and achieves the goal of significantly improving the energy efficiency and area efficiency of the hardware while ensuring that the model accuracy loss is less than 1%.

[0005] To achieve the above-mentioned objectives, the present invention employs the following technical solution:

[0006] The GELU nonlinear activation function approximation circuit in the visual transformer includes: a comparator unit, an approximation calculation unit, and a requantization unit. The comparator unit's input is connected to a densely fully connected layer. The activation value output from the upper multi-head self-attention mechanism is transmitted to the comparator via the densely fully connected layer. It outputs a ReLU mode control signal when the activation value is an extremely large absolute value, a linear fitting mode control signal when the activation value is a small value close to zero, and a polynomial fitting mode control signal when the activation value is other values. The approximation calculation unit, upon receiving the ReLU mode control signal, uses an R... The eLU function approximates the activation value using a piecewise linear function. When a linear fitting mode control signal is received, a linear fitting function is used to approximate the activation value linearly. When a polynomial fitting mode control signal is received, a polynomial fitting function is used to approximate the activation value polynomially. The piecewise linear function approximation result is transmitted to the dense fully connected layer used to obtain the activation value of the lower-level multi-head self-attention mechanism. The requantization unit quantizes the linear or polynomial approximation result output by the approximation calculation unit into integer data output. The quantized integer data is transmitted to the dense fully connected layer used to obtain the activation value of the lower-level multi-head self-attention mechanism.

[0007] As a further optimization scheme for the GELU nonlinear activation function approximation calculation circuit in the vision converter, the ReLU function is y = max(0,x), where x is the activation value and y is the approximate result.

[0008] As a further optimization scheme for the GELU nonlinear activation function approximation calculation circuit in the visual transformer, the linear fitting function is: Where a1 and a2 are linear coefficients, which are determined by minimizing the root mean square error between y and the original GELU nonlinear activation function value.

[0009] As a further optimization scheme for the GELU nonlinear activation function approximation calculation circuit in the visual transformer, the polynomial fitting function is: Among them, A, B, C pos C neg These are the polynomial coefficients.

[0010] As a further optimization scheme for the GELU nonlinear activation function approximation calculation circuit in the visual transformer, a1 and a2 are determined by minimizing the root mean square error between y and the original GELU nonlinear activation function value. A, B, C pos C neg Determined by minimizing the root mean square error between y and the original GELU function value.

[0011] As a further optimization scheme for the GELU nonlinear activation function approximation calculation circuit in the visual transformer, the requantization unit... The linear or polynomial approximation results output by the approximation calculation unit are quantized into integer data output, S q Here, n is the quantization factor, n is the shift, and b is the expression for the linear or polynomial approximation result as b×2. c The last digit is in the form of a decimal, m = 2, b is a decimal in the range (0.5, 1], 2 m This is the magnification factor for the last digit b, and round is the rounding operation.

[0012] As a further optimization of the GELU nonlinear activation function approximation calculation circuit in the vision transformer, the approximation calculation unit includes: a ReLU unit and an approximation unit; the ReLU unit, when receiving the ReLU mode control signal, uses the ReLU function to perform piecewise linear function approximation on the activation value, and the piecewise linear function approximation result is transmitted to the dense fully connected layer used to obtain the activation value of the lower multi-head self-attention mechanism; the approximation unit, when receiving the linear fitting mode control signal, uses the linear fitting function to perform linear approximation on the activation value, and when receiving the polynomial fitting mode control signal, uses the polynomial fitting function to perform polynomial approximation on the activation value.

[0013] As a further optimization of the GELU nonlinear activation function approximation calculation circuit in the visual converter, the ReLU unit includes: an input register and a first selector; the input register is used to store the activation value; the first selector has its control terminal connected to the sign bit data of the activation value, one of its input terminals connected to 0, and the other input terminal connected to the activation value after stripping the sign bit, and outputs the piecewise linear function approximation result.

[0014] As a further optimization of the GELU nonlinear activation function approximation calculation circuit in the visual transformer, the approximation unit includes: a unit controller, a coefficient register, a square register, a first multiplier, an accumulation register, a second multiplier, and an adder; the unit controller receives a linear fitting mode control signal or a polynomial fitting mode control signal and outputs a control signal to select linear or polynomial coefficients; the coefficient register stores linear and polynomial coefficients and outputs linear or polynomial coefficients to the first multiplier according to the control signal output by the unit controller; the square register stores the activation value and the square of the activation value output by the second multiplier; the first multiplier, in the unit controller... When the control signal for selecting linear coefficients is output, the activation value read from the square register is linearly approximated and then output. When the control signal for selecting polynomial coefficients is output by the unit controller, a partial sum is output after polynomial fitting based on the activation value read from the square register and its square. An accumulator register is used to store the partial sum output by the first multiplier after polynomial fitting based on the activation value read from the square register and its square. A second multiplier performs a square operation on the activation value read from the square register and outputs the square of the activation value to the square register. An adder is used to accumulate the output of the first multiplier and the partial sum stored in the accumulator register to output a polynomial approximation result.

[0015] As a further optimization of the GELU nonlinear activation function approximation circuit in the visual transformer, the requantization unit includes: a control unit, a first shifter, a second shifter, an adder, and a second selector; the control unit is used to represent the linear approximation result or the polynomial approximation result as b×2. c Form, perform 2 on the tail value b m Magnified by 10 times, in round(b×2) m When ) = 2,4, the output control signal for selecting mode 1 is given, and the output signal is given in round(b×2). m When ) = 3, the output selects the control signal for mode 2; the first shifter shifts the input linear or polynomial approximation result by n bits; the second shifter shifts the input linear or polynomial approximation result by n+1 bits; the adder has one input connected to the output of the first shifter and the other input connected to the output of the second shifter; the second selector has its control terminal connected to the output of the control unit, one input connected to the output of the first shifter, and the other input connected to the output of the adder. Under the control signal of mode 1, it selects the output of the first shifter; under the control signal of mode 2, it selects the output of the adder.

[0016] The present invention, by adopting the above technical solution, has the following beneficial effects:

[0017] (1) Hardware implementation optimization: The approximate calculation circuit proposed in this invention takes into account the characteristics of hardware design. Through the collaborative work of the comparison unit, the approximate calculation unit and the requantization unit, a pipelined execution path is created. Each unit focuses on its specific task, thereby improving the overall calculation efficiency and realizing an efficient hardware execution path. This optimization reduces the consumption of hardware resources while maintaining the accuracy of the calculation.

[0018] (2) High degree of system modularity: In the approximate calculation circuit proposed in this invention, each module has a clear function and high differentiation between modules. The circuit structure of the comparator, approximate calculation unit and requantization unit can be easily modified and reused according to actual needs. Moreover, the structure is simple and feasible. This modular design makes the system easy to maintain and upgrade, reduces the complexity of development and testing, and also enhances the adaptability of the circuit proposed in this invention. Modification and reuse according to actual needs can also make the circuit proposed in this invention adaptable to other types of deep learning models.

[0019] (3) Improved computational efficiency: The approximate calculation circuit proposed in this invention effectively reduces the overall computational overhead by performing detailed analysis and interval division of activation values ​​and using integer calculation processing. This activation value perception method not only improves the energy efficiency of the visual converter model, but also enhances its performance, making it more suitable for hardware system deployment.

[0020] (4) Balance between accuracy and efficiency: The linear fitting and polynomial fitting methods used in the approximation calculation circuit proposed in this invention minimize the root mean square error between the approximation value and the original GELU nonlinear activation function value in different activation value ranges, thus achieving a good balance between accuracy and hardware efficiency.

[0021] (5) Improved cost-effectiveness: The design of the requantization unit in the approximate calculation circuit proposed in this invention avoids the expensive design of multipliers and reduces hardware costs. In addition, the use of full integer operation further reduces power consumption and improves calculation speed, making the whole system have obvious advantages in cost-effectiveness.

[0022] (6) Compared with existing nonlinear correlation function optimization techniques, which are mostly designed to optimize activation functions such as ReLU, Sigmoid, and Tanh in traditional neural networks, this invention proposes for the first time the design of an approximate calculation circuit for the GELU nonlinear activation function in a visual converter, and proposes a nonlinear activation function optimization method based on integer calculation and activation value awareness. This provides an effective solution for deploying visual converter models in resource-constrained hardware environments, which maintains the performance of the model and significantly reduces the complexity and cost of hardware deployment. This has important practical significance for promoting the widespread application of visual converter technology. Attached Figure Description

[0023] Figure 1 This is a block diagram of the overall hardware circuit for the GELU nonlinear activation function approximation calculation circuit in the visual converter of this invention.

[0024] Figure 2 This is a circuit diagram of the comparator unit in an embodiment of the present invention.

[0025] Figure 3 This is a circuit diagram of the approximate calculation unit in an embodiment of the present invention.

[0026] Figure 4 This is a circuit diagram of the requantization unit in an embodiment of the present invention. Detailed Implementation

[0027] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading the present invention, any modifications of the present invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.

[0028] To achieve the aforementioned objective, this invention employs a GELU nonlinear activation function approximation calculation circuit in a visual transformer, the overall structure of which is as follows: Figure 1 The optimized GELU module includes a comparator unit, an approximation calculation unit, and a requantization unit. The approximation calculation unit comprises a ReLU unit and an approximation unit. The core idea behind this approximation calculation circuit for approximating the GELU nonlinear activation function value lies in the detailed analysis and processing of the activation value. The activation value is divided into different intervals, and the most suitable approximation strategy is applied to each interval to adapt to different hardware execution requirements while maintaining computational accuracy and efficiency.

[0029] In ViT inference, the multi-head self-attention (MSA) activation values ​​obtained from the preprocessing stage of image data are activated by the GELU nonlinear activation function after passing through a dense fully connected layer. The activated values ​​then pass through another dense fully connected layer before being output to the next MSA. In this invention, the activation values ​​output from the upper-layer MSA are first input to the comparator unit of the optimized GELU module of this invention through a dense fully connected layer. The comparator unit's function is to assign an appropriate computation mode to each activation value based on its sign bit and magnitude. These computation modes are determined based on the numerical characteristics of the activation values ​​to effectively perform subsequent computational processing.

[0030] Next, the activation values, after being categorized by the comparator units, are transferred to the approximation calculation units. In the approximation calculation units, the activation values ​​are imported into ReLU units or approximation units according to the calculation mode assigned by the comparator units. Different calculation methods are used for activation values ​​within different ranges to preserve the characteristics of the original GELU nonlinear activation function to the greatest extent possible. This approach not only reduces computational complexity but also improves computational efficiency and hardware friendliness.

[0031] The activation values, after being processed by the approximation unit, enter the requantization unit. In this stage, a controller determines the mode selection in the quantization formula based on the activation value data. This mode selection is crucial because it determines how to effectively quantize the results of the approximation unit to meet subsequent hardware execution requirements. The quantized result is then output to a densely fully connected layer, providing input for the multi-head attention mechanism in the next layer.

[0032] Specifically, for activation value data that has passed through a dense fully connected layer and entered the optimized GELU module, it will first enter such... Figure 2 In the comparator unit shown, the first step is to strip the sign bit of the activation value data, converting it into a 7-bit unsigned number. This 7-bit unsigned number is then compared with the interval threshold data pre-stored in the threshold register in the 7-bit unsigned number comparator. The comparison result includes 00, 01, and 10. This result is then combined with the previously stripped sign bit of the activation value data to form a final 3-bit comparison result, corresponding to... Figure 2 The table below shows the following: 000 corresponds to the ReLU mode with maximum activation values ​​in the positive range; 100 corresponds to the ReLU mode with maximum activation values ​​in the negative range; 001 corresponds to the polynomial fitting mode with intermediate activation values ​​in the positive range; 101 corresponds to the polynomial fitting mode with intermediate activation values ​​in the negative range; 010 corresponds to the linear fitting mode with near-zero activation values ​​in the positive range; and 110 corresponds to the linear fitting mode with near-zero activation values ​​in the negative range. The result from the comparator unit then enters the approximation unit, which determines the selection of the approximation unit's calculation mode.

[0033] When the comparator unit outputs 000 or 100, it indicates that the activation value is an extreme magnitude activation value with an extremely large absolute value. Whether it is an activation value with an extremely large absolute value is determined by comparing the value in the pre-given threshold register with the activation value. The ReLU function y = max(0,x) is used to approximate the activation value x using a piecewise linear function to obtain the approximate activation value y. This can improve hardware efficiency because ReLU approximation is easier to implement in hardware.

[0034] When the comparator unit outputs 001 or 101, it indicates that the activation value is other than small or extreme amplitudes. Whether it is another activation value is determined by comparing the value in the pre-defined threshold register with the activation value, using a more accurate polynomial fitting function. The activation value x is approximated by a polynomial to obtain the approximate activation value y, where A, B, C pos C neg These are polynomial coefficients, which are determined by minimizing the root mean square error between y and the original GELU function value through an optimization process.

[0035] When the comparator unit outputs 010 or 110, it indicates a small-amplitude activation value close to zero. Whether it is a small-amplitude activation value close to zero is determined by comparing the value in the pre-defined threshold register with the activation value, using a simple linear fitting function. The activation value x is linearly approximated to obtain the approximate activation value y, where a1 and a2 are linear coefficients. The linear coefficients can be determined by minimizing the root mean square error between y and the original GELU nonlinear activation function value.

[0036] Activation value data enters Figure 3 After the approximate calculation unit shown, the calculation mode will be selected based on the output of the comparator in the previous step. In ReLU mode, as shown... Figure 3 As shown in the ReLU unit, the sign bit of the activation value data stored in the input register serves as the selection flag for the selector. According to the ReLU function calculation formula, if the sign bit is 1, meaning the input activation value data is negative, then 0 is directly selected as the output; otherwise, the activation value itself is output. In approximate calculation mode, such as... Figure 3 As shown in the approximation unit, the input activation value enters the square register, while the coefficient register stores the fitting coefficients for both polynomial and linear fitting. The specific coefficient used depends on the calculation mode of the comparator output, and the unit controller of the approximation unit adjusts the selection of fitting coefficients according to this mode. The fitting coefficients are pre-defined, obtained through the preceding optimization process. For polynomial fitting, all A, B, and C... pos C neg All four fitting coefficients are stored in the coefficient register. For linear fitting, only the fitting coefficients of the first-order term are used; for the quadratic term, a value of 0 is simply assigned to the coefficient register. It should be noted that the approximation unit's output is 32 bits. To prevent the output bit width from being too large, it needs to be quantized to an 8-bit output. The output of the approximation unit then enters the requantization unit for the next step of requantization, realizing full integer operations in the entire approximation calculation process.

[0037] In such Figure 4In the requantization unit shown, a portion of the data from the approximation unit enters the requantization unit. "Partially entering the requantization unit" means that not all the calculation results from the approximation unit will be requantized. For the ReLU unit, its result is already 8 bits and does not require requantization; the result requiring requantization is the 32-bit data calculated by the approximation unit. The data requiring requantization first enters the requantization unit control unit, which represents the data to be quantized as b×2. c In this form, let b be the tail value of the data to be quantized, and magnify the tail value b by 2. m In this embodiment, m = 2, and b is a decimal in the range (0.5, 1), b × 2 m The value of is [2,3,4], according to b×2 m The value determines the output mode of the selector, round(b×2) m When ) = 2,4, select mode 1, round(b×2) m When ) = 3, select mode 2, that is S q This is the quantization factor that can be achieved through shifting and addition. In mode 1, it corresponds to... The input data is shifted n bits before entering the selector; in mode 2, the corresponding... The input data, after being shifted by n bits and then shifted by n+1 bits, is added and then fed into the selector. The shift-and-add module performs requantization calculation, and the result, after passing through the selector, is 8-bit requantized data. The data output from the ReLU unit or the requantization unit is then passed through a dense fully connected layer to obtain the activation value for the lower-layer multi-head self-attention mechanism. This completes the design of the GELU activation function approximation calculation circuit in the visual transformer of this invention. The requantization method ensures that the optimized GELU activation value is always limited to the 8-bit range, allowing for effective cost reduction, avoiding expensive multiplier design, and improving the cost-effectiveness of the hardware design.

[0038] The above embodiments use ViT as an example to illustrate the approximate calculation circuit proposed in this invention. The circuit proposed in this invention is applicable to various vision transformer models, including but not limited to ViT, Data-efficient Image Transformers (DeiT), and Swing Transformer (Swin) models, and can effectively improve the energy efficiency and area efficiency of the hardware implementation of the nonlinear activation function of the vision transformer model with extremely low precision loss.

Claims

1. A circuit for approximating the GELU nonlinear activation function in a vision converter, characterized in that, include: The comparator unit's input is connected to a densely fully connected layer. The activation values ​​output by the multi-head self-attention mechanism in the upper layer are transmitted to the comparator unit via the densely fully connected layer. When the activation value is an extremely large absolute value, it outputs a ReLU mode control signal; when the activation value is a small value close to zero, it outputs a linear fitting mode control signal; and when the activation value is any other value, it outputs a polynomial fitting mode control signal. The approximation calculation unit, upon receiving a ReLU mode control signal, performs a piecewise linear function approximation of the activation value using the ReLU function; upon receiving a linear fitting mode control signal, it performs a linear approximation of the activation value using a linear fitting function; and upon receiving a polynomial fitting mode control signal, it performs a polynomial approximation of the activation value using a polynomial fitting function. The piecewise linear function approximation result is transmitted to a dense fully connected layer used to obtain the activation values ​​of the lower-level multi-head self-attention mechanism. Requantization unit, through The linear or polynomial approximation result output by the approximation calculation unit is quantized into integer data output. The quantized integer data output is then transmitted to a dense fully connected layer used to obtain the activation values ​​of the lower-level multi-head self-attention mechanism. As a quantification factor, For shifting, To express the linear approximation result or the polynomial approximation result as The last digit of the form, , It is a decimal in the range (0.5, 1). For the tail value Magnification For rounding operations; The requantization unit includes: The control unit is used to represent the linear or polynomial approximation result as... form Tail value conduct Magnified many times, at Output the control signal to select mode 1. Output the control signal to select mode 2. The first shifter shifts the input linear or polynomial approximation result. Bit, The second shifter shifts the input linear or polynomial approximation result. Bit, An adder, one input of which is connected to the output of the first shifter, and the other input of which is connected to the output of the second shifter, and, The second selector has its control terminal connected to the output terminal of the control unit, one input terminal connected to the output terminal of the first shifter, and the other input terminal connected to the output terminal of the adder. Under the action of the mode 1 control signal, it selects the output of the first shifter, and under the action of the mode 2 control signal, it selects the output of the adder.

2. The GELU nonlinear activation function approximation calculation circuit in the visual transformer according to claim 1, characterized in that, The ReLU function is ,in, The activation value. This is an approximate result.

3. The GELU nonlinear activation function approximation calculation circuit in the visual transformer according to claim 2, characterized in that, The linear fitting function is: ,in, and The coefficients are linear. and By minimizing The root mean square error between the original GELU nonlinear activation function value and the original GELU nonlinear activation function value is determined.

4. The GELU nonlinear activation function approximation calculation circuit in the visual transformer according to claim 3, characterized in that, The polynomial fitting function is: ,in, These are the polynomial coefficients.

5. The GELU nonlinear activation function approximation calculation circuit in the visual transformer according to claim 4, characterized in that, and By minimizing The root mean square error between the original GELU nonlinear activation function value and the original GELU nonlinear activation function value is determined. By minimizing The root mean square error between the original GELU function value and the original GELU function value is determined.

6. The GELU nonlinear activation function approximation calculation circuit in the visual transformer according to claim 1, characterized in that, The approximate calculation unit includes: The ReLU unit, upon receiving a ReLU mode control signal, uses the ReLU function to approximate the activation value using a piecewise linear function. The approximation result is then transmitted to a dense fully connected layer used to obtain the activation values ​​of the lower-level multi-head self-attention mechanism. The approximation unit uses a linear fitting function to approximate the activation value when it receives a linear fitting mode control signal, and uses a polynomial fitting function to approximate the activation value when it receives a polynomial fitting mode control signal.

7. The GELU nonlinear activation function approximation calculation circuit in the visual transformer according to claim 6, characterized in that, The ReLU unit includes: The input register is used to store the activation value; and, The first selector has its control terminal connected to the sign bit data of the activation value, one input terminal connected to 0, and the other input terminal connected to the activation value after removing the sign bit. It outputs a piecewise linear function approximation result.

8. The GELU nonlinear activation function approximation calculation circuit in the visual transformer according to claim 7, characterized in that, The approximation unit includes: The unit controller is used to receive linear fitting mode control signals or polynomial fitting mode control signals and output control signals to select linear coefficients or polynomial coefficients. The coefficient register is used to store linear coefficients and polynomial coefficients, and outputs linear coefficients or polynomial coefficients to the first multiplier according to the control signal output by the unit controller. The square register is used to store the activation value and the square of the activation value output by the second multiplier. The first multiplier, when the unit controller outputs the control signal to select linear coefficients, performs a linear approximation on the activation value read from the square register and outputs it; when the unit controller outputs the control signal to select polynomial coefficients, it performs polynomial fitting based on the activation value read from the square register and its square and outputs a partial sum. The accumulator register is used to store the partial sum output by the first multiplier after performing polynomial fitting based on the activation value read from the square register and its square. The second multiplier squares the activation value read from the square register and outputs the square of the activation value back to the square register; and, The adder is used to accumulate the output of the first multiplier and the partial sum stored in the accumulator register, and outputs a polynomial approximation result.

Images