Nonlinear processing method in Transform model and hardware accelerator thereof

Abstract

The embodiment of the invention provides a non-linear processing method in a Transform model and a hardware accelerator thereof, the non-linear processing method is suitable for the hardware accelerator, and the hardware accelerator is configured to convert a non-linear function into basic logic operation which can be realized by the hardware accelerator. The nonlinear processing method comprises the following steps: acquiring quantized input data and a scaling factor in a quantization process; basic logic operation is carried out on the quantized input data and the scaling factor to obtain nonlinear output data, and the nonlinear output data are used for reasoning of a Transform model. According to the non-linear processing method in the Transform model and the hardware thereof, the calculation time of the non-linear function can be effectively shortened, reasoning is accelerated, and the hardware utilization rate is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of data processing, and specifically, to a nonlinear processing method in a Transformer model and a hardware accelerator thereof. Background Art

[0002] The existing Transformer model uses a lot of nonlinear functions in the reasoning process. For example, the model uses the LayerNorm function to ensure numerical stability and accelerate model convergence. The Softmax function is used for numerical normalization and classification, and the GELU function and ReLU function are used to maintain the nonlinearity of the model. At present, many computing platforms such as central processing units (CPUs) and graphics processing units (GPUs) are not friendly to the calculation of the above nonlinear functions. Summary of the invention

[0003] In view of this, an embodiment of the present application provides a nonlinear processing method in a Transformer model and a hardware accelerator thereof, which can effectively reduce the calculation time of nonlinear functions, accelerate reasoning, and improve hardware utilization.

[0004] In a first aspect, a nonlinear processing method in a Transformer model is provided, wherein the nonlinear processing method is applicable to a hardware accelerator, wherein the hardware accelerator is configured to convert a nonlinear function into a basic logical operation that can be implemented by the hardware accelerator, and wherein the nonlinear processing method comprises: obtaining quantized input data and a scaling factor in a quantization process; performing the basic logical operation on the quantized input data and the scaling factor to obtain nonlinear output data, wherein the nonlinear output data is used for reasoning of the Transformer model.

[0005] In a possible implementation, the nonlinear function is a softmax function, and the basic logical operation includes a maximum value query operation, an exponential approximation operation, a shift multiplication and addition operation, and an integer division operation.

[0006] In a possible implementation, the nonlinear function is a GELU function, and the basic logic operations include shift-multiply-add operations, arithmetic operations, maximum value query operations, exponential approximation operations, and integer division operations.

[0007] In a possible implementation, the nonlinear function is a LayerNorm function, and the basic logic operation includes a shift-multiply-add operation, an integer division operation, and an arithmetic operation.

[0008] In a possible implementation manner, the nonlinear processing method further includes: performing quantization processing on the input data.

[0009] In a second aspect, a hardware accelerator in a Transformer model is provided, characterized in that the hardware accelerator is configured to convert a nonlinear function into a basic logical operation that can be implemented by the hardware accelerator, and the hardware accelerator includes: an acquisition module, used to obtain quantized input data and a scaling factor in a quantization process; a nonlinear processing module, used to perform the basic logical operation on the quantized input data and the scaling factor to obtain nonlinear output data, and the nonlinear output data is used for reasoning of the Transformer model.

[0010] In a possible implementation, the nonlinear processing module includes an arithmetic operation unit, an integer divider, a shift multiplication and addition unit, an exponential approximation operation unit, and a maximum value query unit.

[0011] In a possible implementation, the hardware accelerator further includes: a configuration register for configuring different operation orders for each operation unit in the nonlinear processing module according to the type of the nonlinear function, and the type of the nonlinear function includes a softmax function, a GELU function, or a LayerNorm function.

[0012] In a possible implementation, the hardware accelerator is a field programmable gate array (FPGA).

[0013] The nonlinear processing method in the Transformer model provided in the embodiment of the present application and its hardware accelerator, through the hardware accelerator, first converts various nonlinear functions into basic logical operations that can be implemented by the hardware accelerator, and then when nonlinear processing is required on the input data, the converted basic logical operations can be performed on the quantized input data and the scaling factors in the quantization process, so that nonlinear output data can be quickly obtained for the reasoning of the Transformer model. The nonlinear processing method of the embodiment of the present application can effectively reduce the calculation time of nonlinear functions, accelerate reasoning, and improve hardware utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the drawings without paying creative work.

[0015] Figure 1A schematic block diagram of a nonlinear processing method in a Transformer model provided in an embodiment of the present application.

[0016] Figure 2 A schematic block diagram of a hardware accelerator in a Transformer model provided in an embodiment of the present application.

[0017] Figure 3 This is a specific implementation block diagram of the nonlinear data module in the hardware accelerator provided in an embodiment of the present application. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0019] The Transformer model is a deep learning model based on the attention mechanism. Usually, the Transformer model consists of two parts: the encoder and the decoder. The encoder is composed of multiple identical layers stacked together, mainly including two sub-layers: multi-head self-attention mechanism and feedforward neural network. The decoder is also composed of multiple identical layers stacked together, mainly including three sub-layers: multi-head self-attention mechanism, multi-head attention mechanism and feedforward neural network.

[0020] The application scenarios of the Transformer model are very wide, for example, natural language processing, including machine translation, text generation, question-answering systems, sentiment analysis, and named entity recognition. Another example is speech processing, including speech recognition and speech synthesis. Another example is image and video processing, including understanding video content to generate descriptions. Another example is the recommendation system, which makes accurate recommendations based on the user's historical behavior and preferences.

[0021] In the Transformer model, nonlinear functions play a crucial role. For example, in a feedforward neural network, by introducing nonlinear activation functions (such as the ReLU function, etc.), the model complexity can be increased, the gradient vanishing problem can be alleviated, the model generalization ability can be improved, and feature fusion can be achieved. For another example, the LayerNorm function is usually used in the layer normalization operation. By normalizing the input of a layer to obtain the output, it helps to control the numerical stability of the model and accelerate the training convergence.

[0022] At present, many computing platforms such as CPU and GPU are not friendly to the calculation of nonlinear functions. How to effectively reduce the calculation time of nonlinear functions and accelerate reasoning is the top priority in the current application of Transformer models.

[0023] Therefore, the embodiment of the present application introduces a nonlinear processing method in a Transformer model, which is applied to a hardware accelerator, that is, various nonlinear functions are first converted into basic logical operations that can be implemented by the hardware accelerator through the hardware accelerator, and then when nonlinear processing is required on the input data, the converted basic logical operations can be performed on the quantized input data and the scaling factors in the quantization process, so that nonlinear output data can be quickly obtained for the reasoning of the Transformer model. The nonlinear processing method of the embodiment of the present application can effectively reduce the calculation time of nonlinear functions, accelerate reasoning, and improve hardware utilization.

[0024] Figure 1 FIG. 1 is a schematic block diagram of a nonlinear processing method 100 for a Transformer model according to an embodiment of the present application. The nonlinear processing method 100 is applicable to a hardware accelerator, which is configured to convert a nonlinear function into a basic logical operation that can be implemented by the hardware accelerator. Figure 1 As shown, the nonlinear processing method includes part or all of the following contents.

[0025] S110, obtaining quantized input data and a scaling factor in the quantization process.

[0026] S120, performing the basic logical operation on the quantized input data and the scaling factor to obtain nonlinear output data, where the nonlinear output data is used for reasoning of the Transformer model.

[0027] A hardware accelerator is a specially designed hardware device or integrated circuit that is used to accelerate specific computing tasks or functions to improve the performance of a computer system. Hardware accelerators can usually use specialized circuits and architectures to efficiently complete some time-consuming computing tasks. For example, a hardware accelerator can be a digital signal processor (DSP), a field programmable gate array (FPGA), or an application-specific integrated circuit (ASIC).

[0028] The hardware processor of the embodiment of the present application can perform nonlinear processing of data at a higher speed than the software processor, thereby significantly improving the overall performance of the computer system. In addition, since the hardware accelerator can also consume less power when completing nonlinear processing, this is particularly important for mobile devices and embedded systems. Finally, assigning nonlinear processing tasks to the hardware accelerator can also reduce the burden on the main processor (such as the CPU), allowing it to focus on other tasks and improve the overall efficiency of the system.

[0029] It should be noted that the nonlinear processing method in the Transformer model of the embodiment of the present application is particularly suitable for FPGA.

[0030] The following will explain in detail how to convert nonlinear functions into basic logical operations that can be implemented by hardware processors.

[0031] First, using numerical approximation, e x The calculation of is approximately the base 2 calculation, and the latter can be performed using shift operations, which effectively reduces the implementation complexity, as follows:

[0032]

[0033] I·log2 e≈I·(1.0111) b =(I+(I>>1)-(I>>4)) (Formula 2).

[0034] Among them, S is the scaling factor of the input data during the quantization process, and I is the quantized input data.

[0035] To simplify the description, p =(I+(I>>1)-(I>>4)), then the above formula can be converted to:

[0036]

[0037] S.I. p =(qr) b (Formula 4).

[0038] Through fixed-point multiplication, we can get the integer part and the decimal part of the product, which are recorded as q and r respectively. For the integer part q, we can directly perform shift calculation, while for the decimal part r, we need to use Taylor expansion to linearize it. Therefore, the above formula can be converted to:

[0039]

[0040] Next, we need to rewrite each nonlinear function.

[0041] The softmax function, also known as the normalized exponential function, has the following mathematical expression: The rewriting process is as follows:

[0042]

[0043] Where N is the total number of input data, I i is any one of the N quantized input data. max =max{I1,I2,…,I N}, the exponential calculation operation can use the above shift calculation operation, but please note that due to I i -I max ≤0, so we introduce the output scaling factor S out ,therefore The overall algorithm is as follows:

[0044] I Δ =(I i -I max ) (Formula 7);

[0045] I pΔ =(I Δ +(I Δ >>1)-(I Δ >>4)) (Formula 8);

[0046] S.I. Δ =(qr) b (Formula 9);

[0047]

[0048] S out =1 / 2 N (Formula 11);

[0049]

[0050] Furthermore, the softmax function can be rewritten as the following formula:

[0051] I max =max{I1,I2,…,I N} (Formula 13);

[0052]

[0053] Wherein, IntDiv(·) is an integer division operation.

[0054] That is, for the softmax function, the converted basic logical operations may include a maximum value query operation, an exponential approximation operation, a shift multiplication and addition operation, and an integer division operation.

[0055] The GELU function, or Gaussian Error Linear Unit function, is an activation function widely used in deep learning. The rewriting process of the GELU function is as follows:

[0056] GELU(x i )≈GELU(SI i )≈SI i σ(1.702SI i ) (Formula 16);

[0057]

[0058] Based on the above formulas 7-12, we continue to rewrite formulas 16-17, and we can get the overall algorithm of the GELU function as follows:

[0059] I max =max{I1,I2,…,I N} (Formula 18);

[0060]

[0061]

[0062] Wherein, IntDiv(·) is an integer division operation.

[0063] That is, for the GELU function, the converted basic logical operations may include shift multiplication and addition operations, arithmetic operations, maximum value query operations, exponential approximation operations, and integer division operations.

[0064] Finally, for the Layernorm function, its expression is:

[0065] Among them, γ and β are weight coefficients.

[0066] Using Newton's iteration method for linear approximation, the expression is rewritten as follows:

[0067]

[0068] Where N is the total number of input data, I j is any one of the N quantized input data, and γ and β are the quantized weight coefficients.

[0069] Furthermore, by introducing a scaling factor, the Layernorm function is finally rewritten as follows:

[0070]

[0071] S Out =1 / 2 N (Formula 28).

[0072] That is, for the Layernorm function, the converted basic logic operations may include shift multiplication and addition operations, integer division operations, and arithmetic operations.

[0073] After rewriting various nonlinear functions, further, the related register transfer level (RTL) circuit can be designed and verified on a hardware accelerator. For example, verification is performed on an FPGA to generate the final hardware circuit, which may include an arithmetic operation unit, an integer divider, a shift multiply-add unit, a maximum value query unit, and an exponential approximation operation unit.

[0074] In some embodiments, a quantization operation may be performed on the input data on the hardware accelerator. That is, the nonlinear processing method 100 may further include: performing a quantization operation on the input data.

[0075] That is, a general processor, such as a CPU, can send linear input data to a hardware accelerator, and the hardware accelerator can quantize the linear input data to obtain quantized input data. The scaling factor in the quantization process can also be calculated on the hardware accelerator.

[0076] Specifically, the quantization process can adopt a symmetric quantization method, that is, converting the original floating point number x into an interval-symmetric signed integer type I, and the formula is:

[0077]

[0078] Among them, round(·) is the rounding operator, x is the floating point number to be quantized (i.e., input data), m is the quantization interval, k is the number of quantization bits, clip(·) is the truncation function, which makes the quantized data fall within the interval (-m, m), and finally, S is the scaling factor. After quantization, all nonlinear functions can be expressed as the following formula, but since these nonlinear functions do not satisfy the commutative law, scaling factors need to be introduced in the calculation.

[0079] f(R)≈f(S·I), f(S·I)≠Sf(I) (Formula 31).

[0080] Quantizing the input data can reduce the data storage space while ensuring accuracy.

[0081] It should be noted that the above quantization process is applicable to all function inputs and weight parameters in the nonlinear function, that is, the input data of the embodiment of the present application also includes weight parameters.

[0082] By performing quantization operations on input data on the hardware accelerator, the power consumption of the general-purpose processor can be further reduced and the overall efficiency of the system can be improved without occupying too much hardware circuit area.

[0083] In other embodiments, the quantization process of the input data may also be implemented in a general-purpose processor, and the general-purpose processor directly sends the quantized input data to the hardware accelerator.

[0084] Based on the same concept, the embodiment of the present application also provides a hardware accelerator in a Transformer model, wherein the hardware accelerator is configured to convert a nonlinear function into a basic logical operation that can be implemented by the hardware accelerator. Figure 2 As shown, the hardware accelerator 200 includes part or all of the following contents.

[0085] The acquisition module 210 is used to acquire the quantized input data and the scaling factor in the quantization process.

[0086] The non-linear processing module 220 is used to perform the basic logical operation on the quantized input data and the scaling factor to obtain non-linear output data, and the non-linear output data is used for reasoning of the Transformer model.

[0087] In some embodiments, Figure 3 As shown, the nonlinear processing module 220 includes an arithmetic operation unit 221 , an integer divider 222 , a shift multiplication and addition unit 223 , an exponential approximation operation unit 224 and a maximum value query unit 225 .

[0088] For example, if the nonlinear function is a softmax function, the nonlinear processing unit 220 includes a maximum value query unit 225 , an exponential approximation operation unit 224 , a shift multiply-add unit 223 and an integer divider 222 .

[0089] For another example, if the nonlinear function is a GELU function, the nonlinear processing unit 220 includes a shift-multiply-add unit 223 , an arithmetic operation unit 221 , a maximum value query unit 225 , an exponential approximation operation unit 224 and an integer divider 222 .

[0090] For another example, the nonlinear function is a LayerNorm function, and the nonlinear processing unit 220 includes a shift multiplication and addition unit 223 , an integer divider 222 , and an arithmetic operation unit 221 .

[0091] In some embodiments, the hardware accelerator can also interact with the general-purpose processor so that the hardware accelerator can be configured to different pipeline levels and computing modes. Figure 3 As shown, the hardware accelerator 200 may further include: a configuration register 230. That is, all nonlinear functions correspond to a set of basic logic operation units generated by the hardware accelerator. However, the basic logic operation units executed by different nonlinear functions and their execution order are different. At this time, when it is necessary to execute the nonlinear processing corresponding to a specific nonlinear function, the configuration register 230 in the hardware accelerator can be used to execute the basic logic operation units corresponding to the specific nonlinear function in a specific order.

[0092] Taking the GELU function as an example, before starting the calculation, the configuration register 230 will configure the data flow of the architecture to complete the corresponding algorithm. First, input I i is loaded into the first shift multiply add unit 223, completing I p =(I+(I>>1)-(I>>4)). Then it is input to the arithmetic operation unit 221 to perform constant multiplication calculation I G =1.702I p Afterwards G is loaded into the maximum value query unit 225, and after the maximum value is obtained, it is compared with I max The exponential approximation calculation unit 224 is input together to obtain the output Finally, the final output is obtained through the integer divider 222 and written into the memory.

[0093] Although the present application has been described with reference to preferred embodiments, various modifications may be made thereto and parts thereof may be replaced with equivalents without departing from the scope of the present application. In particular, the various technical features mentioned in the various embodiments may be combined in any manner as long as there are no structural conflicts. The present application is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A nonlinear processing method in a Transformer model, characterized in that: The nonlinear processing method is applicable to a hardware accelerator, and the hardware accelerator is configured to convert a nonlinear function into a basic logical operation that can be implemented by the hardware accelerator. The nonlinear processing method includes: Get the quantized input data and the scaling factor during the quantization process; The basic logical operation is performed on the quantized input data and the scaling factor to obtain nonlinear output data, wherein the nonlinear output data is used for reasoning of the Transformer model.

2. The nonlinear processor method according to claim 1, characterized in that The nonlinear function is a softmax function, and the basic logical operations include a maximum value query operation, an exponential approximation operation, a shift multiplication and addition operation, and an integer division operation.

3. The nonlinear processing method according to claim 1, characterized in that: The nonlinear function is a GELU function, and the basic logic operations include shift multiplication and addition operations, arithmetic operations, maximum value query operations, exponential approximation operations, and integer division operations.

4. The nonlinear processing method according to claim 1, characterized in that: The nonlinear function is a LayerNorm function, and the basic logic operations include shift multiplication and addition operations, integer division operations, and arithmetic operations.

5. The nonlinear processing method according to claim 1, characterized in that: The nonlinear processing method further comprises: The input data is quantized.

6. The nonlinear processing method according to claim 1, characterized in that: The nonlinear processing method further comprises: The quantized input data is obtained from a general purpose processor.

7. A hardware accelerator in a Transformer model, characterized in that: The hardware accelerator is configured to convert a nonlinear function into a basic logic operation that can be implemented by the hardware accelerator, and the hardware accelerator includes: An acquisition module, used for acquiring quantized input data and a scaling factor in the quantization process; A nonlinear processing module is used to perform the basic logical operation on the quantized input data and the scaling factor to obtain nonlinear output data, and the nonlinear output data is used for reasoning of the Transformer model.

8. The hardware accelerator according to claim 7, characterized in that: The nonlinear processing module includes an arithmetic operation unit, an integer divider, a shift multiplication and addition unit, an exponential approximation operation unit, and a maximum value query unit.

9. The hardware accelerator according to claim 8, characterized in that: The hardware accelerator also includes: A configuration register is used to configure different operation orders for each operation unit in the nonlinear processing module according to the type of the nonlinear function, and the type of the nonlinear function includes a softmax function, a GELU function or a LayerNorm function.

10. The nonlinear processing method according to any one of claims 1 to 6, characterized in that: The hardware accelerator is a field programmable gate array FPGA.

Images