Method for converter-based bidirectional encoder representation network and related product thereof

Abstract

The present disclosure discloses a method for a converter-based bidirectional encoder representation network and related products thereof, wherein the related products include at least a device and a computer readable storage medium. The apparatus may be included in a computing processing device of a combined processing device, which may include one or more data processing devices. The combined processing device can also comprise an interface device and other processing devices. And the computing processing device interacts with other processing devices to jointly complete computing operation specified by a user. The combined processing device can further comprise a storage device, and the storage device is connected with the equipment and the other processing devices and used for storing data of the equipment and the other processing devices. According to the scheme, the hardware efficiency can be improved, repeated calculation is reduced, and calculation power waste is reduced.

Description

Technical Field

[0001] The present disclosure generally relates to the field of artificial intelligence. More specifically, the present disclosure relates to a method, apparatus, integrated circuit device, and computer-readable storage medium for a Bidirectional Encoder Representation Based on Transformer (BERT) network. Background Art

[0002] The BERT network is mainly composed of a multi-head self-attention (MHA) module and a feedforward neural network (FFN) module, both of which involve the calculation of activation functions including transcendental functions. Specifically, the MHA module involves the calculation of the Softmax activation function, while the FFN module involves the calculation of the Gaussian error linear unit (GeLU) activation function. In the quantized inference (QAT) scenario, the input needs to be quantized from the floating point type to the int8 type for matrix multiplication. However, when implementing the transcendental function calculations of these two modules on hardware, the efficiency of the hardware is usually lower than that of matrix operations and vector operations. In the int8 scenario, taking the MHA module as an example, after batch matrix multiplication, all possible results are in the 256 discrete values ​​of [-128,127]. This means that there will be a lot of repeated calculations, resulting in a waste of computing power. Therefore, how to effectively implement the calculation of activation functions in the BERT network, especially in the MHA and FFN modules, to improve hardware efficiency, reduce repeated calculations and reduce computing power waste, has become a technical problem that needs to be solved urgently. Summary of the invention

[0003] In order to at least solve one or more of the technical problems mentioned above, the present disclosure proposes in multiple aspects a solution for optimizing the calculation of activation functions (including transcendental functions therein) in a Bidirectional Encoder Representation based on Transformer (BERT) network.

[0004] In a first aspect, the present disclosure provides a method for a Bidirectional Encoder Representation Based on Transformer (BERT) network, comprising: receiving a discrete input value for calculating a nonlinear activation function when performing operations related to the calculation of the nonlinear activation function in the BERT network; searching an activation table for a corresponding discrete output value based on the discrete input value; and outputting the discrete output value as a result for calculating the nonlinear activation function.

[0005] In a second aspect, the present disclosure provides a device for a transformer-based bidirectional encoder representation (BERT) network, comprising: a processor; and a memory on which a software program for a transformer-based bidirectional encoder representation (BERT) network is stored, and when the software program is executed by the processor, the method described in the first aspect is implemented.

[0006] In a third aspect, the present disclosure provides an integrated circuit device, comprising the apparatus according to the second aspect.

[0007] In a fourth aspect, the present disclosure provides a board comprising the integrated circuit device according to the third aspect.

[0008] In a fifth aspect, the present disclosure provides a computer-readable storage medium having stored thereon a software program for a Bidirectional Coded Representation Based Transformer (BERT) network, which, when executed by a processor, implements the method according to the first aspect.

[0009] Through the optimized transcendental function calculation method based on the bidirectional encoder representation of the converter (BERT) network provided above, the disclosed embodiment obtains the corresponding discrete output value by pre-calculating the discrete input value and uses the kernel function for calculation, which can improve the hardware efficiency, reduce repeated calculations and reduce the waste of computing power. Further, in some embodiments, by caching the activation table on-chip, the related operations of nonlinear activation function calculation (especially the transcendental function therein) can be accelerated. Furthermore, in some embodiments, by allocating multiple discrete input values ​​to multiple computing cores for pre-calculation, the computing parallelism and efficiency can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] By reading the detailed description below with reference to the accompanying drawings, the above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood. In the accompanying drawings, several embodiments of the present disclosure are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein:

[0011] Figure 1 The calculation process of the self-attention mechanism in the prior art is shown;

[0012] Figure 2 The hardware implementation process of the Softmax activation function in the prior art is shown;

[0013] Figure 3 The calculation process of the FFN module in the prior art is shown;

[0014] Figure 4 A simplified flowchart of the method disclosed herein for a BERT network is shown;

[0015] Figure 5 A detailed flowchart of the method disclosed in the present disclosure for a BERT network is shown;

[0016] Figure 6 A schematic diagram showing discrete activation of the Softmax activation function in the MHA of the present disclosure is shown;

[0017] Figure 7 A schematic diagram showing the calculation process of the Softmax activation function disclosed herein;

[0018] Figure 8 A schematic diagram showing discrete activation of the GeLU activation function disclosed herein;

[0019] Fig. 9 A schematic diagram showing the calculation process of FFN under discrete activation of the GeLU activation function disclosed herein;

[0020] Fig.10 An exemplary schematic diagram of the table lookup of FFN under discrete activation of the GeLU activation function disclosed in the present invention is shown;

[0021] Fig.11 is a schematic diagram showing the structure of a board according to an embodiment of the present disclosure; and

[0022] Fig.12 is a schematic diagram showing the structure of a combined processing device in a chip according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.

[0024] It should be understood that the terms "include" and "comprising" used in the specification and claims of the present disclosure indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections thereof.

[0025] It should also be understood that the terms used in this disclosure are only for the purpose of describing specific embodiments and are not intended to limit the disclosure. As used in this disclosure and claims, the singular forms of "a", "an", and "the" are intended to include the plural forms unless the context clearly indicates otherwise. It should also be further understood that the term "and / or" used in this disclosure and claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations.

[0026] As used in this specification and claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0027] As described in the background technology section, the BERT network is mainly composed of a multi-head self-attention (MHA) module and a feedforward neural network (FFN) module. The MHA module uses the Softmax activation function, while the FFN module uses the Gaussian error linear unit (GeLU) activation function. Both modules involve complex nonlinear operations. In the quantized inference (QAT) scenario, the input of the BERT network is quantized from floating point type to int8 type for matrix multiplication operations. However, when implementing the nonlinear function calculations in the MHA and FFN modules on hardware, the hardware efficiency is usually lower than matrix multiplication and vector operations. Therefore, how to effectively implement the calculation of nonlinear functions in the MHA and FFN modules of the BERT network to improve hardware efficiency and reduce repeated calculations and waste of computing power has become a technical problem that needs to be solved urgently.

[0028] The solution disclosed in this invention aims to solve this technical problem and proposes a hardware implementation solution for a nonlinear activation function based on discrete values. This solution constructs an activation table from discrete input values ​​to corresponding output values ​​and stores the table in advance on the chip. When performing nonlinear activation function calculations, the result is obtained by directly looking up the table, replacing complex multiplication, exponential and division operations, so as to achieve the purpose of computing acceleration. In addition, the solution disclosed in this invention also adopts technical means such as subtracting the maximum value from the result of matrix multiplication and multi-core parallel computing, which further improves the computational efficiency of the nonlinear activation function, reduces the memory requirements and the difficulty of table lookup, and makes it possible for the efficient implementation of the BERT network on a dedicated chip. In order to facilitate the understanding of the solution disclosed in this invention, the following will first be combined with the attached Figure 1-Figure 3 The implementation process of the prior art is described exemplarily.

[0029] Figure 1 FIG. 1 shows a calculation process 100 of the self-attention mechanism in the prior art. Figure 1As shown, in this process, query and key are quantized at steps S102 and S104 respectively. The goal of the quantization operation is to convert the floating-point query and key to integer types, such as int8, for efficient matrix operations. This step is called "quant". As known to those skilled in the art, Query, Key and Value are the basic components of the self-attention mechanism, which originated from the field of information retrieval. In information retrieval, Query is a query or question, Key is an index in a database, and Value is the content corresponding to Key. In the self-attention mechanism, Query, Key and Value are all vectors, and they come from different transformations of the input data. In actual operation, the input data is first mapped to the three spaces of Query, Key and Value. Specifically, the "query" vector is used to request or query information, while the "key" vector is used to represent each part of the input data. By calculating the degree of match between the query and each key (usually using a dot product), a weight distribution can be obtained. This weight distribution is then used to weighted sum the "values" to obtain the output.

[0030] Returning to the calculation process, after quantization, the query and key are matrix multiplied ("batchmatmul") in step S106. In this step, the dot product of the query and key is calculated by matrix multiplication to obtain the attention score, which is used to represent the similarity between the query and the key. Next, in step S108, the result of the matrix multiplication is dequantized. The dequantization operation converts the integer data back to floating-point data so that the data can continue to participate in the next floating-point operation.

[0031] Then, the result of the dequantization is input into the Softmax activation function in step S110. The Softmax activation function is a transcendental function that can convert any real-valued vector input into a probability distribution. In this way, each attention score is converted into a corresponding attention weight.

[0032] Next, in step S112, the output result of the Softmax activation function is quantized to obtain a first quantized value. At the same time, in step S114, the "value" input is also quantized to obtain a second quantized value. Thereafter, in step S116, the first quantized value and the second quantized value are matrix multiplied. This step implements the weighted summation in the attention mechanism, that is, the "value" is weighted according to the weight. Then, in step S118, the result of the matrix multiplication is dequantized again to obtain the final output result. Through this series of calculation processes, the self-attention mechanism in the prior art is realized, and the calculation efficiency is improved through quantization and dequantization operations.

[0033] In the above calculation process, the calculation formula of the Softmax activation function is:

[0034] Where x is the input vector, for example x=(x1,x2,...,x n ), e xi Represents that for each element x in the input vector x i Find the index, represents the exponential sum of all elements in the x vector. The hardware implementation of the Softmax(x) function is described below.

[0035] Figure 2 FIG. 2 shows a hardware implementation process 200 of the Softmax activation function in the prior art. Figure 2 As shown in , the implementation process includes, in step S202, inputting integer int8 data, whose scale is N and C. Here, N can represent "BatchSize", that is, the number of samples in a batch of data, and "C" can represent the dimension of the feature. For example, in the BERT network, C can represent the length of the word embedding vector.

[0036] Next, in step S204, dequant(float)(N,C) is executed, that is, a floating-point dequantization operation is performed on the C features of each of the N samples. Thereafter, in step S206, max(N,) is executed, that is, the maximum value of the C features of each sample is found. In step S208, sub max(N,C) is executed, that is, the corresponding maximum value is subtracted from the C features of each sample. In step S210, exp()(N,C) is executed, that is, an exponential function calculation is performed on the C features of each sample. Next, in step S212, sum(N,) is executed, that is, a summation operation is performed on the C features of each sample. Thereafter, in step S214, recip(N,) is executed, that is, a reciprocal operation is performed on the summed result of each sample. In step S216, mul(N,C) is executed, that is, the C features of each sample are multiplied by the corresponding reciprocal result. Finally, in step S218, quant(int8)(N,C) is executed, that is, the int8 quantization operation is performed on the C features of each sample to obtain the output result of the Softmax activation function.

[0037] Figure 3 FIG. 3 shows a calculation process 300 of the FFN module in the prior art. Figure 3 As shown, it illustrates the calculation process of the feedforward neural network (FFN) module in the prior art. First, in steps S302 and S304, the FFN module receives the original input data and the output of the first layer neural network weight parameter filter1 respectively. Then, these two parts of data are quantized (quant) in steps S306 and S308, for example, converted from floating point numbers to int8 types. Then, in step S310, matrix multiplication (matmul) is performed to perform matrix multiplication on the quantized input data and the quantized output of filter1. In step S312, the result of the matrix multiplication is dequantized (dequant), for example, converted from int8 type back to floating point numbers. In step S314, bias addition (add bias) and Gaussian error linear unit (GeLU) activation function are performed. In step S318, the output of step S314 is quantized. In steps S316 and S320, the FFN module receives the output of the second layer neural network weight parameter filter2 and performs a quantization operation on it. After that, in step S322, matrix multiplication is performed, that is, the quantized output of step S318 and the quantized output of filter2 are matrix multiplied. Finally, in step S324, the result of the matrix multiplication is dequantized, and in step S326, bias addition is performed. Through such a calculation process, the FFN module completes the processing of the input data.

[0038] although Figure 1 , 2 and Figure 3 The calculation and hardware implementation process of the Softmax activation function and the FFN module in the prior art are shown, but these two modules have some efficiency issues when they are actually executed on hardware. For example, when implementing these two modules, the graphics processing unit (GPU) basically follows the above process, but for the calculation of the transcendental functions involved, such as the Softmax activation function and the GeLU activation function, the hardware efficiency is usually lower than that of performing matrix operations and vector operations.

[0039] Especially in the int8 quantization scenario, the above problem is more prominent. Taking the MHA module as an example, after completing the batch matrix multiplication operation, all possible result values ​​are only in the 256 discrete values ​​of [-128, 127], even if the actual amount of data is far more than 256. If the subsequent dequantization and exponential function operation exp(x) is regarded as a function f(x), then a discrete int8 value corresponds to a result of f(x). Since the actual amount of data is greater than 256, this leads to a large number of repeated calculations, resulting in a waste of computing power.

[0040] Therefore, in order to improve hardware efficiency, reduce repeated calculations and save computing power when implementing these calculation processes, the solution disclosed herein proposes the use of an activation table to store pre-calculated transcendental function results. These transcendental functions include but are not limited to the Softmax activation function in the MHA module and the GeLU activation function in the FFN module. When performing quantized reasoning, the corresponding function results can be directly found from the activation table, thereby avoiding repeated calculations. The solution disclosed herein greatly improves the efficiency of transcendental function calculations in the BERT network, especially in the MHA and FFN modules, while reducing the waste of computing resources.

[0041] Figure 4 A simplified flow chart of a method 400 of the present disclosure for a BERT network is shown. Figure 4 As shown, in step S402, when performing operations related to nonlinear activation function calculation in the BERT network, discrete input values ​​for nonlinear activation function calculation are received. In some embodiments, these discrete input values ​​can be used for nonlinear activation function calculation in various modules of the BERT network, for example, a self-attention mechanism module or a feedforward neural network module.

[0042] Next, at step S404, the corresponding discrete output values ​​are found in the activation table according to these discrete input values. In some embodiments, the activation table disclosed herein is pre-calculated and stored on the chip, and contains the activation function output values ​​corresponding to all possible discrete input values. Through such an innovative approach, when performing nonlinear activation function calculations in the BERT network, these pre-calculated results can be directly found in the activation table, thereby avoiding the nonlinear activation function calculations with higher computational costs at runtime.

[0043] Finally, in step S406, the discrete output value is output as the result of the nonlinear activation function calculation. In some embodiments, this discrete output value will be further input into other modules of the BERT network to complete subsequent calculation steps.

[0044] Through the method 400 as described above, the scheme disclosed herein can significantly optimize the calculation process of the nonlinear activation function of the BERT network. First, by looking up the pre-calculated activation function output values ​​at runtime instead of calculating them at runtime, the scheme disclosed herein can significantly reduce the computational burden and improve computational efficiency. Secondly, by storing the activation table on the chip, the scheme disclosed herein can further increase the search speed and further improve the overall computational efficiency. In addition, the scheme disclosed herein processes discrete input values ​​and uses pre-calculated activation function output values, so complex floating-point operations can be avoided at runtime in some computing scenarios, thereby reducing computational complexity.

[0045] Figure 5 Detailed flowchart of the method 500 of the present disclosure for a BERT network is shown. Figure 5 As shown, the method 500 includes, in step S502, obtaining corresponding discrete output values ​​by pre-calculating discrete input values ​​to form an activation table. In some embodiments, in this step, a kernel function can be used to distribute multiple discrete input values ​​to multiple computing cores for pre-calculation to obtain the corresponding discrete output values.

[0046] In step S504, the activation table is cached on-chip for use in related operations of nonlinear activation function calculation. Next, in step S506, discrete input values ​​for nonlinear activation function calculation are received. In some embodiments, these discrete input values ​​can be inputs to the Softmax activation function in the self-attention mechanism module of the BERT network and / or the Gaussian error linear unit (GeLU) activation function in the feedforward neural network module of the BERT network.

[0047] In step S508, the corresponding discrete output value is found in the activation table. Finally, in step S510, the discrete output value is output as the result of the nonlinear activation function calculation. In some embodiments, the discrete output value will be used as the result of the nonlinear activation function calculation and can be further input into other modules of the BERT network to complete subsequent calculation steps.

[0048] In some embodiments, the nonlinear activation function disclosed herein may be a Softmax activation function in a self-attention mechanism module of a BERT network. In this scenario, method 500 may include receiving multiple output values ​​calculated by matrix multiplication as discrete input values ​​in the self-attention mechanism module, and finding corresponding discrete output values ​​in an activation table according to these discrete input values, and then outputting these discrete output values ​​as results for calculating the Softmax activation function. In some embodiments, a maximum value subtraction operation is performed on the result value calculated by the matrix multiplication, and multiple output values ​​obtained after the maximum value subtraction operation are used as discrete input values.

[0049] In other embodiments, the nonlinear activation function may be a GeLU activation function in a feedforward neural network module of a BERT network. In this scenario, method 500 may include receiving a plurality of output values ​​calculated by the first matrix multiplication as discrete input values ​​in the feedforward neural network module, and finding corresponding discrete output values ​​in an activation table according to the discrete input values, and then outputting the discrete output values ​​as results for calculating the GeLU activation function.

[0050] In addition, in some embodiments, method 500 may include using a kernel function to distribute multiple discrete input values ​​to multiple computing cores for pre-calculation. For example, when performing operations related to the calculation of the Softmax activation function, this may include performing dequantization and exponential function calculation on multiple discrete input values. When performing operations of the GeLU activation function, this may include performing dequantization, adding bias terms, calculating GeLU activation function values, and quantization operations on multiple discrete input values.

[0051] Based on the above Figure 5 According to the description, those skilled in the art can understand that the disclosed solution can effectively process nonlinear activation function calculations in the BERT network by using pre-computed activation tables and processing discrete input values. This processing method can not only improve the computational efficiency and accuracy, but also improve the efficiency of quantization and dequantization calculations. Specifically, by reducing the need for floating-point operations, the method can reduce the demand for hardware computing resources, thereby improving computing efficiency while ensuring computational accuracy, and further improving the performance of the BERT network and the quality of computational results.

[0052] Figure 6 Schematic diagram of Softmax discrete activation in MHA of the present disclosure is shown. Figure 6 As shown, the virtual box 601 shows 256 values ​​from -128 to 127, which are all possible result values ​​of the matrix multiplication (batch matmul) of the quantized query and key. Next, a dequantization operation is performed in the virtual box 602 to restore these values ​​to their original floating-point values. After the dequantization operation, in the virtual box 603, the exponential function exp(x) operation is performed on these floating-point values. This is because in the self-attention mechanism, the Softmax activation function needs to be applied to the input value, and part of the Softmax activation function is the exponential operation. Finally, the resulting floating-point numbers are obtained in the virtual box 604, and these resulting floating-point numbers are the output values ​​after the Softmax activation function is processed. Since dequantization and exponential operations are both reversible deterministic operations, it can be known that each quantized integer input value corresponds to a unique floating-point output value after dequantization and exponential operations.

[0053] It can be seen that the Softmax discrete activation process is essentially a one-to-one mapping relationship. In view of this, the disclosed solution proposes to pre-calculate such a mapping relationship and store it in the form of an activation table. When calculating the actual nonlinear activation function, the corresponding output value can be directly found in the activation table without the need for complex inverse quantization and exponential function calculations, thereby greatly improving the calculation efficiency.

[0054] Figure 7 Schematic diagram showing the calculation process of the Softmax activation function disclosed in the present invention. Figure 7 As shown, first, in step S702, integer int8 data is input, and its scale is N and C. Then, in step S704, max(N,) is executed, that is, the maximum value of the C features of each sample is found. Then, in step S706, sub max(N,C) is executed, that is, the C features of each sample are respectively subtracted from their corresponding maximum values.

[0055] Different from Figure 2 In step S708, a table lookup operation is performed through the solution disclosed in the present invention, namely, "lookup intable(float)(N,C)", to find the values ​​corresponding to the C features of each sample in the activation table. It can be seen that this step replaces Figure 2 The dequantization and exponential (exp) operations in Figure 2 dequant(float)(N,C)→exp()(N,C) in

[0056] Next, in step S710, sum(N,) is executed, that is, a sum operation is performed on the C features of each sample. Thereafter, in step S712, recip(N,) is executed, that is, a reciprocal operation is performed on the summed result of each sample. In step S714, mul(N,C) is executed, that is, the C features of each sample are multiplied by the corresponding reciprocal result, that is, a matrix multiplication operation.

[0057] Finally, in step S716, quant(int8)(N,C) is executed, that is, the int8 quantization operation is performed on the C features of each sample to obtain the output result of the Softmax activation function. In this way, through the table lookup strategy, the method disclosed in the present invention not only greatly improves the calculation efficiency when calculating the Softmax activation function, but also ensures the accuracy of the calculation result.

[0058] Figure 8 Schematic diagram of discrete activation of GeLU activation function disclosed in the present invention is shown. Figure 8 As shown, 256 values ​​from -128 to 127 are shown in the virtual box 801. These values ​​are all possible values ​​obtained by performing a matrix multiplication (matmul) operation on the quantized input and filter 1 (filter1). Next, a dequantization (dequant) operation is performed in the virtual box 802 to restore these values ​​to the original floating-point values. At this time, these values ​​can be added to the 4*hidden_size values ​​of the bias, and this step is performed in the virtual box 803. It should be noted that, unlike the MHA module, an operation of adding bias is added after the matrix multiplication operation. Assuming that the dimension of the input is (n, hidden_size) and the dimension of filter 1 is (hidden_size, 4*hidden_size), the dimension of the bias is (4*hidden_size, 1).

[0059] After the matrix multiplication operation is completed, as shown in the dashed box 801, the possible values ​​are in the 256 numbers [-128, 127]. After dequantization, each of these 256 numbers may be added to 4*hidden_size values ​​of bias, so that the GeLU activation function needs to calculate the activation values ​​of 256*4*hidden_size discrete values.

[0060] Then, the operation of the GeLU activation function is performed in the dashed box 804. At the dashed box 805, a floating point value can be obtained, and a quantization operation is performed at the dashed box 806. This process ensures the accuracy and efficiency of the calculation, and also avoids the calculation error caused by a large number of floating point operations.

[0061] It is worth noting that although the input value undergoes multiple operations such as quantization, dequantization, adding bias, and GeLU activation function, in each step, the input value and the output value maintain a one-to-one correspondence. This is because these operations are deterministic and do not introduce any randomness. For example, the dequantization operation simply converts the quantized integer value back to its original floating-point value, while adding bias and applying the GeLU activation function are both based on a fixed calculation method.

[0062] It is based on this one-to-one correspondence that the solution disclosed herein proposes that the output values ​​corresponding to all possible input values ​​can be calculated in advance, and then these results are stored in a table, which is the activation table disclosed herein. In the actual calculation process, it is only necessary to find the corresponding output value in the activation table according to the input value, which can avoid a large number of floating-point operations, thereby greatly improving the calculation efficiency. At the same time, since the results in the activation table are pre-calculated and accurately stored, the accuracy of the calculation results can also be guaranteed.

[0063] Fig. 9 Schematic diagram showing the calculation process of FFN under discrete activation of the GeLU activation function disclosed in the present invention. Fig. 9 As shown, first, in steps S902 and S904, the original input data input (for example, int8 data) and the output of the first layer neural network weight parameter filter1 are received. Then, in steps S906 and S908, the aforementioned original input data input and the output of filter1 are quantized (quantized) respectively. Based on the result after the quantization operation, in step S910, a matrix multiplication (matmul) operation is performed on the input data after the quantization operation and the quantized output of filter1.

[0064] Different from Figure 3 In step S912, a table lookup operation, i.e., "lookup intable", is performed through the solution disclosed in the present invention to find the value corresponding to the result of the matrix multiplication on the activation table after the dequant operation, the add bias operation, the GeLU activation function, and the quant operation. It can be seen that this step replaces the above Figure 3 The dequantization, bias addition, Gaussian error linear unit (GeLU) activation function and quantization (quant) operation in the Figure 3"dequant→add bias&gelu→quant" in . Then, in steps S914 and S916, the output of the second-layer neural network weight parameter filter2 is received and quantized. Thereafter, in step S918, the quantized output of step S912 and the quantized output of filter2 are matrix multiplied. Finally, in step S920, the result of the matmul operation is dequantized (dequant), and in step S922, a bias addition (add bias) operation is performed to obtain the output result based on the GeLU activation function. Based on this, the table lookup strategy not only greatly improves the calculation efficiency, but also ensures the accuracy of the calculation results.

[0065] Fig.10 FIG. 4 shows an exemplary schematic diagram of the table lookup of FFN under discrete activation of the GeLU activation function disclosed in the present invention. Fig.10 exemplarily shows a schematic diagram of a lookup table with n=4 and hidden_size=3 in the input dimension, wherein the left side of the figure shows all possible values ​​obtained by performing a matrix multiplication (matmul) operation on the quantized input and filter 1 (filter1), and the value range is 256 values ​​from -128 to 127. The right side of the figure shows a lookup table, each row of which corresponds to the 256 values ​​from -128 to 127 obtained by the aforementioned matmul operation, and each column corresponds to the column where the 256 values ​​from -128 to 127 obtained by the matmul operation are located.

[0066] For example, in an exemplary scenario, assuming that the value of the left side is -125, and the corresponding value after the dequant operation, add bias operation, GeLU activation function and quant operation can be calculated by looking up the lookup table shown on the right side of the figure. First, find the row where -125 is located in the lookup table, and the figure shows -125 in the fourth row of the lookup table. Next, determine the number of columns where -125 is located on the left side (for example, the figure shows that -125 is in the third column), so that the third column can be found in the lookup table, such as shown in the A position in the lookup table. That is, when the value of the left side is calculated to be -125, and the corresponding value after the dequant operation, add bias operation, GeLU activation function and quant operation, the value of the fourth row and third column in the lookup table on the right can be found, and the value is the output result. Similarly, when the value of the left side is calculated to be 126, and the corresponding value after the dequant operation, add bias operation, GeLU activation function and quant operation, the value of the eighth column of the second to last row in the lookup table on the right can be found, and the value is the output result.

[0067] It can be understood that for the Softmax discrete activation algorithm in the MHA disclosed in the present invention, since it does not include the addbias operation, its corresponding lookup table contains one column. When searching through the lookup table, it is only necessary to determine the row where the 256 values ​​from -128 to 127 obtained by the matmul operation are located, and the corresponding value can be found, and the value is the output result. It can be seen that the embodiment of the present disclosure obtains the output value by directly looking up the table, avoiding a large number of repeated calculations, thereby greatly improving the calculation efficiency.

[0068] Through the above description in combination with the accompanying drawings, it can be understood that the present disclosure also discloses a device for a bidirectional encoder representation based on a converter (BERT) network, which includes a processor and a memory. The memory stores a software program for a bidirectional coded representation based on a converter (BERT) network, and when the software program is executed by the processor, the method steps and / or processing flow described in combination with the accompanying drawings are implemented.

[0069] Fig.11 FIG. 1 is a schematic diagram showing the structure of a board 1100 according to an embodiment of the present disclosure. Fig.11 As shown, the board 1100 includes a chip (or "processing chip") 1101, which is a system-on-chip (SoC), or system on chip, integrated with one or more combined processing devices, which are artificial intelligence computing units that support various deep learning and machine learning algorithms to meet the intelligent processing needs in complex scenarios in the fields of computer vision, speech, natural language processing, data mining, etc. In particular, deep learning technology is widely used in the field of cloud intelligence. A notable feature of cloud intelligence applications is the large amount of input data, which places high demands on the storage capacity and computing power of the platform. The board 1100 of this embodiment is suitable for cloud intelligence applications and has huge off-chip storage, on-chip storage, and a large amount of computing power.

[0070] The chip 1101 is connected to an external device 1103 via an external interface device 1102. The external device 1103 is, for example, a server, a computer, a camera, a display, a mouse, a keyboard, a network card, or a WIFI interface. The data to be processed can be transmitted from the external device 1103 to the chip 1101 via the external interface device 1102. The calculation result of the chip 1101 can be transmitted back to the external device 1103 via the external interface device 1102. According to different application scenarios, the external interface device 1102 can have different interface forms, such as a PCIe interface.

[0071] The board 1100 also includes a storage device 1104 for storing data, which includes one or more storage units 1105. The storage device 1104 is connected to the control device 1107 and the chip 1101 through a bus and transmits data. The control device 1106 in the board 1100 is configured to regulate the state of the chip 1101. To this end, in an application scenario, the control device 1106 may include a single-chip microcomputer (Micro Controller Unit, MCU). In the application scenario of the bidirectional encoder representation network solution based on the converter disclosed in the present invention, a driver can be run in the control device and it includes a device of the embodiment of the present disclosure. When the aforementioned driver is controlled to run by the control device, the device executes the method steps and / or processing flow described above in combination with the accompanying drawings, so that the tasks in each task flow are sent to the processing chip for execution.

[0072] Fig.12 1 is a schematic diagram showing the structure of the combined processing device 1200 in the chip 1101 according to an embodiment of the present disclosure. Fig.12 As shown in FIG. 1 , the combined processing device 1200 includes an integrated circuit device 1201 , an interface device 1202 , a processing device 1203 , and a DRAM 1204 .

[0073] The integrated circuit device 1201 is configured to perform user-specified operations, and is mainly implemented as a single-core intelligent processor or a multi-core intelligent processor to perform deep learning or machine learning calculations. It can interact with the processing device 1203 through the interface device 1202 to jointly complete the user-specified operations.

[0074] The interface device 1202 is used to transmit data and control instructions between the integrated circuit device 1201 and the processing device 1203. For example, the integrated circuit device 1201 can obtain input data from the processing device 1203 via the interface device 1202 and write it into the storage device on the integrated circuit device 1201. Further, the integrated circuit device 1201 can obtain control instructions from the processing device 1203 via the interface device 1202 and write them into the control cache on the integrated circuit device 1201. Alternatively or optionally, the interface device 1202 can also read data in the storage device of the integrated circuit device 1201 and transmit it to the processing device 1203.

[0075] The processing device 1203 is a general processing device that performs basic controls including but not limited to data handling, starting and / or stopping the integrated circuit device 1201. Depending on the implementation, the processing device 1203 can be a central processing unit (CPU), a graphics processing unit (GPU) or one or more types of processors in other general and / or special processors, which include but are not limited to digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc., and their number can be determined according to actual needs. As mentioned above, only with respect to the integrated circuit device 1201 disclosed in the present invention, it can be regarded as having a single-core structure or a homogeneous multi-core structure. However, when the integrated circuit device 1201 and the processing device 1203 are integrated and considered together, the two are regarded as forming a heterogeneous multi-core structure.

[0076] DRAM 1204 is used to store data to be processed, and is a DDR memory, which is usually 16G or larger in size, and is used to store data of the integrated circuit device 1201 and / or the processing device 1203. In one or more implementation scenarios, the memory management solution disclosed herein can be applied to the management and maintenance of the DDR, thereby realizing the reuse or recycling operation of events. In this case, the board disclosed herein can be regarded as the device side of the artificial intelligence computing system.

[0077] It should be noted that, for the purpose of simplicity, the present disclosure describes some methods and embodiments thereof as a series of actions and combinations thereof, but those skilled in the art will appreciate that the scheme of the present disclosure is not limited by the order of the actions described. Therefore, based on the disclosure or teaching of the present disclosure, those skilled in the art will appreciate that some of the steps therein may be performed in other orders or simultaneously. Further, those skilled in the art will appreciate that the embodiments described in the present disclosure may be regarded as optional embodiments, i.e., the actions or modules involved therein may not necessarily be necessary for the implementation of one or some of the schemes of the present disclosure. In addition, depending on the different schemes, the present disclosure also has different focuses on the description of some embodiments. In view of this, those skilled in the art may appreciate the parts that are not described in detail in a certain embodiment of the present disclosure, and may also refer to the relevant descriptions of other embodiments.

[0078] In terms of specific implementation, based on the disclosure and teaching of this disclosure, those skilled in the art can understand that several embodiments disclosed in this disclosure can also be implemented by other methods not disclosed herein. For example, with respect to the various modules in the device or apparatus embodiments described above, this article divides them on the basis of considering logical functions, and there may be other division methods in actual implementation. For another example, multiple modules or components can be combined or integrated into another system, or some features or functions in the modules or components can be selectively disabled. With respect to the connection relationship between different modules or components, the connection discussed in the above text in conjunction with the accompanying drawings can be a direct or indirect coupling between modules or components. In some scenarios, the aforementioned direct or indirect coupling involves a communication connection using an interface, wherein the communication interface can support electrical, optical, acoustic, magnetic or other forms of signal transmission.

[0079] In the present disclosure, the modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules. The aforementioned components or modules may be located in the same location or distributed on multiple systems. In addition, according to actual needs, some or all of the modules may be selected to achieve the purpose of the scheme described in the embodiments of the present disclosure. In addition, in some scenarios, multiple modules in the embodiments of the present disclosure may be integrated into one module or each module may exist physically separately.

[0080] In some implementation scenarios, the above modules can be implemented in the form of software program modules. If implemented in the form of software program modules and sold or used as independent products, the modules can be stored in a computer-readable memory. Based on this, when the scheme of the present disclosure is embodied in the form of a software product (such as a computer-readable storage medium), the software product can be stored in a memory, which may include several instructions to enable a computer device (such as a personal computer, a server or a network device, etc.) to perform some or all of the steps of the method described in the embodiment of the present disclosure. The aforementioned memory may include, but is not limited to, various media that can store program codes, such as a USB flash drive, a flash drive, a read-only memory ("Read Only Memory", abbreviated as ROM), a random access memory ("Random Access Memory", abbreviated as RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0081] In some other implementation scenarios, the above modules can also be implemented in the form of hardware, that is, a specific hardware circuit, which may include a digital circuit and / or an analog circuit. The physical implementation of the hardware structure of the circuit may include but is not limited to physical devices, and the physical devices may include but are not limited to devices such as transistors or memristors. In view of this, the various devices described herein (such as integrated circuit devices or other processing devices) can be implemented by appropriate hardware processors, such as CPU, GPU, FPGA, DSP and ASIC. Furthermore, the aforementioned storage unit or storage device may be any appropriate storage medium (including magnetic storage medium or magneto-optical storage medium, etc.), which may be, for example, resistive random access memory ("Resistive Random Access Memory", abbreviated as RRAM), dynamic random access memory ("Dynamic Random Access Memory", abbreviated as DRAM), static random access memory ("Static Random Access Memory", abbreviated as SRAM), enhanced dynamic random access memory ("Enhanced Dynamic Random Access Memory", abbreviated as "EDRAM"), high bandwidth memory ("High Bandwidth Memory", abbreviated as "HBM"), hybrid memory cube ("Hybrid Memory Cube", abbreviated as "HMC"), ROM and RAM, etc.

[0082] The foregoing content can be better understood in accordance with the following terms:

[0083] Clause A1. A method for a Bidirectional Encoder Representation from Transformer (BERT) network, comprising:

[0084] When performing an operation related to calculation of a nonlinear activation function in a BERT network, receiving a discrete input value for calculation of the nonlinear activation function;

[0085] Finding a corresponding discrete output value in an activation table according to the discrete input value; and

[0086] The discrete output value is output as a result of calculating the nonlinear activation function.

[0087] Clause A2. The method according to clause A1, before the nonlinear activation function calculation related operation, the method further includes:

[0088] Precalculating the discrete input values ​​to obtain the corresponding discrete output values ​​to form the activation table; and

[0089] The activation table is cached on-chip for use in operations related to the calculation of the non-linear activation function.

[0090] Clause A3. The method of clause A2, wherein obtaining the corresponding discrete output value by precalculating the discrete input value comprises:

[0091] The kernel function is used to distribute the multiple discrete input values ​​to multiple computing cores for pre-computation, so as to obtain the corresponding discrete output values.

[0092] Item A4. A method according to any one of Items A1-A3, wherein the nonlinear activation function includes a Softmax activation function in a self-attention mechanism module of a BERT network and / or a Gaussian error linear unit (GeLU) activation function in a feed forward neural network module of a BERT network.

[0093] Clause A5. The method according to clause A4, when performing operations related to the calculation of the Softmax activation function, the method includes:

[0094] Receiving, in the self-attention mechanism module, a plurality of output values ​​calculated by matrix multiplication as the discrete input values;

[0095] Finding a corresponding discrete output value in the activation table according to the discrete input value; and

[0096] The discrete output value is output as a result of the Softmax activation function calculation.

[0097] Clause A6. The method according to clause A5, wherein receiving, in the self-attention mechanism module, a plurality of output values ​​calculated by matrix multiplication as the discrete input values ​​comprises:

[0098] Performing a subtraction operation on the result value obtained by the matrix multiplication calculation; and

[0099] The multiple output values ​​obtained after subtracting the maximum value are used as the discrete input values.

[0100] Clause A7. The method according to clause A4, when performing the operation of the GeLU activation function, the method includes:

[0101] Receiving, in the feedforward neural network module, a plurality of output values ​​calculated by the first matrix multiplication as the discrete input values;

[0102] Finding a corresponding discrete output value in the activation table according to the discrete input value; and

[0103] The discrete output value is output as a result of the GeLU activation function calculation.

[0104] Clause A8. The method according to clause A3, wherein when performing operations related to the Softmax activation function calculation, using a kernel function to distribute the multiple discrete input values ​​to multiple computing cores for pre-calculation includes: performing inverse quantization and exponential function calculation on the multiple discrete input values.

[0105] Clause A9. A method according to clause A3, wherein when performing the operation of the (GeLU) activation function, a kernel function is used to distribute the multiple discrete input values ​​to multiple computing cores for pre-calculation, including: performing inverse quantization on the multiple discrete input values, adding bias terms, calculating GeLU activation function values ​​and quantization operations.

[0106] Clause A10. A device for a Bidirectional Encoder Representation from a Transformer (BERT) network, comprising:

[0107] Processor; and

[0108] A memory having stored thereon a software program for a Bidirectional Coded Representation Based Transformer (BERT) network, which, when executed by the processor, implements the method according to any one of clauses A1-A9.

[0109] Clause A11. An integrated circuit device comprising the apparatus according to clause A10.

[0110] Item A12. A board comprising the integrated circuit device according to Item A11.

[0111] Item A13. A computer-readable storage medium having stored thereon a software program for a Bidirectional Coded Representation Based on Transformer (BERT) network, which, when executed by a processor, implements the method described in any one of Items A1-A9.

[0112] Although multiple embodiments of the present disclosure have been shown and described herein, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Those skilled in the art may think of many changes, modifications, and alternatives without departing from the thought and spirit of the present disclosure. It should be understood that in the process of practicing the present disclosure, various alternatives to the embodiments of the present disclosure described herein may be adopted. The attached claims are intended to define the scope of protection of the present disclosure, and therefore cover equivalents or alternatives within the scope of these claims.

Claims

1. A method for a Bidirectional Encoder Representation from Transformer (BERT) network, characterized in that include: When performing an operation related to calculation of a nonlinear activation function in a BERT network, receiving a discrete input value for calculation of the nonlinear activation function; Finding a corresponding discrete output value in an activation table according to the discrete input value; as well as The discrete output value is output as a result of calculating the nonlinear activation function.

2. The method according to claim 1, characterized in that Before the nonlinear activation function calculation related operations, the method further includes: Precalculating the discrete input values ​​to obtain the corresponding discrete output values ​​to form the activation table; and The activation table is cached on-chip for use in operations related to the calculation of the non-linear activation function.

3. The method according to claim 2, characterized in that Wherein obtaining the corresponding discrete output value by precalculating the discrete input value comprises: The kernel function is used to distribute the multiple discrete input values ​​to multiple computing cores for pre-computation, so as to obtain the corresponding discrete output values.

4. The method according to any one of claims 1 to 3, characterized in that: The nonlinear activation function includes a Softmax activation function in a self-attention mechanism module of a BERT network and / or a Gaussian error linear unit (GeLU) activation function in a feed forward neural network module of a BERT network.

5. The method according to claim 4, characterized in that When performing operations related to the calculation of the Softmax activation function, the method includes: Receiving, in the self-attention mechanism module, a plurality of output values ​​calculated by matrix multiplication as the discrete input values; Finding a corresponding discrete output value in the activation table according to the discrete input value; and The discrete output value is output as a result of the Softmax activation function calculation.

6. The method according to claim 5, characterized in that Wherein receiving a plurality of output values ​​calculated by matrix multiplication as the discrete input values ​​in the self-attention mechanism module comprises: Performing a subtraction operation on the result value obtained by the matrix multiplication calculation; and The multiple output values ​​obtained after subtracting the maximum value are used as the discrete input values.

7. The method according to claim 4, characterized in that When performing the operation of the GeLU activation function, the method includes: Receiving, in the feedforward neural network module, a plurality of output values ​​calculated by the first matrix multiplication as the discrete input values; Finding a corresponding discrete output value in the activation table according to the discrete input value; and The discrete output value is output as a result of the GeLU activation function calculation.

8. The method according to claim 3, characterized in that Wherein, when performing operations related to the calculation of the Softmax activation function, using a kernel function to distribute the multiple discrete input values ​​to multiple calculation cores for pre-calculation includes: performing inverse quantization and exponential function calculation on the multiple discrete input values.

9. The method according to claim 3, characterized in that: When executing the operation of the (GeLU) activation function, using the kernel function to distribute the multiple discrete input values ​​to multiple computing cores for pre-calculation includes: performing inverse quantization on the multiple discrete input values, adding bias terms, calculating the GeLU activation function value and quantization operation.

10. A device for a Bidirectional Encoder Representation from Transformer (BERT) network, characterized in that include: processor; as well as A memory having a software program for a Bidirectional Coded Representation Based on Transformer (BERT) network stored thereon, wherein when the software program is executed by the processor, the method according to any one of claims 1 to 9 is implemented.

11. An integrated circuit arrangement comprising the apparatus according to claim 10.

12. A card comprising the integrated circuit device according to claim 11.

13. A computer-readable storage medium having stored thereon a software program for a Bidirectional Coded Representation Based Transformer (BERT) network, which, when executed by a processor, implements the method according to any one of claims 1 to 9.

Images