A method to reduce the consumption of computing power by floating-point exponential operations

Abstract

The present invention discloses a linear lookup table construction method and a query method. The present invention establishes a linear lookup table for the result of exponential operation, and combines quantitative parameters such as scale factor to search and confirm the endpoints of each segment. The present invention can adapt to a wider range of activations, and in extreme cases, obtain a better computing speed than a one-to-one lookup table, while greatly reducing the consumption of computing power, and can effectively improve the efficiency of natural exponential operations in deep learning.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a method for reducing the consumption of computing power by floating-point exponential operations. Background Art

[0002] The activation function is an important component in neural networks. It is used to introduce nonlinear properties so that neural networks can learn and represent more complex functional relationships. The main function of the activation function is to perform nonlinear transformations on the output of the neural network. In a neural network, the output of each neuron is obtained by processing the input through the activation function. The introduction of the activation function enables the neural network to learn and represent nonlinear functional relationships, thereby improving the expressive power of the model.

[0003] Take the Softmax function as an example. As a classic activation function, the Softmax function is widely used in the field of deep learning. Initially, Softmax was used to convert the output of the model into a probability distribution for classification prediction. Later, in fields such as natural language processing and computer vision, the attention mechanism was proposed and widely used in various structures to improve the performance of the model. The attention mechanism enables the model to better handle long sequences or large-size images by assigning different weights to different parts of the input. In the attention mechanism, the Softmax function is often used to calculate the normalization of weights.

[0004] Neural network quantization is a model compression method that aims to convert floating-point models into full-integer models of target bits. It can significantly reduce the model size, reduce storage costs and improve computing speed at the same time. However, quantization usually cannot solve the computing power requirements brought by Softmax. Taking the attention mechanism as an example, the attention structure mainly consists of two major operations: matrix operations and activation. The Softmax operation is used to calculate the attention weight. The calculation of Softmax is usually performed element by element, and the calculation of each element depends on the values ​​of other elements. This dependency makes it difficult to parallelize the Softmax operation because each element must be calculated sequentially. In contrast, matrix operations can be more easily parallelized because each element in the matrix can be calculated independently. From the definition of the Softmax function, it can be seen that a large number of floating-point exponential operations are involved in the calculation process. Especially in the attention mechanism, the number of Softmax floating-point operations is proportional to the number of tokens. When the number of tokens is large, the computing power consumption is extremely large, and it is difficult to improve efficiency through parallelization. Although these exponential operations bring excellent performance, they also bring huge computing resource overhead, and have become the performance bottleneck in the attention mechanism and various Transformer structures.

[0005] The existing quantization technology solutions for activation are mainly the following:

[0006] 1. Do not quantize the calculation process. Insert a quantization node directly after the activation node to convert the activated floating-point output into a fixed-point output.

[0007] 2. Simplify the activation function into a form that is easier to implement.

[0008] a) Use a linear function to fit the exponential function.

[0009] First, convert the original exponential operation as follows:

[0010]

[0011] Among them, z represents the integer part of xloge, and r represents the mantissa part. , within this range , thus avoiding the index operation.

[0012] b) Use a quadratic function to fit the exponential function.

[0013] make ,in , , therefore, the form of the exponential function can be converted to:

[0014]

[0015] Constructing a quadratic function , use the least squares method to fit the unknown parameters.

[0016] c) Replace the summation of softmax directly with the maximum value.

[0017]

[0018] 3. Build a segmented lookup table or a one-to-one lookup table for fixed-point and floating-point numbers.

[0019] The above technologies have the following disadvantages:

[0020] 1. Do not quantize the calculation process, directly insert a quantization node after the activation node to convert the activated floating-point output into a fixed-point output. Although this method also produces fixed-point output in the end, the intermediate calculation process still cannot avoid floating-point exponential operations, so there is no optimization effect on the calculation complexity.

[0021] 2. Simplify the activation function into a form that is easier to implement. Fitting with a linear function or using the maximum value instead of the sum operation will lead to a large quantization error. The classification model only uses softmax in the last layer, so the error has little impact on the classification task, but for the multi-layer attention structure, it can be predicted that the error will accumulate layer by layer, affecting the final task accuracy. Although the method of fitting with a quadratic function avoids exponential operations, it still does not avoid floating-point multiplication, and its complexity is still higher than that of integer operations.

[0022] 3. Construct a segmented lookup table for fixed-point floating-point numbers, and multiply the results of each segment to obtain the final output. Although this method avoids floating-point exponential operations, it introduces four floating-point multiplications, which are still operations performed at the floating-point level. Summary of the invention

[0023] In view of the above-mentioned deficiencies in the prior art, the present invention provides a method for reducing the consumption of computing power by floating-point exponential operations, which solves the problem that floating-point exponential operations consume a large amount of computing resources.

[0024] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is:

[0025] A method for constructing a linear lookup table is provided, which comprises the following steps:

[0026] S1. Set the input range of the linear lookup table , get the first quantization parameter ;

[0027] S2, yes The data range is uniformly sampled to obtain sampling points; where n is the width of the first fixed-point integer;

[0028] S3, according to the sampling point and the first quantization parameter input , generate all possible first fixed-point integer numbers of input from small to large, and store all possible first fixed-point integer numbers of input in the candidate array;

[0029] S4, calculating the scale factor m according to the length of the linear lookup table;

[0030] S5, according to the scale factor m and the first quantization parameter Calculate the second quantization parameter ;

[0031] S6. Calculate the minimum interpolation coefficient p of the input data in the candidate array that meets the current range according to the scale factor m and the total number of sampling points, and align the first table entry start in the local table with the interpolation coefficient p;

[0032] S7, taking the first entry start in the local table as the reference point, and taking the current second quantization parameter Calculate the subsequent entries for the interval size, and update the first entry start and subsequent entries to the corresponding positions in the temporary table Y;

[0033] S8. Convert the temporary table Y into a second fixed-point integer according to the quantization rule and store it into the linear lookup table, thereby completing the construction of the linear lookup table.

[0034] A linear lookup table query method is provided, which comprises the following steps:

[0035] A1. Input fixed-point integer Multiply each element in by the scale factor m to get a fixed-point integer ;

[0036] A2. Get fixed-point integer The first 8 bits of the binary representation of are used as the index ind of the linear lookup table, and the last 8 bits are used as the interpolation coefficient p;

[0037] A3. Using index ind and interpolation coefficients Calculate the function value corresponding to the corresponding element ;

[0038] A4. According to the function value corresponding to the corresponding element Compute floating point results , complete the query; its expression is:

[0039]

[0040] in , is the input range of the linear lookup table.

[0041] The beneficial effects of the present invention are:

[0042] 1. This linear lookup table construction method only uses integer operations, which can avoid the computational resource consumption of floating-point exponential operations, improve the reasoning speed of the quantized network, and enable the network to use less linear lookup table storage space to ensure the calculation accuracy of complex functions. It can effectively improve the overall reasoning performance of the model and reduce the storage resources required in the calculation.

[0043] 2. The present invention avoids floating-point exponential operations, saves computing resources, and requires less storage space than a one-to-one lookup table, saving storage resources. The present invention can effectively improve the efficiency of natural exponential operations involved in deep learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 This is a flow chart of the linear lookup table construction method;

[0045] Figure 2 It is a schematic diagram for the present invention to be examined in the embodiments. DETAILED DESCRIPTION

[0046] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0047] like Figure 1 As shown, the linear lookup table construction method includes the following steps:

[0048] S1. Set the input range of the linear lookup table , get the first quantization parameter ;

[0049] S2, yes The data range is uniformly sampled to obtain sampling points; where n is the width of the first fixed-point integer;

[0050] S3, according to the sampling point and the first quantization parameter input , generate all possible first fixed-point integer numbers of input from small to large, and store all possible first fixed-point integer numbers of input in the candidate array;

[0051] S4, calculating the scale factor m according to the length of the linear lookup table;

[0052] S5, according to the scale factor m and the first quantization parameter Calculate the second quantization parameter ;

[0053] S6. Calculate the minimum interpolation coefficient p of the input data in the candidate array that meets the current range according to the scale factor m and the total number of sampling points, and align the first table entry start in the local table with the interpolation coefficient p;

[0054] S7, taking the first entry start in the local table as the reference point, and taking the current second quantization parameter Calculate the subsequent entries for the interval size, and update the first entry start and subsequent entries to the corresponding positions in the temporary table Y;

[0055] S8. Convert the temporary table Y into a second fixed-point integer according to the quantization rule and store it into the linear lookup table, thereby completing the construction of the linear lookup table.

[0056] The expression for generating the first fixed-point integer of all possible inputs is:

[0057]

[0058] in Indicates the i-th sampling point The corresponding first fixed-point integer; Represents the rounding function.

[0059] The calculation formula of the scale factor m is:

[0060]

[0061] in represents the rounding function; is the length of the linear lookup table.

[0062] Second quantization parameter The calculation expression is:

[0063] .

[0064] The calculation expression of the interpolation coefficient p of the minimum input data in the candidate array that meets the current range is:

[0065]

[0066] in Represents the first fixed-point integer corresponding to the starting point in the candidate array; Represents the remainder operation.

[0067] The expression for aligning the first entry start in the local table with the interpolation coefficient p is:

[0068]

[0069] in Indicates the first sampling point The position of the corresponding first fixed-point integer in the local table, that is, the position of the first element in the local table.

[0070] The calculation expression of the subsequent table items is:

[0071]

[0072] in is the i-th entry.

[0073] The expression for converting the temporary table Y into the second fixed-point integer and storing it in the linear lookup table is:

[0074]

[0075] in Represents the i-th element stored in the linear lookup table; represents the rounding function; is the i-th entry in the temporary table Y.

[0076] The query method of the linear lookup table comprises the following steps:

[0077] A1. Input fixed-point integer Multiply each element in by the scale factor m to get a fixed-point integer ;

[0078] A2. Get fixed-point integer The first 8 bits of the binary representation of are used as the index ind of the linear lookup table, and the last 8 bits are used as the interpolation coefficient p;

[0079] A3. Using index ind and interpolation coefficients Calculate the function value corresponding to the corresponding element , whose expression is:

[0080]

[0081] in Indicates the data corresponding to index ind in the linear lookup table; Indicates the data corresponding to the index ind+1 in the linear lookup table; k is the alignment parameter used when the temporary storage table Y is converted into the second fixed-point integer, and the value in this embodiment is 16; / / is the integer division symbol;

[0082] A4. According to the function value corresponding to the corresponding element Compute floating point results , complete the query; its expression is:

[0083]

[0084] in , is the input range of the linear lookup table.

[0085] In the common table lookup method, the query object is mainly 8-bit or less integer data, so the lookup table is usually constructed in a one-to-one manner. However, when the quantization granularity is small or the bit width of the output data is large, if the lookup table is still constructed in a one-to-one manner, it will cause resource waste in insensitive areas and excessive lookup table storage space requirements.

[0086] The query method proposed by the present invention is as follows Figure 2 As shown, Figure 2The figure shows the lookup table created by this linear lookup table construction method. The red triangle represents the queried value, and the black square represents the actual value. The horizontal axis represents the query to be searched, and the height of each yellow column represents the actual data stored in the lookup table item.

[0087] In one embodiment of the present invention, taking the reasoning of the complex function softmax as an example, the comparison between this method and other more advanced quantization acceleration methods is shown in Table 1. All algorithms selected in the experiment are implemented based on C++ language, and the program is compiled using g++. The processor platform is Intel(R) Xeon(R) Silver 4210 CPU @ 2.20GHz. A set of randomly generated tensors of size [1000, 1] are used. The calculation reasoning is performed respectively using the methods in the table, and the reasoning time and L2 error of each algorithm are recorded as evaluation indicators to compare the performance of each algorithm. It can be seen that this method achieves the best balance between reasoning time and reasoning error.

[0088] Table 1: Inference test results of applying various methods to softmax

[0089]

[0090] In summary, the present invention establishes a linear lookup table for the result of exponential operation, and combines quantitative parameters such as scale factor to search and confirm the endpoints of each segment. The present invention can adapt to a wider range of activations, and in extreme cases, obtain a better operation speed than a one-to-one lookup table, while greatly reducing the consumption of computing power.

Claims

1. A method for reducing the consumption of computing power by floating-point exponential operations, characterized in that: The following steps are involved: S1. Set the input range of the linear lookup table , get the first quantization parameter ; S2, yes The data range is uniformly sampled to obtain sampling points; where n is the width of the first fixed-point integer; S3, according to the sampling point and the first quantization parameter input , generate all possible first fixed-point integer numbers of input from small to large, and store all possible first fixed-point integer numbers of input in the candidate array; S4, calculating the scale factor m according to the length of the linear lookup table; S5, according to the scale factor m and the first quantization parameter Calculate the second quantization parameter ; S6. Calculate the minimum interpolation coefficient p of the input data in the candidate array that meets the current range according to the scale factor m and the total number of sampling points, and align the first table entry start in the local table with the interpolation coefficient p; S7, taking the first entry start in the local table as the reference point, and taking the current second quantization parameter Calculate the subsequent entries for the interval size, and update the first entry start and subsequent entries to the corresponding positions in the temporary table Y; S8. Convert the temporary table Y into a second fixed-point integer according to the quantization rule and store it into the linear lookup table, thereby completing the construction of the linear lookup table. S9, input fixed-point integer Multiply each element in by the scale factor m to get a fixed-point integer ; S10, get fixed-point integer The first 8 bits of the binary representation of are used as the index ind of the linear lookup table, and the last 8 bits are used as the interpolation coefficient p; S11. Using index ind and interpolation coefficient Calculate the function value corresponding to the corresponding element ; S12, according to the function value corresponding to the corresponding element Compute floating point results , the query is completed, that is, the floating-point exponential operation reduces the computing power consumption of the computer system; Its expression is: in , is the input range of the linear lookup table; Use index ind and interpolation coefficient p to calculate the function value corresponding to the corresponding element The expression is: ； in Indicates the data corresponding to index ind in the linear lookup table; Indicates the data corresponding to index ind+1 in the linear lookup table; k is the alignment parameter used when the temporary storage table Y is converted to the second fixed-point integer; / / is the integer division symbol.

2. The method for reducing the consumption of computing power by floating-point exponential operations according to claim 1, characterized in that: The expression for generating the first fixed-point integer of all possible inputs is: ； in Indicates the i-th sampling point The corresponding first fixed-point integer; Represents the rounding function.

3. The method for reducing the consumption of computing power by floating-point exponential operations according to claim 1, characterized in that: The calculation formula of the scale factor m is: ； in represents the rounding function; is the length of the linear lookup table.

4. The method for reducing the consumption of computing power by floating-point exponential operations according to claim 1, characterized in that: Second quantization parameter The calculation expression is: 。 5. The method for reducing the consumption of computing power by floating-point exponential operations according to claim 1, characterized in that: The calculation expression of the interpolation coefficient p of the minimum input data in the candidate array that meets the current range is: ； in Represents the first fixed-point integer corresponding to the starting point in the candidate array; Represents the remainder operation.

6. The method for reducing the consumption of computing power by floating-point exponential operations according to claim 1, characterized in that: The expression for aligning the first entry start in the local table with the interpolation coefficient p is: ； in Indicates the first sampling point The position of the corresponding first fixed-point integer in the local table, that is, the position of the first element in the local table.

7. The method for reducing the consumption of computing power by floating-point exponential operations according to claim 1, characterized in that: The calculation expression of the subsequent table items is: ； in is the i-th entry.

8. The method for reducing the consumption of computing power by floating-point exponential operations according to claim 1, characterized in that: The expression for converting the temporary table Y into the second fixed-point integer and storing it in the linear lookup table is: ； in Represents the i-th element stored in the linear lookup table; represents the rounding function; is the i-th entry in the temporary table Y.

Images