Approximation method for softmax function calculation and neural network applying this approximation method for softmax function calculation

Abstract

An approximation method for softmax function calculation and a neural network are provided that can reduce calculation time and energy consumption. [Solution] The approximation method of softmax function calculation converts the input value of a k-dimensional vector into the output value of an m-dimensional vector, and includes an exponential approximation calculation sequence, an additive calculation sequence, and a division calculation sequence. The exponential approximation calculation sequence performs a leaky normalized linear unit (Leaky ReLU) calculation on the input value of the k-dimensional vector to obtain a normalized function calculation value, then performs a polynomial function calculation of a predetermined degree based on the normalized function calculation value to obtain an exponential approximation value, and repeats the exponential approximation calculation sequence for another input value to obtain another exponential approximation value. The additive calculation sequence obtains a sum value by adding the exponential approximation value and another exponential approximation value. The division calculation sequence obtains an output value of the m-dimensional vector by dividing at least one exponential approximation value among the exponential approximations obtained by the exponential approximation calculation sequence by the sum value.

Description

[Technical field]

[0001] The present invention relates to an approximation method for softmax function calculation and a neural network to which this approximation method for softmax function calculation is applied, and in particular to an approximation method for softmax function calculation used in a classification machine of an artificial intelligence deep learning model and a neural network to which this approximation method for softmax function calculation is applied. [Background technology]

[0002] Artificial intelligence (AI) is a technology that usually represents human intelligence through ordinary computer programs. The most important part of AI is the mathematical or computational model that mimics the structure and function of biological neural networks in machine learning and cognitive science, and is an approximation of artificial neural networks that estimate functions. Common neural network models include convolutional neural networks and recurrent neural networks (CNN and RNN), and in recent years, the Transformer model has been developed, and the Transformer model has gradually replaced convolutional neural networks and recurrent neural networks (CNN and RNN) and has become the most popular deep learning model.

[0003] As shown in Fig. 1, most of the above neural models include at least a feature learning machine 81 and a classification machine 82. A softmax function 821 is usually adopted in the classification machine 82 of the neural model, and outputs at least one output value between 0 and 1.

[0004] The expansion formula for the conventional softmax function is given by Equation 1.

[0005]

number

[0006] Generally, calculations using the softmax function (Formula 1) convert input values ​​of an i-dimensional vector into output values ​​of a j-dimensional vector, where each output value of the j-dimensional vector is usually a number between 0 and 1, and the sum of these numbers is 1.

[0007] In addition, most of the softmax function calculations related to commercially available GPUs (e.g., Nvidia) use the softmax function of Formula 1, and the input value is in float32 format to realize the calculation of the softmax function. However, when actually calculating the exponential function, there are many degrees of polynomials, and the input value is in float32 format, so there is a problem that a large amount of numerical calculations need to be processed in the calculation process of the classification machine, which consumes time and energy. Therefore, it is a very important task to provide a simple softmax function calculation and make the calculation process of the neural network classification machine time-saving and energy-saving. Summary of the Invention [Problem to be solved by the invention]

[0008] An object of the present invention is to provide an approximation method for softmax function calculation that can reduce calculation time and energy consumption. Another object of the present invention is to provide a neural network that applies this approximation method for softmax function calculation that can reduce calculation time and energy consumption. [Means for solving the problem]

[0009] In order to achieve the above object, the input value of a k-dimensional vector is converted into an output value of an m-dimensional vector according to an approximation method of softmax function calculation of the present invention. The approximation method of softmax function calculation includes an exponential approximation calculation sequence in which the input value of the k-dimensional vector is subjected to leaky normalized linear unit calculation (leaky ReLU) to obtain a normalized function calculation value, a certain degree polynomial function calculation is performed based on the normalized function calculation value to obtain an exponential approximation value, and the sequence is repeated with another input value to obtain another exponential approximation value, an additive calculation sequence in which the sum of the exponential approximation value and the other exponential approximation value is obtained, and a division calculation sequence in which at least one exponential approximation value among the exponential approximations obtained in the exponential approximation calculation sequence is divided by the sum to obtain an output value of an m-dimensional vector.

[0010] In one embodiment, in the exponential approximation calculation sequence, the input values ​​are first subjected to a clamp function calculation, and then further subjected to a leaky ReLU calculation.

[0011] In one embodiment, in the additive calculation sequence, the sum further includes a guard value to ensure that the absolute value of the sum is greater than zero.

[0012] In one embodiment, the constant degree polynomial function calculation is a second to fifth degree polynomial calculation.

[0013] In one embodiment, the exponential approximation calculation sequence is repeated until each input value undergoes leaky ReLU calculation to obtain a normalization function calculation value, and then a certain degree of polynomial function calculation is performed based on the normalization function calculation value to obtain an exponential approximation value corresponding to each input value. The additive calculation sequence obtains a total value by adding up all the exponential approximations. The division calculation sequence obtains an output value of an m-dimensional vector corresponding to a plurality of k-dimensional vectors by dividing each exponential approximation value obtained in the exponential approximation calculation sequence by the total value.

[0014] In one embodiment, the input value is an integer number.

[0015] Also, based on the neural network to which the approximation method of softmax function calculation of the present invention is applied, the classification machine of the neural network is provided with a softmax function calculation module that converts an input value of a k-dimensional vector into an output value of an m-dimensional vector. The softmax function calculation module includes an exponential approximation calculation unit that performs a leaky normalized linear unit function (LeakyReLU function) calculation on an input value of the k-dimensional vector to obtain a normalized function calculation value, performs a polynomial function calculation of a certain degree based on the normalized function calculation value to obtain an exponential approximation value, and then repeats the leaky normalized linear unit function calculation and the polynomial function calculation of the certain degree on another input value to obtain another exponential approximation value, an addition calculation unit that obtains a sum value by adding the exponential approximation value and the other exponential approximation value, and a division calculation unit that divides at least one of the exponential approximations obtained by the exponential approximation calculation unit by the sum value to obtain an output value of the m-dimensional vector.

[0016] In another embodiment, the exponential approximation calculation unit first performs a clamp function calculation on the input value, and then performs a leaky ReLU calculation.

[0017] In another embodiment, in the summation unit, the sum further includes a protection value to ensure that the absolute value of the sum is greater than zero.

[0018] In another embodiment, the constant degree polynomial function calculation is a second to fifth degree polynomial calculation.

[0019] In another embodiment, the exponential approximation calculation unit performs leaky ReLU calculation for each input value to obtain a normalization function calculation value, and then performs a certain degree polynomial function calculation using the normalization function calculation value until an exponential approximation value corresponding to each input value is obtained. The addition calculation unit obtains a total value by adding all the exponential approximations. The division calculation unit divides each exponential approximation value obtained by the exponential approximation calculation unit by the total value to obtain an output value of an m-dimensional vector corresponding to a plurality of k-dimensional vectors.

[0020] In one embodiment, the input value is an integer number. Effect of the Invention

[0021] In the approximation method of the softmax function calculation of the present invention, the exponential function e (xk) is restricted to low-order polynomials (e.g., quadratic polynomials), and clamp functions and leaky ReLU calculations are employed, so that the exponential function e (xk) When calculating, the calculation result shown by the dashed curve can be approximated to the higher order calculation result shown by the solid curve, in other words, the output errors of the softmax function calculation formula of the present invention are all small.

[0022] In particular, all elements of the input vector are integers, and the exponential function e (xk) Since is restricted to low-order polynomials (e.g., quadratic polynomials), the amount of calculation is significantly reduced compared to the conventional float32 format and calculations of higher-order polynomials, which shortens calculation time and reduces energy consumption. [Brief description of the drawings]

[0023] [Figure 1] FIG. 1 is an explanatory diagram of a basic structure of a conventional neural network. [Diagram 2] FIG. 11 is a comparison diagram between the quadratic polynomial function calculation result of the preferred embodiment of the present invention and the conventional exponential calculation result. [Diagram 3] FIG. 13 is a flow diagram of adding leaky ReLU calculation to the exponential approximation calculation sequence of a preferred embodiment of the present invention. [Figure 4] FIG. 2 is a block diagram illustrating an approximation method for softmax function calculation in a preferred embodiment of the present invention. [Diagram 5] FIG. 11 is a comparison diagram between the calculation result of the exponential approximation calculation sequence of the preferred embodiment of the present invention and the conventional exponential calculation result; [Figure 6] FIG. 13 is a flow diagram of an exponential approximation calculation sequence according to a preferred embodiment of the present invention, further including a clamp function calculation. [Figure 7]FIG. 11 is a block diagram illustrating another approximation method for softmax function calculation in a preferred embodiment of the present invention. [Figure 8] FIG. 11 is a comparison diagram between the calculation result of another exponential approximation calculation sequence of the preferred embodiment of the present invention and the conventional exponential calculation result; [Figure 9] 4 is a comparison table of output values ​​of the softmax function formula generated by adopting different exponential function formulas in the preferred embodiment of the present invention; [Figure 10] In a preferred embodiment of the present invention, different exponential function formulas are adopted, and FIG. 9 is a comparison table of the output values ​​of the softmax function formula generated by different input vectors. [Figure 11] In a preferred embodiment of the present invention, different exponential function formulas are adopted, and a protection value is added to one of the softmax function formulas to obtain a softmax function output value. [Figure 12] 11 and 12. In a preferred embodiment of the present invention, the exponential function formula and the softmax function formula similar to those in FIG. 11 are respectively used, and the input vector similar to that in FIG. 9 is used to generate a comparison table of output values ​​of the softmax function formula. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0024] The approximation method for softmax function calculation of the present invention and a neural network to which this approximation method for softmax function calculation is applied will be described below with reference to the drawings.

[0025] Before describing the embodiment of the present invention in detail, it is first necessary to note that in this embodiment, the softmax function can convert the input value of a k-dimensional vector into the output value of an m-dimensional vector, and therefore the softmax function of this embodiment can be expressed as Equation 2.

[0026]

number

[0027] In the calculation of the softmax function in Equation 2 above, the most difficult and time-consuming calculation is to calculate the input value of the k-dimensional vector as the exponential function exp(x) (i.e., e (xk) ) is calculated. (xk) The value is generally calculated using the Taylor expansion formula, i.e., Equation 3.

[0028]

number

[0029]

number

[0030] Next, as shown in FIG. 2, the exponential function value exp(x) calculated by the high-order polynomial shown in Equation 3 is represented by a solid curve, and the exponential function value (2nd-taylor exp(x)) calculated by the second-order Taylor polynomial shown in Equation 4 is represented by a dashed curve. As shown in FIG. 2, if the calculation order of the entire Taylor index expansion formula (Equation 3) is reduced to a low-order polynomial calculation in order to simplify the exponential calculation, for example, by limiting it to a second-order polynomial calculation, the exponential function value (2nd-taylor exp(x)) represented by the dashed curve as shown in FIG. 2 can be obtained. However, as can be seen from FIG. 2, there is a large difference between the calculation result of the second-order polynomial (shown by the dashed line) and the calculation result of the high-order polynomial (shown by the solid line).

[0031] This simplifies the exponential calculation of the softmax function by using the exponential function e (xk) When limiting to a quadratic polynomial calculation, in order to make the simplified calculation result closer to the calculation result before simplification, it is necessary to improve the calculation method of Equation 4 and make the calculation result closer to the solid curve shown in Figure 2.

[0032] As shown in Figures 3 and 4, the approximation method of the softmax function calculation of the present invention includes an exponential approximation calculation sequence S1, an addition calculation sequence S2, and a division calculation sequence S3. The exponential approximation calculation sequence S1 performs a leaky ReLU function calculation step S11 and a quadratic polynomial exponential approximation calculation step S12 shown in Figure 3. Also, as shown in Figure 4, the output result of the above formula 2 can be obtained by these calculation processes.

[0033] Furthermore, as shown in FIG. 3, in this embodiment, the leaky ReLU is used to calculate Equation 4, and Equation 4 can be expressed as Equation 5.

[0034]

number

[0035] By using the leaky ReLU, the normalization function calculation value L1 is obtained in step S11. By substituting the normalization function calculation value L1 into Equation 5, Equation 5 can be expressed as Equation 6. By the calculation of Equation 6, the exponential approximation calculation value exp1(X k That is, by the calculations in steps S11 and S12, the exponential approximation calculation value exp1(X k ) can be calculated. The result of this calculation is expressed as an exponential function e (xk) If we restrict to a quadratic polynomial and adopt leaky ReLU calculation, X k When is negative, the approximation (shown by the dashed line) approaches the solid curve more closely.

[0036]

number

[0037] The calculation results of Equation 5 and Equation 6 are better than those of Equation 4, but as shown in Figure 5, X kWhen X is negative or greater than 1, the approximation result will have a certain degree of discrepancy. In order to solve the problem of the certain degree of discrepancy, in a preferred embodiment of the present invention, as shown in FIG. 6 and FIG. 7, the exponential approximation calculation sequence S1' is as follows: k After performing a clamp function calculation step S10, a leaky normalized linear unit function calculation step S11' is performed, and then a quadratic polynomial exponential approximation calculation step S12' is performed. That is, the calculation sequence is as shown in FIG. 6, and the quadratic polynomial exponential approximation calculation formula may be expressed as Equation 7.

[0038]

number

[0039] When a clamp function and leaky ReLU are used, a normalization function calculation value L2 is obtained in step S11′ by the calculation in step S10. When the normalization function calculation value L2 is substituted into Equation 7, Equation 7 can be expressed as Equation 8.

[0040]

number

[0041] In particular, if F(X k )=LeakyReLU(Clamp(X k , min, max) and taking into account the quadratic polynomial coefficients at the same time, the exponential approximation calculation value of the present invention can be expressed by a general formula such as Formula 9.

[0042]

number

[0043] In other words, when the normalization function calculation value L1 or the normalization function calculation value L2 is substituted into Equation 9, Equation 6 can be expressed as Equation 10, and Equation 8 can be expressed as Equation 11.

[0044]

number

[0045]

number

[0046] As can be seen from Figure 8, the exponential function e (xk) When we restrict to a quadratic polynomial and use clamp function calculation and leaky ReLU calculation, that is, when we use Equation 8 or Equation 11, the calculation result shown by the dashed curve is X k If x is a negative number or is in a range greater than 1, it approaches the high-order calculation result shown by the solid curve. Furthermore, according to the exponential calculation of Equation 8 or Equation 11, the exponential approximation calculation value exp2(X k In other words, the exponential approximation calculation value exp2(X k ) can be calculated.

[0047] The actual calculation of the approximation method of the softmax function calculation of the present invention will be described in detail below with reference to Figure 9. It is particularly noted that the softmax function calculation (i.e., Equation 2) of this embodiment is actually performed by adopting the softmax function calculation layer of the Transformer model.

[0048] Furthermore, as shown in Figure 9, when the input vector value is [-2,0,8], if the exponential function formula is Formula 3 and the softmax function calculation formula is Formula 2, then by substituting each vector value of the input vector value into Formula 3, we get e (-2) = 0.13, e (0) =1, e (8) = 2980. By substituting the results of each calculation in Equation 3 into Equation 2, the calculation value (output value) of the softmax function calculation formula (Equation 2) shown in Figure 9 is obtained, and the output vector is [0.000045, 0.000335, 0.999619].

[0049] As shown in Fig. 9, when the input vector value is [-2, 0, 8], if the exponential function formula is Formula 10 and the softmax function formula is Formula 2, the coefficients of Formula 10 are a = 1, b = 2, c = 1, and after each vector of the input vector is substituted into Formula 10 and calculated, the output vector of the calculation value (output value) of the softmax function formula (Formula 2) is [0.003040, 0.012158, 0.984802]. As can be seen from Fig. 9, for an input vector amount with a relatively large reaction rate (for example, inputting "8" in this embodiment), the reaction error is within 1 to 2%.

[0050] However, in the above explanation, if the exponential function formula is Formula 10, the softmax function formula is Formula 2, and the coefficients of Formula 10 are a=1, b=2, and c=1, when the input vector value is [-4,-4,-4], the calculation value (output value) of the softmax function formula (Formula 2) cannot be calculated, as shown in Figure 10. The reason for this phenomenon is that the denominator of the softmax function formula (Formula 2) is close to 0.

[0051] To solve the problem that the denominator of the softmax function calculation formula may approach 0 and therefore the calculation cannot be performed normally, please refer to Fig. 7. In this embodiment, the approximation method of the softmax function calculation of the present invention not only includes the exponential approximation calculation sequence S1', the addition calculation sequence S2 and the division calculation sequence S3, but also includes a protection value calculation sequence S21 in which the calculation value of the addition calculation sequence S2 is added to the protection value eps after the calculation is completed. Thus, the softmax function calculation formula of the present invention can be expressed as Equation 12. In addition, the function of the protection value eps is to ensure that the denominator of Equation 2 is not other than 0 or does not approach 0. If the denominator of Equation 2 is 0 or approaches 0, the softmax' (xk)m The calculations yield no results.

[0052]

number

[0053] Next, as shown in Fig. 11, if the exponential function formula is Formula 11, the softmax function formula is Formula 12, and the coefficients of Formula 11 are a=1, b=2, and c=1, even if the input vector value is [-4,-4,-4], the calculation value (output value) of the softmax function formula (Formula 12) can be calculated as shown in Fig. 11. Also, if the exponential function formula is Formula 11, the softmax function formula is Formula 12, and the coefficients shown in Formula 11 are a=1, b=2, and c=1, and the eps shown in Formula 12 is eps=1, even if the input vector value is [-2,0,8], the calculation value (output value) of the softmax function formula (Formula 12) can be calculated as [0.007007,0.016016,0.976977] as shown in Fig. 12. As can be seen by comparing with the output value 0.984802 (using Equation 2) shown in Figure 12, for input vector quantities with relatively large reaction rates (for example, inputting "8" in this embodiment), the reaction error is kept within 1 to 2%.

[0054] To summarize the above, in the approximation method for softmax function calculation of the present invention, the exponential function e (xk) is restricted to low-order polynomials (e.g., quadratic polynomials), and clamp functions and leaky ReLU calculations are employed, so that the exponential function e (xk) Whether it is calculated using Equation 8 or Equation 11, the calculation result shown by the dashed curve can be approximated to the high-order calculation result shown by the solid curve. In addition, in the approximation method of the softmax function calculation of the present invention, if the softmax function calculation formula is adopted as Equation 12 and the coefficients shown in Equation 11 are appropriately adjusted (e.g., a=1, b=2, c=1), the output errors of the softmax function calculation formula of the present invention for the main response values ​​are all small.

[0055] In particular, in this embodiment, all the elements of the input vector are integers, and the exponential function e (xk) Since is restricted to low-order polynomials (e.g., quadratic polynomials), the amount of calculation is significantly reduced compared to calculations using the conventional float32 format and higher-order polynomials, resulting in shorter calculation times and reduced energy consumption.

[0056] Another embodiment of the present invention provides a neural network that uses this approximation method of softmax function calculation. The specific description of the neural network that uses this approximation method of softmax function calculation of the present invention is almost the same as the above-mentioned method, so it is omitted here. The only thing to be noted is that in another embodiment of the present invention, the neural network is not limited to the Transformer model neural network.

[0057] The foregoing is intended to be illustrative, not limiting, of the present invention. Any modifications or variations that do not depart from the spirit and scope of the present invention should be within the scope of the appended claims. [Industrial Applicability]

[0058] The present invention relates to an approximation method for softmax function calculation used in a classification machine of an artificial intelligence deep learning model, and a neural network to which this approximation method for softmax function calculation is applied. [Explanation of symbols]

[0059] 81 Feature Learning Machine 82 Sorting Machine S1, S1' exponential approximation calculation order S10 Clamp function calculation step S11, S11' Leaky Regularized Linear Unit Function (Leaky ReLU) calculation steps S12, S12' Second-order polynomial exponential approximation calculation steps S2 additive calculation order S21 Protection value calculation order S3 Division calculation order L1 normalization function calculation value L2 normalization function calculation value

Claims

1. A neural network comprising: A softmax function calculation module is provided for converting the input value of a k-dimensional vector into an output value of an m-dimensional vector in the neural network classification machine; The softmax function calculation module: an exponential approximation calculation unit that obtains a normalized function calculation value by performing a leaky normalized linear unit function calculation on an input value of the k-dimensional vector, and then performs a polynomial function calculation of a fixed degree based on the normalized function calculation value to obtain an exponential approximation value, and then repeats the leaky normalized linear unit function calculation and the polynomial function calculation of the fixed degree on another input value to obtain another exponential approximation value; an addition calculation unit that obtains a sum value by adding the exponential approximation value and the other exponential approximation value; and a division calculation unit that divides at least one of the exponential approximations obtained by the exponential approximation calculation unit by the sum value to obtain an output value of the m-dimensional vector.

2. 2. The neural network according to claim 1, wherein said exponential approximation calculation unit performs a clamp function calculation on an input value first, and then performs said leaky normalized linear unit function calculation.

3. 3. The neural network of claim 2, wherein said sum further includes a guard value to ensure that the absolute value of said sum is greater than zero.

4. 2. The neural network of claim 1, wherein said constant degree polynomial calculation is a second to fifth degree polynomial calculation.

5. The neural network of claim 1, wherein the exponential approximation calculation unit performs the leaky normalized linear unit function calculation for each input value to obtain each normalized function calculation value, and then performs a polynomial function calculation of a certain degree using each normalized function calculation value to obtain an exponential approximation value corresponding to each input value; the addition calculation unit obtains a sum value by adding up all the exponential approximations; and the division calculation unit divides all the exponential approximations of the exponential approximations obtained by the exponential approximation calculation unit by the sum value to obtain an output value of an m-dimensional vector corresponding to a plurality of k-dimensional vectors.

6. 2. The neural network of claim 1, wherein said input values ​​are integer type numeric values.

Images