Normalized exponential operation approximation method and neural network applying same

Abstract

The invention discloses a normalized exponential operation approximation method, which is used for converting an input value of a k-dimensional vector into an output value of an m-dimensional vector, and comprises an exponential approximation operation program for carrying out leakage linear rectification function operation on one input value of the k-dimensional vector to obtain a rectification function operation value, performing polynomial function operation of a certain order according to the rectification function operation value to obtain an exponential approximation value, and repeating the exponential approximation operation program on another input value to obtain another exponential approximation value; the addition operation program adds up the index approximate value and the other index approximate value to obtain a total value; and a division operation program dividing at least one of the index approximations obtained in the index approximation operation program by the sum value to obtain an output value of the m-dimensional vector.

Description

Technical Field

[0001] The present invention relates to a normalized exponential operation approximation method and a neural network using the normalized exponential operation approximation method, and more particularly to a normalized exponential operation approximation method used in a classifier of an artificial intelligence deep learning model and a neural network using the normalized exponential operation approximation method. Background Art

[0002] Artificial intelligence (AI) generally refers to the technology of presenting human intelligence through ordinary computer programs. The most important part of AI is a kind of mathematical model or computational model in the field of machine learning and cognitive science that imitates the structure and function of biological neural networks, and then estimates and approximates functions. Common neural network models include convolutional and recurrent neural networks (CNN and RNN). In recent years, the Transformer model has been developed, and the Transformer model seems to be gradually replacing convolutional and recurrent neural networks (CNN and RNN) and gradually becoming the most popular deep learning model.

[0003] As in Figure 1 As shown in , no matter which neural model is used, the model usually includes at least a feature learning device 81 and a classifier 82. In the classifier 82 of the neural model, a normalized exponential function operation (softmax function) 821 is usually used to output at least one output value between 0 and 1.

[0004] The well-known normalized exponential function is shown below:

[0005]

[0006] Generally speaking, the input value of the i-dimensional vector can be converted to the output value of the j-dimensional vector by using the normalized exponential function (Formula 1), and each output value of the j-dimensional vector is usually a value between 0 and 1, and the sum of their values ​​is 1.

[0007] In addition, currently available GPUs (e.g., Nvidia) mostly use the softmax function of formula 1 for normalized exponential operations, and their input values ​​are in float32 format to implement the normalized exponential function operations. However, when actually operating the exponential function, due to the large number of polynomial orders and the use of float32 format for their input values, a considerable amount of numerical operations must be processed during the classifier operation process, which leads to time-consuming and energy-consuming problems. Therefore, how to provide a relatively simple softmax function operation, thereby saving time and energy during the operation process of the neural network classifier, is indeed an important topic. Summary of the invention

[0008] In view of the above problems, one object of the present invention is to provide a normalized exponential operation approximation method that can reduce the operation time and energy consumption at the same time. Another object of the present invention is to provide a neural network using the approximation method that can reduce the operation time and energy consumption at the same time.

[0009] To achieve the above-mentioned purpose, according to a normalized exponential operation approximation method of the present invention, the input value of a k-dimensional vector is converted into the output value of an m-dimensional vector, and the normalized exponential operation approximation method includes: an exponential approximation operation program, which performs a leaky linear rectification function operation on an input value of the k-dimensional vector to obtain a rectification function operation value, and then performs a polynomial function operation of a certain order based on the rectification function operation value to obtain an exponential approximation value, and then repeats the exponential approximation operation program on another input value to obtain another exponential approximation value; an addition operation program, which adds the exponential approximation value and the other exponential approximation value to obtain a sum value; and a division operation program, which divides at least one of the exponential approximations obtained in the exponential approximation operation program by the sum value to obtain the output value of the m-dimensional vector.

[0010] In one embodiment, in the exponential approximation operation procedure, the input value is firstly subjected to a clamp function operation and then subjected to the leakage linear rectification function operation.

[0011] In one embodiment, in the addition operation procedure, the summed value is further summed with a protection value to ensure that the absolute value of the summed value is greater than zero.

[0012] In one embodiment, the polynomial function operation of a certain order is a polynomial operation of order 2 to 5.

[0013] In one embodiment, the exponential approximation operation procedure is repeated until each input value is subjected to the leakage linear rectification function operation to obtain a rectification function operation value, and then a polynomial function operation of a certain order is performed based on the rectification function operation value to obtain an exponential approximation corresponding to each input value, and the addition operation procedure adds up all the exponential approximation values ​​to obtain a sum value, and the division operation procedure divides each of the exponential approximation values ​​obtained in the exponential approximation operation procedure by the sum value to obtain multiple output values ​​of the m-dimensional vector corresponding to the k-dimensional vector.

[0014] In one embodiment, the input value is an integer value.

[0015] Furthermore, according to a neural network of the present invention that applies the above-mentioned normalized exponential operation approximation method, a normalized exponential operation module is provided in the classifier of the neural network, and the normalized exponential operation module converts the input value of the k-dimensional vector into the output value of the m-dimensional vector, and the normalized exponential operation module includes: an exponential approximation operation unit, which performs a leaky linear rectification function (LeakyReLU function) operation on one input value of the k-dimensional vector to obtain a rectification function operation value, and then performs a polynomial function operation of a certain order according to the rectification function operation value to obtain an exponential approximation value, and then repeats the leaky linear rectification function operation and the polynomial function operation of the certain order on another input value to obtain another exponential approximation value; an addition operation unit, which adds the exponential approximation value and the other exponential approximation value to obtain a summed value; and a division operation unit, which divides at least one exponential approximation value among the exponential approximations obtained in the exponential approximation operation unit by the summed value to obtain the output value in the m-dimensional vector.

[0016] In another embodiment, in the exponential approximation operation unit, the input value is firstly subjected to a clamp function operation and then subjected to the leakage linear rectification function operation.

[0017] In another embodiment, in the adding unit, the summed value is further summed with a protection value to ensure that the absolute value of the summed value is greater than zero.

[0018] In another embodiment, the polynomial function operation of a certain order is a polynomial operation of order 2 to 5.

[0019] In another embodiment, the exponential approximation operation unit is repeated until each input value is subjected to the leakage linear rectification function operation to obtain a rectification function operation value, and then a polynomial function operation of a certain order is performed based on the rectification function operation value to obtain an exponential approximation corresponding to each input value, and the addition operation unit adds up all the exponential approximation values ​​to obtain a summed value, and the division operation unit divides each of the exponential approximation values ​​obtained in the exponential approximation operation unit by the summed value to obtain multiple output values ​​of the m-dimensional vector corresponding to the k-dimensional vector.

[0020] In another embodiment, the input value is an integer value. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is a diagram illustrating the basic architecture of a well-known neural network.

[0022] Figure 2 It is a comparison diagram of the second-order polynomial function calculation results in the preferred implementation of the present invention and the known exponential calculation results.

[0023] Figure 3 It is a flowchart illustrating the addition of leaky linear rectification function operations to the exponential approximation operation program in the preferred implementation of the present invention.

[0024] Figure 4 It is a block diagram illustrating the normalized exponential operation approximation method in a preferred embodiment of the present invention.

[0025] Figure 5 It is a comparison diagram between the calculation results of the exponential approximation calculation program in the preferred embodiment of the present invention and the known exponential calculation results.

[0026] Figure 6 It is a flowchart illustrating the further addition of clamp function calculation to the exponential approximation calculation program in the preferred implementation of the present invention.

[0027] Figure 7 It is a block diagram illustrating another normalized exponential operation approximation method in a preferred embodiment of the present invention.

[0028] Figure 8 It is a comparison diagram between the calculation results of another exponential approximation calculation program in the preferred embodiment of the present invention and the known exponential calculation results.

[0029] Fig. 9 It is a comparison table of softmax function output values ​​generated by using different exponential functions in the preferred implementation of the present invention.

[0030] Fig.10 In the preferred implementation of the present invention, different exponential functions are used, and different from Fig. 9A comparison table of the softmax function output values ​​generated by the input vector.

[0031] Fig.11 It is a comparison table of softmax function output values ​​generated by using different exponential functions and adding a protection value to one of the softmax functions in a preferred implementation of the present invention.

[0032] Fig.12 In the preferred embodiment of the present invention, the same Fig.11 The exponential function and softmax function shown in , and the input vector value uses the same Fig. 9 Similarly, a comparison table of the softmax function output values ​​generated. DETAILED DESCRIPTION

[0033] A normalized exponential operation approximation method and a neural network using the normalized exponential operation approximation method according to a preferred embodiment of the present invention will be described below with reference to the relevant drawings.

[0034] Before describing the embodiments of the present invention in detail, it should be noted that in the present embodiment, the softmax function operation can convert the input value of the k-dimensional vector into the output value of the m-dimensional vector. Therefore, the softmax function in the present embodiment can be expressed as:

[0035]

[0036] k≧m

[0037] In the softmax function operation of the above formula 2, the most difficult and time-consuming operation is to perform the exponential function exp(x) (i.e. ) is calculated, but in actual calculation When the value is calculated, Taylor expansion is generally used, that is, the calculation formula of formula 3 is used for calculation.

[0038]

[0039]

[0040] As mentioned above, Figure 2 As shown, Figure 2 The solid line curve in the figure represents the exponential function value exp(x) calculated according to the high-order polynomial shown in formula 3, while Figure 2 The dashed curve in the figure represents the exponential function value (2nd-Taylor exp(x)) calculated according to the 2nd-Taylor polynomial shown in Formula 4. Figure 2As shown in the figure, if the entire Taylor exponential expansion (Formula 3) operation process is reduced to a lower-order polynomial operation in order to simplify the exponential operation, for example, limited to the second-order polynomial operation, the result will be as shown in Figure 2 The dashed curve shown in the figure shows the exponential function value (2nd-Taylor exp(x)). However, Figure 2 It can be seen that the result of the second-order polynomial operation (shown by the dotted line) is very different from the result of the higher-order operation (shown by the solid line).

[0041] From the above, we can see that if we want to simplify the exponential function operation in the normalized exponential function, and then simplify the exponential function When limited to the operation of 2nd order polynomials, in order to make the simplified operation result close to the unsimplified operation result, the calculation method of formula 4 must be improved so that the calculation result can be closer to Figure 2 The solid line curve shown in .

[0042] Please refer to Figure 3 and Figure 4 As shown, the normalized exponential operation approximation method of the present invention includes an exponential approximation operation program S1, an addition operation program S2, and a division operation program S3. Figure 3 The leaky ReLU function operation step S11 and the second-order polynomial exponential approximation operation step S12 are shown. Figure 4 As shown, the output result of the above formula 2 can be obtained with the help of these operation processes.

[0043] Please refer to Figure 3 As shown, in this embodiment, a leakage linear rectification function is used in the calculation of Formula 4. At this time, the equation of Formula 4 can be expressed as the equation shown in Formula 5.

[0044]

[0045] At this time, by using the leaky linear rectifier function, the rectifier function calculation value L can be obtained in step S11. 1 , using the rectifier function to calculate the value L 1 Substituting into Formula 5, Formula 5 can be expressed as Formula 6. By means of the calculation of Formula 6, the exponential approximate calculation value exp1(x k ). That is to say, through the calculation of step S11 and step S12, Figure 4 The exponential approximation calculation program S1 shown in FIG. 1 calculates the exponential approximation calculation value exp1 (x k The result of this operation is Figure 5 The dashed curve shown is Figure 5 It can be seen that when the exponential function When limited to a 2nd-order polynomial and using a leaky linear rectifier function, at x k When it is a negative number, its approximate calculation result (shown by the dotted line) can be closer to the solid line curve.

[0046]

[0047] Although the calculation results of the above formula 5 or formula 6 are better than the calculation results of formula 4, Figure 5 As shown, when x k When is negative or greater than 1, the approximate calculation result still has a certain degree of deviation. To solve the problem of still having a certain degree of deviation, in the preferred implementation of the present invention, Figure 6 and Figure 7 As shown, in the exponential approximation calculation procedure S1', the input value x can be first k After the clamp function operation step S10 is performed, the leakage linear rectification function operation step S11' is performed, and then the second-order polynomial exponential approximation operation step S12' is performed. Figure 6 The calculation process shown in FIG. 1 is as follows. At this time, the second-order polynomial exponential approximation expression can be expressed as Formula 7.

[0048]

[0049] At this time, by using the clamp function and the leaky linear rectification function, the rectification function calculation value L can be obtained in step S11' through the calculation in step S10. 2 , using the rectifier function to calculate the value L 2 When substituting into equation 7, equation 7 can be expressed as equation 8.

[0050]

[0051] Here, it is worth mentioning that if f(x k )=LeakyReLU(Clamp(x k ,min,max)), and taking the second-order polynomial coefficients into consideration, the exponential approximation calculation value of the present invention can be expressed as the following general formula (Formula 9):

[0052]

[0053] In other words, if the rectifier function value L 1 Or the rectifier function value L 2 When equation 9 is substituted, equation 6 can be expressed as equation 10, and equation 8 can be expressed as equation 11.

[0054]

[0055]

[0056] Depend on Figure 8 It can be seen that when the exponential function When the calculation is limited to the second-order polynomial and the clamp function and the leakage linear rectifier function are used, that is, when the calculation is performed using Formula 8 or Formula 11, the calculation result shown by the dotted curve is k The range of negative numbers or greater than 1 can be closer to the high-order operation result shown by the solid line curve. In other words, the exponential operation of Formula 8 or Formula 11 can be used to obtain the exponential approximate operation value exp2(x k ). In other words, Figure 7 The exponential approximation calculation program S1' shown in FIG. 1 calculates the exponential approximation calculation value exp2 (x k ).

[0057] The following will be Fig. 9 To specifically illustrate the actual operation of a normalized exponential operation approximation method of the present invention, it should be particularly noted that in this embodiment, the normalized function operation (i.e., Formula 2) uses the normalized function operation layer in the Transformer model to perform actual operations.

[0058] Please refer to Fig. 9 As shown, when the input vector value is 〔-2,0,8〕, if the exponential function expression adopts Formula 3 and the normalized function expression adopts Formula 2, then each vector value in the input vector value is substituted into Formula 3 to obtain e (-2) =0.13, e (0) =1, e (8) =2980. Substitute the results of equation 3 into equation 2 to get Fig. 9 The operation value (output value) of the normalized function expression (Formula 2) shown in is the output vector [0.000045, 0.000335, 0.999619].

[0059] Please refer to Fig. 9 As shown, when the input vector value is [-2, 0, 8], if the exponential function adopts Formula 10 and the normalized function expression also adopts Formula 2, where the coefficients a=1, b=2, c=1 shown in Formula 10, and then each vector value in the input vector value is substituted into Formula 10 for operation, the operation value (output value) of the normalized function expression (Formula 2) is [0.003040, 0.012158, 0.984802]. Fig. 9As shown, when the input vector has a relatively large response ratio (such as the input "8" in this embodiment), the response error is within 1-2%.

[0060] However, in the above description, if the exponential function expression is expressed in Formula 10, the normalized function expression is expressed in Formula 2, and the coefficients a=1, b=2, and c=1 shown in Formula 10 are set, if the input vector value is 〔-4,-4,-4〕, then Fig.10 As shown, the operation value (output value) of the normalized function expression (Formula 2) cannot be calculated. This phenomenon is because the denominator in the normalized function expression (Formula 2) may approach 0.

[0061] To solve the problem that the denominator of the normalized function expression may approach 0 and cause abnormal operation, please refer to Figure 7 As shown, in this embodiment, the normalized exponential operation approximation method of the present invention includes not only the exponential approximation operation program S1', the addition operation program S2, and the division operation program S3, but also includes a protection value operation program S21 for adding the calculated value of the addition operation program S2 to the protection value eps after the addition operation program S2 is completed. Based on this, the normalized function expression of the present invention can be expressed as shown in Formula 12. Among them, the role of the protection value eps is to ensure that the denominator of Formula 2 is not equal to 0 or is not close to 0. Because if the denominator of Formula 2 is equal to 0 or close to 0, softmax'(x k ) m The calculation will not produce any result.

[0062]

[0063] k≧m

[0064] As mentioned above, Fig.11 As shown, if the exponential function is expressed in Formula 11, the normalized function expression is expressed in Formula 12, and the coefficients a=1, b=2, and c=1 shown in Formula 11 are set, even if the input vector value is 〔-4,-4,-4〕, then Fig.11 As shown, the operation value (output value) of the normalized function expression (Formula 12) can also be calculated. In addition, if the exponential function expression adopts Formula 11, the normalized function expression adopts Formula 12, and the coefficients a=1, b=2, c=1 shown in Formula 11, and eps=1 shown in Formula 12, even if the input vector value is 〔-2,0,8〕, then Fig.12 As shown, the operation value (output value) of the normalized function expression (Formula 12) can also be calculated as [0.007007, 0.016016, 0.976977]. Fig.12From the output value 0.984802 (using Formula 2), it can be seen that when responding to an input vector with a relatively large response ratio (such as the input "8" in this embodiment), the response error is maintained within 1-2%.

[0065] In summary, in the normalized exponential operation approximation method of the present invention, due to the exponential function is limited to low-order polynomials (e.g., 2nd-order polynomials) and uses a clamp function and a leaky linear rectifier function. Therefore, the exponential function Regardless of whether the operation is performed using Formula 8 or Formula 11, the operation result shown by the dotted curve can reach a high-order operation result similar to that shown by the solid curve. In addition, with respect to the normalized exponential operation approximation method of the present invention, if the normalized function expression adopts Formula 12, and the coefficients shown in Formula 11 are appropriately adjusted (for example, a=1, b=2, c=1), then for the main response values, the output error of the normalized function expression of the present invention is very small.

[0066] In addition, it is worth mentioning that in this embodiment, since the elements of the input vector are all integer types, and the exponential function It is limited to low-order polynomials (such as 2nd-order polynomials), so compared with the known float32 format and high-order polynomial operations, the amount of calculation is greatly reduced, so the purpose of reducing the calculation time and reducing energy consumption can be achieved.

[0067] In another embodiment of the present invention, a neural network using the approximation method is also provided. However, since the specific description of the neural network using the approximation method of the present invention is roughly the same as the aforementioned method, it is omitted here. The only thing to be specifically explained is that in another embodiment of the present invention, the neural network is not limited to the neural network of the Transformer model.

[0068] The above description is for illustrative purposes only and is not intended to be limiting. Any equivalent modifications or changes made thereto without departing from the spirit and scope of the present invention should be included in the scope of the appended claims.

Claims

1. A normalized exponential operation approximation method, which converts a k-dimensional vector input value into an m-dimensional vector output value, the normalized exponential operation approximation method comprising: An exponential approximation operation procedure, performing a leaky linear rectifier function (LeakyReLU function) operation on one input value of the k-dimensional vector to obtain a rectifier function operation value, then performing a polynomial function operation of a certain order based on the rectifier function operation value to obtain an exponential approximation value, and then repeating the exponential approximation operation procedure on another input value of the k-dimensional vector to obtain another exponential approximation value; an addition operation procedure for adding the exponential approximation value and the another exponential approximation value to obtain a summed value; and A division operation procedure divides at least one of the exponential approximation values ​​obtained in the exponential approximation operation procedure by the summed value to obtain an output value in the m-dimensional vector.

2. The method according to claim 1, wherein: In the exponential approximation operation procedure, the input value is first subjected to a clamping function operation and then subjected to the leakage linear rectification function operation.

3. The method according to claim 2, wherein: In the addition operation procedure, the summed value is further summed with a protection value to ensure that the absolute value of the summed value is greater than zero.

4. The method according to claim 1, wherein: The polynomial function operation of a certain order is a polynomial operation of order 2 to 5.

5. The method according to claim 1, wherein: The exponential approximation operation procedure is repeated until each input value is subjected to the leakage linear rectification function operation to obtain each rectification function operation value, and then a polynomial function operation of a certain order is performed based on each rectification function operation value to obtain the exponential approximation value corresponding to each input value, and the addition operation procedure adds up all the exponential approximation values ​​to obtain a total value, and the division operation procedure divides each of the exponential approximation values ​​obtained in the exponential approximation operation procedure by the total value to obtain multiple output values ​​of the m-dimensional vector corresponding to the k-dimensional vector.

6. The method according to claim 1, wherein: The input value is an integer value.

7. A neural network, wherein a classifier of the neural network has a normalized exponential operation module, wherein the normalized exponential operation module converts an input value of a k-dimensional vector into an output value of an m-dimensional vector, wherein the normalized exponential operation module comprises: An exponential approximation operation unit, wherein the exponential approximation operation unit performs a leaky linear rectification function operation on an input value of the k-dimensional vector to obtain a rectification function operation value, then performs a polynomial function operation of a certain order based on the rectification function operation value to obtain an exponential approximation value, and then repeats the leaky linear rectification function operation and the polynomial function operation of the certain order on another input value to obtain another exponential approximation value; an adding unit that adds the exponential approximation value and the other exponential approximation value to obtain a summed value; and A division operation unit divides at least one of the exponential approximation values ​​obtained in the exponential approximation operation unit by the summed value to obtain an output value in the m-dimensional vector.

8. The neural network according to claim 7, wherein: In the exponential approximation operation unit, the input value is firstly subjected to a clamping function operation and then subjected to the leakage linear rectification function operation.

9. The neural network according to claim 8, wherein: The summed value is further summed with a protection value to ensure that the absolute value of the summed value is greater than zero.

10. The neural network according to claim 7, wherein: The polynomial operation of a certain order is a polynomial operation of order 2 to 5.

11. The neural network according to claim 7, wherein: The exponential approximation operation unit repeats the processing until each input value is subjected to the leakage linear rectification function operation to obtain the rectification function operation value, and then the polynomial function operation of a certain order is performed based on the rectification function operation value to obtain the exponential approximation value corresponding to each input value, and the addition operation unit adds up all the exponential approximation values ​​to obtain a summed value, and the division operation unit divides each of the exponential approximation values ​​obtained in the exponential approximation operation unit by the summed value to obtain multiple output values ​​of the m-dimensional vector corresponding to the k-dimensional vector.

12. The neural network according to claim 7, wherein: The input value is an integer value.

Images