Fast and resource-efficient approximation for exponential function
Approximating e^x with a Taylor series using nearest powers of two simplifies neural network computations, reducing hardware size and power consumption while maintaining accuracy, addressing inefficiencies in embedded systems.
Patent Information
- Application Number
- JP2025089125
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2024-05-29
- Filing Date
- 2025-05-28
- Publication Date
- 2025-12-11
AI Technical Summary
The computational complexity and resource-intensive nature of exponential functions, particularly in neural networks, lead to inefficiencies in processing time and hardware requirements, especially in embedded systems with strict size and power constraints.
Approximate the exponential function e^x using a Taylor series expansion with factorials replaced by nearest powers of two, eliminating the need for division operations and reducing circuitry complexity, allowing for miniaturization and reduced power consumption.
This method significantly reduces processing time and hardware resources required, while maintaining accuracy in neural network outputs, particularly in applications like autonomous driving and robotics, by introducing minimal approximation errors that are canceled out.
Smart Images
Figure 2025181779000001_ABST
Abstract
Description
[Technical Field]
[0001] The present invention relates to the calculation of exponential functions in a manner that can be performed more efficiently on a computing platform, thereby saving processing time and allowing for the miniaturization of the computing platform. [Background technology]
[0002] background To evaluate the output of a neuron in a neural network, the inputs to that neuron are aggregated in a weighted sum, and this result is processed into the final output using a nonlinear activation function. A very common activation function is the softmax function. In particular, this activation function is placed in layers where normalized probabilities are needed.
[0003] The cost of calling the softmax function very frequently is that it is computationally expensive. The main reason for the computational complexity is the exponential function e x This requires a large number of floating-point operations or a large look-up table.
[0004] e x to the following format:
number
[0005] Disclosure of the Invention The present invention uses the exponential function e of the argument x. x This method provides a method for calculating an approximation A of
number
[0006] We found that approximating to the nearest power of two results in surprisingly large computational savings because it eliminates the need for expensive hardware implementations of division. In fact, multiplication or division by a power of two is one of the most fundamental computational operations in many hardware architectures and can therefore be implemented by simple bit-shifting operations using very fast, small hardware blocks within the hardware architecture. That is, the hardware platform does not even need to include extensive circuitry capable of performing the division. This circuit savings means that the entire circuitry required to compute the neural network output O can fit within a smaller area on the chip. At the same time, this reduces power consumption, especially leakage energy. Of course, approximating to the nearest power of two introduces some error. However, surprisingly, this is found to introduce only a small error in the desired final result, specifically, the neural network's final output O. That is, in neural networks where many exponents are computed and many approximation errors occur, their effect on the final result is at least partially canceled out.
[0007] In a further particularly advantageous embodiment, the argument x is an integer X q and a non-integer scaling factor Δ x This scaling factor Δ x teeth,
number
number
number
number
number
[0008] In a further particularly advantageous embodiment,
number
number
number
[0009] For example, e x The calculated approximation A of is the element y of the input vector y with m elements. k The following softmax function of
number
number
number
number
[0010] Consider an example where n=3. Here, the approximation A is calculated using the following formula:
number
number
number
number
number
number
number
[0011] As mentioned above, in a further advantageous embodiment, at least one multiplication of a number by a power of 2 for an exponent and / or division of said number by a power of 2 is calculated by bit-shifting the number by the number of bits corresponding to the exponent, thereby avoiding the need for more complex circuitry typically required to perform multiplication or division operations.
[0012] As previously mentioned, the e presented herein x The main use case for approximating e is in computing the output O of a neural network. x the calculated approximation A of the y k ) is used. In particular, when evaluating the output O of such a neural network, e xapproximations of are required a very large number of times. Therefore, the savings in processing time introduced by the simplifications brought about by near-approximations are generally considerable. Furthermore, special structures to perform more complex computational operations are no longer needed. This means that the hardware platform can be miniaturized in terms of the chip area required. This is particularly advantageous for applications in embedded systems such as autonomous driving systems for vehicles and robots, production machines, and quality inspection machines. In these systems, the embedded systems on which neural networks are executed are often subject to strict size and power consumption constraints.
[0013] In a further particularly advantageous embodiment, e x the calculated approximation A of the y k ) is used to calculate the output O of a classifier network and / or the output O of a multi-head attention module of a transformer network for an image or other record of measurement data. These network architectures have been shown to be particularly tolerant to approximation errors introduced by the approximations proposed herein; that is, these approximation errors are unlikely to affect the final output O of the neural network. In particular, in classification networks, it is unlikely that the class with the best classification score will switch to another class due to approximation errors. At the same time, these architectures make particularly extensive use of exponential functions, resulting in more significant savings in overall processing time.
[0014] In many applications of neural networks, not all inputs are equally difficult to process. Particularly in applications involving classification tasks, for some inputs it becomes clear very quickly which will be the final decision, while for other inputs the final decision is not clear until all layers have been processed. This means that when inputs have different difficulty levels, e xThis means that the robustness of the final output O to the approximation errors introduced by the approximation of may vary as well. For less difficult inputs, the power series may be shortened, i.e., a smaller number of terms n may be used. The time saved then increases with higher n. x The degree of difficulty of a particular input may be determined, for example, from the confidence C of the neural network output O.
[0015] Therefore, in a further particularly advantageous embodiment, a confidence C of the output O of the neural network is determined. Depending on whether this confidence C satisfies a predetermined condition, e x The number of terms n used in the subsequent calculation of the approximation A of is changed. In this way, a higher number of terms n may be used only when absolutely necessary, and less processing power is "wasted" on inputs where the final result is already known very early in the process.
[0016] To this end, in particular the number of terms n is controlled so as to be kept at a minimum value sufficient to achieve a predetermined minimum reliability C of the output O.
[0017] As previously mentioned, the e presented herein x A lightweight approximation of e allows for a miniaturization of the hardware platform used to compute the output O of the neural network. Thus, in a particularly advantageous further embodiment, the neural network is xThe output O is implemented on a hardware platform with less memory and / or processing resources than would be required to compute the output O without approximating the value of . This applies to both quantitative and qualitative dimensions. Quantitative means that fewer hardware resources, of which at least one instance would be required regardless of whether the approximations present herein are used, would be required if approximations were used. Qualitative means that for a given type of circuitry that would be required if approximations were not used, an instance would not be required on the hardware platform due to the use of approximations. This qualitative type of miniaturization does not make the hardware platform slower, but it does make certain operations, such as multiplication or division, completely infeasible on the hardware platform.
[0018] In a further particularly advantageous embodiment, the argument x is derived from measurement data acquired using at least one sensor. From the output O of the neural network, an operating signal is calculated. A vehicle, a driver assistance system, a robot, a quality inspection system, a monitoring system, and / or a medical imaging system operates on this operating signal. In this way, the operating signal can be determined faster, and the respective operating technology system requires a lower power and / or less powerful embedded system to process the neural network.
[0019] The method may be wholly or partially computer-implemented and embodied in software. That is, the present invention also relates to a computer program comprising machine-readable instructions that, when executed by one or more computers and / or computational instances, cause the one or more computers and / or computational instances to perform the above method. Control units for vehicles or robots and other embedded systems capable of executing machine-readable instructions are also considered computers herein. Computational instances include virtual machines, containers, or other execution environments that enable the execution of machine-readable instructions in the cloud.
[0020] The non-transitory storage medium and / or downloadable product may comprise a computer program. The downloadable product may be sold online and is an electronic product transferred over a network for immediate fulfillment. One or more computers and / or computing instances may comprise the computer program and / or the non-transitory storage medium and / or downloadable product.
[0021] The present invention will be described below with reference to the drawings, which are not intended to limit the scope of the invention in any way. [Brief explanation of the drawings]
[0022] [Figure 1] 1 illustrates an exemplary embodiment of a method 100 for computing an approximation A of an exponential function ex of an argument x. [Figure 2] FIG. 1 illustrates the simplifications introduced by the approximation according to the method 100. DETAILED DESCRIPTION OF THE INVENTION
[0023] Figure 1 shows the exponential function e of the argument x. x 1 is a schematic flow chart of an embodiment of a method 100 for calculating an approximation A of
[0024] According to block 105, the argument x may be derived from measurement data acquired using at least one sensor.
[0025] In step 110, e x is the argument x raised to the i-th power x i is approximated using a Taylor expansion T around x=0, which includes a predetermined number of terms, n, obtained by dividing by the factorial of i (where i=1, . . . n).
[0026] In step 120, the argument x is an integer X q and a non-integer scaling factor Δ x It is decomposed into a product of
[0027] In step 130, this scaling factor Δ x is the following exponent
number
number
[0028] According to block 131, the following formula
number
number
[0029] In step 140, in the calculation of each term in the Taylor expansion T, the factorial of i is approximated to the nearest power of 2 p(i!). The calculation of all terms in the Taylor expansion T is performed by x provides the calculated approximation of
[0030] In step 150, e xThe calculated approximation A of is the element y of the input vector y with m elements. k The following softmax function of
number
[0031] According to block 151, if A is determined according to block 131 to have the following formula:
number
number
[0032] In step 160, e x the calculated approximation A of the y k ) is used to calculate the output O of the neural network.
[0033] According to Block 161, e x the calculated approximation A of the y k ) is used to compute the output O of the classifier network and / or the output O of the multi-head attention module of the transformer network for the image or other record of measurement data.
[0034] According to block 162, a confidence C of the output O of the neural network may be determined. Then, in block 163, it may be determined whether this confidence C satisfies a predetermined condition, such as whether it is above or below a predetermined threshold. If this is the case (true value 1), according to block 164, e xThe number of terms n used in subsequent calculations of the approximation A of O may be varied. In particular, according to block 164a, the number of terms n may be controlled to be maintained at a minimum value sufficient to achieve a predetermined minimum confidence C of the output O.
[0035] In accordance with block 165, the neural network x The output O may be implemented on a hardware platform with less memory and / or processing resources than would be required to compute the output O without approximating the value of
[0036] In step 170, an operating signal 170a is determined from the output O of the neural network.
[0037] In step 180, the vehicle 50, the driver assistance system 51, the robot 60, the quality inspection system 70, the monitoring system 80, and / or the medical imaging system 90 are operated by the operation signal 170a.
[0038] FIG. 2 illustrates the simplification introduced by the approximation according to the present method 100.
[0039] e x The task of computing x is to lift an argument x, depicted as a container filled with water, to a given height h. This is difficult because a container filled with water is heavy. We can calculate x by multiplying it by the integer x, symbolically represented by a nearly empty container. q and a non-integer scaling factor Δ x By decomposing the container into the integer X, the weight of the container can be significantly reduced. q the following i power of
number
[0040] The approximation requires more than two calculations. The scaling factor Δx The power of i must be calculated and then divided by the factorial i!. The scaling factor Δ x is the following exponent
number
number
[0041] The end result is e x is an approximation A that is close to, but not identical to, the true value of e x height h is not fully reached, and the container e x This is symbolically represented by a lower fill level than
Claims
1. exponential function e of argument x ｘ A method (100) for calculating an approximation A of The method comprises: the argument x to the i-th power x ｉ Using a Taylor expansion T around x=0, which includes a predetermined number of terms n, e is obtained by dividing ｘ approximating (110) - In the calculation of each term, a step (140) of approximating the factorial of i to the nearest power of 2 p(i!); A method (100) comprising:
2. The method further comprises: The argument x is an integer X ｑ and a non-integer scaling factor Δ ｘ and a step (120) of decomposing the The scaling factor Δ ｘ the following exponents [Equation 1] and express it as a power of two with [Equation 2] Step (130) where: The method (100) of claim 1, comprising:
3. The method further comprises: The following formula [Equation 3] The approximate value of is calculated as follows: [Equation 4] and the residual part f (131). The method (100) of claim 2, comprising:
4. The method further comprises: Element y of input vector y having m elements ｋ The following softmax function of [Equation 5] In the calculation of e ｘ Step (150) of using the calculated approximation A of The method (100) of any one of claims 1 to 3, comprising:
5. The S(y ｋ The following terms appear in the numerator and denominator of [Equation 6] The calculation of the two instances of is omitted (151), The method (100) of claim 3 or 4.
6. At least one multiplication of a number by a power of 2 for an exponent and / or division of the number by said power of 2 is calculated by bit-shifting the number by a number of bits corresponding to the exponent, The method (100) of any one of claims 1 to 5.
7. The method further comprises: In calculating the output O of the neural network, ｘ the calculated approximation A of y ... ｋ ) using (160) The method (100) of any one of claims 1 to 6, comprising:
8. The above e ｘ the calculated approximation A of y ... ｋ ) is used to calculate the output O of the classifier network and / or the output O of the multi-head attention module of the transformer network for the image or other record of measurement data (161); The method (100) of claim 7.
9. The method further comprises: - determining (162) a confidence C of the output O of said neural network; In response to the reliability C satisfying a predetermined condition (163), ｘ changing (164) the number of terms n used in subsequent calculations of said approximation A of 9. The method (100) of claim 7 or 8, comprising:
10. The number of terms n is controlled (164a) to be maintained at a minimum value sufficient to achieve a predetermined minimum reliability C of the output O.
10. The method (100) of claim 9.
11. The neural network ｘ 165) implemented on a hardware platform with less memory and / or processing resources than would be required to compute the output O without approximating the value of The method (100) of any one of claims 7 to 10.
12. the argument x is derived (105) from measurement data acquired using at least one sensor; The method further comprises: - determining (170) an operating signal (170a) from the output O of the neural network; - operating (180) a vehicle (50), a driver assistance system (51), a robot (60), a quality inspection system (70), a surveillance system (80), and / or a medical imaging system (90) using said operating signal (170a); 12. The method (100) of any one of claims 7 to 11, comprising:
13. 13. A computer program comprising machine-readable instructions that, when executed by one or more computers and / or computing instances, cause said one or more computers and / or computing instances to perform the method (100) of any one of claims 1 to 12.
14. A non-transitory machine-readable storage medium and / or downloadable product comprising a computer program according to claim 13.
15. One or more computers and / or computing instances comprising the computer program of claim 13 and / or the non-transitory machine-readable storage medium and / or downloadable product of claim 14.