Efficient lookup table-based functions for artificial intelligence (AI) accelerators

Abstract

The present disclosure relates to a method for approximating an activation function, the method comprising: receiving an input value of the activation function; determining that the input value is within a range, wherein the range comprises a set of non-uniform intervals; determining a selected interval including the input value from among the set of non-uniform intervals; retrieving, by a hardware accelerator, values of one or more quadratic interpolation parameters associated with the selected interval from a lookup table (LUT) associated with a type of the activation function; performing quadratic interpolation on the input value using the values of the one or more quadratic interpolation parameters to approximate the input value; and determining a first approximated output of the activation function based on a result of the quadratic interpolation performed on the input value.

Description

Technical Field

[0001] The present application generally relates to an artificial intelligence (AI) accelerator system. More specifically, the present disclosure relates to a system and method for providing an efficient lookup table-based function for an AI accelerator. Background Art

[0002] An artificial intelligence (AI) and / or machine learning (ML) accelerator is a high-performance parallel computing machine specifically designed to efficiently process AI workloads such as neural networks. Given that processing speed and scalability are two key requirements from AI applications, AI accelerators (e.g., neural network processing units) play a key role in delivering the near-instant results that make these applications valuable.

[0003] The above information in the Background section is only for enhancement of understanding of the background of the technology and therefore should not be construed as an admission of the existence or relevance of prior art. Summary of the invention

[0004] This Summary is provided to introduce a selection of features and concepts of embodiments of the present disclosure, which are further described in the Detailed Description below. This Summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. One or more of the described features may be combined with one or more other described features to provide a working device.

[0005] In one or more embodiments, a method for approximating an activation function includes: receiving an input value of the activation function; determining that the input value is within a range, the range including a set of non-uniform intervals; determining a selected interval from the set of non-uniform intervals that includes the input value; retrieving, by a hardware accelerator, values ​​of one or more quadratic interpolation parameters associated with the selected interval from a lookup table (LUT) associated with the type of the activation function; performing quadratic interpolation on the input value using the values ​​of the one or more quadratic interpolation parameters to approximate the input value; and determining a first approximated output of the activation function based on a result of the quadratic interpolation performed on the input value.

[0006] In one or more embodiments, a Remez minimax algorithm is used to determine the non-uniform distribution of the set of non-uniform intervals. In one or more embodiments, the determining the selected interval from the set of non-uniform intervals is based on one or more of the most significant bits (MSBs) of the input values ​​and / or a two's complement encoded input value. In one or more embodiments, the LUT associated with the activation function includes the values ​​of the one or more quadratic interpolation parameters for each interval from among the set of non-uniform intervals. In one or more embodiments, the hardware accelerator stores a plurality of LUTs including parameters associated with quadratic interpolation ranges for a plurality of activation functions.

[0007] In one or more embodiments, the plurality of LUTs are implemented as flip-flops in the hardware accelerator. In one or more embodiments, the method further includes: determining, by the hardware accelerator, that the input value is within one or more linear extrapolation ranges that do not overlap with the range; determining, by the hardware accelerator, an offset value and a slope of the activation function for each of the one or more linear extrapolation ranges; performing, by the hardware accelerator, linear extrapolation of the input value using the offset value and the slope of the activation function for each of the one or more linear extrapolation ranges to approximate the input value; and determining, by the hardware accelerator, a second approximated output of the activation function based on a result of the linear extrapolation performed on the input value. In one or more embodiments, the input value is shifted by a power of two factors within a range of representable values ​​in a fixed point representation of the second approximated output of the activation function.

[0008] In one or more embodiments, a system for approximating an activation function includes: a bit shifter configured to receive an input value of the activation function and perform a shift operation on the input value to calculate a most significant bit (MSB) of the input value; a first device configured to determine whether the input value is within a range including a set of non-uniform intervals, and determine a selected interval from the set of non-uniform intervals that includes the input value based on the MSB of the input value; a second device configured to retrieve values ​​of one or more quadratic interpolation parameters associated with the selected interval from a lookup table (LUT) associated with the activation function; one or more multiply-accumulate scaling circuits configured to perform quadratic interpolation on the input value using the values ​​of the one or more quadratic interpolation parameters to approximate the input value, and determine a first approximated output of the activation function based on a result of the quadratic interpolation performed on the input value.

[0009] In one or more embodiments, the system includes a hardware accelerator and / or an electronic circuit configured by a hard macro intellectual property (IP). In one or more embodiments, the hardware accelerator stores a plurality of LUTs, the plurality of LUTs including parameters associated with quadratic interpolation ranges of a plurality of activation functions. In one or more embodiments, the plurality of LUTs are implemented as flip-flops in the hardware accelerator. In one or more embodiments, the non-uniform distribution of the set of non-uniform intervals is determined using a Remitz minimax algorithm.

[0010] In one or more embodiments, the determining of the selected interval from among the set of non-uniform intervals is based on one or more of a most significant bit (MSB) of the input value and a two's complement encoded input value. In one or more embodiments, the LUT associated with the activation function includes the values ​​of the one or more quadratic interpolation parameters for each interval from among the set of non-uniform intervals.

[0011] In one or more embodiments, the system further includes: a third device configured to determine an offset value and a slope of the activation function for each of the one or more linear extrapolation ranges based on the first device determining that the input value is within one or more linear extrapolation ranges, the one or more linear extrapolation ranges not overlapping the range, wherein the one or more multiply-accumulate scaling circuits are further configured to perform linear extrapolation on the input value using the offset value and the slope of the activation function for each of the one or more linear extrapolation ranges to approximate the input value, and determine a second approximated output of the activation function based on a result of the linear extrapolation performed on the input value. In one or more embodiments, the input value is shifted by a power of two factors within a range of representable values ​​in a fixed point representation of the second approximated output of the activation function.

[0012] In one or more embodiments, a non-transitory computer-readable medium comprising a stored representation of an accelerator circuit, the stored representation of the accelerator circuit, when synthesized by a processor, causes the processor to synthesize a structure of an electronic circuit, the electronic circuit being configured to: receive an input value of an activation function; determine that the input value is within a range, the range comprising a set of non-uniform intervals; determine a selected interval from the set of non-uniform intervals that includes the input value; retrieve values ​​of one or more quadratic interpolation parameters associated with the selected interval from a lookup table (LUT) associated with the activation function; perform quadratic interpolation on the input value using the values ​​of the one or more quadratic interpolation parameters to approximate the input value; and determine a first approximated output of the activation function based on a result of the quadratic interpolation performed on the input value.

[0013] In one or more embodiments, the input value is shifted by a power of two factors within a range of representable values ​​in a fixed point representation of the first approximated output of the activation function. In one or more embodiments, the electronic circuit is further configured to: determine that the input value is within one or more linear extrapolation ranges, the one or more linear extrapolation ranges not overlapping the range; determine an offset value and a slope of the activation function for each of the one or more linear extrapolation ranges; perform linear extrapolation on the input value using the offset value and the slope of the activation function for each of the one or more linear extrapolation ranges to approximate the input value; and determine a second approximated output of the activation function based on a result of the linear extrapolation performed on the input value. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The present disclosure will be more fully understood from the detailed description given below and from the drawings of the embodiments of the present disclosure. The drawings are used to provide an understanding and understanding of the embodiments of the present disclosure and do not limit the scope of the present disclosure to these specific embodiments. In addition, the drawings are not necessarily drawn to scale.

[0015] Figure 1 Diagram illustrating different non-linear activation functions used in AI or ML.

[0016] Figure 2 The diagram illustrates a piecewise approximation method for an activation function of an AI accelerator according to one embodiment of the present disclosure.

[0017] Figures 3A to 3B The present invention is used to implement the Figure 2 The AI ​​accelerator or IP core of the disclosed segmented approximation method discussed. For example, Figures 3A to 3B Illustrate the use of an AI accelerator (e.g., a neural network processing unit) to perform Figure 2 Function of the piecewise approximation method of the present disclosure discussed.

[0018] Figures 4 to 6 The present invention provides some embodiments of the present invention. Figure 2 Exemplary applications of the discussed piecewise approximation methods.

[0019] Figure 7 Flowcharts depicting various processes used during the design and fabrication of integrated circuits according to some embodiments of the present disclosure.

[0020] Figure 8 A diagram depicting an exemplary computer system in which embodiments of the present disclosure may be operated. For example, one or more embodiments, Figure 7 The process can be Figure 8 The neural network processing unit is executed and Figure 8 The neural network processing unit includes Figures 3A to 3B AI accelerator. DETAILED DESCRIPTION

[0021] Aspects of the present disclosure relate to a system and method for providing efficient lookup table based functions for an artificial intelligence (AI) accelerator.

[0022] An artificial intelligence (AI) and / or machine learning (ML) accelerator (e.g., a neural network processing unit) is a high-performance parallel computing machine specifically designed to efficiently process AI workloads (such as neural networks). Given that processing speed and scalability are two key requirements from AI applications, AI accelerators play a key role in delivering near-instant results that make these applications valuable. AI accelerators can be 100 to 1,000 times more efficient than general-purpose computing machines. Due to the high speed in computational data provided by AI accelerators, AI accelerators reduce the latency of the time required to get an answer. This lower latency is particularly important in safety-critical applications such as advanced driver assistance systems (ADAS) where every second counts. AI accelerators may achieve a level of performance speed enhancement that can be almost equal to the number of cores involved. The heterogeneous architecture of AI accelerators accommodates multiple specialized processors to support specific tasks, thereby providing the computing performance required for AI applications.

[0023] There are currently two different AI accelerator spaces, data center and edge. Data centers, especially hyperscale data centers, require massively scalable computing architectures. AI accelerators in data centers can support AI research at significantly faster speeds and scalability than other architectures by delivering more computing, memory, and communication bandwidth. The edge represents the other end of the spectrum. Here, energy efficiency is key and the effective area is limited because intelligence is distributed at the edge of the network rather than in a more centralized location. AI accelerator intellectual property (IP) is integrated into edge system-on-chip (SoC) devices, which, no matter how small, can deliver near-instant results required for interactive programs running on, for example, smartphones or for industrial robots.

[0024] Current state-of-the-art artificial intelligence (AI) and / or machine learning (ML) applications rely on various types of nonlinear activation functions (e.g., ReLU, Sigmoid, Tanh, Swish, etc.). In a neural network, an activation function may be configured to determine whether a neuron of the neural network should be activated. For example, an activation function of a neural network may determine whether an input of the neuron to the neural network is required for the operation of the neural network. For example, an activation function may derive an output from a set of input values ​​fed to a node (or layer) of the neural network. In some cases, these nonlinear activation functions exhibit nonlinear behavior within a limited range and linear behavior outside of the limited range. For example, for input values ​​less than -8, the tanh function returns a value (e.g., possibly quantized) close to -1, while for input values ​​greater than +8, the tanh function returns a value (e.g., possibly quantized) close to +1. However, for input values ​​within the range [-8,8], the tanh function exhibits nonlinear behavior, which is strongest near 0. Thus, for input values ​​less than -8 and greater than +8, tanh may be linear, however, for a limited range of input values ​​between [-8, 8], tanh may be non-linear.

[0025] For example, Figure 1 The diagram illustrates different types of nonlinear activation functions used in AI or ML (e.g., Mish, Tanh, ReLU, Sigmoid, Leaky ReLU, SoftPlus, Bent Identity, and Swish), which exhibit nonlinear behavior within a limited range and linear behavior outside the limited range. Figure 1 The linear and nonlinear parts of the activation functions (e.g., Mish, Tanh, ReLU, Sigmoid, Leaky ReLU, SoftPlus, Bent Identity, and Swish) are also identified in . Figure 1 As shown in Figure 1 The nonlinear activation functions shown in (e.g., Mish, Tanh, ReLU, Sigmoid, Leaky ReLU, SoftPlus, Bent Identity, and Swish) exhibit nonlinear behavior within a limited range and the nonlinear behavior of the activation functions may be strongest near the 0 crossing. However, Figure 1 The nonlinear activation functions shown in (e.g., Mish, Tanh, ReLU, Sigmoid, Leaky ReLU, SoftPlus, Bent Identity, and Swish) exhibit linear behavior towards negative infinity and positive infinity outside the limited range in which they exhibit nonlinear behavior.

[0026] The set of activation functions in AI is not limited to Figure 1 and will grow over time. For large negative and positive values, all of these activation functions are approximately linear functions, such as Figure 1 , as illustrated by the left and right ellipses in . For example, tanh(x) will converge to +1 for large positive input values ​​and to -1 for negative input values ​​towards negative infinity; the swish(x) function will converge to a*x (e.g., "x" is the input and "a" is a constant (e.g., a real or complex number)) for large positive input values ​​and to 0 for negative input values ​​towards negative infinity. However, near the input value of 0 (or some other input value), these functions exhibit nonlinear behavior, as depicted by the middle ellipse in the figure. The nonlinearity of these functions may be stronger in some intervals than in other intervals.

[0027] AI and / or ML interference hardware accelerators require efficient approximations of nonlinear activation functions that exhibit nonlinear behavior within a limited range and linear behavior outside the limited range to save silicon area while maintaining high accuracy and operating frequency. In AI and / or ML applications, AI accelerators may use different activation functions based on the application. The activation functions used in AI or ML applications may be differentiable activation functions so that gradient descent training methods can calculate the parameters of the AI ​​or ML model based on training data. Flexibility in the AI ​​accelerator architecture may be required to take advantage of new activation functions that emerge over time. Differentiable activation functions may also make it possible to have high throughput in AI accelerators (e.g., because gradient descent is easier to perform when differentiable activation functions are used).

[0028] One or more embodiments of the present disclosure use a combination of reprogrammable non-uniform piecewise quadratic approximation and linear approximation to approximate activation functions for AI / ML applications. In other words, one or more embodiments of the present disclosure describe a piecewise approximation method for activation functions for AI / ML. In the approximation method of the present disclosure, quadratic interpolation is used to approximate regions or intervals (e.g., nonlinear intervals) of activation functions that exhibit nonlinear behavior, and linear extrapolation is used for input values ​​less than the minimum interval boundary and input values ​​greater than the maximum interval boundary.

[0029] In the disclosed approximation method, quadratic interpolation is used together with a lookup table (LUT) for fast and efficient calculation. In order to perform quadratic interpolation to approximate the nonlinear region of the activation function, the nonlinear interval of the activation function is divided into non-uniform intervals to ensure a limited lookup table size and efficient software and hardware implementation, because the LUT can store interpolation coefficient values ​​for each of the extreme values ​​of the non-uniform interval.

[0030] In one or more embodiments of the present disclosure, the piecewise approximation technique may be implemented in a semiconductor intellectual property core (SIP core) or IP core or in hardware as a hard macro IP or netlist IP, where intervals and interpolation coefficients for quadratic interpolation may be stored in a lookup table (LUT), where the LUT may store intervals and interpolation coefficients corresponding to a plurality of different functions, and may also be reprogrammed to support a plurality of additional functions while sharing a single hardware implementation. In this case, high accuracy is achieved for quantized inputs and outputs, and the reprogrammable tables may be supplemented by common tables fixed in hardware of the IP core or hard macro IP or netlist IP. For example, an AI accelerator may be configured with an IP core or hard macro IP incorporating the piecewise approximation technique of the present disclosure to approximate nonlinear regions of activation functions for AI and / or ML.

[0031] In one or more embodiments, the piecewise approximation method of the present disclosure can provide an efficient solution to traditional approximation methods for activation functions used in AI / ML by reducing the amount of computation required in traditional approximation methods by performing quadratic interpolation in regions or intervals of the activation function that exhibit nonlinear behavior and using linear extrapolation elsewhere.

[0032] The piecewise approximation method of the present disclosure can be implemented in hardware as a hard macro IP or a netlist IP (for example, in an edge inference engine), and can occupy a small silicon area and provide high throughput. The size of the LUT used for quadratic interpolation in the piecewise approximation method of the present disclosure is small and can therefore be implemented as a trigger, thereby enabling single instruction multiple data (SIMD) to perform activation function calculations. The implementation scheme of the piecewise approximation method of the present disclosure can be flexible because the LUT can be reprogrammed using software. In addition, if the activation function requires, the continuity of the first-order derivative can be guaranteed by using piecewise quadratic interpolation, so gradient descent training of AI / ML is possible. In addition, the non-uniform distribution of the interpolation intervals used in the piecewise approximation method of the present disclosure can ensure high accuracy of the activation function output. In addition, the piecewise approximation method of the present disclosure can be used to approximate functions other than the activation function of the neural network.

[0033] Figure 2 The diagram illustrates a piecewise approximation method for an activation function of a neural network for an AI accelerator according to one embodiment of the present disclosure.

[0034] exist Figure 2 In the method, at 210, an input value x is received.

[0035] Next, at 215, an AI accelerator (e.g., a hardware accelerator) or a hard macro IP (e.g., an electronic circuit configured by a hard macro (IP)) incorporating the piecewise approximation method 200 of the present disclosure can determine whether the input value x is within the range of quadratic interpolation. In other words, at 215, the AI accelerator or the hard macro IP incorporating the piecewise approximation method 200 of the present disclosure can determine whether the input value x is within the boundaries of a set of intervals I during which the activation function exhibits non-linear behavior.

[0036] For example, if it is assumed that there is the set of intervals I and a set of associated limit values L: {l0, l1,..., l N}, then for the input x of the activation function, quadratic interpolation will be performed in the interval i under the following circumstances:

[0037]

[0038] In other words, if the input value x is within the boundaries of the set of intervals I, excluding the right boundary (e.g., l0 ≤ x < l N-1 )(for example, for each non-uniform interval, the right boundary is included in the next interval; however, the input value cannot be included in two intervals), then the parameters of the interval i are used to perform quadratic interpolation to approximate the input x. Otherwise, linear interpolation is performed to approximate the input x (e.g., for x < l0 and for x ≥ l N-1 ).

[0039] In one or more embodiments, the set of intervals I can be the intervals during which the activation function exhibits non-linear behavior. In one or more embodiments, it can be assumed that the set of intervals I incorporates a set of non-uniform intervals, where i represents an interval in the set of intervals I.

[0040] In one or more embodiments, to determine the set of intervals I, the linear and non-linear regions of the activation function can be identified and the associated limit values for the linear and non-linear regions of the activation function can be defined.

[0041] If at 215 it is determined that the input value x is within the boundaries of the set of intervals I (excluding the right boundary), then at 220, the AI accelerator or the hard macro IP can determine the interval i from among the set of intervals I that contains the value of the input x.

[0042] For the interval i in the set of intervals I, the quadratic interpolation of the input x is based on the following expression:

[0043] y = a i + b i x + c i x 2 (2)

[0044] In other words, the quadratic interpolant in interval i is given by parameter a i 、b i 、c i Representation.

[0045] Next, at 225, the parameter a associated with the interval i containing the input x (or input key x) is retrieved from the LUT associated with the input activation function (e.g., the LUT associated with the type of the input activation function). i 、b i 、c i The LUT associated with the type of input activation function incorporates a parameter a for each interval i in the set of intervals I. i 、b i 、c i In the LUT, i may also indicate the index of the LUT, for example, i=lookup(x). The LUT associated with the activation function may be stored in an AI accelerator or a hard macro IP.

[0046] In one or more embodiments, one or more memory devices in an AI accelerator or hard macro IP may store multiple LUTs including coefficients for quadratic interpolation of non-linear regions for different activation functions. In one or more embodiments, before approximating an activation function, multiple LUTs including coefficients for quadratic interpolation of non-linear regions for different activation functions may be determined (e.g., manually or by a processor through manual analysis of the function) and stored in the LUTs. The values ​​stored in the LUTs may be fixed (e.g., hard-coded in a hard macro) or the macro may define a programmable LUT so that a user can later program the LUT with limits and coefficients for approximating other functions.

[0047] To determine the LUT, the intervals where the activation function exhibits nonlinear behavior (e.g., the set of intervals I) are divided into non-uniform intervals. For example, the set of intervals I incorporates a set of non-uniform intervals, where i represents an interval in the set of intervals I. Using a non-uniform distribution of interpolation intervals ensures uniform accuracy of the activation function output. For example, in one or more embodiments, the Remitz minimax algorithm can be used to determine the non-uniform distribution of the interpolation intervals to minimize the approximation error and improve accuracy.

[0048] In one or more embodiments, the non-uniform distribution of intervals may be based on the most significant bits (MSBs) of the quantized input, such as two's complement encoded input values. For example, the 6 most significant input bits may be used to define the boundaries of 16 non-uniform intervals. In one or more embodiments, each interval is characterized by a left limit and a right limit. Using N intervals for approximation requires defining N+1 limit values.

[0049] One or more embodiments of the present disclosure describe performing quadratic interpolation only within the nonlinear region of the activation function to limit the LUT size. The limited size of the LUT can allow high-speed hardware implementation of the piecewise approximation technique of the present disclosure (for example, using SIMD), limit the silicon area used to store the LUT, and reduce the software reconfiguration table overhead. Using quadratic interpolation can also ensure that the resulting approximation is differentiable, which helps gradient descent training methods in AI / ML.

[0050] Therefore, quadratic interpolation is performed on the input x using equation (2).

[0051] To reduce the approximation error in the interpolation range I, a minimax algorithm such as Remitz may be used to approximate the function represented in equation (2) using second-order Chebyshev polynomials.

[0052] The expression of equation (2) can be expressed relative to the interpolation interval o i To rewrite the starting point:

[0053] y = a i ′ + b i ′ (xo i ) + c i (xo i ) 2 (3)

[0054] The rewritten form of equation (2) expressed in equation (3) can reduce the number of bits in the multiplication in quantized applications. In one or more embodiments, i is the left side (eg, left boundary) of the interval (interval i) in which the quadratic interpolation is performed.

[0055] In one or more embodiments, b i ′=b i +2*c i *o i ; and a i '=a i +b i *o i +c i *o i 2

[0056] Equation (3) can be rewritten as Horner's series to improve accuracy:

[0057] y = a i ′ + (b i ′ + c i (xo i )) (xoi ) (4)

[0058] If we assume that d = (x i ), then equation (4) can be rewritten as:

[0059] y = a i ′ + (b i ′ + c i d) d (5)

[0060] The quantized version of equation (5) above may include shift operations represented by powers of two factors within the range of representable values ​​in the fixed point representation of the approximated output of the activation function (e.g., to ensure that intermediate and final values ​​do not overflow and have the desired quantization). For example, the input x and output y are shifted by powers of two factors to ensure that intermediate and final values ​​do not overflow and have the desired quantization. The two (n i and m i ) may be quantization interval dependent and may be determined by the quantization process. Considering the two factors (n i and m i ), the shift operation represented by the power of , equation (5) can be rewritten as:

[0061] y = a i ′ + (b i ′ + c i d 2 ni ) d 2 mi (6)

[0062] In equations (1) to (6), the quadratic interpolation range (eg, the set of intervals I) may be characterized by: 1) I+1 interval limit values ​​(o i ), which specifies the left and right boundaries of each of the interpolation intervals, 2) an offset value (a i ), 3) I first-order derivative value (b i ), 4) I second-order derivative value (c i ), 5) optionally I first-order shift values ​​to (m i ), and 6) optionally I second-order shift values ​​to (n i ).

[0063] Next, at 230, quadratic interpolation is performed on the input x using equation (6).

[0064] If at 215, it is determined that the input value x is outside the quadratic interpolation range (eg, the set of intervals I) (for example, if the input value x is in the range [-∞, l0) or [l N,∞]), then at 235, the ranges [-∞,l0) and [l N ,∞] min and a max , and for the range [-∞,l0) and [l N ,∞] is the gradient or slope b of the activation function min and b max .

[0065] Next, at 240, a linear extrapolation may be performed on the input x to approximate the input x using the following equation:

[0066] y = a min + b min x (7)

[0067] y = a max + b max x (8)

[0068] In equations (7) and (8), the minimum and maximum values ​​represent the left and right ranges. For example, if x is between [-∞, l0), then equation (7) can be used to approximate x, and if x is between [l N ,∞], then equation (8) can be used to approximate x. In one or more embodiments, it is assumed that the minimum value coincides with the left side of the leftmost quadratic interpolation interval (e.g., i0) and the maximum value coincides with the rightmost quadratic interpolation interval (e.g., i N ), equations (7) and (8) can be rewritten as:

[0069] y = a min ′ + b min ′ (x-o0) (9)

[0070] y = a max ′ + b max ′ (xo I ) (10)

[0071] Here, o0 is the left limit of the leftmost quadratic interpolation interval (e.g., i0) and o I is the rightmost quadratic interpolation interval (for example, i N ) is the right limit of .

[0072] The quantized version of the above equation can be expressed as:

[0073] y = a min ′ + b min ′ d 2 nmin (11)

[0074] y = a max ′ + b max ′ d 2 nmax (12)

[0075] The quantized versions of equations (11) and (12) above may include shift operations (e.g., n min 、n max ), which is represented by a power of two factors to ensure that the intermediate and final values ​​do not overflow and have the desired quantization. The left linear extrapolation range and the right linear extrapolation range can be characterized by the following: 1) The left offset value (a min ) and the first-order derivative (b min ), 2) right offset value (a max ) and the first-order derivative (b max ), 3) optionally left-shift the value to (n min ), and 4) optionally right-shift the value to (n max ).

[0076] In one or more embodiments, a min and a max can be respectively expressed for the range of input x [-∞,l0) and [l N ,∞], and b min and b max It can be expressed for the range [-∞,l0) and [l N ,∞] is the gradient or slope of the activation function.

[0077] At 245, the approximated output y of the activation function is determined by performing quadratic interpolation on the input x using equation (6) or performing linear extrapolation on the input x using equations (11) and (12). In one or more embodiments, the input x and output y are shifted by powers of two factors to ensure that intermediate and final values ​​do not overflow and have the desired quantization.

[0078] Figures 3A to 3B Illustrated instructions for performing Figure 2 The AI ​​accelerator or IP core of the disclosed segmented approximation method discussed. For example, Figures 3A to 3B Illustrate the use of an AI accelerator (e.g., a neural network processing unit) to perform Figure 2 Function of the piecewise approximation method of the present disclosure discussed.

[0079] For example, the AI ​​accelerator or IP core 300 includes a bit shifter 310, a device 315 for identifying the interval to which the input value belongs, a device 325 for determining the slope of the input activation function within a specific interval of linear interpolation, a device 320 for storing a LUT associated with one or more activation functions, a multiplexer 330, and two multiply-accumulate scaling devices 335 and 340. It should be understood that the devices 315, 325, and 320 can be combined together in various combinations without departing from the scope of the present disclosure.

[0080] In the AI ​​accelerator or IP core 300 of FIG. 3 , when an input x is received, a bit shift operation is performed on the input x. Figure 2 As discussed, a shift operation can be performed to ensure that the intermediate and final values ​​do not overflow and have the desired quantization. As shown in equation (6), the shift operation can be represented by a power of two factors. These two (n in equation (6) i and m i ) can be quantization interval dependent.

[0081] The device 315 may determine whether the input value x is within the boundaries of the set of intervals I (excluding the right boundary), as discussed with respect to equation (1). The set of intervals I may be intervals where the activation function exhibits nonlinear behavior. In one or more embodiments, it may be assumed that the set of intervals I incorporates a set of non-uniform intervals, where i represents an interval in the set of intervals. In one or more embodiments, a subset of the MSBs of the input x is input to the device 315, because the non-uniform distribution of intervals may be based on the most significant bit (e.g., x_msb) of the quantized input, a two's complement encoded input value. For example, the 6 most significant input bits may be used to define the boundaries of 16 non-uniform intervals.

[0082] If module 315 determines that the input x is within the boundaries of the set of intervals I, then quadratic interpolation is performed using the parameters of interval i representing an interval in the set of intervals I to approximate the input x.

[0083] The means 315 also determines an interval i from among the set of intervals I containing the value of the input x.

[0084] Next, from the device 320, the parameter a associated with the interval i containing the input x (or the input key x) is retrieved from the LUT. i 、b i 、c i The value of to perform quadratic interpolation on the input x based on equation (2) or equation (6). The LUT associated with the activation function may be stored in the device 320 or in another memory or storage unit of the AI ​​accelerator or hard macro IP 300.

[0085] Once the parameters a are retrieved from the LUT to perform quadratic interpolation on the input x i 、b i 、c i , multiplexer 330 may select the input of multiplexer 330 connected to the output of module 320 (eg, when performing quadratic interpolation, there may not be any output from module 325). Multiplexer 330 outputs parameter a from module 320 i 、b i 、c i And the multiplication-accumulation scaling modules 335 and 340 perform quadratic interpolation on the input x based on equations (5) and (6). For example, in one or more embodiments, the multiplication-accumulation scaling module 335 may perform the operation shown in equation (5) (b i ′+c i d) and the multiply-accumulate scaling module 340 may be implemented by multiplying the output from the multiply-accumulate scaling module 335 by “d” and then summing the result with the parameter a i The value of the output y is determined by adding the values ​​of

[0086] In one or more embodiments, the output y may be shifted (eg, see equation (6)) to ensure that intermediate and final values ​​do not overflow and have the desired quantization.

[0087] However, if module 315 determines that the input x is not within the boundaries of the set of intervals I (for example, if the input value x is outside the quadratic interpolation range, that is, x is in the range [-∞, l0) and [l N ,∞]), then linear extrapolation is used to approximate the input x using equations (7) and (8).

[0088] In this case, the device 325 determines the range [-∞, l0) and [l N+1 ,∞] min and a max , and for the range [-∞,l0) and [l N+1 ,∞] is the gradient or slope b of the activation function min and b max .

[0089] Once the parameter a is determined min or a max and b min or b max , the multiplexer 330 outputs the parameter a min or a max and b min or b maxAnd the multiplication-accumulation scaling modules 335 and 340 perform linear interpolation on the input x based on equations (9) and (10). For example, in one or more embodiments, the multiplication-accumulation scaling module 335 may perform the operation shown in equation (9) (b min ′(x-o0)) or as shown in equation (10) max ′(xo I )), and the multiplication-accumulation scaling module 340 can be obtained by combining the output from the multiplication-accumulation scaling module 335 with “a min '" or "a max '" are added to determine the value of the output y.

[0090] In one or more embodiments, the output y may be shifted (eg, see equations (11) and (12)) to ensure that intermediate and final values ​​do not overflow and have the desired quantization.

[0091] Figure 3B Include Figure 3A All components of except the second multiply-accumulate scaling module 340. Figure 3B In the multiplication and accumulation scaling module 335, the multiplication and accumulation scaling module 335 performs the multiplication and accumulation scaling Figure 3A All calculations performed by the multiply-accumulate scaling modules 335 and 340 in (for example, in two stages). Figure 3B In a system of FIG. 1 , during the first round of calculation, the multiplication-accumulation scaling module 335 may execute the following: Figure 3A The output of the first round of calculations performed by the multiplication-accumulation scaling module 335 is fed back to the multiplication-accumulation scaling module 335 via the demultiplexer 345, and during the second round of operations, the multiplication-accumulation scaling module 335 may perform the calculations performed by the multiplication-accumulation scaling module 335. Figure 3A The multiplication-accumulation scaling module 340 performs the calculations.

[0092] In other words, instead of Figure 3A , only one of the multiply-accumulate scaling modules 335 and 340 may be reused to perform all computations of the approximation of the input x. Figure 3BIn an embodiment, after the first multiplication-accumulation scaling operation is performed by the multiplication-accumulation scaling module 335, the first result is fed back to the multiplication-accumulation scaling module 335 as an input via the demultiplexer 345. Then, a second multiplication-accumulation scaling operation is performed by the module 335 based on the first result and the result of the second multiplication-accumulation scaling operation can be output via the demultiplexer 345 as a second result, which can be an approximate final result of the input x. In one or more embodiments, the demultiplexer 345 can select a first output port connected to the input of the multiplication-accumulation scaling module 335 to output the first result, and the demultiplexer 345 can select a second output port connected to the input of the multiplication-accumulation scaling module 335 to output the second result or the approximate final result of the input x.

[0093] For example, in Figure 4 In the Figure 2 The method discussed uses quadratic interpolation to approximate the activation function tanh during the interval [-8, 8) where tanh exhibits nonlinear behavior. For example, the interval [-8, 8) can be divided into multiple non-uniform intervals (e.g., the set of intervals I) and the input x can be approximated using a LUT incorporating parameter values ​​for each interval i in the set of intervals I using equations (1) to (6). However, using equations (7) to (12), the activation function tanh is approximated using linear extrapolation for the intervals [-∞, -8) and [8, ∞]. In this case, a min ′ is -1 and a max ′ has a value of +1. However, b min ' and b max Both values ​​can be zero.

[0094] For example, in Figure 5 In the Figure 2 The method discussed uses quadratic interpolation to approximate the activation function sigmoid during the interval [-16, 16) where sigmoid exhibits nonlinear behavior. For example, the interval [-16, 16) can be divided into multiple non-uniform intervals (e.g., the set of intervals I) and the input x can be approximated using a LUT incorporating parameter values ​​for each interval i in the set of intervals I using equations (1) to (6). However, using equations (7) to (12), the activation function sigmoid is approximated using linear extrapolation for the intervals [-∞, -16) and [16, ∞]. In this case, a min ′ is -1 and a max ′ has a value of +1. However, b min ' and b max Both values ​​can be zero.

[0095] For example, in Figure 6 In the Figure 2 The method discussed uses quadratic interpolation to approximate the activation function swish during the interval [-5, 3) where swish exhibits nonlinear behavior. For example, the interval [-5, 3) can be divided into multiple non-uniform intervals (e.g., the set of intervals I) and the input x can be approximated using a LUT incorporating parameter values ​​for each interval i in the set of intervals I using equations (1) to (6). However, using equations (7) to (12), the activation function swish is approximated using linear extrapolation for the intervals [-∞, -5) and [3, ∞]. In this case, a min The value of ' is 0, a max ′ is 0 and b min ′ is also 0. However, b max The value of ′ is variable in the interval [3,∞].

[0096] One or more embodiments of the present disclosure describe a piecewise approximation method for an activation function for AI / ML. In the approximation method of the present disclosure, quadratic interpolation is used to approximate a region or interval (e.g., a nonlinear interval) of an activation function that exhibits nonlinear behavior, and linear extrapolation is used for input values ​​less than a minimum interval boundary and input values ​​greater than a maximum interval boundary. In the approximation method of the present disclosure, quadratic interpolation is used together with a lookup table (LUT) for fast and efficient calculations. In order to perform quadratic interpolation to approximate the nonlinear region of the activation function, the nonlinear interval of the activation function is divided into non-uniform intervals to ensure a limited lookup table size and efficient software and hardware implementation, because the LUT can store interpolation coefficient values ​​for each of the extreme values ​​of the non-uniform interval.

[0097] Figure 7 The diagram illustrates a set of exemplary processes 700 used during the design, verification and fabrication of a manufactured article (e.g., an integrated circuit) to transform and verify design data and instructions representing the integrated circuit. Each of these processes can be structured and enabled as multiple modules or operations. The term 'EDA' denotes the term 'electronic design automation'. These processes begin with the formation of a product concept 710 using information provided by a designer, which is transformed to form a manufactured article using a set of EDA processes 712. When the design is finally completed, the design is taped out 734, at which time the artwork (e.g., geometric pattern) of the integrated circuit is sent to a fabrication facility to produce a mask set, which is then used to fabricate the integrated circuit. After tape-out, semiconductor die are fabricated 736 and packaging and assembly processes 738 are performed to produce a finished integrated circuit 740.

[0098] The specification for a circuit or electronic structure can range from low-level transistor material layout to a high-level description language. The high-level representation can be used to design circuits and systems using a hardware description language ('HDL') such as VHDL, Verilog, SystemVerilog, SystemC, MyHDL, or OpenVera. The HDL description can be transformed into a logic-level register transfer level ('RTL') description, a gate-level description, a layout-level description, or a mask-level description. Each lower level of representation as a more detailed description adds more useful details to the design description, for example, more details for the modules that contain the description. The lower level of representation as a more detailed description can be computer-generated, derived from a design library, or formed by another design automation process. An example of a specification language for specifying a more detailed description of a lower-level representation language is SPICE, which is used for detailed descriptions of circuits with many analog components. The description at each representation level is enabled for use by the corresponding system of that layer (e.g., a formal verification system). The design process can use Figure 7 The process is described as being enabled by an EDA product (or EDA system).

[0099] During system design 714, the functionality of the integrated circuit to be manufactured is specified. The design can be optimized for desired characteristics such as power consumption, performance, area (physical and / or code line), and reduced cost. The design can be partitioned into different types of modules or components at this stage.

[0100] During logic design and functional verification 716, modules or components in a circuit are specified in one or more description languages ​​and the functional accuracy of the specifications is checked. For example, the components of the circuit can be checked to generate outputs that match the requirements of the specification of the circuit or system being designed. Functional verification can use simulators and other programs such as test bench generators, static HDL checkers, and formal checkers. In some embodiments, special component systems called 'emulators' or 'prototyping systems' are used to accelerate functional verification.

[0101] During synthesis and test design 718, the HDL code is transformed into a netlist. In some embodiments, the netlist may be a graph structure where the edges of the graph structure represent components of the circuit and where the nodes of the graph structure represent how the components are interconnected. Both the HDL code and the netlist are hierarchical manufacturing objects that can be used by EDA products to verify that the integrated circuit performs according to the specified design when manufactured. The netlist can be optimized for the target semiconductor manufacturing technology. In addition, the finished integrated circuit can be tested to verify that the integrated circuit meets the requirements of the specification.

[0102] During netlist verification 720, the netlist may be checked for compliance of timing constraints and correspondence of the HDL code.During design planning 722, the overall floor plan of the integrated circuit may be constructed and analyzed for timing and top-level routing.

[0103] During layout or physical implementation 724, physical placement (positioning of circuit components such as transistors or capacitors) and routing (connection of circuit components through multiple conductors) occur, and selection of cells from a library may be performed to enable specific logic functions. As used herein, the term 'cell' may specify a group of transistors, other components, and interconnects that provide a Boolean logic function (e.g., "and", "or", "not", "exclusive or") or a storage function (e.g., a flip-flop or latch). As used herein, a circuit 'block' may refer to two or more cells. Both cells and circuit blocks may be referred to as modules or components and are enabled both as physical structures and in simulation. Parameters (e.g., size) are specified for selected cells (based on 'standard cells') and may be accessed in a database for use by an EDA product.

[0104] During analysis and extraction 726, circuit functionality is verified at the layout level, which permits improvements to the layout design. During physical verification 728, the layout design is checked to ensure that manufacturing constraints (e.g., DRC constraints, electrical constraints, lithography constraints) are correct and that the circuit system functionality matches the HDL design specifications. During resolution enhancement 730, the geometry of the layout is transformed to improve how the circuit design is manufactured.

[0105] During tape-out, data is formed for use in producing lithographic masks (after applying lithographic enhancements where appropriate).During mask data preparation 732, the 'tape-out' data is used to produce lithographic masks, which are used to produce finished integrated circuits.

[0106] Computer systems (e.g. Figure 8 The storage subsystem of the computer system 800) may be used to store programs and data structures used by some or all of the EDA products described herein, as well as cells for the library and products for using the physical and logical designs described by the library.

[0107] Figure 8 An exemplary machine of a computer system 800 is illustrated within which a set of instructions for causing the machine to perform any one or more of the methodologies discussed herein may be executed. In alternative embodiments, the machine may be connected (e.g., networked) to other machines in a LAN, an intranet, an extranet, and / or the Internet. The machine may operate in the capacity of a server or a client machine in a client-server network environment, as a peer machine in a peer-to-peer (or distributed) network environment, or as a server or a client machine in a cloud computing infrastructure or environment.

[0108] The machine may be a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a cellular phone, a web appliance, a server, a network router, a switch or a bridge, or any machine capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by the machine. Further, while only a single machine is illustrated, the term "machine" shall also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein.

[0109] The exemplary computer system 800 includes a processing device 802, a main memory 804 (e.g., read-only memory (ROM), flash memory, dynamic random access memory (DRAM), such as synchronous DRAM (SDRAM)), a static memory 806 (e.g., flash memory, static random access memory (SRAM), etc.), and a data storage device 818, which communicate with each other via a bus 830.

[0110] The processing device 802 represents one or more processors (e.g., microprocessors, central processing units, etc.). More specifically, the processing device may be a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, or a processor that implements other instruction sets or a processor that implements a combination of instruction sets. The processing device 802 may also be one or more special-purpose processing devices (e.g., application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), digital signal processors (DSPs), network processors, etc.). The processing device 802 may be configured to execute instructions 826 for performing the operations and steps discussed herein.

[0111] The computer system 800 may further include a network interface device 908 to communicate via the network 820. The computer system 800 may also include a video display unit 810 (e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)), an alphanumeric input device 812 (e.g., a keyboard), a cursor control device 814 (e.g., a mouse), a graphics processing unit 822, a signal generating device 816 (e.g., a speaker), a graphics processing unit 822, a video processing unit 828, and an audio processing unit 932. In one or more embodiments, the computer system 800 may also include a neural network processing unit 842. In one or more embodiments, Figure 7 The process may be performed in the neural network processing unit 842 and the neural network processing unit 842 may include Figures 3A to 3B AI accelerator 300.

[0112] The data storage device 818 may include a machine-readable storage medium 824 (also referred to as a non-transitory computer-readable medium) having stored thereon one or more sets of instructions 826 or software embodying any one or more of the methodologies or functions described herein. The instructions 826 may also reside completely or at least partially within the main memory 804 and / or the processing device 802 during execution thereof by the computer system 800, the main memory 804 and the processing device 802 also constituting machine-readable storage media.

[0113] In some embodiments, the instructions 826 include instructions to implement functionality corresponding to the present disclosure. Although the machine-readable storage medium 824 is shown as a single medium in the exemplary embodiment, the term "machine-readable storage medium" should be considered to include a single medium or multiple media (e.g., a centralized or distributed database and / or associated cache memory and server) storing one or more sets of instructions. The term "machine-readable storage medium" should also be considered to include any medium capable of storing or encoding a set of instructions for execution by a machine and causing the machine and processing device 802 to perform any one or more of the methods of the present disclosure. Therefore, the term "machine-readable storage medium" should be considered to include, but not limited to, solid-state memory, optical media, and magnetic media.

[0114] Some portions of the foregoing detailed description have been presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to those skilled in the art. An algorithm may be a sequence of operations leading to a desired result. The operations are those requiring physical manipulation of physical quantities. Such quantities may take the form of electrical or magnetic signals capable of being stored, combined, compared, and otherwise manipulated. Such signals may be referred to as bits, values, elements, symbols, characters, terms, numbers, and the like.

[0115] It should be remembered, however, that all of these terms and similar terms are associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless otherwise specifically stated as apparent from the present disclosure, it should be understood that throughout this description, specific terms refer to actions and processes of computer systems or similar electronic computing devices that manipulate and transform data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system's memories or registers or other such information storage devices.

[0116] The present disclosure also relates to an apparatus for performing the operations described herein. This apparatus may be specially constructed for the intended purpose, or it may include a computer that is selectively activated or reconfigured by a computer program stored in the computer. This computer program may be stored in a computer-readable storage medium, such as but not limited to any type of disk (including floppy disks, optical disks, CD-ROMs, and magneto-optical disks), read-only memory (ROM), random access memory (RAM), EPROM, EEPROM, magnetic or optical cards, or any type of medium suitable for storing electronic instructions, each coupled to a computer system bus.

[0117] The algorithms and displays presented herein are not inherently related to any particular computer or other device. Various other systems may be used with the programs according to the teachings herein, or it may prove convenient to construct more specialized devices to perform the methods. Additionally, the present disclosure is not described with reference to any particular programming language. It will be appreciated that various programming languages ​​may be used to implement the teachings of the present disclosure as described herein.

[0118] The present disclosure may be provided as a computer program product or software, which may include a machine-readable medium having instructions stored thereon, which may be used to program a computer system (or other electronic device) to perform a process according to the present disclosure. A machine-readable medium includes any mechanism for storing information in a form readable by a machine (e.g., a computer). For example, a machine-readable (e.g., computer-readable) medium includes a machine (e.g., computer) readable storage medium, such as a read-only memory ("ROM"), a random access memory ("RAM"), a magnetic disk storage medium, an optical storage medium, a flash memory device, etc.

[0119] The detailed description described below in conjunction with the accompanying drawings is intended to describe some exemplary embodiments of the system and method for an AI accelerator based on an efficient lookup table function provided in accordance with the present disclosure and is not intended to represent the only form in which the present disclosure can be constructed or utilized. The description sets forth the features of the present disclosure in conjunction with the illustrated embodiments. However, it should be understood that the same or equivalent functions and structures may be implemented by different embodiments, which are also intended to be included within the scope of the present disclosure. As indicated elsewhere herein, similar element numbers are intended to indicate similar elements or features.

[0120] It will be understood that although the terms "first," "second," "third," etc., may be used herein to describe various elements, components, regions, layers, and / or sections, these elements, components, regions, layers, and / or sections should not be limited by these terms. These terms are only used to distinguish one element, component, region, layer, or section from another element, component, region, layer, or section. Thus, a first element, component, region, layer, or section discussed herein may be referred to as a second element, component, region, layer, or section without departing from the scope of the present disclosure.

[0121] The terms used herein are for the purpose of describing specific embodiments only and are not intended to limit the present disclosure. As used herein, the terms "substantially," "about," and the like are used as approximate terms rather than as terms of degree, and are intended to account for the inherent deviations of measurements or calculations that those skilled in the art will recognize.

[0122] As used herein, the singular forms "a and an" are also intended to include the plural forms, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising" when used in this specification specify the presence of stated features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. As used herein, the term "and / or" includes any or all combinations of one or more of the associated listed items. When preceding a list of elements, expressions such as "at least one of..." modify the entire list of elements and do not modify the individual elements of the list. In addition, when describing embodiments of the present disclosure, the use of "may" refers to "one or more embodiments of the present disclosure." In addition, the term "exemplary" is intended to refer to an example or illustration. As used herein, the terms "use," "using," and "used" may be considered synonymous with the terms "utilize," "utilizing," and "utilized," respectively.

[0123] In the foregoing disclosure, embodiments of the present disclosure have been described with reference to specific exemplary embodiments of the present disclosure. It will be apparent that various modifications may be made to the present disclosure without departing from the broader spirit and scope of the embodiments of the present disclosure as set forth in the following claims. Where the present disclosure refers to some elements in the singular, more than one element may be depicted in the figures and similar elements are labeled with similar numbers. Therefore, the present disclosure and the drawings should be considered in an illustrative sense rather than a restrictive sense.

Claims

1. A method for approximating an activation function, the method comprising: Receiving an input value of the activation function; determining that the input value is within a range, the range comprising a set of non-uniform intervals; determining a selected interval from among the set of non-uniform intervals that includes the input value; retrieving, by a hardware accelerator, values ​​of one or more quadratic interpolation parameters associated with the selected interval from a lookup table (LUT) associated with the type of the activation function; performing quadratic interpolation on the input value using the values ​​of the one or more quadratic interpolation parameters to approximate the input value; and A first approximated output of the activation function is determined based on a result of the quadratic interpolation performed on the input values. 2 . The method according to claim 1 , wherein the non-uniform distribution of the set of non-uniform intervals is determined using a Remitz minimax algorithm.

3. The method of claim 1 or 2, wherein the determining the selected interval from the set of non-uniform intervals is based on one or more of a most significant bit (MSB) of the input value and a two's complement encoded input value.

4. The method of claim 1, wherein the LUT associated with the activation function comprises the values ​​of the one or more quadratic interpolation parameters for each interval from among the set of non-uniform intervals.

5. The method of claim 1 or 4, wherein the hardware accelerator stores a plurality of LUTs including parameters associated with quadratic interpolation ranges of a plurality of activation functions. The method of claim 5 , wherein the plurality of LUTs are implemented as flip-flops in the hardware accelerator.

7. The method according to claim 1, further comprising: determining, by the hardware accelerator, that the input value is within one or more linear extrapolation ranges, the one or more linear extrapolation ranges not overlapping the range; determining, by the hardware accelerator, an offset value and a slope of the activation function for each of the one or more linear extrapolation ranges; performing, by the hardware accelerator, linear extrapolation on the input value using the offset value and the slope of the activation function for each of the one or more linear extrapolation ranges to approximate the input value; and A second approximated output of the activation function is determined by the hardware accelerator based on a result of the linear extrapolation performed on the input value.

8. The method of claim 7, wherein the input value is shifted by a power of two factors within a range of representable values ​​in a fixed point representation of the second approximated output of the activation function.

9. A system for approximating an activation function, the system comprising: a bit shifter configured to receive an input value of the activation function and perform a shift operation on the input value to calculate a most significant bit (MSB) of the input value; a first device configured to determine whether the input value is within a range that includes a set of non-uniform intervals, and to determine a selected interval from among the set of non-uniform intervals that includes the input value based on the MSB of the input value; a second device configured to retrieve values ​​of one or more quadratic interpolation parameters associated with the selected interval from a lookup table (LUT) associated with the activation function; One or more multiply-accumulate scaling circuits configured to perform quadratic interpolation on the input value using the values ​​of the one or more quadratic interpolation parameters to approximate the input value, and determine a first approximated output of the activation function based on a result of the quadratic interpolation performed on the input value.

10. The system of claim 9, wherein the system comprises a hardware accelerator and / or an electronic circuit configured by a hard macro intellectual property (IP).

11. The system of claim 10, wherein the hardware accelerator stores a plurality of LUTs including parameters associated with quadratic interpolation ranges of a plurality of activation functions.

12. The system of claim 11, wherein the plurality of LUTs are implemented as flip-flops in the hardware accelerator.

13. The system of claim 9, wherein a Remitz minimax algorithm is used to determine the non-uniform distribution of the set of non-uniform intervals.

14. The system of claim 9 or 13, wherein the determining the selected interval from among the set of non-uniform intervals is based on one or more of a most significant bit (MSB) of the input value and a two's complement encoded input value.

15. The system of claim 9, wherein the LUT associated with the activation function includes the values ​​of the one or more quadratic interpolation parameters for each interval from among the set of non-uniform intervals.

16. The system of claim 9, further comprising: A third device configured to determine an offset value and a slope of the activation function for each of the one or more linear extrapolation ranges based on the first device determining that the input value is within the one or more linear extrapolation ranges, and the one or more linear extrapolation ranges do not overlap with the range, wherein the one or more multiplication-accumulation scaling circuits are further configured to perform linear extrapolation on the input value using the offset value and the slope of the activation function for each of the one or more linear extrapolation ranges to approximate the input value, and determine a second approximated output of the activation function based on a result of the linear extrapolation performed on the input value.

17. The system of claim 16, wherein the input value is shifted by a power of two factor within a range of representable values ​​in a fixed point representation of the second approximated output of the activation function.

18. A non-transitory computer-readable medium comprising a stored representation of an accelerator circuit, the stored representation of the accelerator circuit, when synthesized by a processor, causes the processor to synthesize a structure of an electronic circuit, the electronic circuit being configured to: Receive the input value of the activation function; determining that the input value is within a range, the range comprising a set of non-uniform intervals; determining a selected interval from among the set of non-uniform intervals that includes the input value; retrieving values ​​of one or more quadratic interpolation parameters associated with the selected interval from a lookup table (LUT) associated with the activation function; performing quadratic interpolation on the input value using the values ​​of the one or more quadratic interpolation parameters to approximate the input value; and A first approximated output of the activation function is determined based on a result of the quadratic interpolation performed on the input values.

19. The non-transitory computer-readable medium of claim 18, wherein the input value is shifted by a power of two factor within a range of representable values ​​in a fixed point representation of the first approximated output of the activation function.

20. The non-transitory computer readable medium of claim 18, wherein the electronic circuit is further configured to: determining that the input value is within one or more linear extrapolation ranges, the one or more linear extrapolation ranges not overlapping the range; determining an offset value and a slope of the activation function for each of the one or more linear extrapolation ranges; performing linear extrapolation on the input value using the offset value and the slope of the activation function for each of the one or more linear extrapolation ranges to approximate the input value; and A second approximated output of the activation function is determined based on a result of the linear extrapolation performed on the input value.

Images