Hyperbolic functions for machine learning acceleration

Abstract

The present disclosure generally relates to techniques for enhancing recurrent neural networks (RNNs) implemented on integrated circuits. Specifically, approximations of activation functions used in RNNs, such as sigmoid and hyperbolic tangent, can be implemented in integrated circuits, which can result in increased efficiency, reduced latency, increased accuracy, and reduced resource consumption related to implementing machine learning.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is a non-provisional application claiming priority to U.S. Provisional Patent Application Nos. 62 / 532,874 (filed July 14, 2017) and 62 / 555,510 (filed September 7, 2017), both entitled “Hyperbolic Functions for Machine Learning Acceleration,” which are hereby incorporated by reference in their entireties for all purposes. Technical Field

[0003] The present disclosure relates generally to integrated circuits, such as field programmable gate arrays (FPGAs), and more particularly to activation functions implemented on integrated circuits, such as FPGAs, for machine learning. Background Art

[0004] This section is intended to introduce the reader to various aspects of the art that may be related to various aspects of the present disclosure, which are described and / or claimed below. This discussion is believed to be helpful in providing the reader with background information to facilitate a better understanding of the various aspects of the present disclosure. Accordingly, it should be understood that these statements are to be read in this light and not as admissions of prior art.

[0005] Machine learning is becoming an increasingly valuable application area. For example, it can be used in natural language processing, computer vision (e.g., object recognition), bioinformatics, and economics, among other fields and applications. A common class of machine learning techniques is represented by recurrent neural networks (RNNs). While RNNs are increasingly used in real-world applications (e.g., translation, text, and data processing), their use of iterative recursion and possible combinations of large matrix-vector multiplications can introduce complexity. Consequently, RNNs can suffer from latency and accuracy issues, as well as resource utilization issues, particularly related to their use of activation functions. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Various aspects of the present disclosure may be better understood by reading the following detailed description and by referring to the accompanying drawings, in which:

[0007] Figure 1 is a block diagram of a system for implementing a neural network, such as a recurrent neural network (RNN), according to an embodiment;

[0008] Figure 2 is a block diagram of an integrated circuit capable of implementing an RNN according to an embodiment;

[0009] Figure 3 is a graph of a sigmoid function according to an embodiment;

[0010] Figure 4 It shows the embodiment of the Figure 3 A subchart that is part of a chart;

[0011] Figure 5 It is shown that according to an embodiment, single-precision floating point can be used to approximate Figure 3 Block diagram of the S-shaped approximation circuit of the S-shaped function;

[0012] Figure 6 According to the embodiment Figure 3 Second graph of the sigmoid function;

[0013] Figure 7 It is shown that half-precision floating point can be used to approximate Figure 6 Block diagram of the half-precision S-shaped approximation circuit of the S-shaped function;

[0014] Figure 8 yes Figure 7 A second embodiment of a half-precision S-shaped approximation circuit;

[0015] Figure 9 yes Figure 7 A third embodiment of the half-precision S-shaped approximation circuit;

[0016] Figure 10 is a block diagram showing a low-precision S-shaped approximation circuit according to an embodiment;

[0017] Figure 11 is a graph of a hyperbolic tangent function according to an embodiment;

[0018] Figure 12 It is shown that the approximation according to the embodiment Figure 11 A block diagram of a hyperbolic tangent approximation circuit for the hyperbolic tangent function; and

[0019] Figure 13 It is shown that the approximation according to the embodiment Figure 3 The S-shaped function and Figure 11 Block diagram of the circuit for approximating the fused activation function of the hyperbolic tangent function. DETAILED DESCRIPTION

[0020] One or more specific embodiments are described below. In an effort to provide a brief description of these embodiments, not all features of an actual implementation are described in this specification. It should be appreciated that in the development of any such actual implementation, as in any engineering or design project, many implementation-specific decisions may be made to achieve the developer's specific goals, such as compliance with system-related and business-related constraints, which may vary from implementation to implementation. Furthermore, it should be appreciated that such development efforts may be complex and time-consuming, but will nonetheless be a routine matter of design, fabrication, and manufacturing for those of ordinary skill having the benefit of this disclosure.

[0021] As discussed in more detail below, embodiments of the present disclosure generally relate to circuits for enhancing neural networks (e.g., recurrent neural networks (RNNs)) that use activation functions implemented on integrated circuits. Specifically, in certain embodiments, approximations of activation functions used in RNNs (e.g., sigmoid and hyperbolic tangent) can be implemented in integrated circuits (e.g., FPGAs), which can result in increased efficiency, reduced latency, increased accuracy, and reduced resource utilization related to machine learning.

[0022] In view of the above, Figure 1 A block diagram of a system 10 in which machine learning techniques can be implemented is shown. A designer may desire to implement functionality, such as the hyperbolic functions of the present disclosure, on an integrated circuit device 12 (IC, such as a field programmable gate array (FPGA)). The designer may specify a high-level program to be implemented, such as an OpenCL program, which enables the designer to more efficiently and easily provide programming instructions to configure a set of programmable logic cells for the integrated circuit device 12 without requiring specific knowledge of a low-level hardware description language (such as Verilog or VHDL). For example, because OpenCL is quite similar to other high-level programming languages ​​(such as C++), designers of programmable logic familiar with such programming languages ​​may have a simplified learning curve compared to designers who are required to learn an unfamiliar low-level hardware description language to implement new functionality in the IC.

[0023] Designers can use design software 14 (e.g., a version of Intel® Quartus® from Intel Corporation) to implement their high-level design. Design software 14 can use a compiler 16 to convert the high-level program into a low-level description. Compiler 16 can provide machine-readable instructions representing the high-level program to host computer 18 and integrated circuit device 12. Host computer 18 can receive a host program 22, which can be implemented by kernel program 20. To implement host program 22, host computer 18 can pass instructions from host program 22 to integrated circuit device 12 via communication link 24 (which can be, for example, direct memory access (DMA) communication or peripheral component interconnect express (PCIe) communication). In some embodiments, kernel program 20 and host computer 18 can implement the configuration of an RNN 26 on integrated circuit device 12. RNN 26 can include circuits and / or other logic elements and can be configured to implement an activation function.

[0024] Turning now to a more detailed discussion of the integrated circuit device 12, Figure 2 An integrated circuit device 12 is shown, which may be a programmable logic device, such as a field programmable gate array (FPGA) 40. For the purposes of this example, device 40 is referred to as an FPGA, although it should be understood that the device may be any type of programmable logic device (e.g., an application-specific integrated circuit and / or an application-specific standard product). As shown, FPGA 40 may have input / output circuitry 42 for driving signals out of device 40 and for receiving signals from other devices via input / output pins 44. Interconnect resources 46 (e.g., global and local vertical and horizontal conductive lines and buses) may be used to route signals on device 40. Interconnect resources 46 may include fixed interconnects (conductive lines) and programmable interconnects (i.e., programmable connections between corresponding fixed interconnects). Programmable logic 48 may include combinational and sequential logic circuits. For example, programmable logic 48 may include lookup tables, registers, and multiplexers. In various embodiments, programmable logic 48 may be configured to perform custom logic functions. The programmable interconnects associated with the interconnect resources may be considered part of programmable logic 48.

[0025] A programmable logic device (e.g., FPGA 40) may include a programmable element 50 having programmable logic 48. For example, as discussed above, a designer (e.g., a customer) may program (e.g., configure) the programmable logic 48 to perform one or more desired functions. As an example, some programmable logic devices may be programmed by configuring their programmable elements 50 using a mask programming arrangement, which is performed during semiconductor fabrication. Other programmable logic devices may be configured after semiconductor fabrication operations have been completed, for example, by programming their programmable elements 50 using electrical programming or laser programming. In general, the programmable elements 50 may be based on any suitable programmable technology, such as fuses, antifuses, electrically programmable read-only memory technology, random access memory cells, mask-programmable elements, and the like.

[0026] Many programmable logic devices are electrically programmable. Using an electrically programmed arrangement, programmable element 50 may be formed from one or more memory cells. For example, during programming, configuration data is loaded into the memory cells using pins 44 and input / output circuitry 42. In one embodiment, the memory cells may be implemented as random access memory (RAM) cells. The use of RAM-based memory cells described herein is intended to be merely an example. Furthermore, because these RAM cells are loaded with configuration data during programming, they are sometimes referred to as configuration RAM cells (CRAM). Each of these memory cells may provide a corresponding static control output signal that controls the state of an associated logic component in programmable logic 48. For example, in some embodiments, the output signal may be applied to the gate of a metal oxide semiconductor (MOS) transistor in programmable logic 48.

[0027] In some embodiments, the RNN 26 of the integrated circuit device 12 may utilize activation functions, such as a sigmoid function and / or a hyperbolic tangent function, to implement machine learning techniques. Furthermore, in some embodiments, to increase efficiency, reduce footprint (e.g., resources), and / or reduce latency associated with implementing and / or computing the activation function, the integrated circuit device 12 may implement and / or compute the activation function according to the approximations described herein.

[0028] Now go to Figure 3 , as defined by the sigmoid function (σ(x)):

[0029] ,

[0030] This is shown in graph 100 , where vertical axis 102 represents the value of σ(x), and horizontal axis 104 represents the value of the input x to the sigmoid function. As shown by curve 112 (which plots the value of the sigmoid function for each value of input x), the approximation of 1 as the value of σ(x) becomes increasingly accurate as the value of x increases. Thus, positive input cutoff 106 may mark the value of x at which curve 112 approaches saturation at 1 (as represented by asymptote 108). Similarly, negative input cutoff 110 may mark the value of x at which curve 112 approaches saturation at 0. Accordingly, the value of curve 112 may be approximately 1 within positive saturation interval 107, and the value of the curve may be approximately 0 within negative saturation interval 109.

[0031] The location of positive input cutoff 106 (e.g., the boundary of positive saturation interval 107) and the location of negative input cutoff 110 (e.g., the boundary of negative saturation interval 109) may affect the accuracy of the approximation of curve 112. For example, in a single-precision floating-point number representation (e.g., a number representation with a sign bit, 8 exponent bits, and 23 mantissa bits), approximating the value of curve 112 to 1 in positive saturation interval 107 with a positive input cutoff 106 of 15.24 may be accurate to within 2 units in the last place (ULPs) (e.g., the two least significant bits). On the other hand, in a half-precision floating-point number representation (e.g., a number representation with a sign bit, 5 exponent bits, and 10 mantissa bits), a positive input cutoff 106 of 6.23 may be appropriate to approximate the value of curve 112 to the same accuracy (2 ULPs). Accordingly, as will be described in further detail below, any suitable boundaries for the positive saturation interval 107 and the negative saturation interval 109 may be implemented according to suitable accuracy boundaries (eg, 2ULP), number representation formats, etc., used to approximate the sigmoid function.

[0032] Furthermore, the values ​​of the curve 112 along the remaining intervals of the graph 100 (eg, the negative interval 114 and the positive interval 116 ) may be determined via the first identity:

[0033] .

[0034] Therefore, the value of the curve 112 in the positive interval 116 can be approximated based on the value of the curve 112 calculated in the negative interval 114. In order to calculate (e.g., approximate) the value of the curve 112 in the negative interval 114, the negative interval 114 can be divided into a plurality of subintervals 152 of uniform or non-uniform size (e.g., 256, 512, etc.), such as Figure 4 100 ).

[0035] like Figure 4As shown, the piecewise polynomial approximation function (P) (which may include a different polynomial function (eg, P) mapped to each subinterval 152) i 、P i+1 )) can be used to approximate the value of curve 112 over each of subintervals 152. Thus, by combining each of the polynomial functions into a piecewise polynomial approximation function, an approximation of curve 112 along the entire negative interval 114 can be determined.

[0036] The size of each of subintervals 152 may determine the accuracy of the approximation of the sigmoid function. Smaller subintervals 152 may produce a more accurate approximation of curve 112. However, dividing negative interval 114 into smaller subintervals 152 may produce a greater number of subintervals 152 used to cover negative interval 114, and because each polynomial mapped to subintervals 152 may include a set of coefficients, increasing the number of subintervals 152 may increase the number of coefficients used to approximate curve 112. Therefore, the number of subintervals 152 may determine the number of polynomial coefficients to be stored in memory, which may affect the resources (e.g., space occupied) used to implement the approximation. Accordingly, a trade-off may exist between the accuracy of the approximation and the resources used to implement the approximation.

[0037] The degree of the piecewise polynomial can also affect the accuracy of the approximation of the sigmoid function. A higher-degree polynomial can produce a more accurate approximation of the sigmoid function than a lower-degree polynomial. However, a higher-degree polynomial can utilize additional coefficients when compared to a lower-degree polynomial. Furthermore, additional hardware can be used to evaluate higher-degree polynomials, as will be discussed. Therefore, the degree of the polynomial used to approximate the sigmoid function in negative interval 114 can affect both the accuracy of the approximation and the resources used to implement the approximation.

[0038] In some embodiments, as described below, a quadratic polynomial function may be used to approximate the sigmoid function on each subinterval 152 of the negative interval 114. In other embodiments, higher or lower degree polynomial functions may be used in the approximation of the sigmoid function. Thus, any suitable degree polynomial function and any suitable number of subintervals 152 may be used to approximate the sigmoid function within a set accuracy bound (e.g., within 2 ULPs). In any case, each polynomial function included in the piecewise polynomial function may have the same degree, and therefore, each polynomial function included in the piecewise polynomial function may utilize the same number of coefficients.

[0039] Furthermore, in some embodiments, the values ​​of the sigmoid function in the negative interval 114 and the positive interval 116 may be determined based on a second identity:

[0040] .

[0041] Therefore, the values ​​of the curve 112 in the negative interval 114 can be approximated based on the values ​​of the curve 112 calculated in the positive interval 116. For example, the values ​​of the curve 112 can be calculated in the positive interval 116 according to the piecewise polynomial function, as described above with reference to the negative interval 114, and the negative interval 114 can be approximated based on the calculated values ​​of the curve 112 in the positive interval 116 and the second identity. However, in such embodiments, because the range of the curve 112 in the positive interval 116 is different from that in the positive interval 116 (e.g., the positive interval 116 for x=(0, 16) is 2 -1 to about 2 0 ), the floating point representation sigmoid function can have a large dynamic range on the negative interval 114 (e.g., about 2 for the negative interval 114 of x=(-16, 0]). -24 to 2 -1 ), in order to avoid a loss of precision (when compared to an embodiment utilizing the first identity), a higher precision (e.g., double precision) floating point representation may be used. That is, using the second identity of the same precision to determine the value of the S-shape in the negative interval 114 may result in cancellation of values ​​and greater imprecision when compared to employing the first identity to determine the value of curve 112 in the positive interval 116, because negative interval 114 of graph 100 has a greater number of representable values ​​in the floating point representation.

[0042] Now go to Figure 5 , the sigmoid approximation circuit 200 can be used to calculate the sigmoid function according to the above approximation. Therefore, the sigmoid approximation circuit 200 can calculate the approximation of the sigmoid function in the negative interval 114, the positive interval 116, the positive saturation interval 107, and the negative saturation interval 109, and then can select the approximation of the sigmoid function corresponding to the input value x.

[0043] To calculate the approximation of the sigmoid function in the negative interval 114, the sigmoid approximation circuit 200 may implement a piecewise polynomial function, as described above. In some embodiments, the piecewise polynomial function may be a quadratic polynomial function, which may be represented as C0+C1*x+C2*x 2 .

[0044] Accordingly, sigmoid approximation circuit 200 may include coefficient tables 202A-C that may store appropriate sets of coefficients for each of the piecewise polynomial functions (e.g., C0, C1, and C2, respectively). Coefficient tables 202A-C may be indexed by the subinterval 152 (e.g., polynomial function) to which each set of coefficients is mapped. In some embodiments, for example, coefficient tables 202 may include fixed-point indices (e.g., addresses) that map each set of coefficients to a corresponding input (e.g., a value of x) representing a subinterval 152. For example, coefficient tables 202A-C may include 8-bit fixed-point addresses (e.g., 256 unique combinations) that index each set of coefficients into 256 different subintervals 152, and may include 9-bit fixed-point addresses (e.g., 512 unique combinations) that index each set of coefficients into 512 different subintervals 152. Therefore, when the negative interval 114 of the sigmoid function is divided into 512 subintervals, the coefficient table 202 may receive a 9-bit fixed-point input and may output a set of coefficients indexed by the 9-bit fixed-point input.

[0045] The barrel shifter 204 may generate fixed-point inputs that may be uniquely mapped to the subintervals 152. To do so, as shown in the illustrated embodiment, the input x is represented in single-precision floating point, and the barrel shifter 204 may convert x into a fixed-point representation. Thus, for example, in the case of 512 subintervals 152, the barrel shifter 204 may receive as a first input 206 the eight most significant bits (e.g., 1 & fracX (22:15)) of the mantissa of x (e.g., fracX (22:15)) concatenated on the left with an implicit bit (1), and may receive as a second input 208 a shift value. The subtractor 210 may output the shift value (e.g., bias value (127) + 3) as a result of the operation 130 - expX, or subtracting the exponent (expX) of x from 130. The barrel shifter 204 may then right-shift the first input 206 by the shift value received as the second input 208 to generate a 9-bit fixed-point input to the coefficient table 202. In some embodiments, the 9-bit fixed-point input may include 4 integer bits and 5 fraction bits. Accordingly, shifting the first input 206 by 130-expX may align the first input 206 to the correct decimal position in the 9-bit fixed-point format. That is, the value 127 may account for the bias inherently built into the exponent of x, and the value 3 may account for the decimal position of the 9-bit fixed-point, such that an exponent value of 130, for example, may not cause any shifting of the first input 206 because the first input is already aligned with the 9-bit fixed-point format.

[0046] Together with coefficient tables 202A-C, sigmoid approximation circuit 200 can use first and second multiply-add blocks 212A and 212B to implement the result of the piecewise polynomial approximation function. Both first and second multiply-add blocks 212A and 212B can be mapped to (e.g., adapted to) different single-precision hard floating-point digital signal processing (DSP) blocks.

[0047] In some embodiments, regardless of the sign of the input x, the first multiply-add block 212A may receive the exponent of x, the mantissa of x, and the negative sign of x as input (e.g., 1 & expX & fracX). That is, the first multiply-add block 212A may receive -x (e.g., negX) because the piecewise polynomial function can be used to calculate the value of the sigmoid function in the negative interval 114. The first multiply-add block 212A may then multiply negX by the quadratic coefficient (C2) received from the coefficient table 202C. The first multiply-add block 212A may also add the product of negX*C2 to the linear coefficient (C1) received from the coefficient table 202B to output (negX*C2+C1) to the second multiply-add block 212B.

[0048] As described above, the second multiply-add block 212B may receive (negX*C2+C1) as input and may multiply this value by negX, which may also be received as input by the second multiply-add block 212B. The second multiply-add block 212B may then add this product (e.g., negX*(negX*C2+C1)) to the zero-order coefficient (C0) received from the coefficient table 202A. Thus, the output of the second multiply-add block 212B may represent the output of the piecewise polynomial function (P) (e.g., C0+negX*(negX*C2+C1)), which may be rewritten as C2*negX. 2 Accordingly, based on the value of input x and the coefficients indexed by the value of x in coefficient tables 202A-C, first and second multiply-add blocks 212A, 212B are operable to compute an approximation of the sigmoid function in the negative interval 114 based on an appropriate piecewise polynomial function.

[0049] Furthermore, as previously discussed, to determine an approximation of the sigmoid function in the positive interval 116, the first identity may be used. Accordingly, the sigmoid approximation circuit 200 may include a subtractor 214 (e.g., a single-precision floating-point subtractor mapped to a DSP block), which may receive the output of the piecewise polynomial function (P) from the second multiply-add block 212B and may subtract P from 1 to generate an approximation of the sigmoid function in the positive interval 116.

[0050] The sigmoid approximation circuit 200 may also include a multiplexer (mux) 216 configured to receive values ​​of P, 1-P, 1, and 0 (e.g., approximations of the sigmoid function in the negative interval 114, the positive interval 116, the positive saturation interval 107, and the negative saturation interval 109, respectively). That is, the sigmoid approximation circuit 200 may generate an approximation of the value of the sigmoid function for each interval in which the input x resides, and may select the appropriate approximation at mux 216 based on the actual interval in which the input x resides. Thus, mux 216 may receive a select signal to select the approximation of the output of the sigmoid function for that value of x. In some embodiments, the select signal may include the sign of the input x (e.g., signX) and information related to the exponent of x (expX). In such embodiments, the select signal may indicate whether the input x is positive or negative and whether the input x is in the positive saturation interval 107 or the negative saturation interval 109. For example, in an embodiment having a positive input cutoff 106 of 16 and a negative input cutoff 110 of -16, when the absolute value of the mantissa of x is greater than or equal to 1 and less than 2, the selection signal may include a value indicating whether expX is greater than or equal to 4 (e.g., expX>=4), which may indicate whether the value of x is greater than or equal to 16 (e.g., 2 4 =16). Therefore, a selection signal value of 10 may represent a negative value of x having an exponent less than 4 (e.g., a value of x in the negative interval 114), a selection signal value of 00 may represent a positive value of x having an exponent less than 4 (e.g., a value of x in the positive interval 116), a selection signal value of 01 may represent a positive value of x having an exponent greater than or equal to 4 (e.g., a value of x in the positive saturation interval), and a selection signal value of 11 may represent a negative value of x having an exponent greater than or equal to 4 (e.g., a value of x in the negative saturation region). Accordingly, based on the sign of the input x and the exponent of x, the sigmoid approximation circuit 200 may output an appropriate approximation of the sigmoid function for the input x via the mux 216.

[0051] As discussed above, an approximation of the sigmoid function implemented using a half-precision floating-point format (e.g., a number representation format including a sign bit, 5 exponent bits, and 10 mantissa bits) can have different values ​​of x to achieve the same accuracy bound (e.g., 2 ULP) as an approximation of the sigmoid function implemented using a single-precision floating-point format. Accordingly, Figure 6A second graph 250 illustrates a sigmoid function implemented using a half-precision floating-point format. In the half-precision floating-point format, an approximation of the value of the sigmoid function saturating to 1 (e.g., asymptote 108) can be accurate to within 1 ULP for input values ​​of x greater than or equal to 6.98 (as represented by positive input cutoff 106), and an approximation of the value of the sigmoid function saturating to 0 can be accurate to within 1 ULP for input values ​​of x less than or equal to -6.98 (as represented by negative input cutoff 110). However, in some embodiments, positive input cutoff 106 and negative input cutoff 110 can be rounded to 8 and -8, respectively, to round each cutoff value (e.g., 106 and 110) to the nearest power of two, which can facilitate efficient generation of uniformly sized subintervals 152. Furthermore, in such embodiments, a second positive saturation interval 252 marks values ​​of x where x is greater than or equal to 8, and a second negative saturation interval 258 marks values ​​of x where x is less than or equal to -8.

[0052] Furthermore, similar to the approach involving the single-precision floating-point format, to approximate the values ​​of the sigmoid function over the remaining interval of x (e.g., where x is greater than -8 and less than 8), the remaining interval may be segmented into a second positive interval 254 where x is greater than 0 and less than 8 (e.g., x=(0, 8)) and a second negative interval 256 where x is greater than -8 and less than or equal to 0 (e.g., x=(-8, 0]). In some embodiments, because the sigmoid function has a larger dynamic range in the second negative interval 256 than in the second positive interval 254, an approximation of the sigmoid function may be calculated using a piecewise polynomial function calculated across the subintervals 152 of the second negative interval 256, and based on the approximation of the sigmoid function in the second negative interval 256, an approximation of the sigmoid function in the second positive interval 254 may be determined based on the first identity. Furthermore, in some embodiments, when a half-precision implementation can achieve the same accuracy bounds (e.g., within 1 ULP) as a single-precision implementation utilizing a less accurate approximation, the piecewise polynomial function may include a first-degree polynomial for each subinterval 152 of the curve 112.

[0053] Now go to Figure 7The approximation discussed above can be implemented according to the first half-precision sigmoid approximation circuit 300. In the illustrated embodiment, because a first-degree piecewise polynomial function can be sufficient to properly approximate the value of the sigmoid function within appropriate precision bounds, the first half-precision sigmoid approximation circuit 300 may include coefficient tables 202A-B, which may respectively include a zero-degree coefficient (C0) and a first-degree coefficient (C1) mapped to each subinterval 152. To access each indexed (e.g., mapped) coefficient, similar to the sigmoid approximation circuit 200, the first half-precision sigmoid approximation circuit 300 may include a barrel shifter 204 that can convert the input x from a floating-point representation to a fixed-point representation. More specifically, in some embodiments, the barrel shifter 204 may convert the input x into a 6-bit fixed-point value having 3 integer bits and 3 fraction bits. In such embodiments, the 6-bit fixed-point value can be mapped to 64 different subintervals 152; however, in other embodiments, different precision values ​​and numbers of subintervals 152 may be used. Furthermore, to convert the input x into a proper fixed-point value, the barrel shifter 204 may receive the top 5 bits of the mantissa (e.g., fraction) of x (fracX(9:5)) concatenated with an implicit bit (1) (e.g., 1&fracX(9:5)), and may receive a shift value that may be obtained by subtracting the value of the exponent (expX) of x from 2 (e.g., 2-expX) and may be used to right shift 1&fracX to the correct decimal alignment in the 6-bit fixed-point format.

[0054] Furthermore, upon receiving a 6-bit fixed-point value from the barrel shifter 204 that indexes one or more coefficient tables 202, the coefficient tables 202A-B may output a zero-order coefficient (C0) and a first-order coefficient (C1), respectively, to the multiply-add block 212. Because the multiply-add block 212 and / or additional hardware components in the first half-precision sigmoid approximation circuit 300 may operate using single-precision floating-point representation, the coefficient tables 202A-B may store each coefficient (e.g., C0 and C1, respectively) in single-precision floating-point format. Furthermore, because the multiply-add block 212 may also receive an input x as input to generate the polynomial function C0+C1*x, the first half-precision sigmoid approximation circuit 300 may convert the input x from a half-precision floating-point number to a single-precision floating-point number before inputting x to the multiply-add block 212. Accordingly, the first half-precision sigmoid approximation circuit 300 may include a half-precision conversion block 302, which may include circuitry and / or soft logic to convert the input x from a half-precision floating-point number to a single-precision floating-point number. To do so, the half-precision conversion block 302 may update the exponent of x (expX) by the new bias value (e.g., 127-15) and may right-pad the mantissa of x with 13 zeros. Additionally, during the conversion operation, the conversion block 302 may force the sign of x to be negative (e.g., 1).

[0055] Accordingly, the multiply-add block 212 may receive a single-precision value of input x and may multiply the single-precision value of input x by a linear coefficient (C1) to generate C1*x. The multiply-add block may then add C0 to this output to generate an output C1*x+C0, which is the result of a linear polynomial that may approximate the sigmoid function in the second negative interval 256.

[0056] The first half-precision sigmoid approximation circuit 300 can then approximate the sigmoid function in the second positive interval 254 based on the first identity, where the value of σ(-1) is determined by the output C1*x+C0. In the illustrated embodiment, for example, the subtractor 214 can receive the output C1*x+C0 and subtract it from the single-precision value 1 to generate an approximation of the sigmoid function in the second positive interval 254.

[0057] In some embodiments, similar to sigmoid approximation circuit 200, first half-precision sigmoid approximation circuit 300 may include mux 216, which is configured to receive an approximation of the sigmoid function in second positive interval 254 (e.g., 1-(C1*x+C0)), an approximation of the sigmoid function in second negative interval 256 (e.g., C1*x+C0), an approximation of the sigmoid function in second positive saturation interval 252 (e.g., 1), and an approximation of the sigmoid function in second negative saturation interval 258 (e.g., 0). Furthermore, first half-precision sigmoid approximation circuit 300 may approximate the sigmoid function for input x by selecting one of the inputs listed above to mux 216 based on the sign of input x and the value of expX. To determine whether the absolute value of x is greater than or equal to 8 (e.g., to determine whether x is in second negative saturation interval 258 or second positive saturation interval 252), mux 216 may receive a signal indicating whether expX is greater than or equal to 3 or whether bias expX is greater than or equal to 15+3. Thus, a value of x less than or equal to −8 may generate a selection signal of 11, which may select an approximation of the sigmoid signal (e.g., 0) mapped to the second negative saturation interval 258 from mux 216, a value of x greater than −8 and less than 0 may generate a selection signal of 10, which may select an approximation of the sigmoid function mapped to the second negative interval 256 from mux 216, a value of x greater than or equal to 0 and less than 8 may generate a selection signal of 00, which may select an approximation of the sigmoid function mapped to the second positive interval 254 from mux 216, and a value of x greater than or equal to 8 may generate a selection signal of 01, which may select an approximation of the sigmoid function mapped to the second positive saturation interval 252 from mux 216.

[0058] Because each of the outputs of mux 216 can represent a single-precision floating-point value, first half-precision sigmoid approximation circuit 300 may include a single-precision conversion block 304, which may include circuitry and / or soft logic to project the output of mux 216 (the selected approximation of the sigmoid function) from single-precision floating-point to half-precision floating-point. To do so, single-precision conversion block 304 may truncate and / or round the mantissa of the output of mux 216 from 23 bits to 10 bits. Furthermore, to re-bias the exponent of the output of mux 216, single-precision conversion block 304 may subtract a value (e.g., 127-15) from the exponent of the output of mux 216. In some embodiments, single-precision conversion block 304 may also check for exponent overflow and / or underflow and adjust the mantissa of the output of mux 216 accordingly. Thus, the output of first half-precision sigmoid approximation circuit 300 may be represented in half-precision floating-point format.

[0059] In some embodiments, the operations involved in determining the approximation of the sigmoid function in the second positive interval 254 may form a critical path in the first half-precision sigmoid approximation circuit 300. Thus, the operations involving the barrel shifter 204, indexing into the one or more coefficient tables 202, performing multiplication and addition operations at the multiply-add block 212, performing subtraction at the subtractor 214, selecting the output from the mux 216, and projecting the output of the mux 216 to half precision may determine the latency of the architecture of the first half-precision sigmoid approximation circuit 300. Furthermore, the latency contributed by the DSP blocks (e.g., the multiply-add block 212 and the subtractor 214) may have the greatest impact on the overall latency of the sigmoid approximation circuit 200 architecture. Accordingly, in some embodiments, different architectures for approximating the sigmoid function in half-precision format may improve the overall latency contributed by the DSP blocks.

[0060] Now go to Figure 8When compared to the first half-precision sigmoid approximation circuit 300, the second half-precision sigmoid approximation circuit 350 may have a shorter critical path. As shown in the illustrated embodiment, the second half-precision sigmoid approximation circuit 350 may include a first data path 352 that may generate an approximation of the sigmoid function in the second negative interval 256, which may be similar in architecture to the first half-precision sigmoid approximation circuit 300. The second half-precision sigmoid approximation circuit 350 may also include a second data path 354 that may generate an approximation of the sigmoid function in the second positive interval 254. The second data path 354 may be independent of the calculations and / or operations involving the first data path 352. Therefore, because the approximation of the sigmoid function in the second positive interval 254 (e.g., the second data path 354) may not depend on the approximation of the sigmoid function in the second negative interval 256 (e.g., the first data path 352), both the approximation of the sigmoid function in the second positive interval 254 and the approximation of the sigmoid function in the second negative interval 256 may be calculated in parallel. Thus, while the subtractor 214 of the first half-precision sigmoid approximation circuit 300 may calculate an approximation of the sigmoid function in the second positive interval 254 after the approximation of the sigmoid function in the second negative interval 256 is calculated, the second half-precision sigmoid approximation circuit 350 may calculate both approximations substantially simultaneously, thereby reducing the latency of the architecture of the second half-precision sigmoid approximation circuit 350 as compared to the first half-precision sigmoid approximation circuit 300.

[0061] However, because both the first data path 352 and the second data path 354 of the second half-precision sigmoid approximation circuit 350 may each include the coefficient tables 202A-B, the second half-precision sigmoid approximation circuit 350 may utilize additional memory (e.g., resources) to store the coefficient tables 202A-B when compared to the first half-precision sigmoid approximation circuit 300. Accordingly, Figure 9 The illustrated third half-precision sigmoid approximation circuit 400 can improve upon the architecture of the second half-precision sigmoid approximation circuit 350 by reducing the resources used to approximate the sigmoid function. In some embodiments, because the approximation of the sigmoid function in the second positive interval 256 can be easier to compute than the approximation of the sigmoid function in the second negative interval 256, the piecewise polynomial function used in the second data path 354 can include fewer subintervals 152 than the piecewise polynomial function of the first data path 352. For example, in some embodiments, because the dynamic range of the sigmoid function in the second positive interval 254 is smaller than the dynamic range of the sigmoid function in the second negative interval 256, as discussed above, the piecewise polynomial function of the second data path 354 can be implemented in 12 non-uniform subintervals 152 to meet a certain accuracy bound (e.g., 1 ULP). Consequently, one or more coefficient tables of the second data path 354 can include fewer coefficients, which can utilize less memory.

[0062] Because the second data path 354 may include fewer coefficients from the coefficient tables 202A-B, the decoding lookup table (LUT) 402 may map the output of the barrel shifter to an index of a coefficient in the coefficient tables 202A-B. In some embodiments, for example, the approximation of the sigmoid function in the second negative interval 256 may utilize 64 coefficients or 64 subintervals 152 from each coefficient table 202A-B of the first data path 352, while the approximation of the sigmoid function in the second positive interval 254 may utilize 12 coefficients or 12 subintervals from each coefficient table 202A-B of the second data path 354. In such embodiments, the coefficient tables 202A-B of the first data path 352 may receive a 6-bit index (e.g., 64 possible combinations) to uniquely map each of the 64 coefficients to a corresponding subinterval 152 in the second negative interval 256, while a 4-bit index (e.g., 16 possible combinations) may appropriately map each of the 12 coefficients of the coefficient tables 202A-B in the second data path 354 to a corresponding subinterval 152 in the second positive interval 254. Thus, as Figure 9 As shown, the third half-precision sigmoid approximation circuit 400 may include a decoding LUT 402 that may receive the same 6-bit fixed-point output from the barrel shifter 204 that the first data path 352 receives for the input x, and the decoding LUT 402 may map the 6-bit fixed-point value to a 4-bit fixed-point value that may index the appropriate subinterval 152 in the second positive interval 254.

[0063] Furthermore, while the coefficient tables 202A-B may store single-precision coefficients, in some embodiments, the coefficients may be stored in a lower precision format (e.g., half precision). In such embodiments, the coefficient tables 202 may occupy less space and / or use fewer memory resources because each stored coefficient may occupy fewer bits. In the illustrated embodiment, for example, the coefficients may occupy 22 bits, as compared to the 32 bits occupied by the single-precision coefficients, because the trailing (e.g., least significant) 10 bits of the mantissa from the single-precision coefficients may be removed to generate a smaller 22-bit coefficient. However, in order for hardware implemented to manipulate single-precision values ​​(e.g., the multiply-add block 212) to receive and / or operate on the coefficients, the third half-precision sigmoid approximation circuit 400 may convert the coefficients output from the coefficient tables 202A-B into a single-precision format. For example, Figure 9 As shown in the example of , the third half-precision sigmoid approximation circuit 400 may concatenate 10 zeros to the end of the mantissa of the coefficient output by the coefficient table 202 to generate a single-precision coefficient that the multiply-add block 212 may appropriately receive as input.

[0064] Additional architectures can generate approximations of the sigmoid function in floating point precision containing 11 bits (FP11) or less (e.g., low precision). Figure 10Low-precision sigmoid approximation circuitry 450 is provided, which can generate an approximation of a sigmoid function with low precision. Low-precision sigmoid approximation circuitry 450 can include a sigmoid table 452 (e.g., a LUT) that can map input values ​​to outputs according to the sigmoid function. In some embodiments, for example, sigmoid table 452 can contain pre-computed approximations of the sigmoid function for a set of input values ​​and can output an appropriate pre-computed approximation of the sigmoid function according to the received input value. Furthermore, sigmoid table 452 can receive an input comprising a certain number of bits (which may include a sign bit) and can generate an output having at least one bit less than the certain number of bits (e.g., an output without a sign bit). For example, in the illustrated embodiment, sigmoid table 452 can receive a signed 11-bit input and can output an unsigned 10-bit output. Low-precision sigmoid approximation circuitry 450 can then concatenate zero (e.g., a positive sign bit) to the 10-bit output because the value of the sigmoid function is always greater than or equal to zero. Therefore, the sigmoid table 452 may store sigmoid approximation outputs that occupy fewer bits than sigmoid approximation outputs that include a sign bit.

[0065] In addition to the sigmoid function, the hyperbolic tangent function (tanh) is often used as an activation function in machine learning. Accordingly, Figure 11 A third graph 500 shows the tanh function as defined by:

[0066] ,

[0067] Here, second vertical axis 504 represents values ​​of tanh(x), and horizontal axis 104 represents values ​​of x. As shown in second curve 502 (which plots the value of the hyperbolic tangent function for each value of input x), as the value of x increases, the approximation of tanh(x) using 1 becomes increasingly accurate. Therefore, positive input cutoff 106 may mark the value of x at which second curve 502 approaches saturation at 1 (as represented by asymptote 108). Similarly, negative input cutoff 110 may mark the value of x at which second curve 502 approaches saturation at -1 (as represented by negative asymptote 514). Accordingly, the value of second curve 502 may approximate 1 within third positive saturation interval 506, and the value of the curve may approximate -1 within third negative saturation interval 508.

[0068] As discussed with respect to the sigmoid function, the location of the positive input cutoff 106 (e.g., the boundary of the third positive saturation interval 506) and the location of the negative input cutoff 110 (e.g., the boundary of the third negative saturation interval 508) can affect the accuracy of the approximation of the second curve 502. Accordingly, any suitable boundaries for the third positive saturation interval 506 and the third negative saturation interval 508 can be implemented in accordance with suitable accuracy boundaries (e.g., 2 ULP), number representation formats, etc., used to approximate the sigmoid function.

[0069] Furthermore, because the hyperbolic tangent is an odd function (eg, symmetric about the origin), the values ​​of the second curve 502 along the remaining intervals of the third graph 500 (eg, the third negative interval 512 and the third positive interval 510) can be determined via the third identity:

[0070] .

[0071] Thus, the values ​​of the second curve 502 in the third negative interval 512 can be approximated based on the values ​​of the second curve 502 calculated in the third positive interval 510. To calculate (e.g., approximate) the values ​​of the second curve 502 in the third positive interval 510, an odd, fifth-order piecewise polynomial function of the hyperbolic tangent function can approximate the third positive interval 510 across a plurality (e.g., 256, 512, etc.) of subintervals 152. Accordingly, each polynomial in the odd, fifth-order piecewise polynomial function can take the following form:

[0072] ,

[0073] and may have coefficients (eg, C1 , C3 , and C5 ) mapped to corresponding subintervals 152 .

[0074] The size of each of subintervals 152 can determine the accuracy of the approximation of the hyperbolic tangent function. Smaller subintervals 152 can produce a more accurate approximation of second curve 502. However, dividing third positive interval 510 into smaller subintervals 152 can produce a greater number of subintervals 152 used to cover third positive interval 510, and because each polynomial mapped to a subinterval 152 can include a set of coefficients, increasing the number of subintervals 152 can increase the number of coefficients used to approximate second curve 502. Therefore, the number of subintervals 152 can determine the number of polynomial coefficients to be stored in memory, which can affect the resources (e.g., space occupied) used to implement the approximation.

[0075] Now go to Figure 12 , the hyperbolic tangent approximation circuit 550 can be used to calculate the hyperbolic tangent function according to the above approximation. Therefore, the hyperbolic tangent approximation circuit 550 can calculate the approximation of the hyperbolic tangent in the third positive interval 510 and the third positive saturation interval 506, and then can select the approximation of the hyperbolic tangent function corresponding to the input value x.

[0076] To compute an approximation of the hyperbolic tangent function in the third positive interval 510 , the hyperbolic tangent approximation circuit 550 may implement an odd, fifth-degree piecewise polynomial function, as described above.

[0077] Accordingly, the hyperbolic tangent approximation circuit 550 may include coefficient tables 202A-C that may store appropriate sets of coefficients for each of the odd and quintic piecewise polynomial functions (e.g., C1, C3, and C5, respectively). The coefficient tables 202A-C may be indexed by the subinterval 152 (e.g., polynomial function) to which each set of coefficients is mapped. In some embodiments, for example, the coefficient tables 202A-C may include fixed-point indices (e.g., addresses) that map each set of coefficients to a corresponding input (e.g., a value of x) representing a subinterval 152. For example, the coefficient tables 202A-C may include 8-bit fixed-point addresses (e.g., 256 unique combinations) that index each set of coefficients to 256 different subintervals 152.

[0078] The barrel shifter 204 may generate fixed-point inputs that may be uniquely mapped to the subintervals 152. To do so, as shown in the illustrated embodiment, because the input x is represented using single-precision floating point, the barrel shifter 204 may convert x into a fixed-point representation. Thus, for example, in the case of 256 subintervals 152, the barrel shifter 204 may receive as a first input 206 the eight most significant bits (e.g., 1 & fracX (22:15)) of the mantissa of x (e.g., fracX (22:15)) concatenated with the implicit bit (1), and may receive as a second input 208 a shift value. The subtractor 210 may output the shift value (e.g., bias value (127) + 3) as a result of the operation 129 - expX or subtracting the exponent (expX) of x from 130. The barrel shifter 204 may then right-shift the first input 206 by the shift value received as the second input 208 to generate an 8-bit fixed-point input to the coefficient table 202. In such embodiments, the 8-bit fixed-point input may include 3 integer bits and 5 fraction bits. Accordingly, shifting the first input 206 by 129-expX may align the first input 206 to the correct decimal position in the 8-bit fixed-point format. That is, the value 127 may account for the bias inherently built into the exponent of x, and the value 2 may account for the decimal position of the 8-bit fixed-point, such that an exponent value of 129, for example, may not cause any shifting of the first input 206 because the first input is already aligned with the 8-bit fixed-point format.

[0079] Together with coefficient tables 202A-C, hyperbolic tangent approximation circuit 550 can use first multiplication block 552A, first multiplication-add block 212A, second multiplication-add block 212B, and second multiplication block 552B to generate the results of the polynomial function in odd and fifth-degree piecewise polynomial functions. First multiplication block 552A, first multiplication-add block 212A, second multiplication-add block 212B, and second multiplication block 552B can be mapped to (e.g., adapted to) different single-precision hard floating-point DSP blocks, respectively.

[0080] In some embodiments, the first multiplication block 552A may multiply the input x by itself to generate x2 The first multiply-add block may then add the appropriate C5 coefficient from coefficient table 202C to x 2 The terms are multiplied, and the result (e.g. C5*x 2 ) is added to the appropriate C3 coefficient received from coefficient table 202B. Thus, the first multiply-add block 212A may output C5*x to the second multiply-add block 212B. 2 + C3. The second multiply-add block 212B can then convert the output of the first multiply-add block 212A (C5*x 2 +C3) and x generated by the first multiplication block 552A 2 terms, and the result of this multiplication (e.g. x 2 *(C5*x 2 + C3)) is added to the appropriate C1 coefficient received from the first coefficient table 202A. Thus, the second multiply-add block 212B may output C1+x 2 *(C5*x 2 + C3). The second multiplication block 552B may receive the output (C1 + x) from the second multiply-add block 212B. 2 *(C5*x 2 +C3), and can be multiplied with the input x to generate the output x*(C1+x 2 *(C5*x 2 + C3)), which may represent the value of the polynomial function in the odd, fifth-degree polynomial function used to approximate the hyperbolic tangent function in the subinterval 152 containing the input x.

[0081] The hyperbolic tangent approximation circuit 550 may also include a mux 216 configured to receive an approximation of the hyperbolic tangent function in the third positive interval 510 (e.g., the output of the second multiplication block 552B) and an approximation of the hyperbolic tangent function in the third positive saturation interval 506 (e.g., 1), and may select between the approximations based on the interval in which the input value of x resides. Thus, the mux 216 may receive a selection signal to select the appropriate approximation of the output of the hyperbolic tangent function for the value of x. In some embodiments, the selection signal may include information related to the exponent of x (expX). In such embodiments, the selection signal may indicate whether the input x is in the third positive saturation interval 506 or the third positive interval 510. For example, in an embodiment having a positive input cutoff 106 of 8 and a negative input cutoff 110 of -8, the selection signal may include a value indicating whether 130 is greater than or equal to the biased value of expX (e.g., 130>=expX) or a value indicating whether 3 is greater than or equal to the unbiased value of expX, which may indicate whether the value of x is greater than or equal to 8 (e.g., 2 3= 8). Thus, a select signal value of 1 may represent a value of x in the third positive saturation interval 506, and a select signal value of 0 may represent a value of x in the third positive interval 510. Accordingly, based on the exponent of x, the hyperbolic tangent approximation circuit 550 may output, via the mux 216, an appropriate positive approximation of the hyperbolic tangent function for the input x.

[0082] Because the hyperbolic tangent approximation circuit 550 can output a positive approximation of the hyperbolic tangent function (corresponding to the value of x in the third positive interval 510 or the third positive saturation interval 506), the hyperbolic tangent approximation circuit 550 can also include logic and / or circuitry to determine whether x is in the third negative interval 512 or the third negative saturation interval 508. When the input x is in the third negative interval 512 or the third negative saturation interval 508, the hyperbolic tangent approximation circuit 550 can generate an appropriate approximation of the hyperbolic tangent function by taking the negative of the output of mux 216 according to the third identity.

[0083] Although the hyperbolic tangent approximation circuit 550 and the sigmoid approximation circuit 200 may be implemented separately from each other, in some embodiments, such as Figure 13 As shown, the fused activation function approximation circuit 600 can combine them into a single architecture. Because the components (e.g., multiply-add blocks 212A-212B, mux 216, coefficient table 202, barrel shifter 204, etc.) and the general data flow through the hyperbolic tangent approximation circuit 550 and the sigmoid approximation circuit 200 can overlap (e.g., match), the fused activation function approximation circuit 600 can reduce redundant resources involved in implementing the hyperbolic tangent approximation circuit 550 and the sigmoid approximation circuit 200, respectively.

[0084] Both the hyperbolic tangent approximation circuit 550 and the sigmoid approximation circuit 200 include linked pairs of multiply-add blocks 212A-212B configured to evaluate polynomials. In the case of the sigmoid function, the polynomial being evaluated can be expressed as (C0+x*(C1+C2*x)), and in the case of the hyperbolic tangent, the polynomial being evaluated can be expressed as (x*(C1+x 2 *(C3+C5*x 2 ))). Replace x with a variable 2 =y, the evaluated polynomial for the hyperbolic tangent can be rewritten as (x*(C1+y*(C3+C5*y))), and through this expression, the right-hand side of the product (e.g., (C1+y*(C3+C5*y))) matches the structure of the evaluated polynomial for the sigmoid function. Accordingly, the fused activation function approximation circuit 600 can implement the fused data path according to the following function:

[0085]

[0086] Here, the function F(q,z) represents a sigmoid function or a hyperbolic tangent function. If the function F(q,z) is a hyperbolic tangent function, then q = x, z = x 2 , c L =c 1Tanh , c M =c 3Tanh , and c R =c 5Tanh , where c 1Tanh 、c 3Tanh and c 5Tanh are the coefficients of the hyperbolic tangent polynomial, and if the function F(q,z) is sigmoid, then q = 1, z = x, c L =c 0Sigmoid , c M =c 1Sigmoid , and c R =c 2Sigmoid , where c 0Sigmoid 、c 1Sigmoid and c 2Sigmoid are the coefficients of the sigmoid polynomial.

[0087] Thus, in some embodiments, the fused activation function approximation circuit 600 may include a first mux 216A configured to output an appropriate value to generate z. The first mux 216A may receive 1 and an input x as inputs, and may select between these inputs based on a function selection signal. For example, in the illustrated embodiment, a selection signal of 0 corresponds to a hyperbolic tangent function, and a selection signal of 1 corresponds to a sigmoid function. Accordingly, a selection signal of 0 (e.g., hyperbolic tangent) may select x as the output of the first mux 216A, and a selection signal of 1 (e.g., sigmoid) may select 1 as the output of the first mux 216A. As discussed above, when the function F(q,z) is a hyperbolic tangent, z=x 2 , and when the function F(q,z) is sigmoid, z=x. Accordingly, the first multiplication block 552A may receive the output of the first mux 216A and may multiply the output with the input x to generate x or x based on the implemented function. 2 (e.g. z).

[0088] The output of the first multiplication block 552A (eg, z) may then be combined with a set of coefficients (eg, c R and c M ) are fed together into the first multiply-add block 212A. In some embodiments, since c R Can represent c 5Tanh or c 2Sigmoid, so the first coefficient table 202A may store coefficients mapped to each subinterval 152 of both the hyperbolic tangent and sigmoid functions. For example, to approximate the hyperbolic tangent function using 256 subintervals and to approximate the sigmoid function using 256 subintervals 152, the first coefficient table 202A may contain c 5Tanh 256 values ​​to convert a c 5Tanh The value is mapped to each hyperbolic tangent subinterval 152 and can contain 256 c 2Sigmoid Value to be a c 2Sigmoid A value is mapped to each sigmoid subinterval. Thus, the first coefficient table 202A may include 512 total entries. To this end, in the case of 256 coefficients for each function, in order to index the coefficients, the barrel shifter 204 may output an 8-bit index (e.g., 256 possible combinations) to select the appropriate input based on the subinterval of the input x. Thus, similar to the discussion of the operation of the barrel shifter 204 in the above example, the barrel shifter may receive a plurality of most significant bits (e.g., 1 & fracX (22:15)) concatenated with an implicit bit (1) at a first input 206, and may receive a shift value (e.g., 130-expX) at a second input 208. The barrel shifter 204 may then appropriately shift the value received at the first input 206 to align it with a fixed-point format (e.g., a 9-bit fixed-point format). The additional mux 216G may then select the bottom 8 bits (of the 9 fixed-point bits) based on the function select signal if the function select signal corresponds to hyperbolic tangent (0), and may select the top 8 bits (of the 9 fixed-point bits) if the function select signal corresponds to sigmoid (1). Furthermore, since the first coefficient table 202A may contain coefficients for both sigmoid and hyperbolic tangent functions (e.g., 512 total entries), the fused activation function approximation circuit 600 may concatenate the function select signal with the index output by the mux 216G to index either the first half of the coefficient table 202A (which may contain coefficients corresponding to the hyperbolic tangent function) or the second half of the coefficient table 202A (which may contain coefficients corresponding to the sigmoid function) based on the function being approximated. The index output by the additional mux 216G, concatenated with the function select signal, may similarly index the first half of the coefficient table 202A that may include coefficients corresponding to the hyperbolic tangent function. M (For example, c 3Tanh and c 1Sigmoid ) is indexed into the second coefficient table 202B of the value.

[0089] In other embodiments, the fused activation function approximation circuit 600 may include separate coefficient tables 202 for some or all of the coefficients. In such embodiments, for example, the first coefficient table 202A may include exclusively c 5Tanh coefficients, and an additional coefficient table (not shown) may contain c 2Sigmoidvalue. Therefore, the index output by the additional mux 216G can directly index the first coefficient table 202A and the additional coefficient table without a function selection signal juxtaposed therewith. In addition, the fused activation function approximation circuit 600 may include an additional mux 216 (not shown) to select between the coefficients output by the first coefficient table 202A and the coefficients output by the additional coefficient table based on the function selection signal as a selection signal.

[0090] In any case, the first multiply-add block 212A may output (c M +z*c R The second multiply-add block 212B may also select from the third coefficient table 202C (which may include c as described above with reference to the first coefficient table 202A and the second coefficient table 202B). 1Tanh and / or c 0Sigmoid The value of the receiving coefficient (e.g. c L ). Therefore, the second multiply-add block 212B can output c L +z(c M +z*c R ).

[0091] The third multiply-add block 212C may receive 1 or x (e.g., q) as an input from the second mux 216B. The second mux 216B may output a value of 1 or x (e.g., q) based on the function selection signal (e.g., func) received as the selection signal. Thus, a selection signal indicating a hyperbolic tangent function may cause the second mux 216B to output x, and a selection signal indicating a sigmoid function may cause the second mux 216B to output 1. Furthermore, depending on the sign (signX) of the input x and the function selection signal (func), the third multiply-add block 212C may additionally receive the output (e.g., c) of the second multiply-add block 212B from the third multiplexer 216C. L +z(c M +z*c R )) or the negative of the output of the second multiply-add block 212B (e.g., -(c L +z(c M +z*c R In some embodiments, for example, the third mux 216C may receive a selection signal based on an exclusive OR (XOR) of signX and func, or the selection signal may be decoded such that if the function is tanh and the sign of x (signX) is negative or if the function is sigmoid and signX is positive, the third mux 216C may output -(c L +z(c M +z*c R)), and if the function is tanh and signX is positive or if the function is sigmoid and signX is negative, the third mux 216C may output c L +z(c M +z*c R ). Therefore, the multiplication operation in the third multiply-add block 212C can generate F(q, z) or -F(q, z) as input to the addition operation in the third multiply-add block 212C. The addition operation can also receive 1 or 0 as input from the fourth mux 216D based on the function select signal (e.g., func) and the sign of the input x. In the illustrated embodiment, for example, the fourth mux 216D can receive a select signal from the first decoding block 602A (e.g., a LUT). The first decoding block can receive func and signX as inputs and can generate an appropriate select signal for the fourth mux 216D. In some embodiments, for example, the first decoding block 602A can select 0 as the output of the fourth mux 216D for a function select signal indicating a hyperbolic tangent (regardless of the value of signX) and for a combination of a function select signal indicating an sigmoid and a signX value of 1 (e.g., negative x). Furthermore, the first decoding block 602A may select 1 for the fourth mux 216D output for a combination of a function select signal indicating a sigmoid and a signX value of 0 (e.g., a positive x). Accordingly, the third multiply-add block 212C may output an approximation of the hyperbolic tangent in a negative interval (e.g., the third negative interval 512) or a positive interval (e.g., the third positive interval 510), or may output an approximation of the sigmoid in a negative interval (e.g., the negative interval 114) or a positive interval (e.g., the positive interval 116). More specifically, the third multiply-add block 212C may output, for example, 0+F(q,z) as an approximation of the hyperbolic tangent in a positive interval or an approximation of the sigmoid in a negative interval, 0+-F(q,z) as an approximation of the hyperbolic tangent in a negative interval, and 1-F(q,z) as an approximation of the sigmoid in a positive interval.

[0092] Furthermore, the fused activation function approximation circuit 600 may include a fifth mux 216E, which may receive as input the output of the third multiply-add block 212C, along with 1 (which may represent an approximation of the hyperbolic tangent and sigmoid in the positive saturation region), and the output from the sixth mux 216F (which may represent an approximation of the hyperbolic tangent or sigmoid in the negative saturation region). For example, the sixth mux 216F may output 0 based on a function selection signal indicating an approximation of the sigmoid, and may output -1 based on a function selection signal indicating an approximation of the hyperbolic tangent. In either case, the fifth mux 216E may select the appropriate approximation for the hyperbolic tangent or sigmoid function based on a selection signal received from the second decoding block 602B. In some embodiments, the second decoding block 602B may receive the sign of x (signX), a signal indicating whether the exponent of x is greater than or equal to 4 (expX>=4), a signal indicating whether the exponent of x is greater than or equal to 3 (expX>=3), and func. That is, the second decoding block 602B can determine the function to be approximated and the interval in which the input x for a given function is located. Therefore, the fifth mux 216E can receive a first selection signal (e.g., 0) for the hyperbolic tangent approximation in a positive or negative interval (e.g., expX < 3) or the sigmoid approximation in a positive or negative interval (e.g., expX < 4), a second selection signal (e.g., 1) for the hyperbolic tangent or sigmoid approximation in a positive saturation interval, and a third selection signal (e.g., 2) for the hyperbolic tangent or sigmoid approximation in a negative saturation region, and can output an appropriate result based on the received selection signals. Accordingly, for any value of the input x, the fused activation function approximation circuit 600 can generate and select an appropriate approximation of the hyperbolic tangent or sigmoid resulting from the input x.

[0093] The present invention also discloses a set of technical solutions, as follows:

[0094] 1. An integrated circuit device configured to receive an input and configured to approximate an activation function based at least in part on the input, comprising:

[0095] a coefficient table comprising a plurality of coefficients, wherein the plurality of coefficients of the coefficient table are indexed to indices based at least in part on the input, wherein the coefficient table is configured to output coefficients corresponding to the indices, wherein the coefficients represent coefficients in a polynomial that approximates the activation function over a plurality of subintervals in a first interval; and

[0096] A multiply-and-accumulate circuit is configured to evaluate the polynomial using a mathematical operation employing the input and the coefficients to generate a first approximate result.

[0097] 2. The integrated circuit device according to technical solution 1, comprising:

[0098] Multiplexer, configured to receive:

[0099] a first saturation value, wherein the first saturation value comprises a first asymptote value of the activation function;

[0100] a second saturation value, wherein the second saturation value comprises an additional asymptote value of the activation function; and

[0101] the first approximate result;

[0102] Wherein the multiplexer is configured to select an approximation output of the activation function among at least the first saturation value, the second saturation value, and the first approximation result based at least in part on the input.

[0103] 3. An integrated circuit as described in technical solution 1, wherein the input includes a floating-point number with a first precision, and wherein the barrel shifter is configured to receive the input and to generate the index, wherein the index is a fixed-point number with a second precision, wherein the second precision is less than or equal to the first precision.

[0104] 4. An integrated circuit as described in technical solution 1, wherein the activation function includes an S-shape.

[0105] 5. The integrated circuit according to technical solution 4, comprising:

[0106] a subtractor configured to generate a second approximation result based at least in part on the first approximation result, wherein the second approximation result corresponds to an approximation of the activation function in a second interval, wherein the second interval includes a second number of subintervals; and

[0107] A multiplexer is configured to receive the second approximation result and to select an approximation output of the activation function among at least the first approximation result and the second approximation result based at least in part on the input.

[0108] 6. An integrated circuit as described in technical solution 1, wherein the multiplication and addition circuit is configured to use the mathematical operation, the mathematical operation has a first precision, the input has a second precision, the first approximation result has the second precision, and the first precision is greater than or equal to the second precision.

[0109] 7. An integrated circuit as described in technical solution 1, wherein the activation function includes hyperbolic tangent.

[0110] 8. An integrated circuit as described in technical solution 1, wherein the coefficient table is configured to store the coefficients, the coefficients having a lower precision than the precision of the multiplication and addition circuit, and wherein the integrated circuit includes a conversion circuit configured to convert the coefficients from the lower precision to the precision.

[0111] 9. The integrated circuit according to technical solution 1, comprising:

[0112] a second coefficient table comprising a second plurality of coefficients, wherein the second plurality of coefficients of the second coefficient table are indexed to the index, wherein the second coefficient table is configured to output a second coefficient corresponding to the index, wherein the second coefficient represents a second coefficient in a second polynomial that approximates the activation function over a second number of subintervals in a second interval;

[0113] as well as

[0114] a second multiply-accumulate circuit configured to evaluate the second polynomial using the mathematical operation using the input and the second coefficient to generate a second approximation result; and

[0115] A multiplexer is configured to receive the second approximation result and to select an approximation output of the activation function among at least the first approximation result and the second approximation result based at least in part on the input.

[0116] 10. An integrated circuit as described in technical solution 1, wherein the integrated circuit is configured to implement a recursive neural network based at least in part on the first approximation result.

[0117] 11. A tangible, non-transitory, machine-readable medium comprising machine-readable instructions that, when executed by one or more processors, cause the processors to:

[0118] Receive input to the activation function;

[0119] evaluating a piecewise polynomial function to generate a first approximation result corresponding to a first input, wherein a first interval of inputs to the activation function includes the first input, wherein the piecewise polynomial function approximates the activation function over the first interval;

[0120] determining a second approximation result corresponding to a second input using an identity of the activation function and the first approximation result, wherein a second interval of inputs to the activation function includes the second input;

[0121] determining a first saturation value of the activation function corresponding to a third input, wherein a third interval of inputs to the activation function includes the third input;

[0122] determining a second saturation value of the activation function corresponding to a fourth input, wherein a fourth interval of inputs to the activation function includes the fourth input;

[0123] determining whether the first interval, the second interval, the third interval, or the fourth interval includes the input; and

[0124] In response to determining that the first interval includes the input, the first approximation result is selected as an approximation of the activation function on the input.

[0125] 12. The tangible, non-temporary machine-readable medium as described in technical solution 11 comprises machine-readable instructions, which, when executed by one or more processors, cause the processor to divide the first interval into a plurality of sub-intervals, wherein the piecewise polynomial function comprises a polynomial function for each sub-interval.

[0126] 13. A tangible, non-transitory machine-readable medium as described in technical solution 12, wherein the accuracy of the approximate value compared to the corresponding actual value of the activation function is at least partially based on the number of subintervals, the degree of the piecewise polynomial function, or a combination thereof.

[0127] 14. The tangible, non-transitory, machine-readable medium of claim 11, comprising machine-readable instructions that, when executed by one or more processors, cause the processors to:

[0128] In response to determining that the second interval includes the input, selecting the second approximation result as the approximation value;

[0129] In response to determining that the third interval includes the input, selecting the first saturated value as the approximate value; and

[0130] In response to determining that the fourth interval includes the input, the second saturated value is selected as the approximate value.

[0131] 15. A tangible, non-transitory machine-readable medium as described in technical solution 11, wherein the accuracy of the approximate value compared to the corresponding actual value of the activation function is at least partially based on the size of the first interval.

[0132] 16. A fused activation function approximation circuit configured to receive an input and configured to approximate an activation function based at least in part on the input, wherein the activation function selectively includes a first activation function or a second activation function, comprising:

[0133] a first input configured to receive a function selection signal, wherein the function selection signal indicates whether the activation function includes the first activation function or the second activation function;

[0134] a barrel shifter configured to receive the input and configured to generate a fixed-point index based at least in part on the input;

[0135] a coefficient table comprising a plurality of coefficients, wherein the plurality of coefficients of the coefficient table are indexed to the fixed-point index, wherein the coefficient table is configured to output coefficients corresponding to the fixed-point index, wherein the coefficients represent coefficients in a polynomial approximation of the activation function, wherein the polynomial approximation selectively includes a first polynomial function corresponding to the first activation function or a second polynomial function corresponding to the second activation function based at least in part on the function selection signal;

[0136] multiply-add circuitry configured to evaluate the polynomial approximation using a mathematical operation employing the input and the coefficients to generate a first approximation result; and

[0137] Multiplexer, configured to receive:

[0138] a first saturation value, wherein the first saturation value comprises a first asymptote value of the activation function;

[0139] a second saturation value, wherein the second saturation value comprises a second asymptote value of the activation function and is generated based at least in part on the function selection signal; and

[0140] the first approximate result;

[0141] Wherein the multiplexer is configured to select an approximation output of the activation function among at least the first saturation value, the second saturation value, and the first approximation result based at least in part on the input.

[0142] 17. In the fused activation function approximation circuit of technical solution 16, the first polynomial function has a first order, wherein the multiplication and addition circuit is configured to evaluate the second polynomial function based at least in part on the multiplication of the polynomial having the first order and the power of the input, and the second polynomial function has a second order.

[0143] 18. A fusion activation function approximation circuit as described in Technical Solution 16, wherein the first activation function includes an S-shape, and the second activation function includes a hyperbolic tangent.

[0144] 19. A fused activation function approximation circuit as described in technical solution 16, wherein the multiple coefficients include a first subset of coefficients corresponding to the first polynomial function and a second subset of coefficients corresponding to the second polynomial function.

[0145] 20. The fusion activation function approximation circuit according to technical solution 16, comprising:

[0146] a second coefficient table comprising a second plurality of coefficients, wherein the second plurality of coefficients of the coefficient table are indexed to the fixed-point index, wherein the second coefficient table is configured to output a second coefficient corresponding to the fixed-point index, wherein the second coefficient represents a coefficient in the second polynomial function;

[0147] and an additional multiplexer configured to select among at least the second coefficient and the coefficient based at least in part on the function select signal to generate an additional input to the multiply-add circuit, wherein the coefficient represents a coefficient in the first polynomial function, wherein the multiply-add circuit is configured to evaluate the polynomial approximation using the mathematical operation with the input and the additional input to generate the first approximation result.

[0148] Although the embodiments set forth in the present disclosure are susceptible to various modifications and alternative forms, specific embodiments have been shown by way of example in the drawings and have been described in detail herein. However, it should be understood that the present disclosure is not intended to be limited to the particular forms disclosed. The present disclosure is intended to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the present disclosure as defined by the following claims.

[0149] Embodiments of the current application

[0150] The following numbered items define embodiments of the current application.

[0151] Item A1. An integrated circuit device configured to receive an input and configured to approximate an activation function based at least in part on the input, comprising:

[0152] a coefficient table comprising a plurality of coefficients, wherein the plurality of coefficients of the coefficient table are indexed to indices based at least in part on the input, wherein the coefficient table is configured to output coefficients corresponding to the indices, wherein the coefficients represent coefficients in a polynomial that approximates the activation function over a plurality of subintervals in the first interval; and

[0153] A multiply-accumulate circuit is configured to evaluate the polynomial using a mathematical operation using the inputs and the coefficients to generate a first approximate result.

[0154] Item A2. The integrated circuit device of Item A1, comprising:

[0155] Multiplexer, configured to receive:

[0156] a first saturation value, wherein the first saturation value comprises a first asymptote value of the activation function;

[0157] a second saturation value, wherein the second saturation value comprises an additional asymptote value of the activation function; and

[0158] First approximation results;

[0159] Wherein the multiplexer is configured to select an approximation output of the activation function among at least a first saturation value, a second saturation value, and a first approximation result based at least in part on the input.

[0160] Item A3. The integrated circuit of either Item A1 or 2, the input comprising a floating point number having a first precision, and wherein the barrel shifter is configured to receive the input and is configured to generate an index, wherein the index is a fixed point number having a second precision, wherein the second precision is less than or equal to the first precision.

[0161] Item A4. The integrated circuit of any of Items A1, 2, or 3, wherein the activation function comprises a sigmoid.

[0162] Item A5. The integrated circuit of Item A4, comprising:

[0163] a subtractor configured to generate a second approximation result based at least in part on the first approximation result, wherein the second approximation result corresponds to an approximation of the activation function in a second interval, wherein the second interval includes a second number of subintervals; and

[0164] A multiplexer is configured to receive the second approximation result and to select an approximation output of the activation function among at least the first approximation result and the second approximation result based at least in part on the input.

[0165] Item A6. The integrated circuit of any of Items A1, 2, 3, or 4, wherein the multiply-add circuit is configured to use a mathematical operation having a first precision, an input having a second precision, and a first approximation result having the second precision, wherein the first precision is greater than or equal to the second precision.

[0166] Item A7. The integrated circuit of any of Items A1, 2, 3, 4, or 6, wherein the activation function comprises a hyperbolic tangent.

[0167] Item A8. The integrated circuit of any of Items A1, 2, 3, 4, 6, or 7, wherein the coefficient table is configured to store coefficients having a lower precision than a precision of the multiply-add circuit, and wherein the integrated circuit includes a conversion circuit configured to convert the coefficients from the lower precision to the precision.

[0168] Item A9. The integrated circuit of any of Items A1, 2, 3, 4, 6, 7, or 8, comprising:

[0169] a second coefficient table comprising a second plurality of coefficients, wherein the second plurality of coefficients of the second coefficient table are indexed to an index, wherein the second coefficient table is configured to output a second coefficient corresponding to the index, wherein the second coefficient represents a second coefficient in a second polynomial that approximates the activation function over a second number of subintervals in the second interval;

[0170] as well as

[0171] a second multiply-accumulate circuit configured to evaluate the second polynomial using a mathematical operation employing the input and the second coefficient to generate a second approximation result; and

[0172] A multiplexer is configured to receive the second approximation result and to select an approximation output of the activation function among at least the first approximation result and the second approximation result based at least in part on the input.

[0173] Item A10. The integrated circuit of any of Items A1, 2, 3, 4, 6, 7, 8, or 9, wherein the integrated circuit is configured to implement a recurrent neural network based at least in part on the first approximation result.

[0174] Item A11. A tangible, non-transitory, machine-readable medium containing machine-readable instructions that, when executed by one or more processors, cause the processors to:

[0175] Receive input to the activation function;

[0176] evaluating the piecewise polynomial function to generate a first approximation result corresponding to a first input, wherein a first interval of inputs to the activation function includes the first input, wherein the piecewise polynomial function approximates the activation function over the first interval;

[0177] determining a second approximation result corresponding to a second input using an identity of the activation function and the first approximation result, wherein a second interval of inputs to the activation function includes the second input;

[0178] determining a first saturation value of the activation function corresponding to a third input, wherein a third interval of inputs to the activation function includes the third input;

[0179] determining a second saturation value of the activation function corresponding to a fourth input, wherein a fourth interval of inputs to the activation function includes the fourth input;

[0180] determining whether the first interval, the second interval, the third interval, or the fourth interval includes the input; and

[0181] In response to determining that the first interval includes the input, a first approximation result is selected as an approximation of the activation function at the input.

[0182] Item A12. The tangible, non-transitory, machine-readable medium of Item A11, comprising machine-readable instructions that, when executed by one or more processors, cause the processor to divide a first interval into a plurality of subintervals, wherein the piecewise polynomial function comprises a polynomial function for each subinterval.

[0183] Item A13. The tangible, non-transitory, machine-readable medium of Item A12, wherein an accuracy of the approximation compared to a corresponding actual value of the activation function is based at least in part on a number of subintervals, a degree of the piecewise polynomial function, or a combination thereof.

[0184] Item A14. The tangible, non-transitory, machine-readable medium of any of Items A11 or 12, comprising machine-readable instructions that, when executed by one or more processors, cause the processor to:

[0185] In response to determining that the second interval includes the input, selecting a second approximation result as the approximation value;

[0186] In response to determining that the third interval includes the input, selecting the first saturated value as the approximate value; and

[0187] In response to determining that the fourth interval includes the input, the second saturated value is selected as the approximate value.

[0188] Item A15. The tangible, non-transitory machine-readable medium of any of Items A11, 12, or 14, wherein an accuracy of the approximation compared to a corresponding actual value of the activation function is based at least in part on a size of the first interval.

[0189] Item A16. A fused activation function approximation circuit configured to receive an input and configured to approximate an activation function based at least in part on the input, wherein the activation function selectively includes a first activation function or a second activation function, comprising:

[0190] a first input configured to receive a function selection signal, wherein the function selection signal indicates whether the activation function includes a first activation function or a second activation function;

[0191] a barrel shifter configured to receive an input and configured to generate a fixed-point index based at least in part on the input;

[0192] a coefficient table comprising a plurality of coefficients, wherein the plurality of coefficients of the coefficient table are indexed to fixed-point indices, wherein the coefficient table is configured to output coefficients corresponding to the fixed-point indices, wherein the coefficients represent coefficients in a polynomial approximation of an activation function, wherein the polynomial approximation selectively includes a first polynomial function corresponding to the first activation function or a second polynomial function corresponding to the second activation function based at least in part on a function selection signal;

[0193] a multiply-accumulate circuit configured to evaluate the polynomial approximation using a mathematical operation employing the inputs and the coefficients to generate a first approximation result; and

[0194] Multiplexer, configured to receive:

[0195] a first saturation value, wherein the first saturation value comprises a first asymptote value of the activation function;

[0196] a second saturation value, wherein the second saturation value comprises a second asymptote value of the activation function and is generated based at least in part on the function selection signal; and

[0197] First approximation results;

[0198] Wherein the multiplexer is configured to select an approximation output of the activation function among at least a first saturation value, a second saturation value, and a first approximation result based at least in part on the input.

[0199] Item A17. The fused activation function approximation circuit of Item A16, the first polynomial function having a first first denominator, wherein the multiply-add circuit is configured to evaluate a second polynomial function having a second second denominator based at least in part on a multiplication of the polynomial having the first first denominator and a power of the input.

[0200] Item A18. The fused activation function approximation circuit of either Item A16 or 17, wherein the first activation function comprises a sigmoid and the second activation function comprises a hyperbolic tangent.

[0201] Item A19. The fused activation function approximation circuit of any of Items A16, 17, or 18, wherein the plurality of coefficients includes a first subset of coefficients corresponding to a first polynomial function and a second subset of coefficients corresponding to a second polynomial function.

[0202] Item A20. The fused activation function approximation circuit of any of Items A16, 17, 18, or 19, comprising:

[0203] a second coefficient table comprising a second plurality of coefficients, wherein the second plurality of coefficients of the coefficient table are indexed to fixed-point indices, wherein the second coefficient table is configured to output second coefficients corresponding to the fixed-point indices, wherein the second coefficients represent coefficients in a second polynomial function;

[0204] and an additional multiplexer configured to select between at least a second coefficient and the coefficient based at least in part on the function select signal to generate an additional input to a multiply-add circuit, wherein the coefficient represents a coefficient in the first polynomial function, wherein the multiply-add circuit is configured to evaluate the polynomial approximation using a mathematical operation employing the input and the additional input to generate a first approximation result.

[0205] Item A21. The fused activation function approximation circuit of any of Items A16, 17, 18, 19, or 20, wherein the first polynomial function comprises an odd polynomial function.

[0206] Item B1. An integrated circuit device configured to receive an input and configured to approximate an activation function based at least in part on the input, comprising:

[0207] a coefficient table comprising a plurality of coefficients, wherein the plurality of coefficients of the coefficient table are indexed to indices based at least in part on the input, wherein the coefficient table is configured to output coefficients corresponding to the indices, wherein the coefficients represent coefficients in a polynomial that approximates the activation function over a plurality of subintervals in the first interval; and

[0208] A multiply-accumulate circuit is configured to evaluate the polynomial using a mathematical operation using the inputs and the coefficients to generate a first approximate result.

[0209] Item B2. The integrated circuit device of Item B1, comprising:

[0210] Multiplexer, configured to receive:

[0211] a first saturation value, wherein the first saturation value comprises a first asymptote value of the activation function;

[0212] a second saturation value, wherein the second saturation value comprises an additional asymptote value of the activation function; and

[0213] First approximation results;

[0214] Wherein the multiplexer is configured to select an approximation output of the activation function among at least a first saturation value, a second saturation value, and a first approximation result based at least in part on the input.

[0215] Item B3. The integrated circuit of either Item B1 or 2, wherein the input comprises a floating point number having a first precision, and wherein the barrel shifter is configured to receive the input and to generate an index, wherein the index is a fixed point number having a second precision, wherein the second precision is less than or equal to the first precision.

[0216] Item B4. The integrated circuit of any of Items B1, 2, or 3, wherein the activation function comprises a sigmoid.

[0217] Item B5. The integrated circuit of Item B4, comprising:

[0218] a subtractor configured to generate a second approximation result based at least in part on the first approximation result, wherein the second approximation result corresponds to an approximation of the activation function in a second interval, wherein the second interval includes a second number of subintervals; and

[0219] A multiplexer is configured to receive the second approximation result and to select an approximation output of the activation function among at least the first approximation result and the second approximation result based at least in part on the input.

[0220] Item B6. The integrated circuit of any of Items B1, 2, 3, or 4, wherein the multiply-add circuit is configured to use a mathematical operation having a first precision, an input having a second precision, and a first approximation result having the second precision, wherein the first precision is greater than or equal to the second precision.

[0221] Item B7. The integrated circuit of any of Items Bl, 2, 3, 4, or 6, wherein the activation function comprises a hyperbolic tangent.

[0222] Item B8. The integrated circuit of any of Items Bl, 2, 3, 4, 6, or 7, wherein the coefficient table is configured to store coefficients having a lower precision than a precision of the multiply-add circuit, and wherein the integrated circuit includes a conversion circuit configured to convert the coefficients from the lower precision to the precision.

[0223] Item B9. The integrated circuit of any of Items B1, 2, 3, 4, 6, 7, or 8, comprising:

[0224] a second coefficient table comprising a second plurality of coefficients, wherein the second plurality of coefficients of the second coefficient table are indexed to an index, wherein the second coefficient table is configured to output a second coefficient corresponding to the index, wherein the second coefficient represents a second coefficient in a second polynomial that approximates the activation function over a second number of subintervals in the second interval;

[0225] as well as

[0226] a second multiply-accumulate circuit configured to evaluate the second polynomial using a mathematical operation employing the input and the second coefficient to generate a second approximation result; and

[0227] A multiplexer is configured to receive the second approximation result and to select an approximation output of the activation function among at least the first approximation result and the second approximation result based at least in part on the input.

[0228] Item B10. The integrated circuit of any of Items B1, 2, 3, 4, 6, 7, 8, or 9, wherein the integrated circuit is configured to implement a recurrent neural network based at least in part on the first approximation result.

[0229] Item B11. A method for hardware implementation of an approximate activation function, comprising:

[0230] Receive input to the activation function;

[0231] evaluating the piecewise polynomial function to generate a first approximation result corresponding to a first input, wherein a first interval of inputs to the activation function includes the first input, wherein the piecewise polynomial function approximates the activation function over the first interval;

[0232] determining a second approximation result corresponding to a second input using an identity of the activation function and the first approximation result, wherein a second interval of inputs to the activation function includes the second input;

[0233] determining a first saturation value of the activation function corresponding to a third input, wherein a third interval of inputs to the activation function includes the third input;

[0234] determining a second saturation value of the activation function corresponding to a fourth input, wherein a fourth interval of inputs to the activation function includes the fourth input;

[0235] determining whether the first interval, the second interval, the third interval, or the fourth interval includes the input; and

[0236] In response to determining that the first interval includes the input, a first approximation result is selected as an approximation of the activation function at the input.

[0237] Item B12. The hardware implementation of Item B11, comprising dividing the first interval into a plurality of subintervals, wherein the piecewise polynomial function comprises a polynomial function for each subinterval.

[0238] Item B13. The hardware implementation of Item B12, wherein the accuracy of the approximation compared to the corresponding actual value of the activation function is based at least in part on the number of subintervals, the degree of the piecewise polynomial function, or a combination thereof.

[0239] Item B14. The hardware implementation of Item B11 or 12, comprising:

[0240] In response to determining that the second interval includes the input, selecting a second approximation result as the approximation value;

[0241] In response to determining that the third interval includes the input, selecting the first saturated value as the approximate value; and

[0242] In response to determining that the fourth interval includes the input, the second saturated value is selected as the approximate value.

[0243] Item B15. The hardware-implemented method of any of Items B11, 12, or 14, wherein the accuracy of the approximation compared to the corresponding actual value of the activation function is based at least in part on the size of the first interval.

[0244] Item B16. A tangible, non-transitory, machine-readable medium containing machine-readable instructions that, when executed by one or more processors, cause the processors to perform the hardware-implemented method of any of Items B11, 12, 14, or 15.

[0245] Item B17. A fused activation function approximation circuit configured to receive an input and configured to approximate an activation function based at least in part on the input, wherein the activation function selectively includes a first activation function or a second activation function, comprising:

[0246] a first input configured to receive a function selection signal, wherein the function selection signal indicates whether the activation function includes a first activation function or a second activation function;

[0247] a barrel shifter configured to receive an input and configured to generate a fixed-point index based at least in part on the input;

[0248] a coefficient table comprising a plurality of coefficients, wherein the plurality of coefficients of the coefficient table are indexed to fixed-point indices, wherein the coefficient table is configured to output coefficients corresponding to the fixed-point indices, wherein the coefficients represent coefficients in a polynomial approximation of an activation function, wherein the polynomial approximation selectively includes a first polynomial function corresponding to the first activation function or a second polynomial function corresponding to the second activation function based at least in part on a function selection signal;

[0249] a multiply-accumulate circuit configured to evaluate the polynomial approximation using a mathematical operation employing the inputs and the coefficients to generate a first approximation result; and

[0250] Multiplexer, configured to receive:

[0251] a first saturation value, wherein the first saturation value comprises a first asymptote value of the activation function;

[0252] a second saturation value, wherein the second saturation value comprises a second asymptote value of the activation function and is generated based at least in part on the function selection signal; and

[0253] First approximation results;

[0254] Wherein the multiplexer is configured to select an approximation output of the activation function among at least a first saturation value, a second saturation value, and a first approximation result based at least in part on the input.

[0255] Item B18. The fused activation function approximation circuit of Item B17, the first polynomial function having a first order of magnitude, wherein the multiply-add circuit is configured to evaluate a second polynomial function having a second order of magnitude based at least in part on a multiplication of the polynomial having the first order of magnitude and a power of the input.

[0256] Item B19. The fused activation function approximation circuit of either Item B17 or 18, wherein the first activation function comprises a sigmoid and the second activation function comprises a hyperbolic tangent.

[0257] Item B20. The fused activation function approximation circuit of any of Items B17, 18, or 19, wherein the plurality of coefficients includes a first subset of coefficients corresponding to a first polynomial function and a second subset of coefficients corresponding to a second polynomial function.

[0258] Item B21. The fused activation function approximation circuit of any of Items B17, 18, 19, or 20, comprising:

[0259] a second coefficient table comprising a second plurality of coefficients, wherein the second plurality of coefficients of the coefficient table are indexed to fixed-point indices, wherein the second coefficient table is configured to output second coefficients corresponding to the fixed-point indices, wherein the second coefficients represent coefficients in a second polynomial function;

[0260] and an additional multiplexer configured to select between at least a second coefficient and the coefficient based at least in part on the function select signal to generate an additional input to a multiply-add circuit, wherein the coefficient represents a coefficient in the first polynomial function, wherein the multiply-add circuit is configured to evaluate the polynomial approximation using a mathematical operation employing the input and the additional input to generate a first approximation result.

[0261] Item B22. The fused activation function approximation circuit of any of Items B17, 18, 19, 20, or 21, wherein the first polynomial function comprises an odd-degree polynomial function.

[0262] Item C1. An integrated circuit device configured to receive an input and configured to approximate an activation function based at least in part on the input, comprising:

[0263] a coefficient table comprising a plurality of coefficients, wherein the plurality of coefficients of the coefficient table are indexed to indices based at least in part on the input, wherein the coefficient table is configured to output coefficients corresponding to the indices, wherein the coefficients represent coefficients in a polynomial that approximates the activation function over a plurality of subintervals in the first interval; and

[0264] A multiply-accumulate circuit is configured to evaluate the polynomial using a mathematical operation using the inputs and the coefficients to generate a first approximate result.

[0265] Item C2. The integrated circuit device of Item C1, comprising:

[0266] Multiplexer, configured to receive:

[0267] a first saturation value, wherein the first saturation value comprises a first asymptote value of the activation function;

[0268] a second saturation value, wherein the second saturation value comprises an additional asymptote value of the activation function; and

[0269] First approximation results;

[0270] Wherein the multiplexer is configured to select an approximation output of the activation function among at least a first saturation value, a second saturation value, and a first approximation result based at least in part on the input.

[0271] Item C3. The integrated circuit of either Item C1 or 2, wherein the activation function comprises a sigmoid, a hyperbolic tangent, or a combination thereof.

[0272] Item C4. The integrated circuit of any of Items C1, 2, or 3, comprising:

[0273] a subtractor configured to generate a second approximation result based at least in part on the first approximation result, wherein the second approximation result corresponds to an approximation of the activation function in a second interval, wherein the second interval includes a second number of subintervals; and

[0274] A multiplexer is configured to receive the second approximation result and to select an approximation output of the activation function among at least the first approximation result and the second approximation result based at least in part on the input.

[0275] Item C5. The integrated circuit of any of Items C1, 2, 3, or 4, wherein the multiply-add circuit is configured to use a mathematical operation having a first precision, an input having a second precision, and a first approximation result having the second precision, wherein the first precision is greater than or equal to the second precision.

[0276] Item C6. The integrated circuit of any of Items C1, 2, 3, 4, or 5, wherein the coefficient table is configured to store coefficients having a lower precision than a precision of the multiply-add circuit, and wherein the integrated circuit includes a conversion circuit configured to convert the coefficients from the lower precision to the precision.

[0277] Item C7. The integrated circuit of any of Items C1, 2, 3, 4, 5, or 6, comprising:

[0278] a second coefficient table comprising a second plurality of coefficients, wherein the second plurality of coefficients of the second coefficient table are indexed to an index, wherein the second coefficient table is configured to output a second coefficient corresponding to the index, wherein the second coefficient represents a second coefficient in a second polynomial that approximates the activation function over a second number of subintervals in the second interval;

[0279] as well as

[0280] a second multiply-accumulate circuit configured to evaluate the second polynomial using a mathematical operation employing the input and the second coefficient to generate a second approximation result; and

[0281] A multiplexer is configured to receive the second approximation result and to select an approximation output of the activation function among at least the first approximation result and the second approximation result based at least in part on the input.

[0282] Item C8. A tangible, non-transitory, machine-readable medium containing machine-readable instructions that, when executed by one or more processors, cause the processors to:

[0283] Receive input to the activation function;

[0284] evaluating the piecewise polynomial function to generate a first approximation result corresponding to a first input, wherein a first interval of inputs to the activation function includes the first input, wherein the piecewise polynomial function approximates the activation function over the first interval;

[0285] determining a second approximation result corresponding to a second input using an identity of the activation function and the first approximation result, wherein a second interval of inputs to the activation function includes the second input;

[0286] determining a first saturation value of the activation function corresponding to a third input, wherein a third interval of inputs to the activation function includes the third input;

[0287] determining a second saturation value of the activation function corresponding to a fourth input, wherein a fourth interval of inputs to the activation function includes the fourth input;

[0288] determining whether the first interval, the second interval, the third interval, or the fourth interval includes the input; and

[0289] In response to determining that the first interval includes the input, a first approximation result is selected as an approximation of the activation function at the input.

[0290] Item C9. The tangible, non-transitory, machine-readable medium of Item C8, comprising machine-readable instructions that, when executed by one or more processors, cause the processor to divide a first interval into a plurality of subintervals, wherein the piecewise polynomial function comprises a polynomial function for each subinterval.

[0291] Item C10. The tangible, non-transitory, machine-readable medium of any of Items C8 or 9, comprising machine-readable instructions that, when executed by one or more processors, cause the processor to:

[0292] In response to determining that the second interval includes the input, selecting a second approximation result as the approximation value;

[0293] In response to determining that the third interval includes the input, selecting the first saturated value as the approximate value; and

[0294] In response to determining that the fourth interval includes the input, the second saturated value is selected as the approximate value.

[0295] Item C11. A fused activation function approximation circuit configured to receive an input and configured to approximate an activation function based at least in part on the input, wherein the activation function selectively includes a first activation function or a second activation function, comprising:

[0296] a first input configured to receive a function selection signal, wherein the function selection signal indicates whether the activation function includes a first activation function or a second activation function;

[0297] a barrel shifter configured to receive an input and configured to generate a fixed-point index based at least in part on the input;

[0298] a coefficient table comprising a plurality of coefficients, wherein the plurality of coefficients of the coefficient table are indexed to fixed-point indices, wherein the coefficient table is configured to output coefficients corresponding to the fixed-point indices, wherein the coefficients represent coefficients in a polynomial approximation of an activation function, wherein the polynomial approximation selectively includes a first polynomial function corresponding to the first activation function or a second polynomial function corresponding to the second activation function based at least in part on a function selection signal;

[0299] a multiply-accumulate circuit configured to evaluate the polynomial approximation using a mathematical operation employing the inputs and the coefficients to generate a first approximation result; and

[0300] Multiplexer, configured to receive:

[0301] a first saturation value, wherein the first saturation value comprises a first asymptote value of the activation function;

[0302] a second saturation value, wherein the second saturation value comprises a second asymptote value of the activation function and is generated based at least in part on the function selection signal; and

[0303] First approximation results;

[0304] Wherein the multiplexer is configured to select an approximation output of the activation function among at least a first saturation value, a second saturation value, and a first approximation result based at least in part on the input.

[0305] Item C12. The fused activation function approximation circuit of Item C11, the first polynomial function having a first order of magnitude, wherein the multiply-add circuit is configured to evaluate a second polynomial function having a second order of magnitude based at least in part on a multiplication of the polynomial having the first order of magnitude and a power of the input.

[0306] Item C13. The fused activation function approximation circuit of either Item C11 or 12, wherein the first activation function comprises a sigmoid and the second activation function comprises a hyperbolic tangent.

[0307] Item C14. The fused activation function approximation circuit of any of Items C11, 12, or 13, wherein the plurality of coefficients includes a first subset of coefficients corresponding to a first polynomial function and a second subset of coefficients corresponding to a second polynomial function.

[0308] Item C15. The fused activation function approximation circuit of any of Items C11, 12, 13, or 14, comprising:

[0309] a second coefficient table comprising a second plurality of coefficients, wherein the second plurality of coefficients of the coefficient table are indexed to fixed-point indices, wherein the second coefficient table is configured to output second coefficients corresponding to the fixed-point indices, wherein the second coefficients represent coefficients in a second polynomial function;

[0310] and an additional multiplexer configured to select between at least a second coefficient and the coefficient based at least in part on the function select signal to generate an additional input to a multiply-add circuit, wherein the coefficient represents a coefficient in the first polynomial function, wherein the multiply-add circuit is configured to evaluate the polynomial approximation using a mathematical operation employing the input and the additional input to generate a first approximation result.

Claims

1. An integrated circuit device configured to receive an input and configured to approximate an activation function based at least in part on the input, comprising: a coefficient table comprising a plurality of coefficients, wherein the plurality of coefficients of the coefficient table are indexed to indices based at least in part on the input, wherein the coefficient table is configured to output coefficients corresponding to the indices, wherein the coefficients represent coefficients in a polynomial that approximates the activation function over a plurality of subintervals in a first interval; a multiply-and-accumulate circuit configured to evaluate the polynomial using a mathematical operation employing the inputs and the coefficients to generate a first approximate result; as well as A subtractor is configured to generate a second approximation result based on the identity of the activation function and the first approximation result, wherein the second approximation result corresponds to an approximation of the activation function in a second interval, wherein the second interval includes a second number of subintervals.

2. The integrated circuit device according to claim 1, comprising: Multiplexer, configured to receive: a first saturation value, wherein the first saturation value comprises a first asymptote value of the activation function; a second saturation value, wherein the second saturation value comprises an additional asymptote value of the activation function; and the first approximate result; Wherein the multiplexer is configured to select an approximation output of the activation function among at least the first saturation value, the second saturation value, and the first approximation result based at least in part on the input.

3. The integrated circuit device according to claim 1 or 2, wherein: The input comprises a floating point number having a first precision, and wherein the barrel shifter is configured to receive the input and to generate the index, wherein the index is a fixed point number having a second precision, wherein the second precision is less than or equal to the first precision.

4. The integrated circuit device according to claim 1 or 2, wherein: The activation function includes a sigmoid.

5. The integrated circuit device of claim 4 , comprising: A multiplexer is configured to receive the second approximation result and to select an approximation output of the activation function among at least the first approximation result and the second approximation result based at least in part on the input.

6. The integrated circuit device according to claim 1 or 2, wherein: The multiply-add circuit is configured to use the mathematical operation, the mathematical operation having a first precision, the input having a second precision, the first approximation result having the second precision, wherein the first precision is greater than or equal to the second precision.

7. The integrated circuit device according to claim 1 or 2, wherein: The activation function includes hyperbolic tangent.

8. The integrated circuit device according to claim 1 or 2, wherein: The coefficient table is configured to store the coefficients, the coefficients having a lower precision than a precision of the multiply-add circuit, and wherein the integrated circuit includes a conversion circuit configured to convert the coefficients from the lower precision to the precision.

9. The integrated circuit device according to claim 1 or 2, comprising: a second coefficient table comprising a second plurality of coefficients, wherein the second plurality of coefficients of the second coefficient table are indexed to the index, wherein the second coefficient table is configured to output a second coefficient corresponding to the index, wherein the second coefficient represents a second coefficient in a second polynomial that approximates the activation function over a second number of subintervals in a second interval; as well as a second multiply-accumulate circuit configured to evaluate the second polynomial using the mathematical operation using the input and the second coefficient to generate a second approximation result; as well as A multiplexer is configured to receive the second approximation result and to select an approximation output of the activation function among at least the first approximation result and the second approximation result based at least in part on the input.

10. The integrated circuit device according to claim 1 or 2, wherein: The integrated circuit is configured to implement a recurrent neural network based at least in part on the first approximation result.

11. A fused activation function approximation circuit configured to receive an input and configured to approximate an activation function based at least in part on the input, wherein the activation function selectively includes a first activation function or a second activation function, comprising: a first input configured to receive a function selection signal, wherein the function selection signal indicates whether the activation function includes the first activation function or the second activation function; a barrel shifter configured to receive the input and configured to generate a fixed-point index based at least in part on the input; a coefficient table comprising a plurality of coefficients, wherein the plurality of coefficients of the coefficient table are indexed to the fixed-point index, wherein the coefficient table is configured to output coefficients corresponding to the fixed-point index, wherein the coefficients represent coefficients in a polynomial approximation of the activation function, wherein the polynomial approximation selectively includes a first polynomial function corresponding to the first activation function or a second polynomial function corresponding to the second activation function based at least in part on the function selection signal; multiply-add circuitry configured to evaluate the polynomial approximation using a mathematical operation employing the input and the coefficients to generate a first approximation result; and Multiplexer, configured to receive: a first saturation value, wherein the first saturation value comprises a first asymptote value of the activation function; a second saturation value, wherein the second saturation value comprises a second asymptote value of the activation function and is generated based at least in part on the function selection signal; the first approximation result; and a second approximation result determined using the identity of the activation function and the first approximation result, wherein the second approximation result corresponds to an approximation of the activation function in a second interval, wherein the second interval includes a second number of subintervals; Wherein the multiplexer is configured to select an approximation output of the activation function among at least the first saturation value, the second saturation value, the first approximation result, and the second approximation result based at least in part on the input.

12. The fused activation function approximation circuit of claim 11 , wherein the first polynomial function has a first degree, wherein the multiply-add circuit is configured to evaluate the second polynomial function based at least in part on a multiplication of the polynomial of the first degree by a power of the input, the second polynomial function having a second degree.

13. The fused activation function approximation circuit according to claim 11 or 12, wherein: The first activation function comprises a sigmoid, and the second activation function comprises a hyperbolic tangent.

14. The fused activation function approximation circuit according to claim 11 or 12, wherein: The plurality of coefficients includes a first subset of coefficients corresponding to the first polynomial function and a second subset of coefficients corresponding to the second polynomial function.

15. The fused activation function approximation circuit according to claim 11 or 12, comprising: a second coefficient table comprising a second plurality of coefficients, wherein the second plurality of coefficients of the coefficient table are indexed to the fixed-point index, wherein the second coefficient table is configured to output a second coefficient corresponding to the fixed-point index, wherein the second coefficient represents a coefficient in the second polynomial function; and an additional multiplexer configured to select among at least the second coefficient and the coefficient based at least in part on the function select signal to generate an additional input to the multiply-add circuit, wherein the coefficient represents a coefficient in the first polynomial function, wherein the multiply-add circuit is configured to evaluate the polynomial approximation using the mathematical operation with the input and the additional input to generate the first approximation result.

16. The fused activation function approximation circuit according to claim 11 or 12, wherein: The first polynomial function comprises an odd-degree polynomial function.

Images