Deep Neural Network Architecture Using Piecewise Linear Approximation

Abstract

To provide special hardware of a deep layer neural network (DNN) that solves requirements of power consumption, latency and silicon area.SOLUTION: A DNN 100 comprises a logarithmic circuit to: identify an input associated with a logarithm operation; identify a first range that the input falls within; compute a result of the first equation based on a plurality of operands associated with the first equation; and return an output associated with the logarithm operation. The logarithm operation is to be performed by the logarithmic circuit using piecewise linear approximation. The first range is identified from a plurality of ranges associated with a plurality of piecewise linear approximation (PLA) equations for the logarithm operation, and corresponds to a first equation of the plurality of PLA equations. The output is generated based at least in part on the result of the first equation.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates generally to the field of computer architecture and design, and more specifically, but not exclusively, to processing architectures for deep neural networks (DNNs). [Background technology]

[0002] The ever-increasing number of deep learning applications being developed for many different use cases is driving demand for specialized hardware designed for deep neural networks (DNNs). For example, DNNs typically require a significant amount of real-time processing, which often involves multiple layers of complex operations on floating-point numbers, such as convolutional layers, pooling layers, fully connected layers, etc. However, existing hardware solutions for DNNs suffer from various limitations, including high power consumption, long latency, significant silicon area requirements, etc. [Brief description of the drawings]

[0003] The present disclosure is best understood from the following detailed description when read in conjunction with the accompanying drawings. It is emphasized that, in accordance with standard industry practice, the various features are not necessarily drawn to scale, but are used for illustrative purposes only. Any scales, whether explicit or implied, are provided merely to provide an illustration. In other embodiments, the dimensions of the various features may be arbitrarily expanded or reduced for clarity of illustration.

[0004] [Figure 1] 1 illustrates an example embodiment of a deep neural network (DNN) implemented using log and antilog piecewise linear approximation circuits.

[0005] [Figure 2A] 1 illustrates an example embodiment of a unified activation function circuit for a deep neural network (DNN). [Figure 2B]1 illustrates an example embodiment of a unified activation function circuit for a deep neural network (DNN).

[0006] [Figure 3A] 4 shows an example of an activation function of the integrated activation function circuit. [Figure 3B] 4 shows an example of an activation function of the integrated activation function circuit. [Figure 3C] 4 shows an example of an activation function of the integrated activation function circuit. [Figure 3D] 4 shows an example of an activation function of the integrated activation function circuit. [Figure 3E] 4 shows an example of an activation function of the integrated activation function circuit.

[0007] [Figure 4] 1 illustrates an example embodiment of a unified activation function circuit implemented using a modified activation function formula having base-2 exponential terms.

[0008] [Figure 5A] 1 illustrates an example embodiment of a logarithmic circuit implemented using piecewise linear approximation. [Figure 5B] 1 illustrates an example embodiment of a logarithmic circuit implemented using piecewise linear approximation. [Figure 5C] 1 illustrates an example embodiment of a logarithmic circuit implemented using piecewise linear approximation.

[0009] [Figure 6A] 1 illustrates an example embodiment of an anti-logarithm circuit implemented using piecewise linear approximation. [Figure 6B] 1 illustrates an example embodiment of an anti-logarithm circuit implemented using piecewise linear approximation. [Figure 6C] 1 illustrates an example embodiment of an anti-logarithm circuit implemented using piecewise linear approximation.

[0010] [Figure 7] 1 illustrates an example embodiment of an exponential circuit implemented using piecewise linear approximation.

[0011] [Figure 8] 1 shows a flowchart of an example of a processing architecture used to implement an artificial neural network.

[0012] [Figure 9A] 1 illustrates the scalability of an example artificial neural network processing architecture with respect to the number of parallel operations supported. [Figure 9B] 1 illustrates the scalability of an example artificial neural network processing architecture with respect to the number of parallel operations supported.

[0013] [Figure 10A] 1 illustrates one performance aspect of an example artificial neural network processing architecture. [Figure 10B] 1 illustrates one performance aspect of an example artificial neural network processing architecture. [Figure 10C] 1 illustrates one performance aspect of an example artificial neural network processing architecture. [Figure 10D] 1 illustrates one performance aspect of an example artificial neural network processing architecture. [Figure 10E] 1 illustrates one performance aspect of an example artificial neural network processing architecture.

[0014] [Figure 11A] We present examples of DNNs implemented using a traditional activation function and a modified activation function with a base-2 exponential term. [Figure 11B] Here is an example of a DNN implemented using a traditional activation function. [Figure 11C] Here we present an example of a DNN implemented using a modified activation function with base-2 exponential terms.

[0015] [Figure 12A] 1 shows the performance behavior of a DNN implemented using a conventional activation function. [Figure 12B]13 shows the performance aspects of a DNN implemented with a modified activation function. [Figure 13] A comparison of the performance aspects of a DNN implemented with a conventional activation function and a DNN implemented with a modified activation function is shown.

[0016] [Figure 14A] 1 illustrates an example implementation of a computer architecture that can be used in accordance with embodiments disclosed herein. [Figure 14B] 1 illustrates an example implementation of a computer architecture that can be used in accordance with embodiments disclosed herein. [Figure 15] 1 illustrates an example implementation of a computer architecture that can be used in accordance with embodiments disclosed herein. [Figure 16] 1 illustrates an example implementation of a computer architecture that can be used in accordance with embodiments disclosed herein. [Figure 17] 1 illustrates an example implementation of a computer architecture that can be used in accordance with embodiments disclosed herein. [Figure 18] 1 illustrates an example implementation of a computer architecture that can be used in accordance with embodiments disclosed herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] The following disclosure provides many different embodiments, or examples, for implementing different features of the present disclosure. In the following, specific examples of components and arrangements are described to simplify the present disclosure. Of course, these are merely examples and are not intended to be limiting. Furthermore, the present disclosure may repeat reference numbers and / or letters in various examples. This repetition is for the purposes of simplicity and clarity, and does not in itself indicate a relationship between the various embodiments and / or configurations discussed. Different embodiments may have different advantages, and thus a particular advantage is not necessarily a requirement of any embodiment.

[0018] [Deep Neural Network (DNN) Inference Using Log / Anti-Log Piecewise Linear Approximation Circuits] The ever-increasing number of artificial intelligence applications that rely on machine learning (e.g., deep learning) creates a strong demand for specialized hardware designed to implement artificial neural networks (e.g., deep neural networks, convolutional neural networks, feedforward neural networks, recurrent neural networks, etc.) Low power, small area, and high speed hardware are ideal for deep learning applications.

[0019] Specifically, artificial neural networks, such as deep neural networks (DNNs), are implemented with multiple layers of processing nodes or "neurons," such as convolutional layers, pooling layers, fully connected layers, and so on. The nodes in each layer perform computations on a collection of inputs and associated weights (typically represented as vectors) to generate outputs, which are then used as inputs to the nodes in the next layer. The computations performed by the nodes in each layer typically include transforming the inputs based on the associated weights, as well as an activation function that is used to determine whether each node should be "activated." Furthermore, these layers are typically iterated in this manner based on the requirements of a particular application to reach a global minimum.

[0020] Moreover, state-of-the-art DNNs are typically implemented using operations on numbers represented using single-precision (32-bit) floating-point format. DNN inference generally requires a large amount of real-time processing on these floating-point numbers, since it involves multiple complex computation layers such as convolutional layers, pooling layers, fully connected layers, and so on. Furthermore, these complex computations often involve multiplication, and floating-point multipliers are one of the key components in existing DNN solutions. However, floating-point multipliers are extremely costly in terms of power consumption, silicon area, and latency. Furthermore, in some cases, look-up tables (LUTs) are used to simplify DNN operations, but LUTs also require costly silicon area.

[0021] Thus, in some cases, DNN optimization techniques may be utilized to improve the performance and / or reduce the silicon area required for hardware used to implement DNNs and other types of artificial neural networks. However, these DNN optimization techniques typically focus on reducing computational costs by lowering the overall precision and / or reducing the number of basic operations (e.g., limiting the number of convolutional layers, pooling layers, etc.). For example, in some embodiments, the hardware used to implement DNNs may be designed to handle floating-point representations (e.g., from 8-bit quantized floating-point to 16-bit fixed-point representations) that have fewer bits and thus provide lower precision. However, the use of lower precision floating-point representations may incur unacceptable precision losses, especially for larger datasets. Furthermore, DNN optimization techniques that reduce the number of basic computation layers may have adverse effects, such as insufficient convergence time to reach a global minimum during DNN training. Moreover, these various optimizations still require floating-point multipliers, which also have limitations in terms of power, area and performance of the multiplier circuitry.

[0022] In some cases, to ease the requirements of multiplier circuits, the DNN may be implemented using circuits that perform logarithmic, inverse logarithmic and / or exponential calculations using lookup tables (LUTs). For example, in some embodiments, the parabolic curve of the logarithmic, inverse logarithmic and / or exponential operation may be divided into multiple segments, and the individual coefficient values ​​may be pre-calculated using a curve fitting algorithm, and the pre-calculated coefficients may then be stored in a lookup table implemented using a memory component (e.g., ROM). In this scheme, for any point on the curve, ax 2 To calculate +bx+c, the values ​​of coefficients a, b, and c are first fetched from a lookup table, and then the result is calculated using multipliers and adders. However, this approach requires a large amount of silicon area for the associated LUTs and multipliers, and may consume multiple clock cycles (e.g., 5-8 clock cycles) to calculate the above equation.

[0023] Accordingly, the present disclosure describes various embodiments of hardware that can efficiently perform DNN calculations without relying on lookup tables and / or multipliers. With more specific reference to the accompanying drawings, example embodiments that can be used to implement the features and functionality of the present disclosure are described below.

[0024] FIG. 1 illustrates an example embodiment of a deep neural network (DNN) 100 implemented using log and antilog piecewise linear approximation circuits. In the illustrated example, the DNN 100 is implemented using multiple layers 106a-106e, including a first convolutional layer 106a, a max pooling layer 106b, a second convolutional layer 106c, a third convolutional layer 106d, and a fully connected layer 106e. Additionally, the DNN 100 is implemented using a multiplier-less neural network microarchitecture, which uses log and antilog circuits 110, 120 rather than multiplier circuits to perform calculations for the individual DNN layers 106a-106e. Specifically, the logarithm and antilogarithm circuits 110, 120 perform base 2 logarithm (log2) and base 2 antilogarithm (antilog2) calculations that can be leveraged to convert multiplication operations typically required in a given DNN layer 106 to additions. Additionally, the logarithm and antilogarithm circuits 110, 120 perform the log2 and antilog2 calculations using piecewise linear approximations, allowing each calculation to be performed in a single clock cycle and without lookup tables or multipliers. In this manner, the illustrated embodiment reduces DNN processing latency while also eliminating the need for multiplier circuits and lookup tables, thereby significantly reducing the silicon area required for hardware. Example implementations of the logarithm circuit 110 and the antilogarithm circuit 120 are further shown and described in connection with FIGS. 5A-5C and 6A-6C.

[0025] As an example, for a convolutional layer of a DNN (e.g., layers 106a, 106c, 106d of DNN 100), the convolution can generally be expressed as follows (where f(n) and g(n) are floating-point vectors):

number

number

number

number

[0026] As another example, a fully connected layer (e.g., layer 106e of DNN 100) is the last layer of a DNN and is responsible for performing final inferences and decisions. In general, a fully connected layer is similar to a convolutional layer, but typically involves one-dimensional vectors. Thus, a fully connected layer can leverage log2 arithmetic in a similar manner to a convolutional layer to convert multiplication operations into additions. However, because the fully connected layer is the last layer of a DNN, the final output must be in the normal domain rather than the log2 domain.

[0027] To illustrate, a fully connected layer can generally be represented using the following equation:

number

number

number

[0028] In the illustrated embodiment, for example, the DNN 100 is implemented with multiple layers 106a-106e, including a first convolutional layer 106a, a max pooling layer 106b, a second convolutional layer 106c, a third convolutional layer 106d, and a fully connected layer 106e. Each layer 106a-106e performs a calculation using an input (X) 101a-101e together with a weight vector (W) 102a-102d of the given layer to generate a corresponding output (Y) 103a-103f. Furthermore, an initial input vector (X) 101a is provided to the first convolutional layer 106a of the DNN 100, and each remaining layer 106b-106e is provided with the output (Y) 103a-103d of the preceding layer as its input (X) 101b-101e.

[0029] Furthermore, logarithm and antilogarithm circuits 110, 120 implemented using piecewise linear approximation are utilized to perform calculations in each layer 106 of the DNN 100, thus eliminating the need for multiplier circuits and lookup tables while also reducing latency. For example, the logarithm circuit 110 performs a log2 calculation to convert floating-point numbers to fixed-point numbers, thereby allowing complex operations such as floating-point multiplication to be converted to fixed-point addition, and the antilogarithm circuit 120 subsequently performs an antilog2 calculation to convert the fixed-point numbers back to floating-point numbers. Furthermore, the logarithm and antilogarithm circuits 110, 120 perform their respective log2 and antilog2 calculations using piecewise linear approximation, thereby allowing each calculation to be performed in a single clock cycle.

[0030] In the illustrated embodiment, for example, the logarithm circuit 110 is used to convert the original input vector (X) 101a and each weight vector (W) 102a-102d to the log2 domain before they are provided to the DNN 100, while the antilogarithm circuit 120 is used to convert the final output (Y) 103f of the fully connected layer 106e from the log2 domain back to the normal domain. In addition, additional antilog2 and log2 operations (not shown) are also performed throughout the hidden layers (e.g., intermediate layers between the input layer and the output layer) of the DNN 100 to convert between the log2 domain and the normal domain as needed. For example, as explained above, the convolution layer requires that each summation term be converted back to the normal domain prior to its accumulation, and therefore an antilog2 operation must be performed before accumulating each summation term. However, the final output of the hidden layer is subsequently provided to the next layer after being converted back to the log2 domain, still avoiding multiplication operations in the next layer.

[0031] For example, the result of each hidden layer node is typically sent to an activation function that determines whether the node should be "activated," and then the output of the activation function is fed as an input to the next layer. Thus, to avoid multiplication operations in the next layer, the log2 of the hidden layer node's activation function is fed to the next layer. For example, if a hidden layer node performs an antilog2 operation to compute a convolutional component, the result is converted back to the log2 domain before being fed to the activation function. In this manner, the output (Y) computed by each hidden layer node is already in the log2 domain when it is provided as an input (X) to the next layer.

[0032] Thus, the illustrated embodiment provides many advantages, including low latency, high accuracy, and reduced power consumption, with a flexible, small-area hardware design that is highly scalable and portable. For example, in the illustrated embodiment, the DNN 100 is implemented using logarithm and antilogarithm circuits 110, 120 that perform log2 and antilog2 calculations using piecewise linear approximations, thereby eliminating the need for multiplier circuits and lookup tables in the hardware design. In this manner, the illustrated embodiment significantly reduces the required silicon area, power consumption, and latency of the hardware (e.g., by eliminating multipliers and lookup tables), while still providing high accuracy. Specifically, the proposed microarchitecture performs each log2 and antilog2 calculation in a single clock cycle, thereby reducing delays through the data path and thus reducing the overall latency of the hardware.

[0033] The proposed microarchitecture is also highly scalable. Specifically, the flexible implementation of the proposed microarchitecture allows the hardware to be replicated as needed to increase the number of parallel operations supported. For example, the proposed microarchitecture may be implemented with any number of logarithm and antilogarithm circuits 110, 120. In this manner, the proposed microarchitecture can be easily scaled to support the number of parallel operations required by a particular application or use case. The precision of the proposed microarchitecture can also be scaled based on the requirements of the application. For example, if an application requires higher precision, the number of segments in the piecewise linear approximation model used by the logarithm and antilogarithm circuits 110, 120 can be increased to address the precision requirement. In this manner, the proposed microarchitecture is also highly portable as it can be easily ported and / or scaled for any product or form factor including mobile devices (e.g., handheld or wearable devices), drones, servers, and / or any other artificial intelligence solution that requires DNN operations without dependency or modification.

[0034] [DNN activation function circuit using piecewise linear approximation] As an ever-increasing number of products are designed with artificial intelligence (AI) capabilities, there is a strong demand for specialized hardware that can accelerate fundamental AI operations (e.g., neural network activation functions) while remaining versatile enough to support a variety of different implementations and associated algorithms, especially for resource-constrained form factors (e.g., small, low-power edge devices).

[0035] Specifically, the growing popularity of AI solutions that rely on machine learning (e.g., deep learning) has led to a demand for hardware acceleration designed for artificial neural networks (e.g., deep neural networks, convolutional neural networks, forward propagation neural networks, recurrent neural networks, etc.). For example, deep neural networks (DNNs) are implemented using multiple layers of "artificial neurons," which are processing nodes that typically use nonlinear activation functions to determine whether or not they should "activate" in response to a particular input. For example, an activation function is a function that maps inputs to outputs, typically using a nonlinear transformation, to determine whether a particular processing node or "artificial neuron" should activate. The use of activation functions is an important aspect of DNNs, but they can also be extremely computationally intensive.

[0036] There are many different types of activation functions that can be used to implement DNNs, including sigmoid, hyperbolic tangent (Tanh), rectified linear unit (ReLU), leaky ReLU, and Swish, among other examples. The choice of activation function greatly affects the training dynamics and task performance of the DNN. Thus, in some cases, to improve training dynamics and performance, DNNs may be implemented with multiple activation functions within a single neural network. The computational engine of a DNN may also rely on dedicated hardware to implement these activation functions, which typically occupy a significant amount of area on silicon. For example, hardware designed for state-of-the-art DNNs typically handles single-precision (32-bit) floating-point numbers and implements activation functions using a look-up table (LUT) approach. However, the use of a look-up table (LUT) approach to activation functions increases silicon area, power consumption, and latency, each of which continues to increase with increasing numbers of neurons in a DNN. Furthermore, since each activation function requires its own lookup table, the use of multiple activation functions in a single DNN increases the number of required lookup tables, thus further impacting silicon area, power and latency.

[0037] As an example, when using a lookup table approach, the curve of the activation function is typically bounded in an interval [-m, m] (where "m" is a real number), and the bounded curve may then be divided into multiple segments. The values ​​of the individual coefficients may then be pre-calculated using a curve fitting algorithm, and the pre-calculated coefficients may then be stored in a lookup table implemented using a memory component (e.g., ROM). In this scheme, for any point on the curve, ax 2To calculate +bx+c, the values ​​of coefficients a, b, and c are first fetched from a lookup table, and then the result is calculated using multipliers and adders. However, this approach requires a large amount of silicon area for the associated lookup tables and multipliers, and may consume multiple clock cycles (e.g., 5-8 clock cycles) to calculate the above formula.

[0038] To illustrate, a bounded curve spanning the interval [-3, 3], divided into 256 uniform segments with coefficient widths of 64 bits (a: 20 bits, b: 20 bits, c: 24 bits), produces 21 bits of mantissa precision for IEEE-754 single precision floating point numbers. In a given embodiment, this approach requires a 256x64 ROM and computation block with 41,853 and 5,574 composite gates (e.g., NAND equivalent gates), respectively. Scaling down this hardware with lower precision (e.g., 12-bit or 10-bit precision) only saves ROM area. For example, in a given embodiment, the estimated silicon area required for a sigmoid activation function with 10-bit precision is 17,120 composite gates. Furthermore, this area must be further replicated or instantiated based on the number of parallel operations the hardware needs to support.

[0039] Thus, existing hardware used to implement activation functions for DNNs (e.g., hardware implemented using lookup tables) has various drawbacks, including costly silicon area requirements, unsatisfactory power consumption, and long processing latency, among other examples. These drawbacks are further magnified as the hardware is scaled, such as by increasing the number of artificial neurons, parallel operations, and / or activation functions. Furthermore, there is no unified hardware solution that can implement multiple activation functions without using separate hardware blocks and / or lookup tables for each activation function.

[0040] Accordingly, this disclosure describes various embodiments of an integrated hardware solution that supports multiple DNN activation functions without the use of lookup tables, as described in more detail below.

[0041] 2A and 2B illustrate an example embodiment of an artificial neural network (e.g., a deep neural network (DNN)) unified activation function (AF) circuit 200. Specifically, the AF circuit 200 supports multiple DNN activation functions on a single hardware component without relying on lookup tables.

[0042] For example, in the illustrated embodiment, the AF circuit 200 implements each activation function using a novel algorithm that leverages exponential, logarithm to base 2 (log2), and antilogarithm to base 2 (antilog2) calculations implemented with piecewise linear approximations to simplify the required calculations for each activation function. For example, many activation functions are nonlinear functions that involve complex exponential, division, and / or multiplication operations that are typically implemented with costly multiplier circuits (e.g., division may be implemented with a multiplier circuit that multiplies the numerator by the inverse of the denominator). However, the AF circuit 200 leverages log2 and antilog2 calculations to eliminate the complex division and / or multiplication operations required for a given activation function, instead converting them to subtraction and / or addition. Additionally, the AF circuit 200 implements the exponential, log2, and antilog2 calculations with piecewise linear approximations to further simplify the required calculations for the activation functions. As a result, the log2 and antilog2 calculations can be performed in a single clock cycle, while the exponential calculations can be performed in two clock cycles. In this manner, one activation function can be calculated in only five clock cycles, and the basic calculations can be easily pipelined to increase throughput. Thus, the AF circuit 200 utilizes log2, antilog2 and exponential calculations implemented with piecewise linear approximations to simplify the basic calculations of the activation functions, thereby eliminating the need for activation function lookup tables, reducing multiplier circuit requirements and reducing overall latency. This approach translates directly into significant silicon area savings (e.g., due to the elimination of lookup tables and reduced multiplier circuitry) as much fewer synthesis gates are required compared to a typical lookup table approach with similar accuracy.

[0043] In the illustrated embodiment, for example, the AF circuit 200 includes log, antilog, and exponent blocks 210, 220, 230 for performing log2, antilog2, and exponent calculations, respectively, using piecewise linear approximations. For example, in some embodiments, the log, antilog, and exponent blocks 210, 220, 230 may be implemented with 12-bit precision (i.e., 1 sign bit + 8 exponent bits + 12 mantissa bits = 21-bit precision) in the mantissa of an IEEE-754 single precision floating point number using a 16-segment piecewise linear approximation. Example implementations of the log, antilog, and exponent blocks 210, 220, 230 are further shown and described in connection with Figures 5A-5C, 6A-6C, and 7.

[0044] The AF circuit 200 is a configurable circuit that supports the following activation functions: sigmoid, hyperbolic tangent (Tanh), rectified linear unit (ReLU), leaky ReLU, and Swish. However, in other embodiments, the AF circuit 200 may be designed to support any type or number of activation functions. The AF circuit 200 can be configured to use any of the supported activation functions using opcodes. For example, in the illustrated embodiment, the AF circuit 200 uses a 5-bit opcode to select the type of activation function desired by a particular layer or node in the DNN implementation, and the circuit can be reconfigured for other types of activation functions by simply changing the opcode value. In the illustrated embodiment, five opcode bits 202a-202e are shown as Tanh 202a, Sigmoid 202b, Swish 202c, ReLU 202d, and Leaky ReLU 202e, and these individual bit values ​​are set based on the type of activation function desired. Table 1 specifies the hardware configuration of the AF circuit 200 for the various activation functions supported based on the values ​​of the opcode bits 202a-202e. [Table 1: Activation function opcodes] [Table 1]

[0045] The operation of the AF circuit 200 varies depending on which activation function is selected via opcode bits 202a-202e, and the functionality of the AF circuit 200 is therefore further described below in conjunction with Figures 3A-3E, which show the various activation functions supported by the AF circuit 200.

[0046] Figure 3A shows a graph of the sigmoid activation function, which mathematically can be expressed as

number

[0047] In general, for a given input X represented in single precision floating point format, the sigmoid of X, or Sigmoid(X), can be calculated using the following formula:

number

number

number

[0048] 2A and 2B, the AF circuit 200 implements the sigmoid function using the simplified approach described above. For example, if the AF circuit 200 is configured for a sigmoid function, the sigmoid opcode bit (reference number 202b) is set to 1, and the remaining opcode bits of the other activation functions (reference numbers 202a, c, d, e) are set to 0. In this scheme, an input X (reference number 201) is fed into the AF circuit 200, passed through mux 206a and demux 207 to bias block 208, which applies a bias to the input X to convert it to a negative number (-X). The result -X is then sent to exponent block 230 to be multiplied by 1. -X is calculated, and then this result is sent to adder 212 to obtain 1+e -X is calculated. -X The result of this is passed through mux 206d to logarithm block 210b, which then multiplies it by log2(1+e -X ) to calculate

[0049] Separately, subtractor 211 is provided with a constant 1 as its first operand, and with the output of mux 206e as its second operand. In this case, the sigmoid opcode bit 202b provided to mux 206e is set to 1, causing mux 206e to select the constant value 0 as its output. Thus, the constant values ​​1 and 0 are provided to subtractor 211 as respective operands, and thus subtractor 211 calculates 1-0=1. The resulting value 1 is then passed through mux 206f to logarithm block 210a, which then calculates log2(1), which becomes 0.

[0050] Thus, logarithm blocks 210a and 210b are log2(1) and log2(1+e -X ), which are then sent as operands to the adder / subtractor 213. In this case, the adder / subtractor 213 performs the subtraction to obtain log2(1)-log2(1+e -X ), which is then sent to the antilogarithm block 220, where the antilog2 calculation

number

number

[0051] Additionally, as previously mentioned, the exponential, log2 and antilog2 calculations performed by the individual exponential, log and antilog blocks 210-230 of AF circuitry 200 are implemented using piecewise linear approximations to further simplify the calculations required by this alternative equation.

[0052] Figure 3B shows a graph of the hyperbolic tangent (Tanh) activation function, which mathematically can be written as

number

[0053] In general, for a given input X represented in single precision floating point format, the hyperbolic tangent of X, or Tanh(X), can be calculated using the following formula:

number

number

number

[0054] 2A and 2B, the AF circuit 200 implements the Tanh function using the simplified approach described above. For example, when the AF circuit 200 is configured for the Tanh function, the Tanh opcode bit (reference number 202a) is set to 1, and the remaining opcode bits (reference numbers 202b, c, d, e) of the other activation functions are set to 0. In this manner, when an input X (reference number 201) is input to the AF circuit 200, it first passes through a shifter 204, which left-shifts X by one bit to double its value, thus generating an output 2X. Furthermore, since the AF circuit 200 is configured for the Tanh function, the output 2X from the shifter 204 is then sent to the bias block 208 via mux 206a and demux 207. For example, the select signal for mux 206a is based on Tanh opcode bit 202a, which is set to 1, causing mux 206a to select 2X as the output to send to demux 207. In addition, the select signal for demux 207 is based on the output of OR gate 205, which receives as inputs ReLU / Leaky ReLU opcode bits 202d,e, both of which are set to 0, causing demux 207 to route the value 2X to bias block 208.

[0055] Bias block 208 then adds a bias to 2X to convert it to a negative number (−2X), and the resulting value −2X is then sent to exponent block 230, which converts it to a value e -2X The output e from the exponent block 230 -2X is then sent to both subtractor 211 (via mux 206e) and adder 212, which then subtracts 1-e -2X while adder 212 calculates the value of 1+e -2X The outputs from the subtractor 211 and the adder 212 are sent to logarithm blocks 210a and 210b, respectively, which calculate the value of log2(1-e -2X ) and log2(1+e -2X ) value.

[0056] The individual outputs from logarithm blocks 210a and 210b are then sent as operands to an adder / subtractor 213, which performs the subtraction to obtain log2(1-e -2X )-log2(1+e -2X ), which is then sent to the antilogarithm block 220, which calculates the antilog2

number

number

[0057] Additionally, as previously mentioned, the exponential, log2 and antilog2 calculations performed by the individual exponential, log and antilog blocks 210-230 of AF circuitry 200 are implemented using piecewise linear approximations to further simplify the calculations required by this alternative equation.

[0058] FIG. 3C shows a graph of the rectified linear unit (ReLU) activation function, which is mathematically represented as Y=max(0,X). ReLU is a widely used activation function that offers various advantages. Specifically, ReLU is a nonlinear function that avoids the vanishing gradient problem, is less complex than other activation functions and therefore less computationally expensive, and has favorable properties that make DNNs sparse and more efficient (e.g., if its input is negative, the output will be zero, and therefore the corresponding neuron will not be activated). On the other hand, if the output of ReLU becomes zero, the weights cannot be updated during backpropagation, and therefore ReLU can only be used in the hidden layer of a neural network.

[0059] In general, for a given input X represented in single-precision floating-point format, the ReLU of X, or ReLU(X), can be calculated using the following formula:

number

[0060] For example, returning to Figures 2A and 2B, if the AF circuit 200 is configured for a ReLU function, the ReLU opcode bit (reference number 202d) is set to 1, and the remaining opcode bits (reference numbers 202a, b, c, e) of the other activation functions are set to 0. In this scheme, when an input X (reference number 201) is provided to the AF circuit 200, X first goes through mux 206a to demux 207, which then routes X to mux 206c. Separately, a constant value of 1 is also provided to mux 206c (via mux 206b). Furthermore, since the selection signal of mux 206c is based on the sign bit of X, mux 206c selects either X or 0 as its output depending on whether X is positive or negative. Because the output of mux 206c is the final result of the ReLU function, the remaining logic of the AF circuit 200 is bypassed, and thus the output of mux 206c is ultimately used as the final output Y (reference numeral 203) of the AF circuit 200 for the ReLU function.

[0061] Figure 3D shows a graph of the leaky rectified linear unit (Leaky ReLU) activation function, which mathematically can be written as

number

[0062] In general, for a given input X represented in single-precision floating-point format, the leaky ReLU of X, or LeakyReLU(X), can be computed using the following formula:

number

[0063] For example, returning to Figures 2A and 2B, when the AF circuit 200 is configured for a leaky ReLU function, the leaky ReLU opcode bit (reference number 202e) is set to 1, and the remaining opcode bits (reference numbers 202a, b, c, d) of the other activation functions are set to 0. In this scheme, when an input X (reference number 201) is provided to the AF circuit 200, X first goes through mux 206a to demux 207, which then routes X to mux 206c. Separately, a constant value of 0.01 is also provided to mux 206c (via mux 206b). Furthermore, because the selection signal of mux 206c is based on the sign bit of X, mux 206c selects either X or 0.01 as its output, depending on whether X is positive or negative. Since the output of mux 206c is the final result of the leaky ReLU function, the remaining logic of the AF circuit 200 is bypassed, and thus the output of mux 206c is ultimately used as the final output Y (reference numeral 203) of the AF circuit 200 for the leaky ReLU function.

[0064] Figure 3E shows a graph of the Swish activation function, which is mathematically represented as Y = X * Sigmoid(X). In many cases, Swish has been proven to provide higher accuracy than other activation functions (e.g., ReLU).

[0065] In general, for a given input X represented in single precision floating point format, the Swish of X, or Swish(X), can be calculated using the following formula:

number

number

number

[0066] 2A and 2B, the AF circuit 200 implements the Swish function using the simplified approach described above. For example, when the AF circuit 200 is configured for the Swish function, the Swish opcode bit (reference number 202c) is set to 1, and the remaining opcode bits (reference numbers 202a, b, d, e) of the other activation functions are set to 0. In this manner, when an input X (reference number 201) is provided to the AF circuit 200, X goes through mux 206a to demux 207, which then routes X to bias block 208. For example, the select signal of mux 206a is based on Tanh opcode bit 202a, which is set to 0, so mux 206a selects X as the output to send to demux 207. Furthermore, since the select signal of demux 207 is based on the output of OR gate 205, which receives as inputs ReLU / Leaky ReLU opcode bits 202d,e, both of which are set to 0, demux 207 routes the value X to bias block 208.

[0067] Bias block 208 then adds a bias to X to convert it to a negative number (−X), and the resulting value −X is then sent to exponent block 230, which converts it to a value e -X The output of the exponent block 230 is e -X is then sent to adder 212 to become 1+e -X is calculated, and the result is then sent via mux 206d to logarithm block 210b, which then calculates log2(1+e -X ) to calculate

[0068] Alternatively, the selection signal of mux 206f is based on Swish opcode bit 202c, which is set to 1, causing mux 206f to select X as the output to send to logarithm block 210a, which then calculates log2(X).

[0069] The individual outputs from logarithm blocks 210a and 210b are then fed as operands to adder / subtractor 213, which performs the subtraction to obtain log2(X)-log2(1+e -X ), which is then sent to the antilogarithm block 220, which calculates the antilog2

number

number

[0070] Additionally, as previously mentioned, the exponential, log2 and antilog2 calculations performed by the individual exponential, log and antilog blocks 210-230 of AF circuitry 200 are implemented using piecewise linear approximations to further simplify the calculations required by this alternative equation.

[0071] Thus, the illustrated embodiment of the AF circuit 200 of Figures 2A and 2B provides many advantages, including low latency, high accuracy, and reduced power consumption, with a flexible, small-area hardware design that supports multiple activation functions and is highly scalable and portable. Specifically, the AF circuit 200 is an integrated solution that implements multiple DNN activation functions on a single hardware component (e.g., without using separate hardware components for each activation function) without relying on lookup tables. For example, in the illustrated embodiment, the AF circuit 200 is implemented using log, antilog, and exponential circuits 210, 220, 230 that perform log2, antilog2, and exponential calculations using piecewise linear approximations, thereby eliminating the need for lookup tables in the hardware design and reducing the required multiplier circuits.

[0072] In this manner, the illustrated embodiment significantly reduces the required silicon area, power consumption and latency of the hardware while still providing high accuracy. For example, the elimination of lookup tables and reduction of multiplier circuits translates directly into significant silicon area savings, as far fewer synthesis gates are required compared to a typical lookup table approach with similar accuracy. Furthermore, the log2 and antilog2 calculations can be performed in a single clock cycle, while the exponential calculations can be performed in two clock cycles, allowing the activation function to be calculated in only five clock cycles. Furthermore, the basic calculations can be easily pipelined to improve throughput.

[0073] The AF circuit 200 also eliminates the dependency on software to load / program lookup tables associated with different activation functions, because the AF circuit 200 can be configured for different activation functions by simply programming the appropriate opcode. Programming an opcode on the AF circuit 200 is much simpler than programming a lookup table for an activation function, and requires fewer clock cycles.

[0074] The AF circuit 200 is also highly scalable. Specifically, the flexible implementation of the AF circuit 200 allows the underlying hardware to be replicated as needed to increase the number of parallel operations supported. In this manner, the AF circuit 200 can be easily scaled to support the number of parallel operations required by a particular application or use case. The precision of the AF circuit 200 can also be scaled based on the requirements of the application. For example, if an application requires higher precision, the number of segments in the piecewise linear approximation model used by the logarithm, antilogarithm, and exponent circuits 210, 220, 230 can be increased to address the precision requirement. In this manner, the AF circuit 200 is also highly portable, as it can be easily ported and / or scaled for any product or form factor, including mobile devices (e.g., handheld or wearable devices), drones, servers, and / or any other artificial intelligence solution that requires DNN operations without dependency or modification.

[0075] 4 illustrates an alternative embodiment of an integrated activation function (AF) circuit 400 for an artificial neural network (e.g., a deep neural network (DNN)). Specifically, the AF circuit 400 is similar to the AF circuit 200 of FIGS. 2A and 2B, except that the predetermined activation functions are implemented using modified formulas that use powers of 2 rather than powers of the exponential constant e. For illustrative purposes, the original and modified formulas for the sigmoid, swish, and hyperbolic tangent activation functions are provided in Table 2. [Table 2: Modified activation function formula using powers of 2] [Table 2]

[0076] As shown in Table 2, the base-e exponents in the original formula are replaced with base-2 exponents in the modified formula. In this manner, the modified formula still exhibits important nonlinear properties (e.g., curve shape) of the activation function, but the basic activation function hardware can be implemented much more efficiently. Specifically, by replacing the base-e exponents with base-2 exponents, the exponential circuitry is no longer needed in the modified formula, as all of the exponential operations can be performed by the inverse logarithm circuitry. For example, the base-2 inverse logarithm of a variable x is 2 to the power of x, (2 x ), an inverse logarithm circuit that performs an inverse logarithm operation in base 2 can be used to calculate the powers of base 2 that appear in the modified activation function formula.

[0077] Furthermore, by performing the exponentiation operation using an inverse logarithm circuit rather than an exponentiation circuit, both the latency and silicon area of ​​the AF circuit 400 are reduced. By way of comparison, for example, the AF circuit 200 of FIGS. 2A and 2B performs the exponentiation operation using an exponentiation circuit implemented using a piecewise linear approximation (e.g., the exponentiation circuit of FIG. 7), which allows the exponentiation operation to be performed in two clock cycles and requires at least one multiplier. However, the AF circuit 400 performs the exponentiation operation using an inverse logarithm circuit implemented using a piecewise linear approximation (e.g., the inverse logarithm circuit of FIGS. 6A-6C), which allows the base 2 inverse logarithm operation to be performed in a single clock cycle and does not require a multiplier. Thus, by replacing the exponentiation circuit with the inverse logarithm circuit, the overall latency of the AF circuit 400 is reduced by one clock cycle, thereby allowing the activation function to be calculated in only four clock cycles, compared to five clock cycles for the activation function circuit of FIGS. 2A and 2B. Furthermore, the AF circuit 400 no longer requires any multiplier circuitry, and the eliminated exponent circuitry was the only component requiring a multiplier, resulting in significant silicon area savings. For example, the AF circuit 200 of Figures 2A and 2B can be implemented using 8,321 gates, while the AF circuit 400 can be implemented using only 7,221 gates.

[0078] Additionally, similar to the AF circuit 200 of FIGS. 2A and 2B, the AF circuit 400 utilizes base 2 logarithm (log2) and base 2 antilogarithm (antilog2) calculations with piecewise linear approximations to simplify the calculations required for a given activation function. For example, log2 and antilog2 calculations can be used to eliminate complex division and / or multiplication operations required for a given activation function, converting them instead to subtraction and / or addition. Table 3 sets forth the log / antilog equations for the modified sigmoid, swish and hyperbolic tangent activation functions (using powers of 2 instead of e) of Table 2. These log / antilog equations for the modified activation functions are derived in a manner similar to that of the original activation functions as described in connection with FIGS. 3A-3E. [Table 3: Log / antilog versions of modified activation functions (using powers of 2)] [Table 3]

[0079] In the illustrated embodiment, the AF circuit 400 is designed to implement Sigmoid, Swish, Tanh, ReLU and leaky ReLU activation functions. The Sigmoid, Swish and Tanh activation functions are implemented using the log / antilog equations in Table 3, while the ReLU and leaky ReLU activation functions are implemented using the original equations in Figures 3C-3D, since they do not require complex division, multiplication or exponential operations. The rest of the operation of the AF circuit 400 is similar to that of the AF circuit 200 in Figures 2A and 2B.

[0080] [Logarithmic, antilogarithmic and exponential circuits implemented using piecewise linear approximation] 5A-5C illustrate an example embodiment of a logarithmic circuit 500 implemented using piecewise linear approximation. In particular, FIG. 5A illustrates a general implementation of logarithmic circuit 500, and FIGs. 5B and 5C illustrate implementations of certain components of logarithmic circuit 500.

[0081] Logarithm circuit 500 uses a 16-segment piecewise linear approximation to perform the logarithm calculations. In this manner, no lookup table or multiplier circuitry is required for logarithm circuit 500, and the logarithm calculations can be performed in a single clock cycle. The equations used by logarithm circuit 500 to perform the piecewise linear approximation for the logarithm calculations are shown below in Table 4. [Table 4: Piecewise linear approximation equation for log2(1+m)] [Table 4]

[0082] The formulas in Table 4 are designed to calculate or approximate the result of log2(1+m), where m represents the mantissa portion of the single precision floating point input 501. For example, since the mantissa m is always between 0 and 1, and the result of log2(0) is indeterminate, log2(1+m) is calculated instead of log2(m) to avoid the indeterminate result when m is 0. Furthermore, to calculate log2(1+m) using a 16-segment piecewise linear approximation, the potential values ​​of m over the interval [0,1] are divided into 16 different ranges or segments and designated as the range 0 to 15, and a separate equation is defined for each range to approximate the result of log2(1+m). Furthermore, the individual equations are defined using addition and / or subtraction exclusively to any of the following types of operands: m, fractions and / or constant values ​​of m divided by powers of 2. In this scheme, the divisions required by the equations are exclusively by powers of two, and therefore all division operations can be implemented using shifters. Furthermore, the loss of precision resulting from restrictive "shift-based" division is compensated for by the use of constant values ​​that are added and / or subtracted in a given equation. Thus, individual equations can be implemented exclusively using addition, subtraction and / or shift operations, thus eliminating the need for complex multiplication / division circuitry.

[0083] 5A shows the overall logic of a logarithm circuit 500 designed to implement the equations of Table 4. In the illustrated embodiment, a 32-bit single precision floating point number is provided as input 501 to logarithm circuit 500 (e.g., provided via an input register), and logarithm circuit 500 calculates a corresponding 32-bit fixed point number representing the log2 value of input 501 as output 503 (e.g., returned via an output register).

[0084] Input 501 includes a sign bit (input

[31] ), an 8-bit exponent e (input[30:23]), and a 23-bit mantissa m (input[22:0]). Given that the sign of input 501 and the sign of output 503 always match, the sign bit (input

[31] ) of input 501 is fed directly to the corresponding bit (output

[31] ) of output 503. Furthermore, the exponent e (input[30:23]) of input 501 is fed to an 8-bit subtractor 502, which subtracts a bias of 0x7F from the exponent e to generate a corresponding 8-bit unbiased exponent. For example, from a mathematical point of view, when subtracting the bias from the exponent of a floating-point number, the result is always a value equal to log2 of the exponent. The resulting unbiased exponent therefore serves as the integer part (output[30:23]) of the fixed-point number represented in output 503.

[0085] Additionally, the mantissa m of input 501 is used to select a corresponding range and equation from Table 4 that will be used to calculate the fraction field (output[22:0]) of output 503. For example, the four most significant bits of mantissa m (input[22:19]) are provided as inputs to range selection logic 504, which outputs 16 1-bit signals (range[0] through range

[15] ) corresponding to the individual ranges of m in Table 4, such that the signals corresponding to the applicable ranges are set to 1 while the remaining signals are set to 0.

[0086] Based on the output of range selection logic 504, multiplexers (mux) 508a-508c are then used to select operands corresponding to the selected equation from Table 4, which are then provided as inputs to adder / subtractor logic 520. Specifically, muxes 508a-508c select various fractions of mantissa m (e.g., generated using a shift operation), as well as predetermined constant values. For example, mantissa m is provided to a number of shifters 506, which each perform a right shift of m over a predetermined number of bits to generate various fractions of m by powers of 2 (e.g.,

number

number

[0087] Each mux 508a-508c then selects (e.g., based on range selection logic 504) an appropriate output that corresponds to one of the operands of the applicable equations in Table 4, and these outputs are then provided as inputs to adder / subtractor logic 520. In addition, the mantissa m is also provided directly as another input to adder / subtractor logic 520, since it is an operand for all equations in Table 4.

[0088] The adder / subtractor logic 520 then performs the appropriate addition and / or subtraction operations on the various operands provided as inputs, with the result of the calculation then serving as the 23-bit fractional field (output[22:0]) of output 503. An implementation of the adder / subtractor logic 520 is further illustrated and described below in connection with FIG.

[0089] Figure 5B shows one implementation of the constant generation logic 510 of the logarithm circuit 500 of Figure 5A used to generate the constant values ​​required by certain equations in Table 4. In the illustrated embodiment, the constant generation logic 510 includes a shifter 512 for generating a set of constant values ​​that appear throughout the equations in Table 4, multiplexers (mux) 514a, 514b for selecting corresponding constant values ​​of selected equations in Table 4, and an adder 516 for adding the selected constant values.

[0090] In the illustrated embodiment, each shifter 512 is provided as an input with a 23-bit constant value that is either +1 or −1 (e.g., depending on whether the fractional constant that each shifter 512 produces is positive or negative). For example, the 23-bit constant value that is +1 is provided as an input to all but one shifter that produces a positive result, while a 23-bit signed representation that is −1 is provided as an input to the only remaining shifter that produces a negative result (e.g., one that performs a right shift of 9 bits).

number

number

[0091] Furthermore, since each equation in Table 4 requires either zero, one or two of these fractional constants, the appropriate combination of fractional constants for the equation selected from Table 4 is selected using two muxes 514a, 514b. Specifically, the output of the shifter 512 is provided as an input to the two muxes 514a, 514b in the manner shown in FIG. 5B, and a 23-bit constant value of 0 is also provided as an input to each mux 514a, 514b. Each mux 514a, 514b then outputs either a particular fractional constant or a value of 0. In this manner, the muxes 514a, 514b can collectively output zero, one or two of the individual fractional constants generated by the shifter 512, depending on the particular equation selected from Table 4 (e.g., as determined using the range selection logic 504 of the logarithm circuit 500 of FIG. 5A).

[0092] The outputs of muxes 514a, 514b are then provided as inputs to adder 516, which calculates their sum. The result from adder 516 then serves as the final constant value 511 output by constant generation logic 510 of FIG. 5B.

[0093] Figure 5C shows an example implementation of the adder / subtractor logic 520 of the logarithm circuit 500 of Figure 5A used to perform additions and / or subtractions on the operands of an equation selected from Table 4. In the illustrated embodiment, the adder / subtractor logic 520 includes two adders / subtractors 522a, 522b and two OR gates 524a, 524b, which are described in more detail below.

[0094] A first adder / subtractor 522a is provided with as its operands the mantissa m and the outputs of muxes 508b and 508c of logarithm circuit 500. Furthermore, the particular combination of additions and / or subtractions performed on these operands by adder / subtractor 522a is dictated by OR gate 524a. For example, OR gate 524a is provided with signals corresponding to ranges 11 and 15 from Table 4 (e.g., as generated by range selection logic 504 of logarithm circuit 500) as inputs, and the output of OR gate 524a is then provided to adder / subtractor 522a. In this scheme, when the output of OR gate 524a is 0, adder / subtractor 522a sums all of the operands, but when the output of OR gate 524a is 1 (e.g., when mantissa m is in either range 11 or range 15), adder / subtractor 522a subtracts the operand corresponding to the output of mux 508b and adds the remaining operand. In other words, for any range of m in Table 4 except ranges 11 and 15, adder / subtractor 522a outputs the result of m + [output of mux 508b] + [output of mux 508c], but for ranges 11 and 15, adder / subtractor 522a outputs the result of m - [output of mux 508b] + [output of mux 508c].

[0095] The second adder / subtractor 522b is provided with the output of the first adder / subtractor 522a and the output of mux 508a of logarithm circuit 500 as its operands. The output of OR gate 524b indicates whether adder / subtractor 522b performs addition or subtraction on these operands. For example, OR gate 524b is provided with a signal corresponding to ranges 7-15 in Table 4 as an input (e.g., as generated by range selection logic 504 of logarithm circuit 500), and the output of OR gate 524b is then provided to adder / subtractor 522b. In this scheme, when the output of OR gate 524b is 0, adder / subtractor 522b sums both operands, but when the output of OR gate 524b is 1 (e.g., when mantissa m is in the range 7-15), the output of mux 508a is subtracted from the output of first adder / subtractor 522a. In other words, when m is in the range 0-6 of Table 4, adder / subtractor 522b computes [output of first adder / subtractor 522a]+[output of mux 508a], but when m is in the range 7-15, adder / subtractor 522b computes [output of first adder / subtractor 522a]-[output of mux 508a]. The result of second adder / subtractor 522b serves as the final output 521 of adder / subtractor logic 520 of FIG. 5C.

[0096] To illustrate the operation of logarithm circuit 500, a processing flow for example input 501 will be described. Since the sign and exponent fields of input 501 are always processed in the same way regardless of their underlying values, this example focuses on the processing associated with the mantissa m of input 501. In this example, we assume that mantissa m (input[22:0]) of input 501 is within the range 0.125≦m<0.1875, which corresponds to range 2 in Table 4. Thus, logarithm circuit 500 performs the processing according to the corresponding equation for range 2 in Table 4, i.e.

number

[0097] The first operand m, which corresponds to the mantissa field (input[22:0]) of input 501 , is provided directly from the associated bit field of input 501 to adder / subtractor logic 520 .

[0098] The remaining operands of the equations referenced above are provided to adder / subtractor logic 520 by muxes 508a-508c. Specifically, a collection of operands appearing throughout the various equations of Table 4 are provided as inputs to muxes 508a-508c, which then output the particular operand required for the appropriate equation of Table 4. For example, based on the four most significant bits of mantissa m, range selection logic 504 outputs a signal that identifies the particular range of Table 4 that m falls within, and this range signal is used by muxes 508a-508c to select the appropriate operand to be output. In this example, since m is within range 2 of Table 4, range selection logic 504 outputs a range signal, otherwise denoted as range[2] signal, corresponding to range 2. Based on the range[2] signal, mux 508a

number

number

number

number

[0099] In this manner, the following operands are ultimately provided to the adder / subtractor logic 520: m: (supplied directly from input 501),

number

number

number

[0100] 6A-6C illustrate an example embodiment of an inverse logarithm circuit 600 implemented using piecewise linear approximation. Specifically, Figures 6A and 6B illustrate an alternative implementation of the entire inverse logarithm circuit 600, and Figure 6C illustrates the basic adder / subtractor logic 620 of the inverse logarithm circuit 600.

[0101] The inverse logarithm circuit 600 uses a 16-segment piecewise linear approximation to perform the inverse logarithm calculation. In this manner, no lookup table or multiplier circuitry is required for the inverse logarithm circuit 600, and the inverse logarithm calculation can be performed in a single clock cycle. The equations used by the inverse logarithm circuit 600 to perform the piecewise linear approximation for the inverse logarithm calculation are shown in Table 5 below. [Table 5: antilog2(η)=2 η Piecewise linear approximation equation of [Table 5]

[0102] The equation in Table 5 is the antilog2 of η, i.e., 2 to the power of η, or 2 η , where η represents the fractional portion of the fixed-point input number 601 (input[22:0]). For example, the fixed-point input 601 of the antilogarithm circuit 600 is typically derived from an intermediate DNN calculation on the fixed-point output 503 of the logarithm circuit 500 of FIGS. 5A-5C. Furthermore, as discussed above, the fractional portion of the fixed-point output 503 of the logarithm circuit 500 is calculated as log2(1+m) rather than log2(m) to avoid indeterminate results when m=0. Thus, antilog2(η)=2 η The equations in Table 5 for calculating m are designed to produce a value equal to 1+m, as reflected by the constant value +1 in each equation.

[0103] antilog2(η)=2 η To calculate, say, a 16-segment piecewise linear approximation, the potential values ​​of η over the interval [0,1] are divided into 16 different ranges or segments, specified as the range 0 to 15, and the antilog2(η) or 2 ηA separate equation is defined for each range to approximate the result of . Furthermore, each equation is defined using exclusively additions and / or subtractions on any of the following types of operands: η, fractions of η divided by powers of 2, and / or constant values. In this manner, the divisions required by the equations are exclusively by powers of 2, and thus all division operations can be implemented using shifters. Furthermore, the loss of precision resulting from the restrictive "shift-based" division is compensated for by the use of constant values ​​that are added and / or subtracted in a given equation. Thus, each equation can be implemented exclusively using additions, subtractions, and / or shift operations, thus eliminating the need for complex multiplication / division circuitry.

[0104] 6A shows the overall logic of an inverse logarithm circuit 600 designed to implement the equations of Table 5. In the illustrated embodiment, a 32-bit fixed-point number is provided to the inverse logarithm circuit 600 as input 601 (e.g., provided via an input register), and the inverse logarithm circuit 600 calculates a corresponding 32-bit floating-point number representing the antilog2 result of input 601 as output 603 (e.g., returned via an output register).

[0105] Input 601 includes a sign bit (input

[31] ), an 8-bit integer part (input[30:23]), and a 23-bit fractional part (input[22:0]). Given that the sign of input 601 and the sign of output 603 always match, the sign bit (input

[31] ) of input 601 is fed directly to the corresponding bit (output

[31] ) of output 603. The integer part (input[30:23]) of input 601 is fed to an 8-bit adder 602, which adds back a bias of 0x7F to generate an 8-bit biased exponent that serves as the exponent field of floating-point output 603.

[0106] Additionally, the fractional portion of input 601 (input[22:0]), which corresponds to the value of η in Table 5, is used to select from Table 5 the corresponding range and formula used to calculate the mantissa of floating-point output 603 (output[22:0]). For example, the four most significant bits (input[22:19]) of the fractional portion of input 601 are provided as inputs to range selection logic 604, which outputs 16 one-bit signals (range[0] through range

[15] ) corresponding to the respective ranges of η in Table 5, such that the signals corresponding to the applicable ranges are set to 1 while the remaining signals are set to 0. Based on the output of range selection logic 604, multiplexers (mux) 608a through 608d are then used to select predetermined operands, such as the required fraction of η and a fractional constant, required by the corresponding formula in Table 5.

[0107] Specifically, muxes 608a and 608b are used to select the fraction of η required by the corresponding equation in Table 5. For example, the value of η (input [22:0]) is provided to a first set of shifters 606a, which each perform a right shift of η over a predetermined number of bits to produce the various fractions of η by powers of 2 that appear throughout the equations in Table 5 (e.g.,

number

[0108] Similarly, muxes 608c and 608d are used to select the fractional constants required by the corresponding equations in Table 5. For example, a 23-bit constant having a value of 1 is provided to a second set of shifters 606b, which each perform a right shift by a predetermined number of bits to generate the individual fractional constants (e.g.,

number

[0109] Each operand selected by muxes 608a-608d for the corresponding equation in Table 5 is then provided as an input to adder / subtractor logic 620. The value of η (input[22:0]) and a constant value 1 are also provided as inputs to adder / subtractor logic 620 because these values ​​are operands in all of the equations in Table 5. Adder / subtractor logic 620 then performs the appropriate addition and / or subtraction operations on these operands (e.g., based on the output of range selection logic 604) as required by the corresponding equation in Table 5. The result from adder / subtractor logic 620 then serves as the 23-bit mantissa portion of the floating-point output 603 (output[22:0]) of antilogarithm circuit 600. An implementation of adder / subtractor logic 620 is further illustrated and described below in connection with FIG. 6C.

[0110] FIG. 6B illustrates an alternative implementation 600 of an inverse logarithm circuit implemented using piecewise linear approximation. Specifically, whereas the inverse logarithm circuit of FIG. 6A is designed to calculate the inverse logarithm of a fixed-point number as input, the inverse logarithm circuit of FIG. 6B can calculate the inverse logarithm of either a fixed-point number or a floating-point number as input. In this manner, the inverse logarithm circuit of FIG. 6B can calculate the antilog2 of either a fixed-point number in the log2 domain (e.g., derived from the output of the logarithm circuit 500 of FIGS. 5A-5C) or a floating-point number in its original domain. For example, as described in connection with FIG. 4, the activation function (AF) circuit 400 uses the inverse logarithm circuit for two purposes: (1) to convert a fixed-point number in the log2 domain back to a floating-point number in the original domain, and (2) to calculate the power of a base-2 floating-point exponent. Thus, the operands of the inverse logarithm circuit of AF circuit 400 include both fixed-point numbers (e.g., for log2 domain transformations) and floating-point numbers (e.g., for powers of base 2). Thus, the inverse logarithm circuit of AF circuit 400 can be implemented using the inverse logarithm circuit of FIG. 6B, which can process inputs represented as either fixed-point or floating-point numbers.

[0111] The operation of the antilogarithm circuit of FIG. 6B is similar to that of FIG. 6A, except for the way in which it processes the integer or exponent field of input 601 (input[30:23]), as will be described in more detail below.

[0112] Specifically, if input 601 is a fixed-point number, it includes an integer field (input[30:23]) and is provided as one of the inputs to mux 616. The select signal of mux 616 is a binary signal that indicates whether input 601 is a fixed-point number. Thus, the select signal of mux 616 has a value of 1 if input 601 is a fixed-point number, which causes mux 616 to select the integer portion of input 601 (input[30:23]) as its output, which is then provided as an operand to 8-bit adder 602. Adder 602 then adds back a bias of 0x7F to the integer portion of input 601 (input[30:23]) to generate an 8-bit biased exponent, which then serves as the exponent portion of floating-point output 603.

[0113] Alternatively, if input 601 is a floating point number, it will include an exponent field (input[30:23]). The exponent field (input[30:23]) is provided to an 8-bit subtractor 612 which subtracts a bias of 0x7F to generate a corresponding 8-bit unbiased exponent which mathematically represents the log2 value of the exponent. The output of subtractor 612 is then provided to shifter 614 which performs a left shift by the unbiased exponent power of 2 to calculate the corresponding antilog2 value which is then provided as one of the inputs to mux 616. The select signal of mux 616 has a value of 0 when input 601 is a floating point number, causing mux 616 to select the value from shifter 614 as its output. The output of mux 616 is then provided as an operand to 8-bit adder 602 , which adds back a bias of 0x7F to generate an 8-bit biased exponent, which then serves as the exponent portion of floating-point output 603 .

[0114] Figure 6C shows an example implementation of the adder / subtractor logic 620 of the antilogarithm circuit 600 of Figures 6A and 6B used to perform additions and / or subtractions on the operands of the corresponding equations in Table 5. In the illustrated embodiment, the adder / subtractor logic 620 includes three adders / subtractors 622a-622c and three OR gates 624a-624c, which are described in more detail below.

[0115] A first adder / subtractor 622a is provided with the following operands as inputs: the value of η (inputs [22:0]), a constant value of 1, and the output of mux 608a. The particular combination of additions and / or subtractions to be performed on these operands is dictated by OR gate 624a, which is provided with signals in the range [0] to [7] as inputs. In this manner, if η is within the range 0 to 7 of Table 5, the output of OR gate 624a will be 1, causing adder / subtractor 622a to compute η+1-[output of mux 608a], or if η is within the range 8 to 15 of Table 5, the output of OR gate 624a will be 0, causing adder / subtractor 622a to compute η+1+[output of mux 608a],.

[0116] A second adder / subtractor 622b is provided with the following operands as inputs: the output of mux 608c and the output of mux 608d. The particular combination of additions and / or subtractions to be performed on these operands is dictated by an OR gate 624b, which is provided with signals in the ranges [4], [8], [9],

[11] , and

[12] as inputs. In this manner, if η is within ranges 4, 8, 9, 11, or 12 of Table 5, the output of OR gate 624b will be 1, which causes adder / subtractor 622b to compute [output of mux 608c]-[output of mux 608d]. Alternatively, if η is within any of the remaining ranges of Table 5, the output of OR gate 624b will be 0, which causes adder / subtractor 622b to compute [output of mux 608c]+[output of mux 608d].

[0117] A third adder / subtractor 622c receives as input the following operands: the output of adder / subtractor 622a, the output of adder / subtractor 622b, and the output of mux 608b. The particular combination of additions and / or subtractions to be performed on these operands is dictated by OR gate 624c, which receives as input the inverse or NOT values ​​of signals in ranges [2],

[10] ,

[11] ,

[14] , and

[15] . In this manner, if η is within any range other than ranges 2, 10, 11, 14, and 15 in Table 5, the output of OR gate 624c will be 1, which causes adder / subtractor 622c to compute [output of adder / subtractor 622a]-[output of adder / subtractor 622b]-[output of mux 608b]. Alternatively, if η is within the ranges 2, 10, 11, 14, or 15 of Table 5, the output of OR gate 624c will be 0, causing adder / subtractor 622c to calculate [output of adder / subtractor 622a]-[output of adder / subtractor 622b]+[output of mux 608b].

[0118] The result of the third adder / subtractor 622c serves as the final output 621 of the adder / subtractor logic 620 of FIG. 6C.

[0119] 7 illustrates an example embodiment of an exponent circuit 700 implemented using piecewise linear approximation. Specifically, the exponent circuit 700 performs the exponent calculation using a 16-segment piecewise linear approximation. In this manner, no lookup table is required for the exponent circuit 700, only one multiplier is required, and the exponent calculation can be performed in two clock cycles.

[0120] The equations used by the exponent circuit 700 to perform the piecewise linear approximation for the exponent calculation are shown in Table 6 below. [Table 6: Index e x Piecewise linear approximation equation of [Table 6]

[0121] The equations in Table 6 are natural exponential functions e x where e represents a natural exponential constant (e.g., Euler's number) and x represents the 23-bit mantissa (input[22:0]) of a 32-bit floating-point input 701. For example, e x To calculate x using a 16-segment piecewise linear approximation, the potential values ​​of x across the interval [0,1] are divided into 16 distinct ranges or segments, designated as the range 0 to 15, and e x A separate equation is defined for each range to approximate the result of e. Furthermore, each equation is defined using additions and / or subtractions exclusively to the following types of operands: x, 2x, fractions of x divided by powers of 2, and / or constant values. In this manner, the divisions required by the equations are exclusively by powers of 2, and thus all division operations can be implemented using shifters. Furthermore, the loss of precision resulting from the restrictive "shift-based" division is compensated for by the use of constant values ​​that are added and / or subtracted in a given equation. Thus, e x The individual equations of can be implemented using exclusively addition, subtraction and / or shift operations.

[0122] To complete the exponentiation operation, we then use the resulting exponentiation function e , which represents the power of the mantissa portion of input 701 (input [22:0]) of e . x must be multiplied by the value of the exponent part of input 701 of e (input [30:23]). Thus, the exponentiation operation requires only one multiplication operation, and thus exponentiation circuit 700 requires only one multiplier circuit.

[0123] 7 shows the overall logic of an exponent circuit 700. In the illustrated embodiment, the exponent circuit 700 is provided with a 32-bit floating point number as input 701, and the exponent circuit 700 calculates the natural exponential constant e to be raised to the floating point number represented by input 701, i.e., e input , the corresponding 32-bit fixed-point number that corresponds to the input signal is calculated as output 703.

[0124] Floating-point input 701 includes a sign bit (input

[31] ), an 8-bit exponent (input[30:23]), and a 23-bit mantissa (input[22:0]). The sign bit (input

[31] ) of input 701 is fed directly to the corresponding bit (output

[31] ) of output 703, given that the sign of input 701 and the sign of output 703 always match.

[0125] The exponent portion of input 701 (input[30:23]) is fed to an 8-bit subtractor 702, which subtracts a bias of 0x7F to produce an 8-bit unbiased exponent. unbiased_exponent The value of , is then output by mux710. For example, an 8-bit unbiased exponent is 2 8 = 256 potential values, ranging from -128 to +127. The value of e raised to all 256 potential values ​​of the unbiased exponent (e -128 , e -127 , ..., e 0 , e 1 , ..., e 127 ) is pre-computed and provided as a constant input to mux 710. The 8-bit unbiased exponent output by subtractor 702 serves as a select signal for mux 710, which causes mux 710 to select e unbiased_exponent Then, select the precomputed constant input corresponding to the output of mux710 (e unbiased_exponent ) is provided to multiplier 712 as one of its operands, as described further below.

[0126] The mantissa portion of input 701 (input[22:0]) is processed according to the equations in Table 6. For example, exponent circuit 700 generates the natural exponential constant e raised to the mantissa (input[22:0]), i.e., e x , where x represents the mantissa (inputs [22:0]). Additionally, the exponent circuit 700 is designed to compute e x is calculated using piecewise linear approximation based on the equations in Table 6. Specifically, the mantissa part (input [22:0]) of input 701 (corresponding to the value of x in Table 6) is calculated from Table 6 as follows:x For example, the four most significant bits (input[22:19]) of the mantissa of input 701 are provided as inputs to range selection logic 704, which outputs sixteen one-bit signals (range[0] through range

[15] ) corresponding to the respective ranges of x in Table 6, such that the signals corresponding to the applicable ranges are set to 1 while the remaining signals are set to 0. Based on the output of range selection logic 704, multiplexers (mux) 708a through 708d are then used to select the desired operands required by the corresponding equation in Table 6, such as the required fraction of x and the constant values ​​that appear in the equation.

[0127] For example, muxes 708a and 708b are used to select the fraction of x required by the corresponding equation in Table 6. For example, the value of x (input [22:0]) is provided to a first set of shifters 706a, which each perform a right shift of x by a predetermined number of bits to produce various fractions of x by powers of 2 that appear throughout the equations in Table 6 (e.g.,

number

[0128] Similarly, muxes 708c and 708d are used to select the fractional constants and / or constant values, such as the constant value 1, required by a given equation, as required by the corresponding equation in Table 6. For example, a 23-bit constant having a value of 1 is provided to a second set of shifters 706b, which each perform a right shift by a predetermined number of bits to generate individual fractional constants (e.g.,

number

[0129] Each operand selected by muxes 708a-708d for the corresponding equation in Table 6 is then provided as an input to adder / subtractor 720. The value of x (input [22:0]) is also provided to adder / subtractor 720 via either one or two of its inputs, depending on whether the corresponding equation in Table 6 requires an operand of x or 2x. For example, the value of x is always provided directly as one of the inputs to adder / subtractor 720, and may optionally be provided as another input to adder / subtractor 720 via mux 709. Specifically, the value of x and a 23-bit constant 0 are provided as inputs to mux 709, which selects one of these values ​​to provide as an input to adder / subtractor 720. For example, if the corresponding equation in Table 6 requires x rather than 2x as an operand, then mux 709 will select the constant 0 as the output to adder / subtractor 720 because the value of x is already provided directly through another input of adder / subtractor 720. Alternatively, if the corresponding equation in Table 6 requires 2x rather than x as an operand, then mux 709 will select the value of x as the output to adder / subtractor 720 because, as a result, the value of x is provided to adder / subtractor 720 through two inputs.

[0130] Adder / subtractor 720 then performs the appropriate addition and / or subtraction operations on these operands (e.g., based on the output of range selection logic 704) as required by the corresponding equations in Table 6. In this manner, the output of adder / subtractor 720 is e, which is equal to the natural exponential constant e raised to the mantissa of input 701 (input[22:0]). xThis matches the final result.

[0131] The output of mux 710 (e raised to the unbiased exponent of input 701) and the output of adder / subtractor 720 (e raised to the mantissa of input 701) are then provided as operands to multiplier 712, which multiplies these values ​​to produce the integer and fractional portions of the fixed-point output 703 of exponent circuit 700.

[0132] 8 illustrates a flowchart 800 of an example of a processing architecture used to implement an artificial neural network (e.g., a deep neural network (DNN)). Flowchart 800 may be implemented, for example, using embodiments and functions described throughout this disclosure. For example, in some embodiments, flowchart 800 may be implemented using the activation circuit of FIGS. 2A and 2B or 4, the logarithm circuit of FIGS. 5A-5C, the anti-logarithm circuit of FIGS. 6A-6C, and / or the exponential circuit of FIG. 7.

[0133] In this illustrated flowchart, a particular activation function is executed by an activation function circuit designed to accelerate the performance of the activation function. In some embodiments, for example, the activation function circuit may be designed to support multiple types of activation functions commonly used to implement artificial or deep neural networks. Furthermore, the activation function circuit may utilize logarithmic, antilogarithmic and / or exponential circuits implemented with piecewise linear approximations to accelerate the calculations associated with the supported activation functions.

[0134] For example, in some embodiments, the activation function circuit may be implemented on and / or used in connection with a processor, device and / or system that executes an application using an artificial neural network (e.g., a deep neural network, a convolutional neural network, a feedforward neural network, a recurrent neural network, etc.). In this manner, the application may utilize the activation function circuit to accelerate the activation function used to implement the artificial neural network. For example, an application having an artificial neural network may be stored in memory and executed by a processor on a particular device or system. When the application needs to execute an activation function in connection with an operation in the artificial neural network, the application may issue a corresponding instruction or command to the processor and / or the activation function circuit, which may then execute the activation function using the activation function circuit. The result of the activation function may then be provided back to the processor and / or the application and subsequently used by the artificial neural network.

[0135] The flowchart begins at block 802, where an instruction or command to execute a particular activation function is received. The instruction or command may be issued, for example, by an application and received by a processor and / or activation function circuitry. In some embodiments, the instruction or command may identify a desired activation function and any operands or other parameters associated with the activation function.

[0136] Further, in some cases, the selected activation function may include a combination of exponentiation, multiplication, and / or division operations. Thus, the flowchart first proceeds to block 804 to perform any exponentiation operations associated with the activation function. In some embodiments, for example, the exponentiation operations may be performed using piecewise linear approximations to reduce the latency associated with these operations (e.g., using the exponential circuit of FIG. 7). However, in some embodiments, if the exponentiation operation involves base 2, this can be calculated using an inverse logarithm circuit (using a base 2 inverse logarithm operation) instead of the exponential circuit to reduce latency.

[0137] The flowchart then proceeds to block 806 to perform any multiplication and / or division operations associated with the activation function. In some embodiments, for example, the multiplication and / or division operations of the activation function may be performed using a combination of logarithm, inverse logarithm, and addition / subtraction operations. For example, by utilizing the properties of the logarithm function, logarithm and inverse logarithm operations can be used to convert costly multiplication / division operations into addition and / or subtraction. Furthermore, the logarithm and inverse logarithm operations can be performed using piecewise linear approximations to reduce the latency associated with these operations.

[0138] In some embodiments, a logarithm circuit (e.g., the logarithm circuit of FIG. 5A-5C) may be used, for example, to perform a logarithmic operation using a piecewise linear approximation. For example, the logarithm circuit may be designed to calculate the logarithm of an input represented as a floating-point number (e.g., with an exponent and a mantissa), and the logarithm circuit may represent the resulting output as a fixed-point number (e.g., with an integer and a fraction). The logarithm circuit first identifies an input or operand associated with the logarithm operation, and then identifies or determines a particular range within which the input falls for the purpose of the piecewise linear approximation. For example, to perform a logarithmic operation using a piecewise linear approximation, multiple ranges or segments and associated equations are defined. Thus, a corresponding range within which the input falls is identified, and the operands required by the equation for that range are obtained and / or generated. In some embodiments, for example, a given operand may be extracted, obtained and / or calculated using an input, such as a mantissa of the input, and a fractional operand with a denominator that is a power of two, which may be generated using a shift circuit (e.g., by shifting bits to perform a division). An appropriate operand is then selected using one or more multiplexers, and the result of the equation is calculated using the selected operand. For example, the result may be calculated by performing addition and / or subtraction on the individual operands using an adder / subtractor circuit. Furthermore, the exponent of the floating-point input may be converted to an unbiased exponent by subtracting a bias using a subtractor circuit. The output of the logarithmic operation is then generated using the corresponding equation result and the unbiased exponent. For example, the unbiased exponent serves as the integer part of the resulting fixed-point output, and the equation result serves as the fractional part of the fixed-point output.

[0139] Similarly, an inverse logarithm circuit (such as the inverse logarithm circuit of FIGS. 6A-6C) may be used to perform an inverse logarithm operation using a piecewise linear approximation. The inverse logarithm operation may be performed in a similar manner, except using different calculations for the integer / exponential portion and using different equations for the piecewise linear approximation.

[0140] The flowchart then proceeds to block 808 where an activation function result is output based on the result of the exponentiation, multiplication and / or division operation.

[0141] At this point, the flowchart may be complete. However, in some embodiments, the flowchart may resume and / or certain blocks may be repeated. For example, in some embodiments, the flowchart may resume at block 802 to continue executing the activation function.

[0142] [DNN performance] 9A and 9B show the scalability of the described embodiments with respect to the number of parallel operations supported. Specifically, the described embodiments are implemented with a flexible design that allows for replication of the basic hardware to increase the number of parallel operations supported. In this manner, the described embodiments can be scaled as needed to support the number of parallel operations required by a particular application or use case. For example, as shown in FIG. 9A and FIG. 9B, the proposed solution can be scaled from a single operand to n operands by simple replication. Specifically, FIG. 9A shows the proposed solution for a single operand, and FIG. 9B shows the proposed solution for n operands. Furthermore, the proposed solution can be pipelined to reduce latency and increase throughput.

[0143] 10A illustrates the scalability of the described embodiments with respect to accuracy. Specifically, the described embodiments can be scaled to provide various levels of accuracy simply by adjusting the number of segments in the piecewise linear approximation model implemented by the logarithm, antilogarithm and / or exponential circuit. In this manner, the described embodiments can be scaled as needed to provide the accuracy levels required for different applications and use cases. For example, if an application requires higher accuracy, the number of segments included in the piecewise linear approximation model can be increased to accommodate the higher accuracy requirement.

[0144] The number of segments required in the piecewise linear approximation model for various levels of precision is shown in Figure 10A. For example, if an application requires 23-bit precision from a 23-bit mantissa, as shown in Figure 10A, the piecewise linear approximation model should be implemented using at least 1,556 segments. The number of segments used in the piecewise linear approximation can be adjusted in a similar manner to provide any desired level of precision.

[0145] FIG. 10B compares the silicon area requirements of various implementations of logarithm and antilogarithm hardware. Specifically, FIG. 10B compares the number of synthesis gates for a solution implemented using a lookup table method (12-bit precision) with a logarithm and antilogarithm circuit implemented using a piecewise linear approximation (PLA) (either 10-bit or 12-bit precision). As shown in FIG. 10B, the logarithm and antilogarithm circuit implemented using a piecewise linear approximation uses significantly fewer gates than the solution implemented using a lookup table method. For example, the lookup table solution uses 8,800 gates at 12-bit precision, while the piecewise linear approximation (PLA) logarithm and antilogarithm circuits use 350 and 450 gates at 10-bit precision, and 1,048 and 1,348 gates at 12-bit precision, respectively.

[0146] FIG. 10C compares the silicon area requirements of various implementations of activation function hardware (10-bit precision). Specifically, FIG. 10C shows a comparison of the number of synthesis gates for various individual activation functions (e.g., sigmoid, hyperbolic tangent (Tanh) and Swish) implemented using look-up tables (LUTs) with the number of synthesis gates for a unified activation function circuit (e.g., activation function circuit 200 of FIG. 2A and FIG. 2B) that supports multiple activation functions and is implemented using piecewise linear approximation (PLA). As shown in FIG. 10C, for example, the unified circuit uses significantly fewer gates than any of the LUT-based circuits while still supporting many more activation functions. For example, the unified circuit can be implemented using a total of 8,321 synthesis gates, with its simplified equations allowing the exponential, logarithmic and antilogarithmic blocks to be implemented using only 4,387, 1,048 and 1,348 synthesis gates, respectively. In comparison, each LUT-based circuit requires approximately 17,000-18,000 synthesis gates for just one activation function.

[0147] FIG 10D shows the approximation error of log and antilog circuits implemented using piecewise linear approximation (e.g., FIGS. 5A-5C and 6A-6C), and FIG 10E shows the approximation error of an exponential circuit implemented using piecewise linear approximation (e.g., FIG. 7). Specifically, these circuits can be implemented with 0.045% absolute error (ABS) for the individual log2, and antilog2 and exponential calculations, which results in 12 bits of mantissa precision for IEEE-754 single precision floating point numbers (e.g., 1 sign bit + 8 exponent bits + 12 bits of mantissa = 21 bits of precision).

[0148] The total precision of an integrated activation function circuit implemented using logarithm, antilogarithm, and exponential circuits (e.g., AF circuit 200 of FIGS. 2A and 2B) is 10 mantissa bits for IEEE-754 single precision floating point numbers (e.g., 1 sign bit + 8 exponent bits + 10-bit mantissa = 19-bit precision).

[0149] 11A-11C compare the performance of a deep neural network (DNN) implemented with a conventional activation function formula (using a power of the exponential constant e) and a modified activation function formula (using a power of base 2). For example, as discussed above, the AF circuit 200 of FIG. 2A and FIG. 2B and the AF circuit 400 of FIG. 4 both utilize piecewise linear approximation to implement the activation function. However, while the AF circuit 200 implements a conventional activation function, the AF circuit 400 implements a modified activation function that uses a power of 2 rather than a power of the exponential constant e. The performance of these individual approaches is compared in the examples shown in FIG. 11A-11C.

[0150] 11A shows an example of a DNN 1100 that includes an input layer with two input neurons, a single hidden layer with three neurons, and a fully connected layer with one output neuron. For simplicity, the DNN 1100 implements the feature mapping shown in Table 7, which has the same behavior as an XOR gate. Table 7: DNN feature set mapping [Table 7]

[0151] The illustrated example focuses on one of these feature sets, A=1 and B=1, that is expected to have an output of 0. In the illustrated example, input neuron X1 corresponds to feature A and input neuron X2 corresponds to feature B.

[0152] Figure 11B shows how the DNN1100 uses a conventional sigmoid activation function.

number

[0153] For simplicity, the illustrated example adopts a bias of 0. After applying the sigmoid activation function, the updated hidden layer neurons have the following values: H1=0.7310, H2=0.7858, H3=0.6899.

[0154] Next, the fully connected (FC) layer can be calculated as follows: FC=H1*Wh1+H2*Wh2+H3*Wh3=0.7310*0.3+0.7858*0.5+0.6899*0.9=1.235. Applying a sigmoid activation function to the fully connected layer results in FC=0.7746. The error is then calculated as follows: Error = Expected value - Calculated value = 0 - 0.7746 = -0.7746.

[0155] During the backpropagation and weight update phase, the derivative of the sigmoid activation function

number

[0156] Then, for ΔH1, ΔH2 and ΔH3, the following calculation is performed: New Hidden Layer Values ​​= ΔFC * f´ (Hidden Layer Neuron Values ​​without Activation Function) / Hidden Layer Weights: ΔH1=ΔFC*f´(1) / Wh1=-0.13439*0.1966 / 0.3=-0.088, ΔH2=ΔFC*f´(1.3) / Wh2=-0.13439*0.1683=-0.0452, ΔH1=ΔFC*f´(0.8) / Wh3=-0.13439*0.2139=-0.0319.

[0157] After backpropagation, the new hidden layer weights have the following values: Wh1=Wh1+ΔWh1=0.3-0.1838=0.1162, Wh2=Wh2+ΔWh2=0.5-0.1710=0.392, Wh3=Wh3+ΔWh3=0.9-0.1920=0.708.

[0158] Next, the weight updates between the input layer and the hidden layer are calculated as follows: weight delta = hidden layer delta / input: ΔW11=ΔH1 / X1=-0.088 / 1=-0.088, ΔW12=ΔH2 / X1=-0.0452 / 1=-0.0452, ΔW13=ΔH3 / X1=-0.0319 / 1=-0.0319, ΔW21=ΔH1 / X2=-0.088 / 1=-0.088, ΔW22=ΔH2 / X2=-0.0452 / 1=-0.0452, ΔW23=ΔH3 / X2=-0.0319 / 1=-0.0319, New weights: W11=W11+ΔW11=0.8-0.088=0.712, W12=W12+ΔW12=0.4-0.0452=0.3548, W13=W13+ΔW13=0.3-0.0319=0.2681, W21=W21+ΔW21=0.2-0.088=0.112, W22=W22+ΔW22=0.9-0.0452=0.8548, W23=W23+ΔW23=0.5-0.0319=0.4681.

[0159] Figure 11B shows the state of the DNN 1100 after the weights have been updated based on the calculations above. The output of the DNN after this iteration is 0.69, an improvement over the output of 0.77 after the first iteration. Iterations in this manner continue until the loss function gradient reaches an acceptable level.

[0160] FIG. 11C shows that the DNN1100 uses the base e, i.e.

number

[0161] During the forward propagation stage, the weights are randomly selected as follows: W11 = 0.8, W12 = 0.4, W13 = 0.3, W21 = 0.2, W22 = 0.9, W23 = 0.5, Wh1 = 0.3, Wh2 = 0.5 and Wh3 = 0.9. The outputs of the hidden layer nodes (H) are then calculated as follows: H1=X1*W11+X2*W21=0.8+0.2=1, H2=X1*W12+X2*W22=0.4+0.9=1.3, H3=X1*W13+X2*W23=0.3+0.5=0.8.

[0162] For simplicity, the illustrated example adopts a bias of 0. After applying the modified sigmoid activation function, the updated hidden layer neurons have the following values: H1=0.6667, H2=0.7117, H3=0.6351.

[0163] Next, the fully connected (FC) layer can be calculated as follows: FC=H1*Wh1+H2*Wh2+H3*Wh3=0.6667*0.3+0.7117*0.5+0.6351*0.9=1.1272. Applying the modified sigmoid activation function to the fully connected layer results in FC=0.6859. The error is then calculated as follows: Error = Expected value - Calculated value = 0-0.6859 = -0.6859.

[0164] During the backpropagation and weight update phase, the derivative of the modified sigmoid activation function

number

[0165] Then, for ΔH1, ΔH2 and ΔH3, the following calculation is performed: New hidden layer value = ΔFC*f` (hidden layer neuron value without activation function) / hidden layer weight: ΔH1=ΔFC*f´(1) / Wh1=-0.13439*0.1966 / 0.3=-0.05259, ΔH2=ΔFC*f´(1.3) / Wh2=-0.13439*0.1683=-0.02917, ΔH1=ΔFC*f´(0.8) / Wh3=-0.13439*0.2139=-0.01828.

[0166] After backpropagation, the new hidden layer weights have the following values: Wh1=Wh1+ΔWh1=0.3-0.15364=0.1463, Wh2=Wh2+ΔWh2=0.5-0.14402=0.3559, Wh3=Wh3+ΔWh3=0.9-0.16125=0.7387.

[0167] Next, the weight updates between the input layer and the hidden layer are calculated as follows: weight delta = hidden layer delta / input: ΔW11=ΔH1 / X1=-0.05259 / 1=-0.05259, ΔW12=ΔH2 / X1=-0.02917 / 1=-0.02917, ΔW13=ΔH3 / X1=-0.01828 / 1=-0.01828, ΔW21=ΔH1 / X2=-0.02917 / 1=-0.05259, ΔW22=ΔH2 / X2=-0.0452 / 1=-0.02917, ΔW23=ΔH3 / X2=-0.01828 / 1=-0.01828, new weights: W11=W11+ΔW11=0.8-0.05259=0.7474, W12=W12+ΔW12=0.4-0.02917=0.3708, W13=W13+ΔW13=0.3-0.01828=0.2817, W21=W21+ΔW21=0.2-0.05259=0.1474, W22=W22+ΔW22=0.9-0.02917=0.8708, W23=W23+ΔW23=0.5-0.01828=0.4817.

[0168] Figure 11C shows the state of the DNN 1100 after the weights have been updated based on the calculations above (e.g., using a modified sigmoid activation function). The output of the DNN in Figure 11C after this iteration is 0.70, which compares to the output of the DNN in Figure 11B, which is 0.69. Iterations in this manner continue until the loss function gradient reaches an acceptable level.

[0169] Figures 12A and 12B compare the training convergence for the individual DNNs of Figures 11B and 11C. Specifically, Figure 12A shows the convergence speed for the original sigmoid activation function used by the DNN of Figure 11B, and Figure 12B shows the convergence speed for the modified sigmoid activation function (e.g., using a power of base 2 instead of e) used by the DNN of Figure 11C. Figure 13 shows the error rate or accuracy of these approaches.

[0170] [Example of computing architecture] 14A-18 illustrate example implementations of computing environments and architectures that may be used in accordance with embodiments disclosed herein. In various embodiments, for example, these example computer architectures may be used in conjunction with and / or to implement the deep neural network (DNN) processing capabilities described throughout this disclosure.

[0171] Also suitable are other system designs and configurations known in the art for laptops, desktops, handheld PCs, personal digital assistants, engineering workstations, servers, network devices, network hubs, switches, embedded processors, digital signal processors (DSPs), graphics devices, video game devices, set-top boxes, microcontrollers, mobile phones, portable media players, handheld devices, and a variety of other electronic devices. In general, a wide variety of systems or electronic devices that can incorporate a processor and / or other execution logic as disclosed herein are likely to be suitable.

[0172] FIG. 14A is a block diagram illustrating both an exemplary in-order pipeline and an exemplary register renaming, out-of-order issue / execution pipeline, according to an embodiment of the present invention. FIG. 14B is a block diagram illustrating both an exemplary embodiment of an in-order architecture core and an exemplary register renaming, out-of-order issue / execution architecture core to be included in a processor, according to an embodiment of the present invention. The solid lined boxes in FIG. 14A and FIG. 14B indicate the in-order pipeline and the in-order core, and the optionally added dashed lined boxes indicate the register renaming, out-of-order issue / execution pipeline and the core. Given that the in-order aspect is a subset of the out-of-order aspect, the out-of-order aspect will be described.

[0173] In FIG. 14A , a processor pipeline 1400 includes a fetch stage 1402, a length decode stage 1404, a decode stage 1406, an allocation stage 1408, a renaming stage 1410, a scheduling (also known as dispatch or issue) stage 1412, a register read / memory read stage 1414, an execution stage 1416, a write-back / memory write stage 1418, an exception handling stage 1422, and a commit stage 1424.

[0174] 14B illustrates a processor core 1490 including a front end unit 1430 coupled to an execution engine unit 1450, both of which are coupled to a memory unit 1470. The core 1490 may be a reduced instruction set computing (RISC) core, a complex instruction set computing (CISC) core, a very long instruction word (VLIW) core, a hybrid or alternative core type. As yet another option, the core 1490 may be a special purpose core, such as a network or communication core, a compression engine, a co-processor core, a general purpose computing graphics processing unit (GPGPU) core, a graphics core, or the like.

[0175] The front-end unit 1430 includes a branch prediction unit 1432 coupled to an instruction cache unit 1434, which is coupled to an instruction translation lookaside buffer (TLB) 1436, which is coupled to an instruction fetch unit 1438, which is coupled to a decode unit 1440. The decode unit 1440 (or decoder) decodes instructions and may generate as output one or more micro-operations, microcode entry points, microinstructions, other instructions, or other control signals that are decoded from or otherwise reflect or are derived from the original instruction. The decode unit 1440 may be implemented using a variety of different mechanisms. Examples of suitable mechanisms include, but are not limited to, lookup tables, hardware implementations, programmable logic arrays (PLAs), microcode read-only memory (ROM), and others. In one embodiment, core 1490 includes a microcode ROM or other medium (e.g., within decode unit 1440 or else within front end unit 1430) that stores microcode for a given macroinstruction. Decode unit 1440 is coupled to a rename / allocator unit 1452 within execution engine unit 1450.

[0176] The execution engine unit 1450 includes a rename / allocator unit 1452 coupled to a retirement unit 1454 and a set of one or more scheduler units 1456. The scheduler units 1456 represent any number of different schedulers, including reservation stations, central instruction window, etc. The scheduler units 1456 are coupled to a physical register file unit 1458. The physical register file units 1458 each represent one or more physical register files, different ones of which store one or more different data types, such as scalar integer, scalar floating point, packed integer, packed floating point, vector integer, vector floating point, status (e.g., an instruction pointer, which is the address of the next instruction to be executed), etc. In one embodiment, the physical register file unit 1458 comprises a vector register unit, a write mask register unit, and a scalar register unit. These register units may provide architectural vector registers, vector mask registers, and general purpose registers. The physical register file unit 1458 is overlapped by the retirement unit 1454 to illustrate various ways in which register renaming and out-of-order execution may be implemented (e.g., by using a reorder buffer and a retirement register file, by using a future file, a history buffer and a retirement register file, by using a register map and a register pool, etc.). The retirement unit 1454 and the physical register file unit 1458 are coupled to the execution clusters 1460. The execution clusters 1460 include a set of one or more execution units 1462 and a set of one or more memory access units 1464. The multiple execution units 1462 may perform various operations (e.g., shift, add, subtract, multiply) on various types of data (e.g., scalar floating point, packed integer, packed floating point, vector integer, vector floating point).Some embodiments may include some execution units dedicated to a particular function or set of functions, while other embodiments may include only one execution unit or multiple execution units that all execute all functions. The scheduler unit 1456, physical register file unit 1458, and execution clusters 1460 are shown as possibly multiple because certain embodiments generate separate pipelines for certain types of data / operations (e.g., a scalar integer pipeline, a scalar floating point / packed integer / packed floating point / vector integer / vector floating point pipeline, and / or a memory access pipeline, each of which has its own scheduler unit, physical register file unit, and / or execution cluster, and in the case of a separate memory access pipeline, certain embodiments are implemented in which only the execution cluster of this pipeline has a memory access unit 1464). It should also be understood that if separate pipelines are used, one or more of these pipelines may be out-of-order issue / execution and the rest in-order.

[0177] The set of memory access units 1464 is coupled to a memory unit 1470, which includes a data TLB unit 1472 coupled to a data cache unit 1474, which is coupled to a level 2 (L2) cache unit 1476. In one example embodiment, the plurality of memory access units 1464 may include a load unit, a store address unit, and a store data unit, each of which is coupled to the data TLB unit 1472 in the memory unit 1470. The instruction cache unit 1434 is further coupled to a level 2 (L2) cache unit 1476 in the memory unit 1470. The L2 cache unit 1476 is coupled to one or more other levels of cache and ultimately to main memory.

[0178] As an example, an exemplary register renaming, out-of-order issue / execution core architecture may implement pipeline 1400 as follows: 1) the instruction fetch unit 1438 performs the fetch stage 1402 and the length decode stage 1404; 2) the decode unit 1440 performs the decode stage 1406; 3) the rename / allocator unit 1452 performs the allocation stage 1408 and the renaming stage 1410; 4) the scheduler unit 1456 performs the schedule stage 1412; 5) the physical register file unit 1458 and the memory unit 1470 perform the register read / memory read stage 1414 and the execution cluster 1460 performs the execution stage 1416; 6) the memory unit 1470 and the physical register file unit 1458 perform the write back / memory write stage 1418; 7) various units may be involved in the exception handling stage 1422; and 8) the retirement unit 1454 and the physical register file unit 1458 perform the commit stage 1424.

[0179] Core 1490 may support one or more instruction sets (e.g., the x86 instruction set (with some extensions added in newer versions), the MIPS instruction set from MIPS Technologies of Sunnyvale, Calif., the ARM instruction set from ARM Holdings of Sunnyvale, Calif. (with optional extensions such as NEON)), including the instructions described herein. In one embodiment, core 1490 includes logic to support packed data instruction set extensions (e.g., AVX1, AVX2), enabling it to perform operations with packed data used by many multimedia applications.

[0180] It should be appreciated that a core may support multithreading (executing two or more parallel sets of operations or threads) and may do so in a variety of ways, including time-sliced ​​multithreading, simultaneous multithreading (wherein a single physical core provides a logical core for each of the threads that the physical core is simultaneously multithreading), or a combination of these (e.g., time-sliced ​​fetch and decode followed by simultaneous multithreading, such as in Intel® Hyper-Threading Technology).

[0181] Although register renaming is described in the context of out-of-order execution, it should be understood that register renaming may be used in in-order architectures. The illustrated embodiment of the processor also includes separate instruction and data cache units 1434 / 1474 and a shared L2 cache unit 1476, although alternative embodiments may have a single internal cache for both instructions and data, such as a level 1 (L1) internal cache, or multiple levels of internal cache. In some embodiments, the system may include a combination of internal cache and external cache that is external to the core and / or processor. Alternatively, all of the cache may be external to the core and / or processor.

[0182] 15 is a block diagram of a processor 1500 that may have two or more cores, an integrated memory controller, and integrated graphics, according to an embodiment of the invention. The solid lined box in FIG. 15 illustrates a processor 1500 with a single core 1502A, a system agent 1510, and a set of one or more bus controller units 1516, while an optional dashed lined box illustrates an alternative processor 1500 with multiple cores 1502A-1502N, a set of one or more integrated memory controller units 1514 in the system agent unit 1510, and dedicated logic 1508.

[0183] Thus, different implementations of the processor 1500 may include: 1) a CPU with special purpose logic 1508 that is integrated graphics and / or scientific (throughput) logic (which may include one or more cores) and one or more general purpose cores 1502A-1502N (e.g., general purpose in-order cores, general purpose out-of-order cores, a combination of both); 2) a coprocessor with multiple special purpose cores 1502A-1502N intended primarily for graphics and / or scientific (throughput); 3) a coprocessor with multiple general purpose in-order cores 1502A-1502N. Thus, the processor 1500 may be a general purpose processor, coprocessor, or a special purpose processor, such as, for example, a network or communication processor, a compression engine, a graphics processor, a GPGPU (general purpose graphics processing unit), a high throughput multiple integrated core (MIC) coprocessor (containing 30 or more cores), an embedded processor, or the like. The processor may be implemented on one or more chips. The processor 1500 may be a part of one or more substrates and / or may be implemented on one or more substrates using any of a number of process technologies, such as, for example, BiCMOS, CMOS, or NMOS.

[0184] The memory hierarchy includes one or more levels of cache within the cores, one or more shared cache units 1506 or sets of shared cache units 1506, and external memory (not shown) coupled to a set of integrated memory controller units 1514. The set of shared cache units 1506 may include one or more intermediate level caches, such as level 2 (L2), level 3 (L3), level 4 (L4) or other level caches, last level caches (LLCs), and / or combinations thereof. In one embodiment, a ring-based interconnect unit 1512 interconnects the integrated graphics logic 1508, the set of shared cache units 1506, and the system agent unit 1510 / integrated memory controller unit 1514, although alternative embodiments may use any number of well-known techniques to interconnect such units. In one embodiment, coherence is maintained between one or more cache units 1506 and the cores 1502A-1502N.

[0185] In some embodiments, one or more of the cores 1502A-1502N are capable of multi-threading. The system agent 1510 includes components that coordinate the cores 1502A-1502N. The system agent unit 1510 may include, for example, a power control unit (PCU) and a display unit. The PCU may be or include the logic and components necessary to stabilize the power state of the cores 1502A-1502N and the integrated graphics logic 1508. The display unit is for driving one or more externally connected displays.

[0186] The cores 1502A-1502N may be homogenous or heterogeneous with respect to architectural instruction sets, i.e., two or more of the cores 1502A-1502N may be capable of executing the same instruction set, while others may be capable of executing only a subset of that instruction set, or a different instruction set.

[0187] 16, a block diagram of a system 1600 according to one embodiment of the present invention is shown. The system 1600 may include one or more processors 1610, 1615, which are coupled to a controller hub 1620. In one embodiment, the controller hub 1620 includes a graphics and memory controller hub (GMCH) 1690 and an input / output hub (IOH) 1650 (which may be on separate chips), where the GMCH 1690 includes a memory controller and a graphics controller to which the memory 1640 and the coprocessor 1645 are coupled, and the IOH 1650 couples the input / output (I / O) devices 1660 to the GMCH 1690. Alternatively, one or both of the memory controller and the graphics controller are integrated within the processor (as described herein), where the memory 1640 and the coprocessor 1645 are directly coupled to the processor 1610, and the controller hub 1620 is on a single chip with the IOH 1650.

[0188] An additional, and inherently optional, processor 1615 is shown in dashed lines in Figure 16. Processors 1610, 1615 may each include one or more of the processing cores described herein and may be some version of processor 1500.

[0189] The memory 1640 may be, for example, dynamic random access memory (DRAM), phase change memory (PCM), or a combination thereof. In at least one embodiment, the controller hub 1620 communicates with the processors 1610, 1615 via a multi-drop bus, such as a front side bus (FSB), a point-to-point interface, such as a quick path interconnect (QPI), or a similar connection 1695.

[0190] In one embodiment, the coprocessor 1645 is a special purpose processor, such as, for example, a high throughput MIC processor, a network or communication processor, a compression engine, a graphics processor, a GPGPU, an embedded processor, or the like. In one embodiment, the controller hub 1620 may include an integrated graphics accelerator.

[0191] There may be a variety of differences between the physical resources 1610, 1615 with respect to a range of metrics of merit including architecture, microarchitecture, thermal, power consumption characteristics, and the like.

[0192] In one embodiment, the processor 1610 executes instructions that control generic types of data processing operations. The instructions may have coprocessor instructions embedded in them. The processor 1610 recognizes these coprocessor instructions as being of a type that should be executed by the connected coprocessor 1645. The processor 1610 therefore issues these coprocessor instructions (or control signals representing the coprocessor instructions) on a coprocessor bus or other interconnection to the coprocessor 1645. The coprocessor 1645 accepts and executes the received coprocessor instructions.

[0193] Referring now to Figure 17, a block diagram of a more specific first exemplary system 1700 is shown in accordance with one embodiment of the present invention. As shown in Figure 17, the multiprocessor system 1700 is a point-to-point interconnect system including a first processor 1770 and a second processor 1780 coupled via a point-to-point interconnect 1750. The processors 1770 and 1780 may each be some version of the processor 1500. In one embodiment of the present invention, the processors 1770 and 1780 are the processors 1610 and 1615, respectively, and the coprocessor 1738 is the coprocessor 1645. In another embodiment, the processors 1770 and 1780 are the processors 1610 and 1645, respectively.

[0194] Processors 1770 and 1780 are shown containing integrated memory controller (IMC) units 1772 and 1782, respectively. Processor 1770 also includes point-to-point (PP) interfaces 1776 and 1778 as part of its bus controller unit, and similarly, second processor 1780 includes PP interfaces 1786 and 1788. Processors 1770, 1780 may use PP interface circuits 1778, 1788 to exchange information over point-to-point (PP) interface 1750. As shown in FIG. 17, IMCs 1772 and 1782 couple these processors to respective memories, i.e., memories 1732 and 1734, which may be part of a main memory that is locally connected to the respective processors.

[0195] Processors 1770, 1780 may each exchange information with chipset 1790 via respective PP interfaces 1752, 1754 using point-to-point interface circuits 1776, 1794, 1786, 1798. Chipset 1790 may optionally exchange information with coprocessor 1738 via high performance interface 1739. In one embodiment, coprocessor 1738 is a special purpose processor, such as, for example, a high throughput MIC processor, a network or communication processor, a compression engine, a graphics processor, a GPGPU, an embedded processor, or the like.

[0196] A shared cache (not shown) may be included external to either or both processors and connected to the processors via the PP interconnect so that when either processor is placed in a low power mode, local cache information of either or both processors may be stored in the shared cache.

[0197] Chipset 1790 may be coupled to a first bus 1716 via an interface 1796. In one embodiment, first bus 1716 may be a bus such as a Peripheral Component Interconnect (PCI) bus, a PCI Express bus, or another third generation I / O interconnect bus, although the scope of the invention is not limited in such a way.

[0198] As shown in FIG. 17, various I / O devices 1714 may be coupled to the first bus 1716, along with a bus bridge 1718 that couples the first bus 1716 and a second bus 1720. In an embodiment, one or more additional processors 1715, such as a co-processor, a high throughput MIC processor, a GPGPU, an accelerator (e.g., a graphics accelerator or a digital signal processing (DSP) unit), a field programmable gate array, or any other processor, are coupled to the first bus 1716. In an embodiment, the second bus 1720 may be a low pin count (LPC) bus. Various devices may be coupled to the second bus 1720, including, for example, a keyboard and / or mouse 1722, a communication device 1727, and a storage unit 1728, such as a disk drive or other mass storage device, which in an embodiment may contain instructions / code and data 1730. Additionally, audio I / O 1724 may be coupled to the second bus 1720. However, it should be noted that other architectures are possible: for example, instead of the point-to-point architecture of Figure 17, the system may implement a multi-drop bus or other similar architecture.

[0199] Referring now to FIG. 18, a block diagram of an SoC 1800 according to an embodiment of the present invention is shown. Similar elements to FIG. 15 are given similar reference numbers. Again, dashed boxes are optional features on more advanced SoCs. In FIG. 18, an interconnect unit 1802 is coupled to an application processor 1810 including a set of one or more cores 1802A-1802N and a shared cache unit 1806, a system agent unit 1810, a bus controller unit 1816, an integrated memory controller unit 1814, a set of one or more coprocessors 1820 that may include integrated graphics logic, image processors, audio processors, and video processors, a static random access memory (SRAM) unit 1830, a direct memory access (DMA) unit 1832, and a display unit 1840 for coupling to one or more external displays. In one embodiment, the coprocessor 1820 includes a special purpose processor, such as, for example, a network or communication processor, a compression engine, a GPGPU, a high-throughput MIC processor, an embedded processor, or the like.

[0200] The embodiments of the mechanisms disclosed herein may be implemented in hardware, software, firmware or a combination of these implementation techniques.Embodiments of the invention may be implemented as a computer program or program code running on a programmable system comprising at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device and at least one output device.

[0201] Program code, such as code 1730 shown in Figure 17, may be applied to the input instructions to perform the functions described herein and generate output information. The output information may be applied to one or more output devices, in known fashion. For purposes of this application, a processing system includes any system having a processor, such as, for example, a digital signal processor (DSP), a microcontroller, an application specific integrated circuit (ASIC), or a microprocessor.

[0202] The program code may be implemented in a high level procedural or object oriented programming language to communicate with a processing system. The program code may also be implemented in assembly or machine language, if desired. Indeed, the mechanisms described herein are not limited in scope to any particular programming language. In any case, the language may be a compiled or interpreted language.

[0203] One or more aspects of at least one embodiment may be implemented by denotative instructions stored on a machine-readable medium that represent various logic within a processor, which when read by a machine, creates the logic for performing the techniques described herein. These representations, known as "IP cores," may be stored on tangible machine-readable media and supplied to various customers or manufacturing facilities for loading into manufacturing machines that actually create the logic or processor.

[0204] Such machine-readable storage media may include, but are not limited to, non-transitory tangible apparatus of articles manufactured or formed by a machine or device, including storage media such as hard disks, other disks of any type, including floppy disks, optical disks, compact disk read-only memory (CD-ROM), compact disk re-writable (CD-RW), and magneto-optical disks, read-only memory (ROM), random access memory (RAM), such as dynamic random access memory (DRAM), static random access memory (SRAM), semiconductor devices such as erasable programmable read-only memory (EPROM), flash memory, electrically erasable programmable read-only memory (EEPROM), phase change memory (PCM), magnetic or optical cards, or any other type of medium suitable for storing electronic instructions.

[0205] Accordingly, embodiments of the present invention also include non-transitory tangible machine-readable media, such as hardware description languages ​​(HDLs), that include instructions or include design data that define the structures, circuits, devices, processors and / or system features described herein. Such embodiments may also be referred to as program products.

[0206] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various aspects of the disclosure. In this regard, each block in the flowchart or block diagram may represent a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function. It should also be noted that in alternative implementations, the functions noted in the blocks may occur out of the order depicted in the figures. For example, depending on the functionality involved, two blocks shown in succession may in fact be executed substantially simultaneously, or these blocks may sometimes be executed in reverse order or in an alternative order. It should also be noted that each block of the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, may be implemented by a dedicated hardware-based system that executes a particular function or activity, or by a combination of dedicated hardware and computer instructions.

[0207] The disclosure set forth above outlines features of some embodiments so that those skilled in the art may better understand various aspects of the present disclosure. Those skilled in the art should recognize that they may readily use the disclosure as a basis for designing or modifying other processes and structures to carry out the same purpose and / or achieve the same advantages as the embodiments presented herein. Those skilled in the art should also recognize that such equivalent constructions do not depart from the spirit and scope of the present disclosure, and that those skilled in the art may make various changes, substitutions and alterations to the description herein without departing from the spirit and scope of the present disclosure.

[0208] All or a portion of any hardware element disclosed herein may be readily provided in a system-on-chip (SoC) that includes a central processing unit (CPU) package. SoC refers to an integrated circuit (IC) that integrates components of a computer or other electronic system into a single chip. An SoC may include digital, analog, mixed-signal and radio frequency functions, all of which may be provided on a single chip substrate. Other embodiments may include multi-chip modules (MCMs), in which multiple chips are positioned within and configured to tightly interact with one another through a single electronic package. In various other embodiments, the computing functionality disclosed herein may be implemented in one or more silicon cores in application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) and other semiconductor chips.

[0209] The term "processor" or "microprocessor" as used herein should be understood to encompass not only conventional microprocessors (such as Intel's industry leading x86 and x64 architectures), but also graphics processors, matrix processors, and any ASIC, FPGA, microcontroller, digital signal processor (DSP), programmable logic device, programmable logic array (PLA), microcode, instruction set, emulated, or virtual machine processor, or similar "Turing complete" device, combination of devices, or logic element (hardware or software) that permits the execution of instructions.

[0210] It should also be noted that in certain embodiments, some components may be omitted or integrated. In a general sense, the illustrated arrangement should be understood as a logical division, but the physical architecture may include various permutations, combinations, and / or hybridizations of these elements. It is essential to note that an infinite number of possible design configurations may be used to achieve the operational objectives outlined herein. Thus, the associated infrastructure has an infinite number of alternative arrangements, design choices, device possibilities, hardware configurations, software implementations, and equipment options.

[0211] In a general sense, any suitably configured processor can execute instructions associated with data or microcode to perform the operations detailed herein. Any processor disclosed herein can also transform an element or item (e.g., data) from one state or thing to another. In another example, some of the activities outlined herein can be implemented by fixed logic or programmable logic (e.g., software and / or computer instructions executed by a processor), and the elements identified herein can be any type of programmable processor, programmable digital logic (e.g., field programmable gate array (FPGA), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM)), digital logic, software, code, electronic instructions, flash memory, optical disk, CD-ROM, DVD ROM, magnetic or optical card, ASIC including other types of machine-readable media suitable for storing electronic instructions, or any suitable combination of these.

[0212] In operation, the storage device may store information in any suitable type of tangible, non-transitory storage medium (e.g., random access memory (RAM), read only memory (ROM), field programmable gate array (FPGA), erasable programmable read only memory (EPROM), electrically erasable programmable read only memory (EEPROM) or microcode), software, hardware (e.g., processor instructions or microcode), or any other suitable component, device, element or object, as appropriate and based on the particular needs. Additionally, the information tracked, transmitted, received, or stored in the processor may also be provided in any database, register, table, cache, queue, control list, or memory structure, as appropriate and based on the particular needs and implementation, all of which information may be referenced in any suitable time frame. Any memory or storage elements disclosed herein should be construed as being encompassed by the broad meanings of "memory" and "storage device," as appropriate. A non-transitory storage medium herein is expressly intended to encompass any non-transitory dedicated or programmable hardware configured to provide the disclosed operations or to cause a processor to perform the disclosed operations. A non-transitory storage medium also expressly includes a processor having stored thereon hardware coded instructions, and optionally microcode instructions or sequences encoded in hardware, firmware or software.

[0213] Computer program logic implementing all or part of the functionality described herein may be embodied in a variety of forms, including, but by no means limited to, hardware description languages, source code forms, computer executable forms, machine instructions or microcode, programmable hardware, and various intermediate forms (e.g., forms generated by an HDL processor, assembler, compiler, linker, or locator). In one example, source code includes a series of computer program instructions implemented in various programming languages, such as object code, assembly language, or high-level languages ​​for use with various operating systems or operating environments, such as OpenCL, FORTRAN, C, C++, JAVA, or HTML, or implemented in hardware description languages, such as Spice, Verilog, and VHDL. The source code may define and use various data structures and communication messages. The source code may be in a computer executable form (e.g., via an interpreter), or the source code may be converted (e.g., via a translator, assembler, or compiler) into a computer executable form or into an intermediate form, such as bytecode. Where appropriate, any of the above may be used to construct or describe suitable discrete or integrated circuits, whether sequential, combinatorial, state machine, or otherwise.

[0214] In one example, any number of the electrical circuits in the figures may be implemented on a board of the associated electronic device. The board may be a general circuit board that holds various components of the internal electronic system of the electronic device and may also provide connectors for other peripherals. More specifically, the board may provide electrical connections that allow electrical communication with other components of the system. Any suitable processor and memory may be appropriately coupled to the board based on the specific configuration needs, processing requirements, and computing design. Other components such as external storage devices, additional sensors, controllers for audio / video displays, and peripherals may be attached to the board as plug-in cards via cables or may be integrated into the board itself. In another example, the electrical circuits in the figures may be implemented as stand-alone modules (e.g., devices having associated components and circuits configured to perform a specific application or function) or as plug-in modules in application-specific hardware of the electronic device.

[0215] It should be noted that in many of the examples provided herein, interactions may be described in terms of two, three, four, or more electrical components. However, this is done merely for purposes of clarity and illustration. It should be recognized that the system may be integrated or reconfigured in any suitable manner. In addition to similar alternative designs, any of the illustrated components, modules, and elements of the figures may be combined in a variety of possible configurations, all of which are within the broad scope of this specification. In some cases, it may be easier to describe one or more of the functions of a given flow set with reference to only a limited number of electrical elements. It should be recognized that the illustrated electrical circuits and their teachings are easily scalable to accommodate a large number of components and more complex / advanced arrangements and configurations. Thus, the examples provided are not intended to limit the scope or inhibit the broad teachings of the electrical circuits that are potentially applicable to countless other architectures.

[0216] Many other changes, substitutions, variations, modifications and alterations may be ascertained by those skilled in the art, and it is intended that the present disclosure encompass such changes, substitutions, variations, modifications and alterations as fall within the scope of the appended claims.

[0217] [Implementation example] The following examples relate to embodiments described throughout this disclosure.

[0218] One or more embodiments may include an apparatus comprising a logarithmic circuit comprising circuitry for identifying an input associated with a logarithmic operation via an input register, identifying a first range within which the input falls using a range selection circuit, obtaining a plurality of operands associated with a first equation, calculating a result of the first equation based on the plurality of operands using an adder / subtractor circuit, and returning an output associated with the logarithmic operation via an output register, wherein the logarithmic operation is to be performed by the logarithmic circuit using a piecewise linear approximation, wherein the first range is identified from a plurality of ranges associated with a plurality of piecewise linear approximation (PLA) equations of the logarithmic operation, and wherein the first range corresponds to a first equation of a plurality of PLA equations, and wherein the output is generated based at least in part on a result of the first equation.

[0219] In one example embodiment of the apparatus, the logarithmic operation is associated with an artificial neural network operation.

[0220] In one example embodiment of the apparatus, the input comprises a floating point number, the floating point number comprising an exponent portion and a mantissa portion, and the output comprises a fixed point number, the fixed point number comprising an integer portion and a fractional portion.

[0221] In one example embodiment of the apparatus, the plurality of operands includes a mantissa operand and one or more fraction operands, each of which includes a denominator that includes a power of two.

[0222] In one example embodiment of the apparatus, the logarithm circuit further comprises one or more shift circuits for generating the one or more fractional operands.

[0223] In an example embodiment of the apparatus, the logarithm circuit further comprises a subtractor circuit for subtracting the bias from the exponent of the floating-point number to produce an unbiased exponent.

[0224] In an example embodiment of the apparatus, the circuit for returning an output associated with the logarithmic operation via the output register further generates an integer portion of the fixed-point number based on the unbiased exponent, and generates a fractional portion of the fixed-point number based on a result of the first equation.

[0225] In one example embodiment of the apparatus, the logarithm circuit further comprises one or more multiplexers for selecting a plurality of operands associated with the first equation.

[0226] In one example embodiment of the apparatus, an adder / subtractor circuit performs one or more addition or subtraction operations on multiple operands.

[0227] In an example embodiment of the apparatus, the apparatus further includes an inverse logarithm circuit comprising circuitry for identifying a second input associated with the inverse logarithm operation, identifying a second range within which the second input falls, calculating a second result of a second equation based on a second plurality of operands associated with the second equation, and generating a second output associated with the inverse logarithm operation, wherein the inverse logarithm operation is to be performed using piecewise linear approximation by the inverse logarithm circuit, the second range being identified from a second plurality of ranges associated with a second plurality of piecewise linear approximation (PLA) equations of the inverse logarithm operation, and the second range corresponds to a second equation of the second plurality of PLA equations, and the second output is generated based at least in part on a second result of the second equation.

[0228] In an example embodiment of the apparatus, the apparatus further comprises an activation function circuit, the activation function circuit comprising a logarithm circuit and an inverse logarithm circuit, and the activation function circuit further comprises a circuit for receiving an instruction to perform an activation function selected from a plurality of available activation functions, performing one or more multiplication or division operations using one or more logarithm operations and one or more inverse logarithm operations, and generating an activation output associated with the activation function, the activation function including one or more multiplication or division operations, the one or more logarithm operations being performed using the logarithm circuit, and the one or more inverse logarithm operations being performed using the inverse logarithm circuit, and the activation output being generated based at least in part on one or more results of the one or more multiplication or division operations.

[0229] In an example embodiment of the apparatus, the activation function further includes one or more exponential operations, and the activation function circuit further comprises an exponential circuit for performing the one or more exponential operations using piecewise linear approximation.

[0230] One or more embodiments may include a system comprising a memory for storing information associated with an application, a processor for executing one or more instructions associated with the application, and an activation function circuit for performing a plurality of activation functions, the activation function circuit comprising circuitry for receiving instructions to perform an activation function associated with the application, performing one or more multiplication or division operations using one or more logarithmic operations and one or more inverse logarithmic operations, and generating an output associated with the activation function, the activation function being selected from a plurality of activation functions and the activation function comprising one or more multiplication or division operations, the one or more logarithmic operations being performed using a piecewise linear approximation by a logarithm circuit, and the one or more inverse logarithmic operations being performed using a piecewise linear approximation by an inverse logarithm circuit, and the output being generated based at least in part on one or more results of the one or more multiplication or division operations.

[0231] In one example embodiment of the system, the application includes an artificial neural network, and an activation function is associated with the operation of the artificial neural network.

[0232] In an example embodiment of the system, the circuitry for performing one or more multiplication or division operations using one or more logarithm operations and one or more inverse logarithm operations further performs one or more base 2 logarithm operations on one or more operands associated with the one or more multiplication or division operations, performs one or more addition or subtraction operations on one or more results of the one or more base 2 logarithm operations, and performs one or more base 2 inverse logarithm operations on one or more results of the one or more addition or subtraction operations, wherein the one or more base 2 logarithm operations are performed using a piecewise linear approximation, and wherein the one or more base 2 inverse logarithm operations are performed using a piecewise linear approximation.

[0233] In an example embodiment of the system, the activation function further includes one or more exponential operations, and the activation function circuitry further comprises circuitry for performing the one or more exponential operations using a piecewise linear approximation.

[0234] In an example embodiment of the system, the one or more exponentiation operations each include base 2, and the circuitry for performing the one or more exponentiation operations using a piecewise linear approximation further performs the one or more exponentiation operations using one or more base 2 inverse logarithm operations, and the one or more base 2 inverse logarithm operations are performed using a piecewise linear approximation.

[0235] In one example embodiment of the system, the activation functions include a sigmoid function, a hyperbolic tangent function, a swish function, and a rectified linear unit function.

[0236] In one example embodiment of the system, at least one of the sigmoid function, hyperbolic tangent function, or swish function is defined using one or more exponential operations involving exclusively base 2, and the activation function circuitry further includes circuitry for performing the one or more exponential operations using one or more base 2 inverse logarithm operations, where the one or more base 2 inverse logarithm operations are performed using a piecewise linear approximation.

[0237] One or more embodiments may include at least one machine-accessible storage medium storing instructions that, when executed on a machine, cause the machine to receive, with an activation function circuit, instructions to perform an activation function selected from a plurality of available activation functions, perform one or more multiplication or division operations using one or more logarithmic operations and one or more inverse logarithmic operations, and generate an output associated with the activation function, the activation function including one or more multiplication or division operations, the one or more logarithmic operations and the one or more inverse logarithmic operations being performed using a piecewise linear approximation, and the output being generated based at least in part on one or more results of the one or more multiplication or division operations.

[0238] In an example embodiment of the storage medium, the instructions for causing a machine to perform one or more multiplication or division operations using one or more logarithm operations and one or more inverse logarithm operations further cause the machine to perform one or more base 2 logarithm operations on one or more operands associated with the one or more multiplication or division operations, perform one or more addition or subtraction operations on one or more results of the one or more base 2 logarithm operations, and perform one or more base 2 inverse logarithm operations on one or more results of the one or more addition or subtraction operations, wherein the one or more base 2 logarithm operations are performed using piecewise linear approximation, and wherein the one or more base 2 inverse logarithm operations are performed using piecewise linear approximation.

[0239] In an example embodiment of the storage medium, the activation function further includes one or more exponential operations, and the instructions further cause the machine to perform the one or more exponential operations using piecewise linear approximation.

[0240] In one example embodiment of the storage medium, at least one activation function of the plurality of available activation functions is defined using one or more exponentiation operations involving exclusively base 2, and the instructions further cause the machine to perform the one or more exponentiation operations using one or more base 2 inverse logarithm operations, where the one or more base 2 inverse logarithm operations are performed using a piecewise linear approximation.

[0241] One or more embodiments may include a method comprising receiving, by an activation function circuit, instructions to execute an activation function selected from a plurality of available activation functions, the activation function including one or more multiplication or division operations; performing the one or more multiplication or division operations using one or more logarithmic operations and one or more inverse logarithmic operations, the one or more logarithmic operations and the one or more inverse logarithmic operations being performed using a piecewise linear approximation; and generating an output associated with the activation function, the output being generated based at least in part on one or more results of the one or more multiplication or division operations.

[0242] In an example embodiment of the method, the method further includes performing one or more base 2 logarithm operations on one or more operands associated with the one or more multiplication or division operations, where the one or more base 2 logarithm operations are performed using a piecewise linear approximation; performing one or more addition or subtraction operations on one or more results of the one or more base 2 logarithm operations; and performing one or more base 2 inverse logarithm operations on one or more results of the one or more addition or subtraction operations, where the one or more base 2 inverse logarithm operations are performed using a piecewise linear approximation. [Other possible claims] (Item 1) An apparatus comprising a logarithmic circuit, the logarithmic circuit comprising: Identifying, via an input register, an input associated with the logarithmic operation; identifying a first range within which the input falls using a range selection circuit; Calculating a result of a first equation based on a plurality of operands using an adder / subtractor circuit; and a circuit for returning an output associated with said logarithmic operation via an output register; the logarithmic operation is to be performed by the logarithmic circuit using piecewise linear approximation; the first range is identified from a plurality of ranges associated with a plurality of piecewise linear approximation (PLA) equations of the logarithmic operation, and the first range corresponds to a first equation of the plurality of PLA equations; The output is generated based at least in part on the result of the first equation. (Item 2) 2. The apparatus of claim 1, wherein the logarithmic operation is associated with an artificial neural network operation. (Item 3) the input comprises a floating point number, the floating point number comprising an exponent portion and a mantissa portion; and Item 2. The apparatus of item 1, wherein the output comprises a fixed-point number, the fixed-point number comprising an integer portion and a fractional portion. (Item 4) 4. The apparatus of claim 3, wherein the plurality of operands include the mantissa operand and one or more fraction operands, each of the one or more fraction operands including a denominator that includes a power of two. (Item 5) 5. The apparatus of claim 4, wherein the logarithm circuit further comprises one or more shift circuits for generating the one or more fractional operands. (Item 6) 4. The apparatus of claim 3, wherein the logarithm circuit further comprises a subtractor circuit for subtracting a bias from the exponent portion of the floating-point number to generate an unbiased exponent. (Item 7) The circuit for returning the output associated with the logarithmic operation via the output register further comprises: generating an integer portion of the fixed-point number based on the unbiased exponent; and 7. The apparatus of claim 6, further comprising: generating the fractional part of the fixed-point number based on the result of the first equation. (Item 8) 2. The apparatus of claim 1, wherein the logarithm circuit further comprises one or more multiplexers for selecting the plurality of operands associated with the first equation. (Item 9) 2. The apparatus of claim 1, wherein the adder / subtractor circuit performs one or more addition or subtraction operations on the plurality of operands. (Item 10) The antilogarithm circuit further comprises: Identifying a second input associated with the inverse logarithm operation; identifying a second range within which the second input falls; calculating a second result of a second equation based on a second plurality of operands associated with the second equation; and a circuit for generating a second output related to the antilogarithm operation; the antilogarithm operation is to be performed by the antilogarithm circuit using piecewise linear approximation; the second range is identified from a second plurality of ranges associated with a second plurality of piecewise linear approximation (PLA) equations of the inverse logarithm operation, and the second range corresponds to a second equation of the second plurality of PLA equations; Item 2. The apparatus of item 1, wherein the second output is generated based at least in part on the second result of the second equation. (Item 11) The method further includes an activation function circuit, the activation function circuit including the logarithm circuit and the antilogarithm circuit, the activation function circuit including: receiving instructions to execute an activation function selected from a plurality of available activation functions; performing one or more multiplication or division operations using one or more logarithmic operations and one or more antilogarithmic operations; and further comprising a circuit for generating an activation output associated with the activation function; said activation function including said one or more multiplication or division operations; the one or more logarithmic operations are performed using the logarithm circuit, and the one or more inverse logarithmic operations are performed using the inverse logarithm circuit; Item 11. The apparatus of item 10, wherein the activation output is generated based at least in part on one or more results of the one or more multiplication or division operations. (Item 12) The activation function further includes one or more exponential operations; and Item 12. The apparatus of item 11, wherein the activation function circuitry further comprises an exponential circuitry for performing the one or more exponential operations using piecewise linear approximation. (Item 13) 1. A system comprising: a memory for storing information associated with the application; a processor for executing one or more instructions associated with said application; and an activation function circuit for implementing a plurality of activation functions, the activation function circuit comprising: receiving instructions to execute an activation function associated with the application; performing one or more multiplication or division operations using one or more logarithmic operations and one or more antilogarithmic operations; and a circuit for generating an output associated with the activation function; the activation function is selected from the plurality of activation functions, and the activation function includes the one or more multiplication or division operations; the one or more logarithmic operations are performed by a logarithmic circuit using a piece-wise linear approximation, and the one or more inverse logarithmic operations are performed by an inverse logarithmic circuit using a piece-wise linear approximation; The system, wherein the output is generated based at least in part on one or more results of the one or more multiplication or division operations. (Item 14) Item 14. The system of item 13, wherein the application includes an artificial neural network and the activation function is associated with the operation of the artificial neural network. (Item 15) The circuit for performing the one or more multiplication or division operations using the one or more logarithmic operations and the one or more antilogarithmic operations further comprises: performing one or more base 2 logarithm operations on one or more operands associated with said one or more multiplication or division operations; performing one or more addition or subtraction operations on one or more results of said one or more base 2 logarithm operations; and performing one or more base 2 inverse logarithm operations on one or more results of said one or more addition or subtraction operations; wherein the one or more base 2 logarithmic operations are performed using piecewise linear approximations; Item 14. The system of item 13, wherein the one or more base 2 inverse logarithm operations are performed using piecewise linear approximation. (Item 16) The activation function further includes one or more exponential operations; and Item 14. The system of item 13, wherein the activation function circuitry further comprises circuitry for performing the one or more exponential operations using piecewise linear approximation. (Item 17) The one or more exponentiation operations each include a base 2 exponentiation, and 17. The system of claim 16, wherein the circuitry for performing the one or more exponential operations using a piecewise linear approximation further performs the one or more exponential operations using one or more base 2 inverse logarithm operations, wherein the one or more base 2 inverse logarithm operations are performed using a piecewise linear approximation. (Item 18) The above activation functions include The sigmoid function, The hyperbolic tangent function, The swish function, and a normalized linear unit function. (Item 19) At least one of the sigmoid function, the hyperbolic tangent function, or the swish function is defined using one or more exponential operations that include exclusively base 2; and 20. The system of claim 18, wherein the activation function circuit further comprises circuitry for performing the one or more exponential operations using one or more base 2 inverse logarithm operations, the one or more base 2 inverse logarithm operations being performed using piecewise linear approximations. (Item 20) At least one machine-accessible storage medium storing instructions that, when executed on a machine, cause the machine to: receiving instructions by an activation function circuit to perform an activation function selected from a plurality of available activation functions; performing one or more multiplication or division operations using one or more logarithmic operations and one or more antilogarithmic operations; and generating an output associated with said activation function; said activation function including said one or more multiplication or division operations; the one or more logarithmic operations and the one or more antilogarithmic operations are performed using piecewise linear approximations; and At least one machine-accessible storage medium, wherein said output is generated based, at least in part, on one or more results of said one or more multiplication or division operations. (Item 21) The instructions for causing the machine to perform the one or more multiplication or division operations using the one or more logarithmic operations and the one or more antilogarithmic operations further include instructions for causing the machine to: performing one or more base 2 logarithm operations on one or more operands associated with said one or more multiplication or division operations; performing one or more addition or subtraction operations on one or more results of said one or more base 2 logarithm operations; and performing one or more base 2 inverse logarithm operations on one or more results of said one or more addition or subtraction operations; wherein the one or more base 2 logarithmic operations are performed using piecewise linear approximations; 21. The storage medium of item 20, wherein the one or more base 2 inverse logarithm operations are performed using piecewise linear approximation. (Item 22) The activation function further includes one or more exponential operations; and 23. The method of claim 22, further comprising: causing the machine to perform the one or more exponential operations using a piecewise linear approximation. At least one activation function of the plurality of available activation functions is defined using one or more exponential operations that include exclusively base 2, and 21. The storage medium of claim 20, wherein the instructions further cause the machine to perform the one or more exponential operations using one or more base 2 inverse logarithm operations, the one or more base 2 inverse logarithm operations being performed using piecewise linear approximation.

Claims

1. A method for a processor, device and / or system that executes an application using a deep neural network to implement the deep neural network, comprising: receiving an input vector to be processed by the deep neural network, the input vector including one or more floating point input elements, the deep neural network including one or more hidden layers; converting the one or more floating-point input elements to one or more fixed-point input elements; inputting the one or more fixed-point input elements into the one or more hidden layers; computing, by the one or more hidden layers, one or more fixed-point output elements based on the one or more fixed-point input elements; converting the one or more fixed-point output elements to one or more floating-point output elements; and generating an output of the deep neural network using the one or more floating point output elements; Equipped with The method, wherein the deep neural network is implemented using a multiplier-less neural network architecture.

2. Converting the one or more floating-point input elements to the one or more fixed-point input elements comprises: applying a base 2 logarithm function to the one or more floating-point input elements, the base 2 logarithm function outputting the one or more fixed-point input elements; The method of claim 1.

3. Converting the one or more fixed-point output elements to the one or more floating-point output elements comprises: applying a base 2 inverse logarithm function to the one or more fixed-point output elements, the base 2 inverse logarithm function outputting the one or more floating-point output elements. The method of claim 1.

4. receiving a weight vector, the weight vector comprising one or more floating-point weights of the one or more hidden layers; converting the one or more floating-point weights into one or more fixed-point weights; and inputting the one or more fixed-point weights into the one or more hidden layers; the one or more hidden layers compute the one or more fixed-point output elements based on the one or more fixed-point input elements and the one or more fixed-point weights. The method of claim 1.

5. The one or more hidden layers include one or more convolutional layers, one or more pooling layers, or one or more fully connected layers. The method of claim 1.

6. the one or more hidden layers include a plurality of convolutional layers, each of the plurality of convolutional layers having a weight vector, the weight vector including one or more floating-point weights; and computing, by the one or more hidden layers, the one or more fixed-point output elements based on the one or more fixed-point input elements, comprises: converting the one or more floating-point weights into one or more fixed-point weights; and inputting the one or more fixed-point weights into the convolutional layer; In the convolution layer, a convolution is performed on the one or more fixed-point weights. The method according to claim 5.

7. computing, by the one or more hidden layers, the one or more fixed-point output elements based on the one or more fixed-point input elements, performing one or more fixed-point additions on the one or more fixed-point input elements; The method of claim 1.

8. 1. A non-transitory computer-readable medium storing executable instructions for performing operations for implementing a deep neural network, the operations comprising: receiving an input vector to be processed by the deep neural network, the input vector including one or more floating point input elements, the deep neural network including one or more hidden layers; converting said one or more floating-point input elements into one or more fixed-point input elements; inputting said one or more fixed-point input elements into said one or more hidden layers; computing, by the one or more hidden layers, one or more fixed-point output elements based on the one or more fixed-point input elements; converting said one or more fixed-point output elements into one or more floating-point output elements; and generating an output of the deep neural network using the one or more floating point output elements; Equipped with The non-transitory computer-readable medium, wherein the deep neural network is implemented using a multiplier-less neural network architecture.

9. The step of converting the one or more floating-point input elements to the one or more fixed-point input elements comprises: applying a base 2 logarithm function to the one or more floating-point input elements, the base 2 logarithm function outputting the one or more fixed-point input elements; The non-transitory computer-readable medium of claim 8.

10. The step of converting the one or more fixed-point output elements to the one or more floating-point output elements comprises: applying an inverse logarithm base 2 function to the one or more fixed-point output elements, the inverse logarithm base 2 function outputting the one or more floating-point output elements. The non-transitory computer-readable medium of claim 8.

11. The calculation is receiving a weight vector, the weight vector including one or more floating point weights of the one or more hidden layers; converting said one or more floating-point weights into one or more fixed-point weights; and inputting the one or more fixed-point weights into the one or more hidden layers; the one or more hidden layers compute the one or more fixed-point output elements based on the one or more fixed-point input elements and the one or more fixed-point weights. The non-transitory computer-readable medium of claim 8.

12. The one or more hidden layers include one or more convolutional layers, one or more pooling layers, or one or more fully connected layers. The non-transitory computer-readable medium of claim 8.

13. The one or more hidden layers include a plurality of convolutional layers, each of the plurality of convolutional layers having a weight vector, the weight vector including one or more floating-point weights, and the step of computing, by the one or more hidden layers, the one or more fixed-point output elements based on the one or more fixed-point input elements includes: converting said one or more floating-point weights into one or more fixed-point weights; and inputting the one or more fixed-point weights into the convolutional layer; In the convolution layer, a convolution is performed on the one or more fixed-point weights.

13. The non-transitory computer readable medium of claim 12.

14. The step of computing, by the one or more hidden layers, the one or more fixed-point output elements based on the one or more fixed-point input elements comprises: performing one or more fixed-point additions on the one or more fixed-point input elements; The non-transitory computer-readable medium of claim 8.

15. a computer processor for executing computer program instructions; and 1. An apparatus comprising: a non-transitory computer-readable memory storing computer program instructions executable by the computer processor to perform operations for implementing a deep neural network, the operations comprising: receiving an input vector to be processed by the deep neural network, the input vector including one or more floating point input elements, the deep neural network including one or more hidden layers; converting said one or more floating-point input elements into one or more fixed-point input elements; inputting said one or more fixed-point input elements into said one or more hidden layers; computing, by the one or more hidden layers, one or more fixed-point output elements based on the one or more fixed-point input elements; converting said one or more fixed-point output elements into one or more floating-point output elements; and generating an output of the deep neural network using the one or more floating point output elements; Equipped with The apparatus, wherein the deep neural network is implemented using a multiplier-less neural network architecture.

16. The step of converting the one or more floating-point input elements to the one or more fixed-point input elements comprises: applying a base 2 logarithm function to the one or more floating-point input elements, the base 2 logarithm function outputting the one or more fixed-point input elements; 16. The apparatus of claim 15.

17. The step of converting the one or more fixed-point output elements to the one or more floating-point output elements comprises: applying an inverse logarithm base 2 function to the one or more fixed-point output elements, the inverse logarithm base 2 function outputting the one or more floating-point output elements.

16. The apparatus of claim 15.

18. The calculation is receiving a weight vector, the weight vector including one or more floating point weights of the one or more hidden layers; converting said one or more floating-point weights into one or more fixed-point weights; and inputting the one or more fixed-point weights into the one or more hidden layers; the one or more hidden layers compute the one or more fixed-point output elements based on the one or more fixed-point input elements and the one or more fixed-point weights.

16. The apparatus of claim 15.

19. The one or more hidden layers include a plurality of convolutional layers, each of the plurality of convolutional layers having a weight vector, the weight vector including one or more floating-point weights, and the step of computing, by the one or more hidden layers, the one or more fixed-point output elements based on the one or more fixed-point input elements includes: converting said one or more floating-point weights into one or more fixed-point weights; and inputting the one or more fixed-point weights into the convolutional layer; In the convolution layer, a convolution is performed on the one or more fixed-point weights.

16. The apparatus of claim 15.

20. The step of computing, by the one or more hidden layers, the one or more fixed-point output elements based on the one or more fixed-point input elements comprises: performing one or more fixed-point additions on the one or more fixed-point input elements; 16. The apparatus of claim 15.

Images