Deep neural network architecture using piecewise linear approximation

Abstract

The invention discloses a deep neural network architecture using piecewise linear approximation. In one embodiment, an apparatus includes a logarithmic circuit to identify an input associated with a logarithmic operation, where the logarithmic operation is to be performed by the logarithmic circuit using piecewise linear approximation; identifying a first range within which the input falls, where the first range is identified from a plurality of ranges associated with a plurality of piecewise linear approximation (PLA) equations for the logarithmic operation, and where the first range corresponds to a first equation of the plurality of PLA equations; calculating a result of the first equation based on a plurality of operands associated with the first equation; and returning an output associated with the logarithmic operation, where the output is generated based at least in part on the result of the first equation.

Description

This application is a divisional application of the invention patent application with the application date of May 29, 2019, the priority date of June 29, 2018, the application number 201910455910.1, and the title “Deep Neural Network Architecture Using Piecewise Linear Approximation”. Technical Field

[0001] The present disclosure relates generally to the field of computer architecture and design, and more particularly, but not exclusively, to processing architectures for deep neural networks (DNNs). Background Art

[0002] Due to the increasing number of deep learning applications developed for many different use cases, there is a strong demand for specialized hardware designed for deep neural networks (DNNs). For example, DNNs typically require a large amount of real-time processing, which often involves multiple layers of complex operations on floating-point numbers, such as convolutional layers, pooling layers, fully connected layers, etc. However, existing hardware solutions for DNNs suffer from various limitations, including large power consumption, high latency, significant silicon area requirements, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0003] The present disclosure will be better understood from the following detailed description when read in conjunction with the accompanying drawings. It should be emphasized that, in accordance with standard practice in the industry, the various features are not necessarily drawn to scale and are only for illustrative purposes. When scale is shown explicitly or implicitly, it merely provides an illustrative example. In other embodiments, the sizes of the various features may be arbitrarily enlarged or reduced for clarity of discussion.

[0004] Figure 1 An example embodiment of implementing a deep neural network (DNN) using logarithmic and antilogarithmic piecewise linear approximation circuits is presented.

[0005] FIG. 2A to FIG. 2B An example embodiment of a unified activation function circuit for a deep neural network (DNN) is presented.

[0006] FIG. 3A to FIG. 3E Example activation functions showing the uniform activation function circuit.

[0007] Figure 4 An example embodiment of implementing a unified activation function circuit using a modified activation function equation with a base 2 exponential term is presented.

[0008] FIG. 5A to FIG. 5C An example embodiment of implementing a logarithmic circuit using a piecewise linear approximation is presented.

[0009] FIG. 6A to FIG. 6C An example embodiment of implementing an antilogarithmic circuit using piecewise linear approximation is presented.

[0010] Figure 7 An example embodiment of implementing an exponential circuit using a piecewise linear approximation is presented.

[0011] Figure 8 A flow chart showing an example processing architecture for implementing an artificial neural network.

[0012] FIG. 9A to FIG. 9B The scalability of an example processing architecture for artificial neural networks with respect to the number of parallel operations supported is demonstrated.

[0013] FIG. 10A to FIG. 10E Various performance aspects of example processing architectures for artificial neural networks are presented.

[0014] FIG. 11A to FIG. 11C Examples of DNNs implemented using traditional activation functions and modified activation functions with base-2 exponential terms are shown.

[0015] FIG. 12A to FIG. 12B and Fig.13 Various performance aspects of DNNs implemented using traditional activation functions and modified activation functions are demonstrated.

[0016] FIG. 14A to FIG. 14B , Fig.15 , Fig.16 , Fig.17 and Fig.18 An example implementation of a computer architecture that may be used in accordance with the embodiments disclosed herein is presented. DETAILED DESCRIPTION

[0017] The following disclosure provides many different embodiments or examples for implementing the different features of the present disclosure. The following describes specific examples of components and arrangements to simplify the present disclosure. Of course, these are only examples and are not intended to be restrictive. Further, the present disclosure may repeat reference numerals and / or letters in various examples. This repetition is for the purpose of simplicity and clarity, and does not itself specify the relationship between the various embodiments and / or configurations discussed. Different embodiments may have different advantages, and it is not necessarily required that any embodiment has specific advantages. Deep Neural Network (DNN) Inference Using Logarithmic / Antilogarithmic Piecewise Linear Approximation Circuits

[0018] Due to the increasing number of artificial intelligence applications that rely on machine learning (e.g., deep learning), there is a strong demand for specialized hardware designed to implement artificial neural networks (e.g., deep neural networks, convolutional neural networks, feed-forward neural networks, recurrent neural networks, etc.). Low power, low area, and high-speed hardware is ideal for deep learning applications.

[0019] In particular, artificial neural networks such as deep neural networks (DNNs) are implemented using multiple layers of processing nodes or "neurons" such as convolutional layers, pooling layers, fully connected layers, etc. The nodes in each layer compute a set of inputs and associated weights (usually represented as vectors) to generate outputs, which are then used as inputs for nodes in the next layer. The computations performed by the nodes in each layer typically involve transforming the inputs based on the associated weights and an activation function that determines whether each node should be "activated." Further, the layers are typically repeated in this manner based on the requirements of the specific application so as to reach a global minimum.

[0020] In addition, state-of-the-art DNNs are typically implemented using operations on digital values ​​represented using a single-precision (32-bit) floating-point format. DNN inference generally requires a large amount of real-time processing of these floating-point numbers because it involves multiple layers of complex operations such as convolutional layers, pooling layers, fully connected layers, etc. Further, because these complex operations often involve multiplication, floating-point multipliers are one of the key components in existing DNN solutions. However, floating-point multipliers are extremely expensive in terms of power consumption, silicon area, and latency. Further, although lookup tables (LUTs) can be used to simplify DNN operations in some cases, LUTs similarly require expensive silicon area.

[0021] Therefore, in some cases, DNN optimization techniques can be used to improve performance and / or reduce the necessary silicon area of ​​hardware for implementing DNNs and other types of artificial neural networks. However, these DNN optimization techniques are generally focused on reducing the cost of operations by reducing overall precision and / or reducing the number of underlying operations (e.g., limiting the number of convolutional layers, pooling layers, etc.). In some embodiments, for example, the hardware for implementing DNNs can be designed to operate on floating-point representations with fewer bits and thus providing less precision (e.g., from 8-bit quantized floating points to 16-bit fixed-point representations). However, the use of lower-precision floating-point representations leads to unacceptable accuracy losses, especially for larger data sets, in some cases. In addition, DNN optimization techniques that reduce the number of underlying operations or layers may have adverse effects, such as poor convergence time for reaching a global minimum during DNN training. Further, because these various optimizations still require floating-point multipliers, they still suffer from power, area, and performance limitations of multiplier circuits.

[0022] In some cases, a DNN may be implemented using circuitry that uses a lookup table (LUT) to perform logarithmic, antilogarithmic, and / or exponential calculations to alleviate the need for multiplier circuitry. In some embodiments, for example, a parabolic curve of logarithmic, antilogarithmic, and / or exponential operations may be divided into multiple segments, a curve fitting algorithm may be used to pre-calculate values ​​for corresponding coefficients, and the pre-calculated coefficients may then be stored in a lookup table implemented using a memory component (e.g., ROM). In this way, to calculate ax for any point on the curve, 2 +bx+c, first fetching the values ​​of coefficients a, b, and c from a lookup table, and then using multipliers and adders to calculate the result. However, this approach requires significant silicon area for the associated LUTs and multipliers, and it may also consume multiple clock cycles (e.g., 5 to 8 clock cycles) to calculate the above equation.

[0023] Thus, the present disclosure describes various embodiments that can efficiently perform DNN calculations without relying on hardware for lookup tables and / or multipliers. Example embodiments that can be used to implement the features and functions of the present disclosure will now be described in more detail with reference to the accompanying drawings.

[0024] Figure 1 An example embodiment of implementing a deep neural network (DNN) 100 using logarithmic and antilogarithmic piecewise linear approximation circuits is shown. In the example shown, the DNN 100 is implemented using multiple layers 106a to 106e including a first convolutional layer 106a, a maximum pooling layer 106b, a second convolutional layer 106c, a third convolutional layer 106d, and a fully connected layer 106e. In addition, the DNN 100 is implemented using a multiplier-free neural network microarchitecture that utilizes logarithmic circuits 110 and antilogarithmic circuits 120 instead of multiplier circuits to perform calculations for the corresponding DNN layers 106a to 106e. Specifically, the logarithmic circuits 110 and antilogarithmic circuits 120 perform logarithmic (log2) calculations with a base of 2 and antilogarithmic (antilog2) calculations with a base of 2, which can be used to convert multiplication operations that are typically required in certain DNN layers 106 into additions. Further, the logarithmic circuit 110 and the antilogarithmic circuit 120 use piecewise linear approximations to perform the log2 and antilog2 calculations, which enables each calculation to be performed in a single clock cycle and without the use of lookup tables or multipliers. In this way, the illustrated embodiments reduce DNN processing latency while also eliminating the need for multiplier circuitry and lookup tables, which significantly reduces the necessary silicon area of ​​the hardware. Figures 5A to 5C and FIG. 6A to FIG. 6C Example implementations of logarithmic 110 and anti-logarithmic 120 circuits are further shown and described.

[0025] As an example, with respect to the convolutional layer(s) of a DNN (e.g., layers 106a, 106c, 106d of DNN 100), convolution may generally be represented by the following equation (where f(n) and g(n) are floating point vectors): In this equation, each summation term is calculated using multiplication. However, if we take the log2 of both sides of the equation, the equation becomes: Furthermore, if the left side of the equation is defined as y(n), meaning log2(f*g)(n)=y(n), then the equation becomes: However, the above equation is no longer useful for the purpose of convolution, since convolution cannot be performed by accumulating the results of log2 calculations. Therefore, each summation term must be antilog2ed before it is accumulated (e.g., in order to convert each summation term back from the log2 domain to the original domain): In this alternative equation for convolution, each summation term is now calculated using additions. Thus, while the original convolution equation shown above required multiplications to calculate each summation term, this alternative convolution equation requires additions instead of multiplications. Thus, this alternative equation essentially converts the multiplications required by the original convolution equation into additions using log2 (and antilog2) operations. For example, since f(n) and g(n) in the convolution equation are floating point numbers (e.g., IEEE-754 single precision floating point numbers), the mantissa bits are log2 and antilog2 taken while the exponent bits and sign bits are processed separately, as combined with Figures 5A to 5C as well as Figures 6A to 6C In this manner, logarithmic circuitry 110 and antilogarithmic circuitry 120 may be used to perform convolution using such alternative equations rather than the original equations in order to avoid complex floating point multiplication operations.

[0026] As another example, a fully connected layer (e.g., layer 106e of DNN 100) is the last layer of a DNN and is responsible for performing final reasoning and decision making. In general, a fully connected layer is similar to a convolutional layer, but typically involves a single dimensional vector. Therefore, a fully connected layer can utilize log2 calculations in a similar manner to a convolutional layer to convert multiplication operations into additions. However, because the fully connected layer is the last layer of a DNN, the final output should be in the normal domain rather than the log2 domain.

[0027] For demonstration purposes, a fully connected layer can be represented in general using the following equation: As with convolution, we can take the log2 of both sides of the equation to convert the multiplication into an addition in the summation. fcl Substituting into the left side of the equation, the resulting equation becomes: The corresponding summation term can then be Antilog2'd before being accumulated, thus converting the summation term from the log2 domain back to the normal domain: In this way, the final output of the fully connected layer is in the normal domain rather than the log2 domain. Further, the multiplications required by the original equation have been transformed into additions in this alternative equation.

[0028] In the illustrated embodiment, for example, the DNN 100 is implemented using multiple layers 106a to 106e including a first convolutional layer 106a, a maximum pooling layer 106b, a second convolutional layer 106c, a third convolutional layer 106d, and a fully connected layer 106e. Each layer 106a to 106e performs calculations using inputs (X) 101a to 101e and weight vectors (W) 103a to 103d in certain layers, and generates corresponding outputs (Y) 103a to 103f. In addition, the initial input vector (X) 101a is fed into the first layer 106a of the DNN 100, and each of the remaining layers 106b to 106e is fed with the output (Y) 103a to 103d of the previous layer as its input (X) 101b to 101e.

[0029] Further, the logarithmic circuit 110 and the antilogarithmic circuit 120 implemented using piecewise linear approximation are used to perform calculations at each layer 106 of the DNN 100, thereby eliminating the need for multiplier circuits and lookup tables, while also reducing latency. For example, the logarithmic circuit system 110 performs a log2 calculation to convert a floating point number to a fixed point number, which enables complex operations such as floating point multiplication to be converted to fixed point addition, and the antilogarithmic circuit system 120 performs anitlog2 calculations to subsequently convert the fixed point number back to a floating point number. In addition, the logarithmic circuit 110 and the antilogarithmic circuit 120 use piecewise linear approximation to perform corresponding log2 and antilog2 calculations, which enables each calculation to be performed within a single clock cycle.

[0030] In the illustrated embodiment, for example, the logarithmic circuit system 110 is used to convert the original input vector (X) 101a and each weight vector (W) 103a to 103d into the log2 domain before feeding them into the DNN 100, while the antilogarithmic circuit system 120 is used to convert the final output (Y) 103f of the fully connected layer 106e from the log2 domain back to the normal domain. Further, additional antilog2 and log2 operations (not shown) are also performed throughout the hidden layers (e.g., intermediate layers between the input layer and the output layer) of the DNN 100 to convert between the log2 domain and the normal domain when necessary. For example, as explained above, the convolutional layer needs to convert each summation term back to the normal domain before calculating it, and therefore, anitlog2 operations must be performed before accumulating each summation term. However, the final output of the hidden layer is then converted back to the log2 domain and then provided to the next layer in order to continue to avoid multiplication operations in subsequent layers.

[0031] For example, the result of each hidden layer node is usually passed to an activation function that determines whether the node should be "activated", and the output of the activation function is then fed as input to the next layer. Therefore, in order to avoid multiplication operations in the next layer, the log2 of the activation function of the hidden layer node can be provided to the next layer. For example, after the hidden layer node performs an antilog2 operation for the purpose of calculating the convolution component, the result is converted back to the log2 domain and then passed to the activation function. In this way, when the output (Y) calculated by each hidden layer node is provided as input (X) to the next layer, the output is already in the log2 domain.

[0032] Thus, the illustrated embodiments provide many advantages, including low latency, high accuracy, and reduced power consumption, using a flexible, low-area hardware design that is highly scalable and portable. For example, in the illustrated embodiments, the DNN 100 is implemented using a logarithmic circuit 110 and an antilogarithmic circuit 120 that perform log2 and antilog2 calculations using piecewise linear approximation, which eliminates the need for multiplier circuits and lookup tables in the hardware design. In this way, the illustrated embodiments significantly reduce the necessary silicon area (e.g., by eliminating multipliers and lookup tables), power consumption, and latency of the hardware, while still providing high accuracy. Specifically, the proposed microarchitecture performs each log2 and antilog2 calculation in a single clock cycle, which reduces latency through the data path and therefore reduces the overall latency of the hardware.

[0033] The proposed micro-architecture is also highly scalable. Specifically, the flexible implementation of the proposed micro-architecture allows the hardware to be replicated as needed to increase the number of supported parallel operations. For example, any number of logarithmic circuits 110 and anti-logarithmic circuits 120 can be used to implement the proposed micro-architecture. In this way, the proposed micro-architecture can be easily expanded to support the number of parallel operations required for a specific application or use case. The accuracy of the proposed micro-architecture can also be expanded based on application requirements. For example, if the application requires greater accuracy, the number of segments in the piecewise linear approximation model used by the logarithmic circuit system 110 and the anti-logarithmic circuit system 120 can be increased to accommodate the accuracy requirements. In this way, the proposed micro-architecture is still highly portable because it can be easily ported and / or extended for any product or form factor, including mobile devices (e.g., handheld or wearable devices), drones, servers, and / or any other artificial intelligence solutions that require DNN operations without any dependencies or modifications. DNN activation function circuit using piecewise linear approximation

[0034] As the number of products designed with intelligent (AI) capabilities continues to increase, there is a strong demand for specific hardware that can accelerate basic AI operations (e.g., neural network activation functions) while also remaining versatile enough to support a variety of different implementations and associated algorithms, especially for resource-constrained form factors (e.g., small, low-power edge devices).

[0035] In particular, the increasing popularity of AI solutions that rely on machine learning (e.g., deep learning) has led to a demand for hardware acceleration designed for artificial neural networks (e.g., deep neural networks, convolutional neural networks, feedforward neural networks, recurrent neural networks, etc.). For example, deep neural networks (DNNs) are implemented using multiple layers of "artificial neurons," which are typically processing nodes that use non-linear activation functions to determine whether they should each "activate" in response to a specific input. An activation function, for example, is a function that typically uses a non-linear transformation to map an input to an output in order to determine whether a specific processing node or "artificial neuron" should be activated. The use of activation functions is an important aspect of DNNs, but it can also be very computationally intensive.

[0036] There are many different types of activation functions that can be used in the implementation of DNNs, including Sigmoid function, hyperbolic tangent (Tanh) function, rectified linear unit (ReLU function), leaky ReLU function and Swish function, etc. The choice of (multiple) activation functions has a significant impact on the training dynamics and task performance of DNNs. Therefore, in some cases, multiple activation functions within a single neural network can be used to implement DNNs in order to improve training dynamics and performance. The DNN computing engine may also rely on specific hardware for implementing these activation functions, which typically occupies a considerable area on silicon. For example, hardware designed for the latest DNNs typically operates on single-precision (32-bit) floating-point numbers and uses a lookup table (LUT) method to implement activation functions. However, using a lookup table (LUT) method for activation functions increases silicon area, power consumption, and latency, each of which continues to grow as the number of neurons in the DNN increases. In addition, because each activation function requires its own lookup table, using multiple activation functions in a single DNN increases the number of necessary lookup tables, thereby further affecting silicon area, power, and latency.

[0037] As an example, using a lookup table approach, the curve of the activation function is typically defined between the interval [-m, m] (where 'm' is a real number), and the defined curve can then be divided into multiple segments. The values ​​of the corresponding coefficients can then be pre-calculated using a curve fitting algorithm, and the pre-calculated coefficients can then be stored in a lookup table implemented using a memory component (e.g., ROM). In this way, in order to calculate ax for any point on the curve 2 +bx+c, first fetching the values ​​of coefficients a, b, and c from a lookup table, and then using multipliers and adders to calculate the result. However, this approach requires significant silicon area for the associated lookup tables and multipliers, and it may also consume multiple clock cycles (e.g., 5 to 8 clock cycles) to calculate the above equation.

[0038] To demonstrate, a defined curve divided into 256 uniform segments with 64-bit coefficient widths (a: 20 bits, b: 20 bits, c: 24 bits) on the interval [-3, 3] produces 21 bits of mantissa precision for IEEE-754 single-precision floating point numbers. In some embodiments, this approach requires 256×64 ROM and computation blocks including 41,853 and 5,574 synthesis gates (e.g., NAND equivalent gates), respectively. Scaling down this hardware with lesser precision (e.g., 12-bit or 10-bit precision) will only save ROM area. In some embodiments, for example, the estimated silicon area required for a Sigmoid activation function with 10-bit precision is 17,120 synthesis gates. In addition, this area must be further replicated or instantiated based on the number of parallel operations that the hardware needs to support.

[0039] Therefore, existing hardware for implementing DNN activation functions (e.g., hardware implemented using lookup tables) has various disadvantages, including expensive silicon area requirements, poor power consumption, and high processing latency, etc. These disadvantages are further amplified when the hardware is scaled, such as by increasing the number of artificial neurons, parallel operations, and / or activation functions. Further, there is no unified hardware solution that can implement multiple activation functions without using separate hardware blocks and / or lookup tables for each activation function.

[0040] Accordingly, the present disclosure describes various embodiments of a unified hardware solution that supports multiple DNN activation functions without using lookup tables, as further described below.

[0041] FIG. 2A to FIG. 2B An example embodiment of a unified activation function (AF) circuit 200 for an artificial neural network (e.g., a deep neural network (DNN)) is shown. Specifically, the AF circuit 200 provides support for multiple DNN activation functions on a single hardware component without relying on a lookup table.

[0042] In the illustrated embodiment, for example, the AF circuit 200 implements the corresponding activation function using a novel algorithm that utilizes exponential, logarithmic (log2), and antilogarithmic (antilog2) calculations implemented using piecewise linear approximation to simplify the necessary calculations for each activation function. For example, many activation functions are nonlinear functions that involve complex exponential, division, and / or multiplication operations that are typically implemented using expensive multiplier circuit systems (e.g., division can be implemented using a multiplication circuit system that multiplies the numerator by the reciprocal of the denominator). However, the AF circuit 200 utilizes log2 and antilog2 calculations to eliminate the complex division and / or multiplication operations required for certain activation functions and instead converts them into subtractions and / or additions. Further, the AF circuit 200 implements the exponential, log2, and antilog2 calculations using piecewise linear approximation to further simplify the necessary calculations required for the activation function. Therefore, the log2 and antilog2 calculations can be performed in a single clock cycle, while the exponential calculations can be performed in two clock cycles. In this way, the activation function can be calculated in only five clock cycles, and the underlying calculations can be easily pipelined to increase throughput. Thus, the AF circuit 200 simplifies the underlying calculations of the activation function using log2, antilog2, and exponential calculations implemented using piecewise linear approximations, which eliminates the need for lookup tables, reduces multiplier circuitry requirements, and reduces the overall latency of the activation function. This approach directly translates into significant savings in silicon area (e.g., due to the elimination of lookup tables and reduced multiplier circuitry) because it requires a smaller number of synthesis gates than a typical lookup table approach with similar accuracy.

[0043] In the illustrated embodiment, for example, AF circuit 200 includes logarithm block 210, antilogarithm block 220, and exponent block 230 for performing corresponding log2, antilog2, and exponent calculations using piecewise linear approximation. In some embodiments, for example, logarithm block 210, antilogarithm block 220, and exponent block 230 may be implemented using a 16-segment piecewise linear approximation with 12 bits of precision (i.e., 1 sign bit + 8 exponent bits + 12 mantissa bits = 21 bits of precision) in the mantissa of an IEEE-754 single-precision floating point number. Figures 5A to 5C and FIG. 6A to FIG. 6C and Figure 7 Example implementations of logarithm block 210 , antilogarithm block 220 , and exponentiation block 230 are further shown and described.

[0044] The AF circuit 200 is a configurable circuit that supports the following activation functions: Sigmoid function, hyperbolic tangent (Tanh) function, rectified linear unit (ReLU) function, leaky ReLU function, Swish function. In other embodiments, however, the AF circuit 200 can be designed to support any type or number of activation functions. The AF circuit 200 can be configured to use any of the supported activation functions using an opcode. In the illustrated embodiment, for example, the AF circuit 200 uses a 5-bit opcode to select the type of activation function desired by a particular layer or node in the implementation of the DNN, and the circuit can be reconfigured for other types of activation functions by simply changing the opcode value. In the illustrated embodiment, five opcode bits 202a to 202e are designated as Tanh 202a, Sigmoid 202b, Swish 202c, ReLU 202d, and Leaky ReLU 202e, and these corresponding bit values ​​are set based on the type of activation function desired. Table 1 identifies the hardware configuration of the AF circuit 200 for various supported activation functions based on the values ​​of the opcode bits 202a through 202e. Table 1: Activation function opcodes

[0045] The operation of the AF circuit 200 varies depending on which activation function is selected via the opcode bits 202a to 202e. FIG. 3A to FIG. 3E The functionality of the AF circuit 200 is further discussed.

[0046] Figure 3A It is shown that mathematically it is represented as Graph of the Sigmoid activation function. The output (y-axis) of Sigmoid has a range between 0 and 1, and its shape is similar to a smooth step function, which is an important property that makes it useful as a DNN activation function. Specifically, the function is smooth and continuously differentiable, and the gradient is very steep in the interval -4 to 4. This means that a small change in X will cause a large change in Y, which is an important property of backpropagation in DNNs. However, there are also some disadvantages to the Sigmoid function. For example, Sigmoid suffers from the vanishing gradient problem because the function is almost flat in the region outside of +4 and -4, which results in very small gradients and makes it difficult for DNNs to perform course corrections. In addition, because the output ranges from 0 to 1, the output is not symmetric around the origin, which causes the gradient update to proceed in the positive direction.

[0047] In general, for a given input X represented in single-precision floating-point format, the sigmoid of X or Sigmoid(X) can be calculated using the following equation: However, since the above equation requires an expensive division operation, we can use log2 and antilog2 calculations to avoid the division. For example, based on the properties of the logarithmic function, we can take log2 on each side of the equation to turn the division into a subtraction: However, in order to solve for f(X), we must also take the antilog2 of each side of the equation: This alternative equation for the sigmoid function no longer requires division because the division has been replaced with subtraction and log2 / antilog2 calculations. Further, the exponential, log2, and antilog2 calculations can be implemented using piecewise linear approximations to further simplify the calculations required for this alternative equation.

[0048] Therefore, returning to Figures 2A to 2B , the AF circuit 200 implements the Sigmoid function using the simplified method described above. For example, when the AF circuit 200 is configured for the Sigmoid function, the Sigmoid opcode bit (reference numeral 202b) is set to 1, and the remaining opcode bits of the other activation functions (reference numerals 202a, 202c, 202d, 202e) are set to 0. In this way, when the input X (reference numeral 201) is fed back into the AF circuit 200, the input is passed to the bias block 208 through the multiplexing 206a and the demultiplexing 207, and the bias block adds the bias to the input X to convert it into a negative number (-X). The result -X is then passed to the exponent block 230 to calculate e -X , and then pass the result to adder 212 to calculate 1+e -X . 1+e -X The result is passed to the logarithm block 210b via multiplexer 206d, which then calculates log2(1+e -X ).

[0049] Separately, subtractor 211 is provided with a constant value of 1 as its first operand, and the output of multiplexer 206e is provided as its second operand. In this case, since the Sigmoid opcode bit 202b fed to multiplexer 206e is set to 1, multiplexer 206e selects the constant value of 0 as its output. Therefore, constant values ​​1 and 0 are provided as corresponding operands to subtractor 211, and subtractor 211 thus calculates 1-0=1. The resulting value 1 is then passed to logarithm block 210a through multiplexer 206f, which then calculates log2(1) (which is equal to 0).

[0050] Therefore, logarithm blocks 210a and 210b output log2(1) and log2(1+e -X ) and then pass these results as operands to the adder / subtractor 213. In this case, the adder / subtractor 213 performs subtraction to calculate log2(1)-log2(1+e -X ), and then passes the result to antilog block 220, which performs an antilog2 calculation: In this way, the result calculated by the antilog block 220 corresponds to the final result of the Sigmoid function. For example, based on the properties of the logarithmic function discussed above: Therefore, the AF circuit 200 outputs the result of the antilog block 220 as the final output Y of the Sigmoid function (reference numeral 203 ).

[0051] Further, as noted above, the exponential, log2, and antilog2 calculations performed by the corresponding exponential, logarithmic, and antilogarithmic blocks 210 through 230 of the AF circuit 200 are performed using piecewise linear approximations to further simplify the calculations required by this alternative equation.

[0052] Figure 3B It is shown that mathematically it is represented as Graph of the hyperbolic tangent (Tanh) activation function. This function has an output ranging from -1 to 1 and symmetric around the origin, and it also has a steeper gradient than the Sigmoid function, but it still suffers from the vanishing gradient problem.

[0053] In general, for a given input X represented in single-precision floating-point format, the hyperbolic tangent of X, or Tanh(X), can be calculated using the following equation: Since the above equation requires an expensive division operation, it can be solved by comparing it with the above for Figure 3A The sigmoid function can be used in a similar way to avoid division by using log2 and antilog2 calculations. For example, we can take the log2 of each side of the equation to turn the division into a subtraction: Further, to solve for f(X), we can then take the antilog2 of each side of the equation: This alternative equation for the Tanh function no longer requires division because the division has been replaced by subtraction and log2 / antilog2 calculations. Further, the exponential, log2, and antilog2 calculations can be implemented using piecewise linear approximations to further simplify the calculations required for this alternative equation.

[0054] Therefore, returning to Figures 2A to 2B , the AF circuit 200 implements the Tanh function using the simplified method described above. For example, when the AF circuit 200 is configured for the Tanh function, the Tanh opcode bit (reference numeral 202a) is set to 1, and the remaining opcode bits of the other activation functions (reference numerals 202b, 202c, 202d, 202e) are set to 0. In this way, when the input X (reference numeral 201) is fed into the AF circuit 200, the input first passes through the shifter 204, which shifts X left by a single bit to double its value, thereby generating an output 2X. In addition, since the AF circuit 200 is configured for the Tanh function, the output 2X from the shifter 204 is then passed to the deviation block 208 through the multiplexer 206a and the demultiplexer 207. For example, since the selection signal of multiplexer 206a is based on the Tanh opcode bit 202a being set to 1, multiplexer 206a selects 2X as its output which is passed to demultiplexer 207. Further, since the selection signal of demultiplexer 207 is based on the output of OR gate 205 (which is fed with ReLU / Leaky ReLU opcode bits 202d, 202e both set to 0 as inputs), demultiplexer 207 routes the value of 2X to bias block 208.

[0055] The bias block 208 then adds the bias to 2X to convert it to a negative number (-2X), and then passes the resulting value of -2X to the output value e -2X The output e from the exponential block 230 is then -2X is passed to both subtractor 211 (via multiplexer 206e) and adder 212, and subtractor 211 then calculates 1-e -2XThe value of 1+e is calculated by adder 212. -2X The outputs from subtractor 211 and adder 212 are passed to the respective calculation units for log2(1-e -2X ) and log2(1+e -2X ) values ​​of blocks 210a and 210b.

[0056] The corresponding outputs from logarithm blocks 210a and 210b are then passed as operands to adder / subtractor 213 which performs a subtraction to calculate log2(1-e -2X )-log2(1+e -2X ), and the result is then passed to antilog block 220 which performs an antilog2 calculation: In this way, the result calculated by the antilog block 220 corresponds to the final result of the Tanh function. For example, based on the properties of the logarithmic function discussed above: Therefore, the AF circuit 200 outputs the result of the antilog block 220 as the final output Y of the Tanh function (reference numeral 203).

[0057] Further, as noted above, the exponential, log2, and antilog2 calculations performed by the corresponding exponential, logarithmic, and antilogarithmic blocks 210 through 230 of the AF circuit 200 are performed using piecewise linear approximations to further simplify the calculations required by this alternative equation.

[0058] Figure 3C A graph of a rectified linear unit (ReLU) activation function, mathematically represented as Y=max(0,X), is shown. ReLU is a widely used activation function that provides various advantages. Specifically, ReLU is a nonlinear function that avoids the vanishing gradient problem, is less complex than other activation functions and is therefore computationally cheaper, and has good properties that make DNNs sparse and more efficient (e.g., when its input is negative, its output becomes zero, and therefore the corresponding neuron is not activated). On the other hand, when the output of ReLU becomes zero and ReLU can only be used in the hidden layer of the neural network, the weights cannot be updated during backpropagation.

[0059] In general, for a given input X represented in single-precision floating point format, the ReLU of X or ReLU(X) can be calculated using the following equation: The above equation is simple and does not require any expensive calculations, and therefore its implementation is relatively simple since no exponential, log2 or antilog2 calculations need to be utilized.

[0060] For example, return to Figures 2A to 2B , when the AF circuit 200 is configured for the ReLU function, the ReLU opcode bit (reference numeral 202d) is set to 1, and the remaining opcode bits of the other activation functions (reference numerals 202a, 202b, 202c, 202e) are set to 0. In this way, when the input X (reference numeral 201) is fed into the AF circuit 200, X is first passed to the demultiplexer 207 through the multiplexer 206a, and the demultiplexer 207 then routes X to the multiplexer 206c. Separately, a constant value of 1 is also provided to the multiplexer 206c (via the multiplexer 206b). Further, since the selection signal of the multiplexer 206c is based on the sign bit of X, the multiplexer 206c selects X or 0 as its output, depending on whether X is positive or negative. Since the output of the multiplexer 206 c is the final result of the ReLU function, the remaining logic of the AF circuit 200 is bypassed and the output of the multiplexer 206 c ultimately serves as the final output Y (reference numeral 203 ) of the AF circuit 200 for the ReLU function.

[0061] Figure 3D It is shown that mathematically it is represented as Graph of the Leaky Rectified Linear Unit (LeakyReLU) activation function, where a=0.01. Leaky ReLU is an improved variant of ReLU. For example, with ReLU, when the input is negative, the output and the gradient become zero, which causes problems during the update of the weights in backpropagation. When the input is negative, Leaky ReLU solves this problem using a multiplication of the input by a small linear component (0.01), which prevents neurons from dying and also prevents the gradient from becoming zero.

[0062] In general, for a given input X represented in single-precision floating point format, the Leaky ReLU of X or LeakyReLU(X) can be calculated using the following equation: Like ReLU, the equation for Leaky ReLU is simple and does not require any expensive calculations, and therefore its implementation is relatively simple since no exponential, log2, or antilog2 calculations need to be utilized.

[0063] For example, return to Figures 2A to 2B, when the AF circuit 200 is configured for the Leaky ReLU function, the Leaky ReLU opcode bit (reference numeral 202e) is set to 1, and the remaining opcode bits of the other activation functions (reference numerals 202a, 202b, 202c, 202d) are set to 0. In this way, when the input X (reference numeral 201) is fed into the AF circuit 200, X is first passed to the demultiplexer 207 through the multiplexer 206a, and the demultiplexer 207 then routes X to the multiplexer 206c. Separately, a constant value of 0.01 is also provided to the multiplexer 206c (via the multiplexer 206b). Further, since the selection signal of the multiplexer 206c is based on the sign bit of X, the multiplexer 206c selects either X or 0.01 as its output, depending on whether X is positive or negative. Since the output of the multiplexer 206 c is the final result of the LeakyReLU function, the remaining logic of the AF circuit 200 is bypassed and the output of the multiplexer 206 c ends up as the final output Y (reference numeral 203 ) of the AF circuit 200 for the Leaky ReLU function.

[0064] Figure 3E A graph of the Swish activation function, mathematically represented as Y = X*Sigmoid(X), is shown. In many cases, Swish has been shown to provide better accuracy than other activation functions (e.g., ReLU).

[0065] In general, for a given input X represented in single-precision floating-point format, the following equation can be used to calculate Swish of X or Swish(X): Since the above equation requires an expensive division operation, it can be solved by comparing it with the above for Figure 3A The sigmoid function can be used in a similar way to avoid division by using log2 and antilog2 calculations. For example, we can take the log2 of each side of the equation to turn the division into a subtraction: Further, to solve for f(X), we can then take the antilog2 of each side of the equation: This alternative equation for the Swish function no longer requires division because the division has been replaced by subtraction and log2 / antilog2 calculations. Further, the exponential, log2, and antilog2 calculations can be implemented using piecewise linear approximations to further simplify the calculations required for this alternative equation.

[0066] Therefore, returning to Figures 2A to 2B , the AF circuit 200 implements the Swish function using the simplified method described above. For example, when the AF circuit 200 is configured for the Swish function, the Swish opcode bit (reference numeral 202c) is set to 1, and the remaining opcode bits of the other activation functions (reference numerals 202a, 202b, 202d, 202e) are set to 0. In this way, when the input X (reference numeral 201) is fed into the AF circuit 200, X is passed to the demultiplexer 207 through the multiplexer 206a, and the demultiplexer 207 then routes X to the bias block 208. For example, since the selection signal of the multiplexer 206a is based on the Tanh opcode bit 202a being set to 0, the multiplexer 206a selects X as its output passed to the demultiplexer 207. Further, since the selection signal of the demultiplexer 207 is based on the output of the OR gate 205 (the OR gate is fed with the ReLU / Leaky ReLU opcode bits 202d, 202e both set to 0 as input), the demultiplexer 207 routes the value of X to the bias block 208.

[0067] The bias block 208 then adds the bias to X to convert it to a negative number (-X), and then passes the resulting value -X to the output e -X The output of the exponential block 230 is then -X Passed to adder 212 to calculate 1+e -X , and the result is then passed through multiplexer 206d to logarithm block 210b, which then calculates log2(1+e -X ).

[0068] For example, because the select signal of multiplexer 206f is based on the Swish opcode bit 202c being set to 1, multiplexer 206f selects X as the output passed to logarithm block 210a, which then calculates log2(X).

[0069] The corresponding outputs from logarithm blocks 210a and 210b are then passed as operands to adder / subtractor 213 which performs a subtraction to calculate log2(X)-log2(1+e -X ), and the result is then passed to antilog block 220 which performs an antilog2 calculation: In this manner, the result calculated by the antilog block 220 corresponds to the final result of the Swish function. For example, based on the properties of the logarithmic function discussed above: Therefore, the AF circuit 200 outputs the result of the antilog block 220 as the final output Y of the Swish function (reference numeral 203).

[0070] Further, as noted above, the exponential, log2, and antilog2 calculations performed by the corresponding exponential, logarithmic, and antilogarithmic blocks 210 through 230 of the AF circuit 200 are performed using piecewise linear approximations to further simplify the calculations required by this alternative equation.

[0071] therefore, FIG. 2A to FIG. 2B The illustrated embodiment of the AF circuit 200 provides many advantages, including low latency, high accuracy, and reduced power consumption, using a flexible, low-area hardware design that supports multiple activation functions and is highly scalable and portable. Specifically, the AF circuit 200 is a unified solution for implementing multiple DNN activation functions on a single hardware component without relying on lookup tables (e.g., instead of using separate hardware components for each activation function). For example, in the illustrated embodiment, the AF circuit 200 is implemented using a logarithmic circuit 210, an antilogarithmic circuit 220, and an exponential circuit 230 that utilize piecewise linear approximations to perform log2, antilog2, and exponential calculations, which eliminates the need for lookup tables and reduces the required multiplier circuitry in the hardware design.

[0072] In this manner, the illustrated embodiments significantly reduce the necessary silicon area, power consumption, and latency of the hardware, while still providing high accuracy. For example, the elimination of the lookup table and reduced multiplier circuitry translates directly into significant savings in silicon area, since a much smaller number of synthesis gates are required compared to a typical lookup table approach with similar accuracy. Further, log2 and antilog2 calculations can be performed in a single clock cycle, while exponential calculations can be performed in two clock cycles, which enables activation functions to be calculated in only five clock cycles. In addition, the underlying calculations can be easily pipelined to increase throughput.

[0073] The AF circuit 200 also eliminates the reliance on software for loading / programming lookup tables associated with different activation functions, since the AF circuit 200 can be configured for different activation functions by simply programming the appropriate opcodes. Programming opcodes on the AF circuit 200 is much simpler and requires fewer clock cycles than programming lookup tables for activation functions.

[0074] The AF circuit 200 is also highly scalable. Specifically, the flexible implementation of the AF circuit 200 allows the underlying hardware to be replicated as needed to increase the number of parallel operations supported. In this way, the AF circuit 200 can be easily expanded to support the number of parallel operations required for a specific application or use case. The accuracy of the AF circuit 200 can also be expanded based on application requirements. For example, if an application requires greater accuracy, the number of segments in the piecewise linear approximation model used by the logarithmic circuit system 210, the antilogarithmic circuit system 220, and the exponential circuit system 230 can be increased to accommodate the accuracy requirements. In this way, the AF circuit 200 is also highly portable because it can be easily ported and / or extended for any product or form factor, including mobile devices (e.g., handheld or wearable devices), drones, servers, and / or any other artificial intelligence solution that requires DNN operations without any dependencies or modifications.

[0075] Figure 4 An example embodiment of a unified activation function (AF) circuit 400 for an artificial neural network (deep neural network (DNN)) is shown. Specifically, the AF circuit 400 is similar to the one from FIG. 2A to FIG. 2B The AF circuit 200 of FIG. 2 is similar to FIG. 2. For demonstration, the original equations and modified equations for the Sigmoid function, the Swish function, and the hyperbolic tangent activation function are provided in Table 2. Table 2: Modified activation function equation using powers of 2

[0076] As shown in Table 2, the base-2 exponents in the modified equations are used instead of the base-e exponents in the original equations. In this way, the important nonlinear characteristics of the activation function (e.g., the shape of the curve) are still exhibited by the modified equations, but the underlying activation function hardware can be implemented more efficiently. Specifically, by using base-2 exponents instead of base-e exponents, the modified equations no longer require exponential circuits because all exponential operations can now be performed by antilog circuits. For example, since the base-2 antilog of the variable x is equal to 2 raised to the power of x (2 x ), so an antilogarithm circuit that performs an antilogarithm operation with base 2 can be used to calculate the power of base 2 that appears in the modified activation function equation.

[0077] Furthermore, using antilogarithmic circuitry rather than exponential circuitry to perform exponential operations reduces both latency and silicon area of ​​AF circuit 400. By way of comparison, for example, FIG. 2A to FIG. 2B The AF circuit 200 uses an exponential circuit implemented using a piecewise linear approximation (e.g., Figure 7 The exponential operation is performed by using an exponential circuit that can perform the exponential operation in two clock cycles and requires at least one multiplier. However, the AF circuit 400 uses an antilog circuit implemented using a piecewise linear approximation (e.g., FIG. 6A to FIG. 6C The exponential operation is performed by using an antilogarithm circuit (an antilogarithm circuit) which can perform an antilogarithm operation with a base of 2 in a single clock cycle and does not require a multiplier. Therefore, by using the antilogarithm circuit system instead of the exponential circuit system, the overall latency of the AF circuit 400 is reduced by one clock cycle, thus reducing the latency of the AF circuit 400 by 1 clock cycle. FIG. 2A to FIG. 2B The activation function can be calculated in only four clock cycles, compared to five clock cycles of the activation function circuit of FIG. Further, AF circuit 400 no longer requires any multiplier circuitry, which results in significant silicon area savings because the eliminated exponential circuit is the only component that requires a multiplier. For example, although FIG. 2A to FIG. 2B AF circuit 200 may be implemented using 8,321 gates, but AF circuit 400 uses only 7,221 gates.

[0078] In addition, FIG. 2A to FIG. 2B Similar to the AF circuit 200 of FIG. 4 , the AF circuit 400 utilizes base 2 logarithm (log2) and base 2 antilogarithm (antilog2) calculations using piecewise linear approximation to simplify the necessary calculations for certain activation functions. For example, log2 and antilog2 calculations can be used to eliminate complex division and / or multiplication operations required by certain activation functions and instead convert them into subtraction and / or addition. The logarithmic / antilogarithmic equations for the modified Sigmoid function, Swish function, and hyperbolic tangent activation function from Table 2 (which use powers of 2 instead of powers of e) are provided in Table 3. These logarithmic / antilogarithmic equations for the modified activation functions are derived in a manner similar to the original activation functions, such as in conjunction with FIG. 3A to FIG. 3E Described. Table 3: Logarithmic / anti-logarithmic versions of modified activation functions (using powers of 2)

[0079] In the illustrated embodiment, the AF circuit 400 is designed to implement Sigmoid, Swish, Tanh, ReLU, and Leaky ReLU activation functions. The Sigmoid, Swish, and Tanh activation functions are implemented using the logarithmic / antilogarithmic equations from Table 3, while the FIG. 3C to FIG. 3D The ReLU and Leaky ReLU activation functions are implemented using their original equations because they do not require any complex division, multiplication, or exponential operations. The operation of the AF circuit 400 is similar to FIG. 2A to FIG. 2BThe AF circuit 200 is similar in other respects. Use piecewise linear approximation to implement logarithmic, antilogarithmic, and exponential circuits

[0080] FIG. 5A to FIG. 5C An example embodiment of implementing the logarithmic circuit 500 using piecewise linear approximation is shown. Specifically, Figure 5A shows an overall implementation of the logarithmic circuit 500, and Figure 5B and 5C An implementation of certain components of logarithmic circuit 500 is shown.

[0081] The logarithmic circuit 500 uses a 16-segment piecewise linear approximation to perform logarithmic calculations. In this way, the logarithmic circuit 500 does not require any lookup tables or multiplier circuits, and the logarithmic calculations can be performed in a single clock cycle. The equations used by the logarithmic circuit 500 to perform piecewise linear approximation for logarithmic calculations are shown in Table 4 below. Table 4: Piecewise linear approximation equation for log2(1+m)

[0082] The equations in Table 4 are designed to calculate or approximate the result of log2(1+m), where m represents the mantissa portion of the single-precision floating-point input 501. For example, because the mantissa m is always bounded between 0 and 1, and because the result of log2(0) is undefined, log2(1+m) is calculated instead of log2(m) to avoid undefined results when m is 0. In addition, in order to calculate log2(1+m) using a 16-segment piecewise linear approximation, the possible values ​​of m on the interval [0, 1] are divided into 16 different ranges or segments designated as ranges 0 to 15, and a separate equation is defined for each range to approximate the result of log2(1+m). Further, the corresponding equation is exclusively defined using addition and / or subtraction for any of the following types of operands: m, m divided by a fraction of a power of 2, and / or a constant value. In this way, the only division required by the equation is only by a power of 2, and therefore all division operations can be implemented using shifters. Further, the precision loss caused by limited "shift-based" division is compensated by using constant values ​​that are added and / or subtracted in certain equations. Thus, the corresponding equations can be implemented exclusively using addition, subtraction, and / or shift operations, thereby eliminating the need for complex multiplication / division circuitry.

[0083] Figure 5AThe overall logic of a logarithm circuit 500 designed to implement the equation of Table 4 is shown. In the illustrated embodiment, the logarithm circuit 500 is provided with a 32-bit single-precision floating point number as input 501 (e.g., provided via an input register), and the logarithm circuit 500 calculates a corresponding 32-bit fixed point number as output 503 (e.g., returned via an output register), the fixed point number representing the log2 value of the input 501.

[0084] Input 501 includes a sign bit (input

[31] ), an 8-bit exponent e (input [30:23]), and a 23-bit mantissa m (input [22:0]). Assuming that the sign of input 501 always matches the sign of output 503, the sign bit of input 501 (input

[31] ) is directly fed into the corresponding bit of output 503 (output

[31] ). In addition, the exponent e (input [30:23]) of input 501 is fed into an 8-bit subtractor 502, which subtracts the bias 0x7F from the exponent e to generate a corresponding 8-bit unbiased exponent. From a mathematical perspective, for example, subtracting the bias from the exponent of a floating point number always results in a value equal to the log2 of the exponent. Therefore, the resulting unbiased exponent is used as the integer portion of the fixed point number represented in output 503 (output [30:23]).

[0085] In addition, the mantissa m of input 501 is used to select a corresponding range and equation from Table 4 to be used to calculate the fraction field (output [22:0]) of output 503. For example, the four most significant bits of the mantissa m (input [22:19]) are provided as input to the range selection logic 504, which outputs sixteen 1-bit signals (range [0] to range

[15] ) corresponding to the corresponding ranges of m from Table 4, such that the signals corresponding to the applicable ranges are set to 1 and the remaining signals are set to 0.

[0086] Based on the output of the range selection logic 504, multiplexers (mux) 508a to 508c are then used to select operands corresponding to the selected equation from Table 4, and these operands are then provided as inputs to the adder / subtractor 520. Specifically, the multiplexers 508a to 508c select between various fractions of the mantissa m (e.g., generated using a shift operation) and certain constant values. For example, the mantissa m is fed into a plurality of shifters 506, which each perform a right shift of m by a certain number of bits in order to calculate various fractions of m divided by a power of 2 that appear throughout the equations in Table 4 (e.g., ). Then Figure 5AThe outputs of these shifters 506 are fed as inputs to corresponding multiplexers 508a to 508c in the manner shown in FIG. 4. In addition, a constant value of 0 is provided as one of the inputs to multiplexer 508b because, for certain equations that do not otherwise require an operand from multiplexer 508b, that value is output by multiplexer 508b. Finally, certain equations from Table 4 (e.g., and / or ) The constant value required is generated by the constant generation logic 510 and further provided as an input to the multiplexer 508c. Figure 5B Implementations of constant generation logic 510 are further shown and described.

[0087] Each multiplexer 508a to 508c then selects the appropriate output corresponding to one of the operands for the applicable equations from Table 4 (e.g., based on the range selection logic 504), and then provides these outputs as inputs to the adder / subtractor 520. Further, the mantissa m is also provided directly to the adder / subtractor 520 as another input, since the mantissa is an operand for all equations from Table 4.

[0088] The adder / subtractor 520 then performs the appropriate addition and / or subtraction operations on the various operands provided as inputs, and the calculated results are then used as the 23-bit fraction field (output [22:0]) of output 503. Figure 5C An implementation of adder / subtractor 520 is further shown and described.

[0089] Figure 5B Shows from Figure 5A 4. An example implementation of a constant generation logic 510 of a logarithmic circuit 500 for generating constant value(s) required for certain equations from Table 4 is shown. In the illustrated embodiment, the constant generation logic 510 includes: a shifter 512 for generating a set of constant values ​​that appear throughout the equations from Table 4; multiplexers (mux) 514a, 514b for selecting corresponding constant value(s) for a selected equation from Table 4; and an adder 516 for adding the selected constant value.

[0090] In the illustrated embodiment, a 23-bit constant value of +1 or -1 is provided as input to the corresponding shifter 512 (e.g., depending on whether each shifter 512 generates a positive or negative fractional constant). For example, the 23-bit constant value of +1 is provided as input to all shifters except the shifter that generates a positive result, while the 23-bit sign representation of -1 is provided as input to the only remaining shifter that generates a negative result (e.g., performing a 9-bit right shift to generate a positive result). Each shifter 512 then performs a right shift by a certain number of bits to generate a corresponding fractional constant that appears throughout the equations from Table 4 (e.g., ).

[0091] Furthermore, since the corresponding equation from Table 4 requires zero, one, or two of these fractional constants, two multiplexers 514a, 514b are used to select the appropriate combination of fractional constants for the selected equation from Table 4. Specifically, Figure 5B The output of the shifter 512 is provided as an input to two multiplexers 514a, 514b in the manner shown in Table 4, and a 23-bit constant value of 0 is also provided as an input to each multiplexer 514a, 514b. Each multiplexer 514a, 514b then outputs either a specific fractional constant or a value of 0. In this manner, the multiplexers 514a, 514b can collectively output zero, one, or two of the corresponding fractional constants generated by the shifter 512, depending on the specific equation selected from Table 4 (e.g., as used from Figure 5A The range selection logic 504 of the logarithmic circuit 500 determines the value of FIG.

[0092] The outputs of multiplexers 514a, 514b are then provided as inputs to adder 516 which calculates their sum. The result from adder 516 is then used as the Figure 5B The constant generation logic 510 outputs a final constant value 511 .

[0093] Figure 5C Shows from Figure 5A 4 is an example implementation of adder / subtractor logic 520 of the logarithm circuit 500 for performing addition and / or subtraction on operands of a selected equation from Table 4. In the illustrated embodiment, the adder / subtractor logic 520 includes two adder / subtractors 522a, 522b and two OR gates 524a, 524b as further described below.

[0094] The first adder / subtractor 522a is provided with the mantissa m and the outputs of the multiplexers 508b and 508c of the logarithmic circuit 500 as its operands. In addition, the specific combination of addition and / or subtraction performed by the adder / subtractor 522a on these operands is specified by the OR gate 524a. For example, the OR gate 524a is provided with signals corresponding to the ranges 11 and 15 from Table 4 as inputs (e.g., as generated by the range selection logic 504 of the logarithmic circuit 500), and the output of the OR gate 524a is then fed into the adder / subtractor 522a. In this way, when the output of the OR gate 524a is 0, the adder / subtractor 522a adds all operands together, but when the output of the OR gate 524a is 1 (for example, the mantissa m falls within the range 11 or the range 15), the adder / subtractor 522a subtracts the operand corresponding to the output of the multiplexer 508b and adds the remaining operand. In other words, for any range of m from Table 4 except the ranges 11 and 15, the adder / subtractor 522a outputs the result of m+the output of the multiplexer 508b+the output of the multiplexer 508c, but for the ranges 11 and 15, the adder / subtractor 522a outputs the result of m-the output of the multiplexer 508b+the output of the multiplexer 508c.

[0095] Then, the second adder / subtractor 522b is provided with the output of the first adder / subtractor 522a and the output of the multiplexer 508a of the logarithmic circuit 500 as its corresponding operand. The output of the OR gate 524b specifies whether the adder / subtractor 522b performs addition or subtraction on these operands. For example, the OR gate 524b is provided with a signal corresponding to the range 7 to 15 from Table 4 as an input (e.g., as generated by the range selection logic 504 of the logarithmic circuit 500), and then the output of the OR gate 524b is fed into the adder / subtractor 522b. In this way, when the output of the OR gate 524b is 0, the adder / subtractor 522b adds the two operands together, but when the output of the OR gate 524b is 1 (e.g., the mantissa m falls within the range 7 to 15), the output of the multiplexer 508a is subtracted from the output of the first adder / subtractor 522a. In other words, when m falls within the range of 0 to 6 from Table 4, the adder / subtractor 522b calculates the output of the first adder / subtractor 522a + the output of the multiplexer 508a, but when m falls within the range of 7 to 15, the adder / subtractor 522b calculates the output of the first adder / subtractor 522a - the output of the multiplexer 508a. The result from the second adder / subtractor 522b is used as Figure 5C The final output 521 of the adder / subtractor 520 logic.

[0096] To illustrate the operation of logarithm circuit 500, the processing flow will be described for example input 501. Since the sign field and exponent field of input 501 are always processed in the same manner regardless of their underlying values, this example focuses on the processing associated with the mantissa m of input 501. In this example, it is assumed that the mantissa m of input 501 (input [22:0]) falls within the range 0.125 ≤ m < 0.1875 corresponding to range 2 of Table 4. Therefore, logarithm circuit 500 will execute the corresponding equation for range 2 from Table 4, which is Logarithm circuit 500 begins by processing raw input 501 to generate and / or obtain corresponding operands of the equation referenced above, and logarithm circuit 500 then provides these operands to adder / subtractor 520 to calculate the result of the equation.

[0097] The first operand m corresponding to the mantissa field (input [22:0]) of input 501 is fed directly from the associated bit field of input 501 to adder / subtractor logic 520 .

[0098] The remaining operands of the equation referenced above are provided to adder / subtractor logic 520 via multiplexers 508a through 508c. Specifically, the set of operands that appear throughout the various equations from Table 4 are provided as inputs to multiplexers 508a through 508c, and multiplexers 508a through 508c then output the specific operands required by the appropriate equation from Table 4. For example, based on the four most significant bits of the mantissa m, range select logic 504 outputs a signal identifying the specific range within which m from Table 4 falls, and the range signal is then used by multiplexers 508a through 508c to select the appropriate operand to output. In this example, since m falls within range 2 from Table 4, range select logic 504 outputs a range signal corresponding to range 2, otherwise labeled as a range [2] signal. Based on the range [2] signal, multiplexer 508a selects (m>>2) as its output, multiplexer 508b selects 0 as its output, and multiplexer 508c selects constant 511 generated by constant generation logic 510 as its output. Figure 5B The constant generation logic 510, for example, the range [2] signal causes multiplexers 514a and 514b to select (1>>6) and (-1>>9) as its corresponding output, and then these values ​​are added together by adder 516, and the resulting constant 511 is output by constant generation logic 510. In addition, multiplexer 508c selects this constant 511 as its output, and the constant has a corresponding value

[0099] In this manner, the following operands are finally provided to adder / subtractor 520: m (provided directly from input 501) (Provided by multiplexer 508a) 0 (provided by multiplexer 508b) and (provided by multiplexer 508c). The range [2] signal causes adder / subtractor 520 to perform addition on all of these operands, thus computing the equation for range 2 from Table 4 as The resulting value is therefore used as the 23-bit fraction field (output [22:0]) in output 503 generated by logarithm circuit 500.

[0100] FIG. 6A to FIG. 6C An example embodiment of implementing the antilog circuit 600 using piecewise linear approximation is shown. Specifically, Fig. 6A and 6B An alternative embodiment of the overall antilog circuit 600 is shown, and Figure 6C The underlying adder / subtractor logic 620 of the antilog circuit 600 is shown.

[0101] The antilog circuit 600 uses a 16-segment piecewise linear approximation to perform the antilog calculation. In this way, the antilog circuit 600 does not require any lookup tables or multiplier circuits, and the antilog calculation can be performed in a single clock cycle. The equations used by the antilog circuit 600 to perform the piecewise linear approximation for the logarithmic calculation are shown in Table 5 below. Table 5: antilog2(η)=2 η The piecewise linear approximation equation of

[0102] The equations in Table 5 are designed to calculate or approximate η to the power of 2 or 2 η The result of antilog2 of η, where η represents the fractional portion of the fixed-point input number 601 (input [22:0]). For example, it is common to Figures 5A to 5C The fixed-point input 601 of the antilogarithm circuit 600 is derived from the intermediate DNN calculation performed on the fixed-point output 503 of the logarithm circuit 500. In addition, as discussed above, the fractional part of the fixed-point output 503 of the logarithm circuit 500 is calculated as log2(1+m) instead of log2(m) to avoid undefined results when m=0. Therefore, the calculation for calculating antilog2(η)=2 from FIG.η The equations for are designed to produce a value equal to 1+m, as reflected by the constant value of +1 in each equation.

[0103] In order to calculate antilog2(η)=2 using 16-segment piecewise linear approximation η , for example, dividing the possible values ​​of η on the interval [0, 1] into 16 different ranges or segments designated as the range 0 to 15, and defining a separate equation for each range to approximate antilog2(η) or 2 η Result. Further, the corresponding equation is defined exclusively using addition and / or subtraction for any of the following types of operands: n, fractions of n divided by powers of 2, and / or constant values. In this way, the only divisions required by the equations are just divisions by powers of 2, and thus all division operations can be implemented using shifters. Further, the loss of precision caused by limited "shift-based" divisions is compensated by using constant values ​​that are added and / or subtracted in certain equations. Therefore, the corresponding equations can be implemented exclusively using addition, subtraction, and / or shift operations, thus eliminating the need for complex multiplication / division circuitry.

[0104] Fig. 6A The overall logic of an antilog circuit 600 designed to implement the equation of Table 5 is shown. In the illustrated embodiment, the antilog circuit 600 is provided with a 32-bit fixed point number as input 601 (e.g., provided via an input register), and the antilog circuit 600 calculates a corresponding 32-bit floating point number as output 603 (e.g., returned via an output register), the floating point number representing the antilog2 result of the input 601.

[0105] Input 601 includes a sign bit (input

[31] ), an 8-bit integer (input [30:23]), and a 23-bit fraction (input [22:0]). Assuming that the sign of input 601 always matches the sign of output 603, the sign bit (input

[31] ) of input 601 is fed directly into the corresponding bit (output

[31] ) of output 603. The integer portion (input [30:23]) of input 601 is fed to an 8-bit adder 602, which adds back a bias of 0x7F to generate an 8-bit biased exponent used as the exponent field of the floating point output 603.

[0106] In addition, the fractional portion of input 601 (input [22:0]) corresponding to the value of n in Table 5 is used to select a corresponding range and equation from Table 5 to be used to calculate the mantissa (output [22:0]) of floating point output 603. For example, the four most significant bits (input [22:19]) of the fractional portion of input 601 are provided as input to range selection logic 604, which outputs sixteen 1-bit signals (range [0] to range

[15] ) corresponding to the corresponding ranges of n from Table 5, such that the signals corresponding to the applicable ranges are set to 1 and the remaining signals are set to 0. Based on the output of range selection logic 604, multiplexers (mux) 608a to 608d are then used to select certain operands required for the corresponding equation from Table 5, such as the necessary fraction n and integer constants.

[0107] Specifically, multiplexers 608a and 608b are used to select the fraction n required by the corresponding equation from Table 5. For example, the value of n (input [22:0]) is fed into a first set of shifters 606a, which each perform a right shift of n by a certain number of bits in order to calculate each fraction of n divided by a power of 2 that appears throughout the equations from Table 5 (e.g., ). Then Fig. 6A The outputs of these shifters 606a are fed as inputs to multiplexers 608a and 608b in the manner shown in FIG. Multiplexers 608a and 608b then select the specific fraction η required by the corresponding equation from Table 5, which is determined based on the output of range selection logic 604.

[0108] Similarly, multiplexers 608c and 608d are used to select the fractional constants required for the corresponding equations from Table 5. For example, a 23-bit constant having a value of 1 is fed to a second set of shifters 606b, which each perform a right shift by a certain number of bits to generate the corresponding fractions that appear throughout the equations from Table 5. Then with Fig. 6A The outputs of these shifters 606b are fed as inputs to multiplexers 608c and 608d in the manner shown in FIG. Multiplexers 608c and 608d then select the specific constant fraction required by the corresponding equation from Table 5, which is determined based on the output of range selection logic 604.

[0109] The corresponding operands selected by multiplexers 608a to 608d for the corresponding equations from Table 5 are then provided as inputs to adder / subtractor 620. The value of n (input [22:0]) and the constant value 1 are also provided as inputs to adder / subtractor 620, since these values ​​are operands in all equations from Table 5. Adder / subtractor 620 then performs appropriate addition and / or subtraction operations on these operands as required by the corresponding equations from Table 5 (e.g., based on the output of range selection logic 604). The result from adder / subtractor 620 is then used as the 23-bit mantissa portion (output [22:0]) of the floating point output 603 of antilog circuit 600. The following combination Figure 6C An implementation of adder / subtractor 620 is further shown and described.

[0110] Figure 6B An alternative implementation of the antilog circuit 600 using a piecewise linear approximation is shown. Specifically, when Fig. 6A The antilogarithm circuit is designed to calculate the antilogarithm of a floating point number when it is input. Figure 6B The antilogarithm circuit is capable of computing the antilogarithm of a fixed-point or floating-point number as input. In this way, Figure 6B The antilog circuit can calculate the log2 domain (for example, from Figures 5A to 5C The antilog2 of a fixed-point number in the original domain or a floating-point number in the original domain. Figure 4 As described, the activation function (AF) circuit 400 uses an antilog circuit for the following two purposes: (1) converting a fixed-point number in the log2 domain back to a floating-point number in the original domain; and (2) calculating a floating-point exponent raised to a power of 2. Therefore, the operands of the antilog circuit of the AF circuit 400 include both fixed-point numbers (e.g., for log2 domain conversion) and floating-point numbers (e.g., for powers of 2). Therefore, the antilog circuit of the AF circuit 400 can be used. Figure 6B The antilogarithm circuit of the AF circuit 400 is implemented by an antilogarithm circuit that is capable of processing inputs represented as fixed-point numbers or floating-point numbers.

[0111] Figure 6B The antilogarithmic circuit operates similarly to Fig. 6A operation, except for how it handles the integer or exponent field of input 601 (inputs [30:23]), as further described below.

[0112] Specifically, if input 601 is a fixed-point number, it will include an integer field (input [30:23]) provided as one of the inputs to multiplexer 616. The select signal of multiplexer 616 is a binary signal indicating whether input 601 is a fixed-point number. Therefore, when input 601 is a fixed-point number, the select signal of multiplexer 616 will have a value of 1, which causes multiplexer 616 to select the integer portion of input 601 (input [30:23]) as its output, which is then provided as an operand to 8-bit adder 602. Adder 602 then adds the bias 0x7F back to the integer portion of input 601 (input [30:23]) to generate an 8-bit bias exponent, which is then used as the exponent portion of floating-point output 603.

[0113] Alternatively, if input 601 is a floating point number, it will contain an exponent field (input [30:23]). The exponent field (input [30:23]) is provided to an 8-bit subtractor 612 from which a bias of 0x7F is subtracted to generate a corresponding 8-bit unbiased exponent that mathematically represents the log2 value of the exponent. The output of subtractor 612 is then fed back to shifter 614, which performs a left shift of 2 未偏置_指数 to calculate the corresponding antilog2 value, which is then provided as one of the inputs to multiplexer 616. When input 601 is a floating point number, the select signal of multiplexer 616 will have a value of 0, which causes multiplexer 616 to select the value from shifter 614 as its output. The output of multiplexer 616 is then fed as an operand to 8-bit adder 602 which adds back the bias 0x7F to generate an 8-bit bias exponent, which is then used as the exponent portion of the floating point output 603.

[0114] Figure 6C Shown Fig. 6A and 6B 5 is an example implementation of adder / subtractor logic 620 of the antilogarithmic circuit 600 for performing addition and / or subtraction on operands of corresponding equations from Table 5. In the illustrated embodiment, the adder / subtractor logic 620 includes three adder / subtractors 622a-622c and three OR gates 624a-624c as further described below.

[0115] The first adder / subtractor 622a is provided with the following operands as inputs: the value of n (input [22:0]), a constant value of 1, and the output of the multiplexer 608a. The specific combination of addition and / or subtraction performed on these operands is specified by an OR gate 624a, which is fed with a signal in the range [0] to the range [7] as input. In this way, when n falls within the range 0 to 7 from Table 5, the output of the OR gate 624a will be 1, which causes the adder / subtractor 622a to calculate: n+1-the output of the multiplexer 608a. Alternatively, when n falls within the range 8 to 15, the output of the OR gate 624a will be 0, which causes the adder / subtractor 622a to calculate: n+1+the output of the multiplexer 608a.

[0116] The second adder / subtractor 622b is provided with the following operands as inputs: the output of multiplexer 608c and the output of multiplexer 608d. The specific combination of addition and / or subtraction performed on these operands is specified by an OR gate 624b, which is fed with signals of range [4], range [8], range [9], range

[11] , and range

[12] as inputs. In this way, when n falls within the ranges 4, 8, 9, 11 to 12 from Table 5, the output of OR gate 624a will be 1, which causes the adder / subtractor 622b to calculate: the output of multiplexer 608c-the output of multiplexer 608d. Alternatively, when n falls within any of the remaining ranges from Table 5, the output of OR gate 624a will be 0, which causes the adder / subtractor 622b to calculate: the output of multiplexer 608c+the output of multiplexer 608d.

[0117] The third adder / subtractor 622c is provided with the following operands as inputs: the output of adder / subtractor 622a, the output of adder / subtractor 622b, and the output of multiplexer 608b. The specific combination of addition and / or subtraction performed on these operands is specified by an OR gate 624c, which is fed with the inverse or negation of the signals of range [2], range

[10] , range

[11] , range

[14] , and range

[15] as inputs. In this way, when n falls within any range from Table 5 other than range 2, 10, 11, 14, and 15, the output of OR gate 624c will be 1, which causes adder / subtractor 622c to calculate: the output of adder / subtractor 622a - the output of adder / subtractor 622b - the output of multiplexer 608b. Alternatively, when n falls within the range of 2, 10, 11, 14 or 15 from Table 5, the output of OR gate 624c will be 0, which causes adder / subtractor 622c to calculate: output of adder / subtractor 622a-output of adder / subtractor 622b+output of multiplexer 608b.

[0118] The result from the third adder / subtractor 622c is used as Figure 6C The final output 621 of the adder / subtractor logic 620.

[0119] Figure 7 An example embodiment of implementing the exponential circuit 700 using piecewise linear approximation is shown. Specifically, the exponential circuit 700 uses a 16-segment piecewise linear approximation to perform exponential calculations. In this way, the exponential circuit 700 does not require any lookup tables and only requires one multiplier, and the exponential calculation can be performed within two clock cycles.

[0120] The equations used by the exponential circuit 700 to perform the piecewise linear approximation for the exponential calculation are shown in Table 6 below. Table 6: Index e x The piecewise linear approximation equation of

[0121] The equations in Table 6 are designed to calculate or approximate the natural exponential function e x , where e represents the natural exponential constant (e.g., Euler's number) and x represents the 23-bit mantissa of the 32-bit floating point input 701 (input [22:0]). To calculate e using a 16-segment piecewise linear approximation x , for example, by dividing the possible values ​​of x on the interval [0, 1] into 16 different ranges or segments designated as the range 0 to 15, and defining a separate equation for each range to approximate e x Further, the corresponding equation is defined exclusively using addition and / or subtraction for any of the following types of operands: x, 2x, fractions of x divided by powers of 2, and / or constant values. In this way, the only divisions required by the equation are just divisions by powers of 2, and thus all division operations can be implemented using shifters. Further, the loss of precision caused by limited "shift-based" divisions is compensated by using constant values ​​that are added and / or subtracted in certain equations. Thus, addition, subtraction, and / or shift operations can be used exclusively to implement the equation. x The corresponding equation of .

[0122] To complete the exponential operation, the resulting e must then be x The value of e is multiplied by the value of the exponential portion of input 701 (input [30:23]) raised to the power of e. x represents the mantissa portion of input 701 (input [22:0]) raised to the power of e. Therefore, only one multiplication operation is required for the exponentiation operation, and thus the exponentiation circuit 700 requires only a single multiplier circuit.

[0123] Figure 7 The overall logic of the exponentiation circuit 700 is shown in FIG. In the illustrated embodiment, the exponentiation circuit 700 is provided with a 32-bit floating point number as input 701, and the exponentiation circuit 700 calculates a corresponding 32-bit fixed point number as output 703, the fixed point number being the power of the floating point number represented by the input 701 or e. 输入 Corresponding.

[0124] The floating point input 701 includes a sign bit (input

[31] ), an 8-bit exponent (input [30:23]), and a 23-bit mantissa (input [22:0]). Assuming that the sign of the input 701 always matches the sign of the output 703, the sign bit (input

[31] ) of the input 701 is directly fed into the corresponding bit (output

[31] ) of the output 703.

[0125] The exponent portion of input 701 (input [30:23]) is fed into an 8-bit subtractor 702 which subtracts the bias 0x7F to generate an 8-bit unbiased exponent. The unbiased exponent raised to the natural exponential constant e or e is then output by multiplexer 710. 无偏差_指数 For example, an 8-bit unbiased exponent has a range of -128 to +127. 8 = 256 possible values. Precompute all 256 possible powers of the unbiased exponent of e (e -128 、e -127 ,…,e 0 、e 1 ,…,e 127 ) and feeds it as a constant input to multiplexer 710. The 8-bit unbiased exponent output by subtractor 702 is used as a select signal for multiplexer 710, which causes multiplexer 710 to select the value of e 无偏差_指数 The corresponding pre-calculated constant input. Then the output of multiplexer 710 (e 无偏差_指数 ) is fed to multiplier 712 as one of its operands, as discussed further below.

[0126] The mantissa portion (input [22:0]) of input 701 is processed according to the equation from Table 6. For example, exponentiation circuit 700 is designed to calculate the mantissa (input [22:0]) raised to the power of a natural exponential constant e or e x , where x represents the mantissa (input [22:0]). In addition, based on the equation from Table 6, the exponent circuit 700 uses a piecewise linear approximation to calculate e x Specifically, the mantissa portion of input 701 (output [22:0]) (corresponding to the value of x in Table 6) is used to select from Table 6 the value to be used in calculating e xFor example, the four most significant bits of the mantissa portion of input 701 (input [22:19]) are provided as inputs to range selection logic 704, which outputs sixteen 1-bit signals (range [0] to range

[15] ) corresponding to the corresponding ranges of x from Table 6, such that the signals corresponding to the applicable ranges are set to 1 and the remaining signals are set to 0. Based on the output of range selection logic 704, multiplexers (mux) 708a to 708d are then used to select certain operands required for the corresponding equations from Table 6, such as the necessary fraction x and constant values ​​that appear in the equations.

[0127] For example, multiplexers 708a and 708b are used to select the fraction x required by the corresponding equation from Table 6. For example, the value of x (input [22:0]) is fed into a first set of shifters 706a, which each perform a right shift of x by a certain number of bits in order to calculate each fraction of x divided by a power of 2 that appears throughout the equations from Table 6 (e.g., ). Then Figure 7 The outputs of these shifters 706a are fed as inputs to multiplexers 708a and 708b in the manner shown in. Multiplexers 708a and 708b then select the specific fraction x required by the corresponding equation from Table 6, which is determined based on the output of range selection logic 704.

[0128] Similarly, multiplexers 708c and 708d are used to select constant values ​​required for corresponding equations from Table 6, such as fractional constants and / or a constant value of 1 required for certain equations. For example, a 23-bit constant having a value of 1 is fed to a second set of shifters 706b, which each perform a right shift of a certain number of bits to generate corresponding fractions that appear throughout the equations from Table 6 (e.g., ). Then Figure 7 The outputs of these shifters 706b are fed as inputs to multiplexers 708c and 708d in the manner shown in FIG. 7. Further, a constant value of 1 (which is required by some equations from Table 6) is also fed as one of the inputs to multiplexer 708d. Multiplexers 708c and 708d then select a combination of specific constants required by the corresponding equations from Table 6, which is determined based on the output of range selection logic 704.

[0129] The corresponding operands selected by multiplexers 708a to 708d for the corresponding equations from Table 6 are then provided as inputs to adder / subtractor 720. The value of x (input [22:0]) is also provided to adder / subtractor 720 through one or both of its inputs, depending on whether the corresponding equation from Table 6 requires an operand of x or an operand of 2x. For example, the value of x is always provided directly as one of the inputs of adder / subtractor 720, and in some cases, it may also be provided as another input of adder / subtractor 720 through multiplexer 709. Specifically, the value of x and a 23-bit constant of 0 are provided as inputs to multiplexer 709, and multiplexer 709 selects one of these values ​​as input to adder / subtractor 720. For example, if the corresponding equation from Table 6 requires x instead of 2x as an operand, then multiplexer 709 selects a constant of 0 as its output to adder / subtractor 720 because the value of x has been directly provided through another input of adder / subtractor 720. Alternatively, if the corresponding equation from Table 6 requires 2x instead of x as an operand, then multiplexer 709 selects the value of x as its output to adder / subtractor 720 because this results in the value of x being provided to adder / subtractor 720 through two inputs.

[0130] The adder / subtractor 720 then performs the appropriate addition and / or subtraction operations on these operands as required by the corresponding equation from Table 6 (e.g., based on the output of the range selection logic 704). In this manner, the output of the adder / subtractor 720 is summed with e x The final result corresponds to the e x Equal to the mantissa of input 701 (input [22:0]) raised to the power of the natural exponential constant e.

[0131] The output of multiplexer 710 (the unbiased exponent power of e's input 701) and the output of adder / subtractor 720 (the mantissa power of e's input 701) are then provided as operands to multiplier 712, which multiplies these values ​​together to generate the integer and fractional parts of the fixed-point output 703 of the exponent circuit 700.

[0132] Figure 8 A flowchart 800 of an example processing architecture for implementing an artificial neural network (e.g., a deep neural network (DNN)) is shown. For example, the flowchart 800 can be implemented using the embodiments and functions described throughout this disclosure. For example, in some embodiments, the flowchart 800 can be implemented using the following: Figures 2A to 2B or Figure 4 The activation circuit Figures 5A to 5C The logarithmic circuit, Figures 6A to 6C The antilog circuit and / or Figure 7 The exponential circuit.

[0133] In the flowchart shown, a specific activation function is performed by an activation function circuit designed to accelerate the performance of the activation function. In some embodiments, for example, the activation function circuit can be designed to support multiple types of activation functions that are commonly used to implement artificial or deep neural networks. In addition, the activation function circuit can utilize logarithmic, anti-logarithmic and / or exponential circuits implemented using piecewise linear approximation to accelerate calculations associated with supported activation functions.

[0134] In some embodiments, for example, the activation function circuit can be implemented on a processor, device, and / or system that executes an application with an artificial neural network (e.g., a deep neural network, a convolutional neural network, a feedforward neural network, a recurrent neural network, etc.) and / or used in combination with the processor, the device, and / or the system. In this way, the application can utilize the activation function circuit to accelerate the activation function used to implement the artificial neural network. For example, an application with an artificial neural network can be stored in a memory and executed by a processor on a specific device or system. When the application needs to execute the activation function in conjunction with the operation in the artificial neural network, the application can issue corresponding instructions or commands to the processor and / or the activation function circuit, and the processor can then use the activation function circuit to execute the activation function. The result of the activation function can then be provided back to the processor and / or the application and then used by the artificial neural network.

[0135] The flowchart begins at block 802 where an instruction or command for executing a specific activation function is received. The instruction or command may, for example, be issued by an application and received by a processor and / or activation function circuit. In some embodiments, the instruction or command may identify the desired activation function and any operands or other parameters associated with the activation function.

[0136] Further, in some cases, the selected activation function may include a combination of exponential, multiplication, and / or division operations. Thus, the flowchart first proceeds to block 804 for performing any exponential operations associated with the activation function. In some embodiments, for example, the exponential operations may be performed using piecewise linear approximations to reduce the overhead associated with these operations (e.g., using Figure 7 However, in some embodiments, if the exponential operation involves a base 2, it may be calculated using an antilog circuit (using an antilog operation with base 2) instead of an exponential circuit to reduce latency.

[0137] The flow chart then proceeds to block 806 for performing any multiplication and / or division operations associated with the activation function. In some embodiments, for example, a combination of logarithmic, antilogarithmic, and addition / subtraction operations may be used to perform multiplication and / or division operations for the activation function. For example, by exploiting the properties of the logarithmic function, logarithmic and antilogarithmic operations may be used to convert expensive multiplication / division operations into additions and / or subtractions. Further, the logarithmic and antilogarithmic operations may be performed using piecewise linear approximations to reduce latency associated with these operations.

[0138] In some embodiments, for example, a logarithmic circuit (e.g., Figures 5A to 5C A logarithmic circuit) can be used to perform logarithmic operations using piecewise linear approximation. For example, a logarithmic circuit can be designed to calculate the logarithm of an input represented as a floating point number (e.g., with an exponent and a mantissa), and the logarithmic circuit can represent the resulting output as a fixed point number (e.g., with an integer and a fraction). The logarithmic circuit first identifies the input or operand associated with the logarithmic operation, and then identifies or determines the specific range within which the input falls for the purpose of piecewise linear approximation. For example, multiple ranges or segments and associated equations are defined for performing logarithmic operations using piecewise linear approximation. Therefore, the corresponding range within which the input falls is identified, and the operands required for the range equations are obtained and / or generated. In some embodiments, for example, certain operands can be extracted, obtained and / or calculated using the input, such as the mantissa of the input, and a fractional operand with a denominator that is a power of 2, which can be generated using a shift circuit (e.g., by shifting to perform division). One or more multiplexers are then used to select the appropriate operand, and the result of the equation is calculated using the selected operand. For example, the result can be calculated by using an adder-subtractor circuit to perform addition and / or subtraction on the corresponding operands. In addition, the exponent of the floating-point input can be converted into an unbiased exponent by using a subtractor circuit to subtract the bias. The result from the corresponding equation and the unbiased exponent is then used to generate the output of the logarithmic operation. For example, the unbiased exponent is used as an integer for the resulting fixed-point output, and the result of the equation is used as a fraction for the fixed-point output.

[0139] Similarly, an antilog circuit (e.g. Figures 6A to 6C An antilogarithm circuit) can be used to perform an antilogarithm operation using a piecewise linear approximation. The antilogarithm operation can be implemented in a similar manner, except that a different calculation is used for the integer / exponent portion and a different equation is used for the piecewise linear approximation.

[0140] The flowchart then proceeds to block 808 for outputting the result of the activation function based on the result of the exponential, multiplication, and / or division operation.

[0141] At this point, the flow chart can be complete. However, in some embodiments, the flow chart can restart and / or repeat certain blocks. For example, in some embodiments, the flow chart can start at block 802 to continue executing the activation function. DNN Performance

[0142] FIG. 9A to FIG. 9B The scalability of the described embodiments with respect to the number of parallel operations supported is demonstrated. Specifically, the described embodiments are implemented using a flexible design that enables duplication of the underlying hardware in order to increase the number of parallel operations supported. In this way, the described embodiments can be expanded as necessary to support the number of parallel operations required for a specific application or use case. For example, as described by Figures 9A to 9B As shown, the proposed solution can be simply replicated to extend it from a single operand to n operands. Specifically, Fig.9A shows the proposed solution for a single operand, while Fig. 9B The same solution for n operands is shown. Further, the proposed solution can be pipelined to reduce latency and improve throughput.

[0143] Fig. 10A The scalability of the described embodiments with respect to accuracy is demonstrated. Specifically, the described embodiments can be extended to provide different accuracy levels by simply adjusting the number of segments in the piecewise linear approximation model implemented by the logarithmic, antilogarithmic and / or exponential circuits. In this way, the described embodiments can be extended as necessary to provide the accuracy levels required by different applications and use cases. For example, if an application requires higher accuracy, the number of segments included in the piecewise linear approximation can be increased to accommodate the higher accuracy requirements.

[0144] Fig. 10A The number of segments required for different accuracy levels in the piecewise linear approximation model is shown. Fig. 10A As shown, for example, if an application requires 23 bits of precision in addition to a 23-bit mantissa, then at least 1,556 segments should be used to implement the piecewise linear approximation model. The number of segments used for the piecewise linear approximation can be adjusted in a similar manner to provide any necessary level of precision.

[0145] Fig. 10B The silicon area requirements of various implementations of logarithmic and antilogarithmic hardware are compared. Specifically, Fig. 10B The number of synthesis gates in the logarithmic and antilogarithmic circuits is shown to be comparable between a solution implemented using a lookup table approach (with 12-bit precision) and a solution implemented using a piecewise linear approximation (PLA) (with 10-bit or 12-bit precision). Fig. 10BAs shown in , the logarithm and antilogarithm circuits implemented using piecewise linear approximation use significantly fewer gates than the solution implemented using the lookup table method. For example, the lookup table solution uses 8,800 gates for 12-bit precision, while the piecewise linear approximation (PLA) logarithm and antilogarithm circuits use 350 and 450 gates for 10-bit precision and 1,048 and 1,348 gates for 12-bit precision, respectively.

[0146] Fig. 10C The silicon area requirements of various implementations of activation function hardware (with 10-bit precision) are compared. Specifically, Fig. 10C The number of synthesis gates for various individual activation functions (e.g., Sigmoid, Tanh, and Swish) implemented using a lookup table (LUT) is shown compared to a unified activation function circuit (e.g., Figures 2A to 2B The comparison of the number of synthesis gates of the activation function circuit 200). Fig. 10C As shown in , for example, the unified circuit uses significantly fewer gates than any of the LUT-based circuits, yet it still supports multiple activation functions. For example, the unified circuit can be implemented using a total of 8,321 synthesis gates because its simplified equations enable the exponential, logarithmic, and antilogarithmic blocks to be implemented using only 4,387, 1,048, and 1,348 synthesis gates, respectively. By comparison, each LUT-based circuit requires approximately 17,000 to 18,000 synthesis gates for only a single activation function.

[0147] Fig. 10D The use of piecewise linear approximation to implement logarithmic and antilogarithmic circuits (e.g., Figures 5A to 5C and Figures 6A to 6C ), and Fig.10E Exponential circuits implemented using piecewise linear approximation are shown (e.g., Figure 7 Specifically, for the corresponding log2 and antilog2 and exponent calculations, these circuits can achieve an absolute error (ABS) of 0.045%, for IEEE-754 single-precision floating point numbers (e.g., 1 sign bit + 8 exponent bits + 12-bit mantissa = 21 bits of precision), which these circuits convert to 12-bit precision of the mantissa.

[0148] For IEEE-754 single-precision floating point numbers (e.g., 1 sign bit + 8 exponent bits + 10 mantissa bits = 19 bits of precision), a unified activation function circuit (e.g., Figures 2A to 2B The overall accuracy of the AF circuit 200) is 10 mantissa bits.

[0149] Figures 11A to 11C A comparison of the performance of a deep neural network (DNN) implemented using a traditional activation function equation (with an exponential constant e) and a modified activation function equation (using a power of 2) is presented. For example, as discussed above, to implement the activation function, Figures 2A to 2B The AF circuit 200 and Figure 4 Both AF circuits 200 and 400 utilize piecewise linear approximation. However, AF circuit 200 implements a conventional activation function, while AF circuit 400 implements a modified activation function that uses powers of 2 instead of powers of the exponential constant e. Figures 11A to 11C The performance of these corresponding methods is compared in the presented examples.

[0150] Fig.11A An example DNN 1100 is shown, which includes an input layer with two input neurons, a single hidden layer with three neurons, and a fully connected layer with one input neuron. For simplicity, DNN 1100 implements the feature mapping with the same behavior of the XOR gate shown in Table 7. Table 7: Mapping of DNN feature sets Feature A Feature B Output 0 0 0 0 1 1 1 0 1 1 1 0

[0151] The example shown focuses on one of these feature sets that is expected to have an output of 0, namely, A = 1 and B = 1. In the example shown, input neuron X1 corresponds to feature A, and input neuron X2 corresponds to feature B.

[0152] Fig. 11B shows the processing performed when implementing DNN 1100 using the traditional Sigmoid activation function: During the forward propagation phase, the weights are randomly selected as follows: W11 = 0.8, W12 = 0.4, W13 = 0.3, W21 = 0.2, W22 = 0.9, WW23 = 0.5, Wh1 = 0.3, Wh2 = 0.5, and Wh3 = 0.9. The output of the hidden layer node (H) is then calculated as follows: H1=X1*W11+X2*W21=0.8+0.2=1; H2=X1*W12+X2*W22=0.4+0.9=1.3; H3=X1*W13+X2*W23=0.3+0.5=0.8.

[0153] For simplicity, a bias of 0 is used in the examples shown. After applying the Sigmoid activation function, the updated hidden layer neurons have the following values: H1=0.7310; H2=0.7858; H3=0.6899.

[0154] The fully connected (FC) layer can then be calculated as: FC = H1*Wh1 + H2*Wh2 + H3*Wh3 = 0.7310*0.3 + 0.7858*0.5 + 0.6899*0.9 = 1.235. After applying the Sigmoid activation function to the fully connected layer, FC = 0.7746. The error is then calculated as follows: Error = Expected - Calculated = 0 - 0.7746 = -0.7746.

[0155] During the backward propagation and weight update phase, the derivative of the Sigmoid activation function is used And perform the following calculations: ΔFC = f'(FC value without activation function)*error = f'(1.235)*(-0.7746)=-0.13439; ΔFC = hidden layer neurons * hidden layer weight = H1*Wh1+H2*Wh2+H3*Wh3; ΔWh1=ΔFC / H1=-0.13439 / 0.7310=-0.1838; ΔWh2=ΔFC / H2=-0.13439 / 0.7858=-0.1710; ΔWh3=ΔFC / H3=-0.13439 / 0.6899=-0.1920.

[0156] The following calculations are then performed with respect to ΔH1, ΔH2, and ΔH3: New hidden layer value = ΔFC*f` (hidden layer neuron value without activation function) / hidden layer weight: ΔH1 = ΔFC*f` (1) / Wh1 = -0.13439*0.1966 / 0.3 = -0.088; ΔH2=ΔFC*f`(1.3) / Wh2=-0.13439*0.1683=-0.0452; ΔH1=ΔFC*f`(0.8) / Wh3=-0.13439*0.2139=-0.0319.

[0157] After back-propagation, the new hidden layer weights have the following values: Wh1=Wh1+ΔWh1=0.3-0.1838=0.1162; Wh2=Wh2+ΔWh2=0.5-0.1710=0.392; Wh3=Wh3+ΔWh3=0.9-0.1920=0.708.

[0158] Then calculate the weight update between the input layer and the hidden layer as follows: Weight increment = hidden layer increment / input: ΔW11=ΔH1 / X1=-0.088 / 1=-0.088; ΔW12=ΔH2 / X1=-0.0452 / 1=-0.0452; ΔW13=ΔH3 / X1=-0.0319 / 1=-0.0319; ΔW21=ΔH1 / X2=-0.088 / 1=-0.088; ΔW22=ΔH2 / X2=-0.0452 / 1=-0.0452; ΔW23=ΔH3 / X2=-0.0319 / 1=-0.0319; New weights: W11=W11+ΔW11=0.8-0.088=0.712; W12=W12+ΔW12=0.4-0.0452=0.3548; W13=W13+ΔW13=0.3-0.0319=0.2681; W21=W21+ΔW21=0.2-0.088=0.112; W22=W22+ΔW22=0.9-0.0452=0.8548; W23=W23+ΔW23=0.5-0.0319=0.4681.

[0159] Fig. 11B The state of DNN 1100 is shown after the weights have been updated based on the above calculations. After this iteration, the output of the DNN is 0.69, which is an improvement over the output of 0.77 after the first iteration. Iterations continue in this manner until the loss function gradient reaches an acceptable level.

[0160] Fig. 11C 1 shows the processing performed when the DNN 1100 is implemented using a modified Sigmoid activation function having an exponential term with a base of 2 instead of a base of e:

[0161] During the forward propagation phase, the weights are randomly selected as follows: W11 = 0.8, W12 = 0.4, W13 = 0.3, W21 = 0.2, W22 = 0.9, WW23 = 0.5, Wh1 = 0.3, Wh2 = 0.5, and Wh3 = 0.9. The output of the hidden layer node (H) is then calculated as follows: H1=X1*W11+X2*W21=0.8+0.2=1; H2=X1*W12+X2*W22=0.4+0.9=1.3; H3=X1*W13+X2*W23=0.3+0.5=0.8.

[0162] For simplicity, a bias of 0 is used in the examples shown. After applying the modified Sigmoid activation function, the updated hidden layer neurons have the following values: H1=0.6667; H2=0.7117; H3=0.6351.

[0163] The fully connected (FC) layer can then be calculated as: FC = H1*Wh1 + H2*Wh2 + H3*Wh3 = 0.6667*0.3 + 0.7117*0.5 + 0.6351*0.9 = 1.1272. After applying the modified Sigmoid activation function to the fully connected layer, FC = 0.6859. The error is then calculated as follows: Error = Expected - Calculated = 0 - 0.6859 = -0.6859.

[0164] During the backward propagation and weight update phase, the derivative of the modified Sigmoid activation function is used And perform the following calculations: ΔFC = f'(FC value without activation function)*error = f'(1.1272)*(-0.6859)=-0.1024; ΔFC = hidden layer neurons * hidden layer weight = H1*Wh1+H2*Wh2+H3*Wh3; ΔWh1=ΔFC / H1=-0.1024 / 0.6667=-0.15364; ΔWh2=ΔFC / H2=-0.1024 / 0.7117=-0.14402; ΔWh3=ΔFC / H3=-0.1024 / 0.6351=-0.16125.

[0165] The following calculations are then performed with respect to ΔH1, ΔH2, and ΔH3: New hidden layer value = ΔFC*f` (hidden layer neuron value without activation function) / hidden layer weight: ΔH1=ΔFC*f`(1) / Wh1=-0.13439*0.1966 / 0.3=-0.05259; ΔH2=ΔFC*f`(1.3) / Wh2=-0.13439*0.1683=-0.02917; ΔH1=ΔFC*f`(0.8) / Wh3=-0.13439*0.2139=-0.01828.

[0166] After back-propagation, the new hidden layer weights have the following values: Wh1=Wh1+ΔWh1=0.3-0.15364=0.1463; Wh2=Wh2+ΔWh2=0.5-0.14402=0.3559; Wh3=Wh3+ΔWh3=0.9-0.16125=0.7387.

[0167] Then calculate the weight update between the input layer and the hidden layer as follows: Weight increment = hidden layer increment / input: ΔW11=ΔH1 / X1=-0.05259 / 1=-0.05259; ΔW12=ΔH2 / X1=-0.02917 / 1=-0.02917; ΔW13=ΔH3 / X1=-0.01828 / 1=-0.01828; ΔW21=ΔH1 / X2=-0.02917 / 1=-0.05259; ΔW22=ΔH2 / X2=-0.0452 / 1=-0.02917; ΔW23=ΔH3 / X2=-0.01828 / 1=-0.01828; New weights: W11=W11+ΔW11=0.8-0.05259=0.7474; W12=W12+ΔW12=0.4-0.02917=0.3708; W13=W13+ΔW13=0.3-0.01828=0.2817; W21=W21+ΔW21=0.2-0.05259=0.1474; W22=W22+ΔW22=0.9-0.02917=0.8708; W23=W23+ΔW23=0.5-0.01828=0.4817.

[0168] Fig. 11C1 shows the state of DNN 1100 after the weights have been updated based on the above calculations (e.g., using a modified Sigmoid activation function). After this iteration, Fig. 11C The output of the DNN is 0.70, which is equivalent to Fig. 11B The output of the DNN is 0.69. Iterations continue in this way until the loss function gradient reaches an acceptable level.

[0169] Figures 12A to 12B Shows the Fig. 11B and 11C Comparison of the training convergence of the corresponding DNN. Specifically, Fig. 12A Shown by Fig. 11B The convergence speed of the original Sigmoid activation function used by the DNN is Fig. 12B Shown by Fig. 11C The convergence rate of a modified Sigmoid activation function (e.g., using powers of 2 instead of e) used by a DNN. Fig.13 The percentage error or accuracy of these methods is shown. Example Compute Architecture

[0170] Figure 14 to Fig.18 Example implementations of computer environments and architectures that can be used according to the embodiments disclosed herein are presented. In various embodiments, for example, these example computer architectures can be used in conjunction with and / or to implement the deep neural network (DNN) processing functions described throughout this disclosure.

[0171] Other system designs and configurations known in the art for laptop computers, desktop computers, handheld PCs, personal digital assistants, engineering workstations, servers, network devices, network hubs, switches, embedded processors, digital signal processors (DSPs), graphics devices, video game devices, set-top boxes, microcontrollers, cellular phones, portable media players, handheld devices, and various other electronic devices are also suitable. In general, a wide variety of systems or electronic devices that can incorporate a processor and / or other execution logic as disclosed herein are generally suitable.

[0172] Fig.14A is a block diagram illustrating both an exemplary in-order pipeline and an exemplary register renaming out-of-order issue / execution pipeline according to embodiments of the present invention. Fig. 14B is a block diagram illustrating both an exemplary embodiment of an in-order architecture core and an exemplary register renaming out-of-order issue / execution architecture core to be included in a processor in accordance with an embodiment of the present invention. Figures 14A to 14BThe solid line boxes in show the in-order pipeline and in-order core, while the optional addition of dashed line boxes shows the register renaming out-of-order issue / execution pipeline and core. The out-of-order aspects will be described assuming that the in-order aspects are a subset of the out-of-order aspects.

[0173] exist Fig.14A , the processor pipeline 1400 includes a fetch stage 1402, a length decode stage 1404, a decode stage 1406, an allocation stage 1408, a rename stage 1410, a scheduling (also known as dispatch or issue) stage 1412, a register read / memory read stage 1414, an execute stage 1416, a write back / memory write stage 1418, an exception handling stage 1422, and a commit stage 1424.

[0174] Fig. 14B A processor core 1490 is shown that includes a front end unit 1430 coupled to an execution engine unit 1450, and both the execution engine unit and the front end unit are coupled to a memory unit 1470. The core 1490 may be a reduced instruction set computing (RISC) core, a complex instruction set computing (CISC) core, a very long instruction word (VLIW) core, or a hybrid or alternative core type. As yet another option, the core 1490 may be a special purpose core, such as, for example, a network or communication core, a compression engine, a coprocessor core, a general purpose computing graphics processing unit (GPGPU) core, a graphics core, etc.

[0175] The front end unit 1430 includes a branch prediction unit 1432 coupled to an instruction cache unit 1434, which is coupled to an instruction translation lookaside buffer (TLB) 1436, which is coupled to an instruction fetch unit 1438, which is coupled to a decode unit 1440. The decode unit 1440 (or decoder) can decode instructions and generate one or more micro-operations, microcode entry points, microinstructions, other instructions, or other control signals as outputs, which are decoded from the original instructions, or otherwise reflect the original instructions, or are derived from the original instructions. The decode unit 1440 can be implemented using a variety of different mechanisms. Examples of suitable mechanisms include, but are not limited to: a lookup table, a hardware implementation, a programmable logic array (PLA), a microcode read-only memory (ROM), etc. In one embodiment, the core 1490 includes a microcode ROM or other medium that stores microcode for certain macroinstructions (e.g., in the decode unit 1440 or within the front end unit 1430). Decode unit 1440 is coupled to rename / allocator unit 1452 in execution engine unit 1450 .

[0176] The execution engine unit 1450 includes a rename / allocator unit 1452 coupled to 1454 and a set of one or more scheduler units 1456. The (multiple) scheduler units 1456 represent any number of different schedulers, including reservation stations, central instruction windows, etc. The (multiple) scheduler units 1456 are coupled to the (multiple) physical register file units 1458. Each of the (multiple) physical register file units 1458 represents one or more physical register files, where different physical register files store one or more different data types, such as scalar integers, scalar floating points, packed integers, packed floating points, vector integers, vector floating points, status (e.g., an instruction pointer addressing the next instruction to be executed), etc. In one embodiment, the (multiple) physical register file units 1458 include vector register units, write mask register units, and scalar register units. These register units can provide architectural vector registers, vector mask registers, and general purpose registers. The physical register file(s) unit 1458 overlaps with the retirement unit 1454 to demonstrate various ways in which register renaming and out-of-order execution may be implemented (e.g., using reorder buffer(s) and retirement register file(s); using future stack(s), history buffer(s) and retirement register file(s); using register map and register pool; etc.). The retirement unit 1454 and the physical register file(s) unit 1458 are coupled to the execution cluster(s) 1460. The execution cluster(s) 1460 includes a set of one or more execution units 1462 and a set of one or more memory access units 1464. The execution unit 1462 may perform various operations (e.g., shifts, additions, subtractions, multiplications), and on various types of data (e.g., scalar floating point, packed integer, packed floating point, vector integer, vector floating point). While some embodiments may include multiple execution units dedicated to specific functions or sets of functions, other embodiments may include only one execution unit or include multiple execution units that all perform all functions. The scheduler unit(s) 1456, physical register file(s) unit(s) 1458, and execution cluster(s) 1460 are shown as possibly plural because some embodiments create separate pipelines for certain types of data / operations (e.g., scalar integer pipelines, scalar floating point / packed integer / packed floating point / vector integer / vector floating point pipelines, and / or memory access pipelines, each of which has its own scheduler unit, physical register file(s) unit(s), and / or execution cluster(s), and in the case of separate memory access pipelines, certain embodiments are implemented in which only the execution cluster of this pipeline has the memory access unit(s) 1464). It should also be understood that where separate pipelines are used, one or more of these pipelines may be out-of-order issue / execution pipelines, and the rest are in-order pipelines.

[0177] The set of memory access units 1464 are coupled to a memory unit 1470, which includes a data TLB unit 1472 coupled to a data cache unit 1474, which is coupled to a level 2 (L2) cache unit 1476. In an exemplary embodiment, the memory access unit 1464 may include a load unit, a store address unit, and a store data unit, each coupled to the data TLB unit 1472 in the memory unit 1470. The instruction cache unit 1434 is further coupled to a level 2 (L2) cache unit 1476 in the memory unit 1470. The L2 cache unit 1476 is coupled to one or more other levels of cache and ultimately to the main memory.

[0178] By way of example, an exemplary register renaming, out-of-order issue / execution core architecture may implement pipeline 1400 as follows: 1) instruction fetch 1438 performs fetch and length decode stages 1402 and 1404; 2) decode unit 1440 performs decode stage 1406; 3) rename / allocator unit 1452 performs allocate stage 1408 and rename stage 1410; 4) (multiple) scheduler units 1456 perform schedule stage 1412; 5) (multiple) physical register file units 1458 and memory units 1470 perform register read / memory read stage 1414; execution cluster 1460 performs the execute stage 1416; 6) memory unit 1470 and the (multiple) physical register file units 1458 perform write back / memory write stage 1418; 7) various units may be involved in exception handling stage 1422; and 8) retirement unit 1454 and the (multiple) physical register file units 1458 perform commit stage 1424.

[0179] Core 1490 may support one or more instruction sets (e.g., the x86 instruction set (with some extensions that have been added with newer versions); the MIPS instruction set of MIPS Technologies, Inc. of Sunnyvale, California; the ARM instruction set of ARM Holdings, Inc. of Sunnyvale, California (with optional additional extensions, such as NEON)), including the instruction(s) described herein. In one embodiment, core 1490 includes logic to support packed data instruction set extensions (e.g., AVX1, AVX2), allowing operations used by many multimedia applications to be performed using packed data.

[0180] It should be understood that a core may support multithreading (executing two or more sets of operations or threads in parallel) and may accomplish such multithreading in a variety of ways, including time-division multithreading, simultaneous multithreading (where a single physical core provides a logical core for each of the threads that the physical core is simultaneously multithreading), or a combination thereof (e.g., time-division fetching and decoding and subsequent processing such as Hyper-Threading (Simultaneous Multithreading).

[0181] Although register renaming is described in the context of out-of-order execution, it should be understood that register renaming can be used in an in-order architecture. Although the illustrated embodiment of the processor also includes separate instruction and data cache units 1434 / 1474 and a shared L2 cache unit 1476, alternative embodiments may have a single internal cache for both instructions and data, such as a level 1 (L1) internal cache, or multiple levels of internal cache. In some embodiments, the system may include a combination of internal caches and external caches external to the core and / or processor. Alternatively, all caches may be external to the core and / or processor.

[0182] Fig.15 is a block diagram of a processor 1500 that may have more than one core, may have an integrated memory controller, and may have integrated graphics in accordance with an embodiment of the present invention. Fig.15 The solid line box in the figure shows a processor 1500 having a single core 1502A, a system agent 1510, and a set of one or more bus controller units 1516, while the optional addition of the dashed line box shows a processor 1500 having multiple cores 1502A to 1502N, a set of one or more integrated memory controller units 1514 in the system agent unit 1510, and dedicated logic 1508.

[0183] Thus, different implementations of processor 1500 may include: 1) a CPU, where dedicated logic 1508 is integrated graphics and / or scientific (throughput) logic (which may include one or more cores), and cores 1502A to 1502N are one or more general purpose cores (e.g., general purpose in-order cores, general purpose out-of-order cores, a combination of both); 2) a coprocessor, where cores 1502A to 1502N are a large number of dedicated cores intended primarily for graphics and / or scientific (throughput); and 3) a coprocessor, where cores 1502A to 1502N are a large number of general purpose in-order cores. Thus, processor 1500 may be a general purpose processor, a coprocessor, or a special purpose processor, such as, for example, a network or communication processor, a compression engine, a graphics processor, a GPGPU (general purpose graphics processing unit), a high throughput integrated many-core (MIC) coprocessor (including 30 or more cores), an embedded processor, etc. The processor may be implemented on one or more chips. Processor 1500 may be part of one or more substrates and / or may be implemented on one or more substrates using any of a variety of processing technologies such as, for example, BiCMOS, CMOS, or NMOS.

[0184] The memory hierarchy includes one or more levels of cache within the core, a set or one or more shared cache units 1506, and external memory (not shown) coupled to the set of integrated memory controller units 1514. The set of shared cache units 1506 may include one or more mid-level caches, such as level 2 (L2), level 3 (L3), level 4 (L4), or other levels of cache, a last level cache (LLC), and / or combinations thereof. While in one embodiment, a ring-based interconnect unit 1512 interconnects the integrated graphics logic 1508, the set of shared cache units 1506, and the system agent unit 1510 / (multiple) integrated memory controller units 1514, alternative embodiments may use any number of known techniques for interconnecting such units. In one embodiment, coherency is maintained between the one or more cache units 1506 and the cores 1502A to 1502N.

[0185] In some embodiments, one or more of the cores 1502A to 1502N are capable of multithreading. System agent 1510 includes those components that coordinate and operate cores 1502A to 1502N. System agent unit 1510 may include, for example, a power control unit (PCU) and a display unit. The PCU may be or include logic and components required for regulating the power state of cores 1502A to 1502N and integrated graphics logic 1508. The display unit is used to drive one or more externally connected displays.

[0186] The cores 1502A-1502N may be homogeneous or heterogeneous with respect to the architectural instruction set; that is, two or more of the cores 1502A-1502N may be capable of executing the same instruction set, while other cores may be capable of executing only a subset of the instruction set or a different instruction set.

[0187] Now refer to Fig.16 , a block diagram of a system 1600 according to one embodiment of the present invention is shown. The system 1600 may include one or more processors 1610, 1615 coupled to a controller hub 1620. In one embodiment, the controller hub 1620 includes a graphics memory controller hub (GMCH) 1690 and an input / output hub (IOH) 1650 (which may be on separate chips); the GMCH 1690 includes memory and a graphics controller, to which memory 1640 and a coprocessor 1645 are coupled; and the IOH 1650 couples an input / output (I / O) device 1660 to the GMCH 1690. Alternatively, one or both of the memory and the graphics controller are integrated within the processor (as described herein), and the memory 1640 and the coprocessor 1645 are directly coupled to the processor 1610 and the controller hub 1620 with the IOH 1650 in a single chip.

[0188] Fig.16 The optional nature of the additional processor 1615 is indicated by dashed lines in FIG. 16 . Each processor 1610 , 1615 may include one or more of the processing cores described herein and may be some version of processor 1500 .

[0189] The memory 1640 may be, for example, a dynamic random access memory (DRAM), a phase change memory (PCM), or a combination of both. For at least one embodiment, the controller hub 1620 communicates with the processor(s) 1610, 1615 via a multi-drop bus (e.g., a front side bus (FSB), a point-to-point interface such as a Quick Path Interconnect (QPI), or a similar connection 1695).

[0190] In one embodiment, coprocessor 1645 is a special purpose processor such as, for example, a high throughput MIC processor, a network or communication processor, a compressor engine, a graphics processor, a GPGPU, an embedded processor, etc. In one embodiment, controller hub 1620 may include an integrated graphics accelerator.

[0191] There may be various differences between the physical resources 1610 , 1615 in terms of a range of metrics including architectural, microarchitectural, thermal, power consumption characteristics, etc.

[0192] In one embodiment, processor 1610 executes instructions that control general types of data processing operations. Coprocessor instructions may be embedded in the instructions. Processor 1610 identifies these coprocessor instructions as being of a type that should be executed by an attached coprocessor 1645. Therefore, processor 1610 issues these coprocessor instructions (or control signals representing coprocessor instructions) on a coprocessor bus or other interconnect to coprocessor 1645. Coprocessor(s) 1645 accept and execute the received coprocessor instructions.

[0193] Now refer to Fig.17 , a block diagram of a first more specific exemplary system 1700 according to an embodiment of the present invention is shown. Fig.17 As shown in FIG. 1 , multiprocessor system 1700 is a point-to-point interconnect system and includes a first processor 1770 and a second processor 1780 coupled via a point-to-point interconnect 1750. Processor 1770 and processor 1780 may each be a version of processor 1500. In one embodiment of the present invention, processor 1770 and processor 1780 are processor 1610 and processor 1615, respectively, and coprocessor 1738 is coprocessor 1645. In another embodiment, processor 1770 and processor 1780 are processor 1610 and coprocessor 1645, respectively.

[0194] Processor 1770 and processor 1780 are shown as including integrated memory controller (IMC) units 1772 and 1782, respectively. Processor 1770 also includes point-to-point (PP) interfaces 1776 and 1778 as part of its bus controller unit; similarly, second processor 1780 includes PP interfaces 1786 and 1788. Processors 1770, 1780 can exchange information via PP interface 1750 using point-to-point (PP) circuits 1778, 1788. Fig.17 As shown in FIG. 1 , IMCs 1772 and 1782 couple the processors to corresponding memories, namely, memory 1732 and memory 1734 , which may be portions of main memory locally attached to the corresponding processors.

[0195] Processors 1770, 1780 may each exchange information with chipset 1790 via separate PP interfaces 1752, 1754 using point-to-point interface circuits 1776, 1794, 1786, 1798. Alternatively, chipset 1790 may exchange information with coprocessor 1738 via high performance interface 1739. In one embodiment, coprocessor 1738 is a special purpose processor such as, for example, a high throughput MIC processor, a network or communication processor, a compressor engine, a graphics processor, a GPGPU, an embedded processor, or the like.

[0196] A shared cache (not shown) may be included in either processor or external to both processors but connected to the processors via the PP interconnect so that if the processors are placed in a low power mode, local cache information of either or both processors may be stored in the shared cache.

[0197] Chipset 1790 may be coupled to first bus 1716 via interface 1796. In one embodiment, first bus 1716 may be a peripheral component interconnect (PCI) bus, or a bus such as PCI Express or another third generation I / O interconnect bus, but the scope of the present invention is not limited in this regard.

[0198] like Fig.17 As shown in , various I / O devices 1714 and a bus bridge 1718 may be coupled to the first bus 1716, which couples the first bus 1716 to a second bus 1720. In one embodiment, one or more additional processors 1715 (such as a coprocessor, a high throughput MIC processor, a GPGPU, an accelerator (such as, for example, a graphics accelerator or a digital signal processing (DSP) unit), a field programmable gate array, or any other processor) are coupled to the first bus 1716. In one embodiment, the second bus 1720 may be a low pin count (LPC) bus. In one embodiment, various devices may be coupled to the second bus 1720, including, for example, a keyboard and / or mouse 1722, communication devices 1727, and a storage unit 1728 such as a disk drive or other mass storage device, which in one embodiment may include instruction / code data 1730. Further, an audio I / O 1724 may be coupled to the second bus 1720. It should be noted that other architectures are possible. For example, instead of Fig.17 In a point-to-point type architecture, the system may implement a multi-drop bus or other such architecture.

[0199] Now refer to Fig.18 , shows a block diagram of a SoC 1800 according to an embodiment of the present invention. Fig.15 Similar elements in FIG. 1 have the same reference numerals. Also, dashed boxes are optional features for more advanced SoCs. Fig.18, interconnect unit(s) 1802 are coupled to: application processor 1810, which includes a set of one or more cores 1802A to 1802N and shared cache unit(s) 1806; system agent unit 1810; bus controller unit(s) 1816; integrated memory controller unit(s) 1814; a set of one or more coprocessors 1820, which may include integrated graphics logic, image processor, audio processor, and video processor; static random access memory (SRAM) unit 1830; direct memory access (DMA) unit 1832; and a unit 1840 for coupling to one or more external displays. In one embodiment, coprocessor(s) 1820 are special purpose processors, such as, for example, network or communication processors, compression engines, GPGPUs, high throughput MIC processors, embedded processors, etc.

[0200] The embodiments of the mechanisms disclosed herein may be implemented in hardware, software, firmware, or a combination of these implementation approaches. Embodiments of the present invention may be implemented as a computer program or program code executed on a programmable system, the programmable system comprising at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device.

[0201] Such as Fig.17 The program code of the displayed code 1730 can be applied to input instructions to perform the functions described herein and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, a processing system includes any system having a processor, such as, for example, a digital signal processor (DSP), a microcontroller, an application specific integrated circuit (ASIC), or a microprocessor.

[0202] The program code can be implemented in a high-level program or object-oriented programming language to communicate with the processing system. If necessary, the program code can also be implemented in an assembly or machine language. In fact, the scope of the mechanism described herein is not limited to any specific programming language. In any case, the language can be a compiled or interpreted language.

[0203] One or more aspects of at least one embodiment may be implemented by representative instructions stored on a machine-readable medium, which represent various logic within a processor, and when read by a machine, the instructions cause the machine to fabricate logic for performing the techniques described herein. Such representations, known as "IP cores," may be stored on a tangible machine-readable medium and provided to various customers or manufacturing facilities to load into a fabrication machine that actually fabricates the logic or processor.

[0204] Such machine-readable storage media may include, but are not limited to: a non-transitory tangible arrangement of items manufactured or formed by a machine or device, including storage media such as a hard disk; any other type of disk, including floppy disks, optical disks, compact disk-read only memory (CD-ROM), compact disk-rewritable (CD-RW), and magneto-optical disks; semiconductor devices, such as read-only memory (ROM); random access memory (RAM), such as dynamic random access memory (DRAM), static random access memory (SRAM); erasable programmable read-only memory (EPROM); flash memory; electrically erasable programmable read-only memory (EEPROM); phase change memory (PCM); magnetic or optical cards; or any other type of medium suitable for storing electronic instructions.

[0205] Therefore, embodiments of the present invention also include non-transitory tangible machine-readable media containing instructions or containing design data (such as hardware description language (HDL)), which defines the structures, circuits, devices, processors and / or system features described herein. Such embodiments may also be referred to as program products.

[0206] The flow charts and block diagrams in the accompanying drawings show the architecture, functions and operations of possible implementations of the system, method and computer program product according to various aspects of the present disclosure. In this regard, each frame in the flow chart or block diagram can represent a part of a module, a fragment or a code, which contains one or more executable instructions for implementing (multiple) specified logical functions. It should also be noted that in some alternative embodiments, the functions indicated in the frame may not occur in the order marked in the figure. For example, depending on the functions involved, the two frames shown in succession can in fact be executed substantially simultaneously, or these frames can sometimes be executed in reverse order or in an alternative order. It will also be noted that each frame of the block diagram and / or flow chart and the combination of frames in the block diagram and / or flow chart can be implemented by a combination of a system based on special hardware or special hardware and computer instructions that perform a specific function or action.

[0207] The foregoing disclosure summarizes the features of several embodiments so that those skilled in the art can better understand the various aspects of the present disclosure. It should be understood by those skilled in the art that they can easily use the present disclosure as a basis for designing or modifying other processes and structures for performing the same purpose and / or achieving the same advantages of the embodiments described herein. It should also be recognized by those skilled in the art that such equivalent constructions will not deviate from the spirit and scope of the present disclosure, and that they can make various changes, substitutions and transformations herein without departing from the spirit and scope of the present disclosure.

[0208] All or part of any hardware element disclosed herein can be easily provided in a system on chip (SoC) including a central processing unit (CPU) package. SoC represents an integrated circuit (IC) that integrates the components of a computer or other electronic system into a single chip. SoC can include digital signals, analog signals, mixed signals, and radio frequency functions, all of which can be provided on a single chip substrate. Other embodiments may include a multi-chip module (MCM), in which multiple chips are located in a single electronic package and are configured to interact closely with each other through the electronic package. In various other embodiments, the computing functions disclosed herein can be implemented in one or more silicon cores in application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), and other semiconductor chips.

[0209] As used throughout this specification, the term "processor" or "microprocessor" should be understood to include not only conventional microprocessors (such as industry-leading x86 and x64 architectures), but also includes graphics processors, matrix processors, and any ASIC, FPGA, microcontroller, digital signal processor (DSP), programmable logic device, programmable logic array (PLA), microcode, instruction set, emulation or virtual machine processor, or any similar "Turing-complete" device, combination of devices, or logic element (hardware or software) that allows execution of instructions.

[0210] It is also noted that in certain embodiments, some of the components may be omitted or combined. In a general sense, the arrangements depicted in the accompanying drawings should be understood as logical partitions, while the physical architecture may include various permutations, combinations, and / or hybrids of these elements. It should be noted that countless possible design configurations may be used to achieve the operational goals outlined herein. Therefore, the associated infrastructure has a large number of alternative arrangements, design choices, device possibilities, hardware configurations, software implementations, and equipment options.

[0211] In a general sense, any appropriately configured processor can execute instructions associated with data or microcode to implement the operations detailed herein. Any processor disclosed herein can transform an element or article (e.g., data) from one state or thing to another state or thing. In another example, some of the activities outlined herein can be implemented using fixed logic or programmable logic (e.g., software and / or computer instructions executed by a processor), and the elements identified herein can be some type of programmable processor; programmable digital logic (e.g., field programmable gate array (FPGA), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM)); or an ASIC including digital logic, software, code, electronic instructions; flash memory; optical disk; CD-ROM; DVD ROM; magnetic card or optical card; other types of machine-readable media suitable for storing electronic instructions; or any suitable combination thereof.

[0212] In operation, where appropriate and based on specific needs, the storage device may store information in any suitable type of tangible, non-transitory storage medium (e.g., random access memory (RAM), read-only memory (ROM), field programmable gate array (FPGA), erasable programmable read-only memory (EPROM), electrically erasable programmable ROM (EEPROM), or microcode); software, hardware (e.g., processor instructions, or microcode); or in any other suitable component, device, element, or object. In addition, based on specific needs and implementations, information tracked, sent, received, or stored in the processor may be provided in any database, register, table, cache, queue, control list, or storage structure, all of which may be referenced in any suitable time frame. Any of the memories or storage elements disclosed herein should be interpreted as being included within the broad terms 'memory' and 'storage'. The non-transitory storage medium herein is explicitly intended to include any non-transitory dedicated or programmable hardware configured to provide the disclosed operations or to cause the processor to perform the disclosed operations. The non-transitory storage medium also explicitly includes a processor having stored thereon hardware-encoded instructions and, optionally, micro-code instructions or sequences encoded in hardware, firmware, or software.

[0213] The computer program logic that implements all or part of the functions described herein is embodied in various forms, including but not limited to: hardware description language, source code form, computer executable form, machine instructions or microcode, programmable hardware, and various intermediate forms (e.g., forms generated by HDL processors, assemblers, compilers, linkers, or locators). In an example, the source code includes a series of computer program instructions implemented in the following languages: various programming languages ​​used with various operating systems or operating environments, such as object code, assembly language, or high-level languages ​​such as OpenCL, FORTRAN, C, C++, JAVA, or HTML; or hardware description languages ​​such as Spice, Verilog, and VHDL. The source code can define and use various data structures and communication messages. The source code can be in a computer executable form (e.g., via an interpreter), or the source code can be converted into a computer executable form (e.g., via a converter, assembler, or compiler) or converted into an intermediate form such as byte code. Where appropriate, any of the above contents can be used to construct or describe appropriate discrete circuits or integrated circuits, whether in a sequential, combined, state machine, or other manner.

[0214] In one example, any number of circuits of the accompanying drawings can be implemented on a board of an associated electronic device. The board can be a general circuit board that can hold various components of the internal electronic system of the electronic device, and further provides connectors for other peripheral devices. More specifically, the board can provide an electrical connection, through which other components of the system can communicate electrically. Any appropriate processor and memory can be appropriately coupled to the board based on specific configuration needs, processing requirements and computing design. Other components such as external storage devices, additional sensors, controllers for audio / video display, and peripheral devices can be attached to the board as plug-in cards via cables, or integrated into the board itself. In another example, the circuit of the accompanying drawings can be implemented as an independent module (e.g., a device with associated components and circuit systems configured to perform specific applications or functions) or implemented as a plug-in module in the dedicated hardware of the electronic device.

[0215] Note that, through the numerous examples provided herein, interactions can be described with respect to two, three, four or more electrical components. However, this has been accomplished only for the purpose of clarity and example. It should be understood that the system can be merged or reconfigured in any appropriate manner. Together with similar design alternatives, any of the components, modules and elements shown in the accompanying drawings can be combined in various possible configurations, all of which are within the broad protection scope of this specification. In some cases, by only citing a limited number of electrical components, one or more functions in the function of a given flow set can be more easily described. It should be understood that the circuits of the accompanying drawings and their teachings are easily scalable and can adapt to a large number of components and more complex / precise arrangements and configurations. Therefore, the examples provided should not limit the scope of protection to or constrain the broad teachings of the circuit to potentially be applied to a large number of other architectures.

[0216] Numerous other changes, substitutions, variations, alterations and modifications may occur to those skilled in the art, and it is intended that this disclosure embrace all such changes, substitutions, variations, alterations and modifications that fall within the scope of the appended claims. Example Implementations

[0217] The following examples relate to embodiments described throughout this disclosure.

[0218] One or more embodiments may include a device comprising: a logarithmic circuit, wherein the logarithmic circuit includes a circuit system for the following operations: identifying an input associated with a logarithmic operation via an input register, wherein the logarithmic operation is to be performed by the logarithmic circuit using a piecewise linear approximation; identifying a first range within which the input falls using a range selection circuit system, wherein the first range is identified from a plurality of ranges associated with a plurality of piecewise linear approximation (PLA) equations for the logarithmic operation, and wherein the first range corresponds to a first equation in the plurality of PLA equations; obtaining a plurality of operands associated with the first equation; calculating a result of the first equation using an adder-subtractor circuit system based on the plurality of operands; and returning an output associated with the logarithmic operation via an output register, wherein the output is generated based at least in part on the result of the first equation.

[0219] In an example embodiment of the apparatus, said logarithmic operation is associated with an artificial neural network operation.

[0220] In an example embodiment of the apparatus: the input comprises a floating point number, wherein the floating point number comprises an exponent and a mantissa; and the output comprises a fixed point number, wherein the fixed point number comprises an integer and a fraction.

[0221] In an example embodiment of the apparatus, the plurality of operands includes the mantissa and one or more fractional operands, wherein the one or more fractional operands each include a denominator comprising a power of two.

[0222] In an example embodiment of the apparatus, the logarithm circuit further comprises one or more shift circuits for generating the one or more fractional operands.

[0223] In an example embodiment of the apparatus, the logarithm circuit further comprises a subtractor circuit for subtracting a bias from the exponent of the floating point number to generate an unbiased exponent.

[0224] In an example embodiment of a device, the circuit system for returning the output associated with the logarithm operation via the output register is further used to: generate the integer of the fixed-point number based on the unbiased exponent; and generate the fraction of the fixed-point number based on the result of the first equation.

[0225] In an example embodiment of the apparatus, wherein the logarithm circuit further comprises one or more multiplexers for selecting the plurality of operands associated with the first equation.

[0226] In an example embodiment of the apparatus, wherein the adder-subtractor circuitry is to perform one or more addition or subtraction operations on the plurality of operands.

[0227] In an example embodiment of the device, the device further includes an antilogarithm circuit, wherein the antilogarithm circuit includes a circuit system for the following operations: identifying a second input associated with an antilogarithm operation, wherein the antilogarithm operation is to be performed by the antilogarithm circuit using a piecewise linear approximation; identifying a second range within which the second input falls, wherein the second range is identified from a second plurality of ranges associated with a second plurality of piecewise linear approximation (PLA) equations for the antilogarithm operation, and wherein the second range corresponds to a second equation in the second plurality of PLA equations; calculating a second result of the second equation based on a second plurality of operands associated with the second equation; and generating a second output associated with the antilogarithm operation, wherein the second output is generated based at least in part on the second result of the second equation.

[0228] In an example embodiment of the device, the device further includes an activation function circuit, wherein the activation function circuit includes the logarithmic circuit and the antilogarithmic circuit, and wherein the activation function circuit further includes a circuit system for: receiving an instruction for executing an activation function selected from a plurality of available activation functions, wherein the activation function includes one or more multiplication or division operations; performing the one or more multiplication or division operations using one or more logarithmic operations and one or more antilogarithmic operations, wherein the one or more logarithmic operations are performed using the logarithmic circuit and the one or more antilogarithmic operations are performed using the antilogarithmic circuit; and generating an activation output associated with the activation function, wherein the activation output is generated based at least in part on one or more results of the one or more multiplication or division operations.

[0229] In an example embodiment of the apparatus: the activation function further comprises one or more exponential operations; and the activation function circuit further comprises an exponential circuit for performing the one or more exponential operations using a piecewise linear approximation.

[0230] One or more embodiments may include a system comprising: a memory for storing information associated with an application; a processor for executing one or more instructions associated with the application; and an activation function circuit for executing multiple activation functions, wherein the activation function circuit includes a circuit system for the following operations: receiving an instruction for executing an activation function associated with the application, wherein the activation function is selected from the multiple activation functions, and wherein the activation function includes one or more multiplication or division operations; performing the one or more multiplication or division operations using one or more logarithmic operations and one or more antilogarithmic operations, wherein the one or more logarithmic operations are performed by the logarithmic circuit using a piecewise linear approximation, and wherein the one or more antilogarithmic operations are performed by the antilogarithmic circuit using a piecewise linear approximation; and generating an output associated with the activation function, wherein the output is generated at least in part based on one or more results of the one or more multiplication or division operations.

[0231] In an example embodiment of the system, the application comprises an artificial neural network, and wherein the activation function is associated with the operation of the artificial neural network.

[0232] In an example embodiment of the system, the circuit system for performing the one or more multiplication or division operations using the one or more logarithmic operations and the one or more antilogarithmic operations is further used to: perform one or more base-2 logarithmic operations on one or more operands associated with the one or more multiplication or division operations, wherein the one or more base-2 logarithmic operations are performed using a piecewise linear approximation; perform one or more addition or subtraction operations on one or more results of the one or more base-2 logarithmic operations; and perform one or more base-2 antilogarithmic operations on one or more results of the one or more addition or subtraction operations, wherein the one or more base-2 antilogarithmic operations are performed using a piecewise linear approximation.

[0233] In an example embodiment of the system: the activation function further comprises one or more exponential operations; and the activation function circuit further comprises circuitry for performing the one or more exponential operations using a piecewise linear approximation.

[0234] In an example embodiment of the system: the one or more exponential operations each include a base of 2; and the circuit system for performing the one or more exponential operations using piecewise linear approximation is further used to perform the one or more exponential operations using one or more antilog operations with base 2, wherein the one or more antilog operations with base 2 are performed using piecewise linear approximation.

[0235] In an example embodiment of the system, the plurality of activation functions include: a Sigmoid function; a hyperbolic tangent function; a Swish function; and a rectified linear unit function.

[0236] In an example embodiment of the system: at least one of the Sigmoid function, the hyperbolic tangent function, or the Swish function is defined using one or more exponential operations that do not exclusively include base 2; and the activation function circuit further includes a circuit system for performing the one or more exponential operations using one or more antilog operations with base 2, wherein the one or more antilog operations with base 2 are performed using piecewise linear approximation.

[0237] One or more embodiments may include at least one at least one machine-accessible storage medium having a plurality of instructions stored thereon, wherein the instructions, when executed on a machine, cause the machine to: receive, by an activation function circuit, instructions for executing an activation function selected from a plurality of available activation functions, wherein the activation function comprises one or more multiplication or division operations; perform the one or more multiplication or division operations using one or more logarithmic operations and one or more antilogarithmic operations, wherein the one or more logarithmic operations and the one or more antilogarithmic operations are performed using piecewise linear approximation; and generate an output associated with the activation function, wherein the output is generated based at least in part on one or more results of the one or more multiplication or division operations.

[0238] In an example embodiment of the storage medium, the instructions that cause the machine to perform the one or more multiplication or division operations using the one or more logarithmic operations and the one or more antilogarithmic operations further cause the machine to: perform one or more base-2 logarithmic operations on one or more operands associated with the one or more multiplication or division operations, wherein the one or more base-2 logarithmic operations are performed using a piecewise linear approximation; perform one or more addition or subtraction operations on one or more results of the one or more base-2 logarithmic operations; and perform one or more base-2 antilogarithmic operations on one or more results of the one or more addition or subtraction operations, wherein the one or more base-2 antilogarithmic operations are performed using a piecewise linear approximation.

[0239] In an example embodiment of the storage medium: the activation function further comprises one or more exponential operations; and the instructions further cause the machine to perform the one or more exponential operations using a piecewise linear approximation.

[0240] In an example embodiment of the storage medium: at least one activation function of the plurality of available activation functions is defined using one or more exponential operations that do not exclusively include base 2; and the instructions further cause the machine to perform the one or more exponential operations using one or more antilog operations with base 2, wherein the one or more antilog operations with base 2 are performed using piecewise linear approximation.

[0241] One or more embodiments may include a method comprising: receiving, by an activation function circuit, an instruction for executing an activation function selected from a plurality of available activation functions, wherein the activation function comprises one or more multiplication or division operations; performing the one or more multiplication or division operations using one or more logarithmic operations and one or more antilogarithmic operations, wherein the one or more logarithmic operations and the one or more antilogarithmic operations are performed using piecewise linear approximation; and generating an output associated with the activation function, wherein the output is generated based at least in part on one or more results of the one or more multiplication or division operations.

[0242] In an example embodiment of the method, the method further includes performing one or more base-2 logarithm operations on one or more operands associated with the one or more multiplication or division operations, wherein the one or more base-2 logarithm operations are performed using a piecewise linear approximation; performing one or more addition or subtraction operations on one or more results of the one or more base-2 logarithm operations; and performing one or more base-2 inverse logarithm operations on one or more results of the one or more addition or subtraction operations, wherein the one or more base-2 inverse logarithm operations are performed using a piecewise linear approximation.

Claims

1. A device comprising: first circuitry configured to operate on a floating point input to compute a base-2 logarithm of the input; second circuitry configured to operate on the base-2 logarithm of the input to compute an intermediate result; as well as A third circuit system is configured to apply a power function with base 2 to the intermediate result to generate an output of the floating point operation.

2. The device according to claim 1, wherein: The floating point operations are operations in a machine learning model.

3. The device as claimed in claim 2, wherein: The operation in the machine learning model is an activation function.

4. The device according to claim 1, wherein: The input has a floating point format, and the first circuitry is configured to compute the base-2 logarithm of the input by performing a calculation on a mantissa portion of the input.

5. The device according to claim 4, wherein: Performing a calculation on the mantissa portion of the input includes applying a piecewise approximation to the mantissa portion of the input.

6. The device according to claim 5, wherein: Applying the piecewise approximation includes using one or more coefficients stored in a piecewise approximation table.

7. The device according to claim 6, wherein: Applying the piecewise approximation further includes retrieving the one or more coefficients from the piecewise approximation table based on a range that includes the mantissa portion of the input.

8. The device of claim 4, wherein: The first circuitry is further configured to perform a separate calculation on an exponential portion of the input.

9. The device of claim 8, wherein: The first circuitry is configured to perform the separate calculation on the exponential portion of the input by subtracting a bias from the exponential portion of the input to calculate an unbiased exponent.

10. The device of claim 1, wherein: The second circuit system is configured to further operate on another input to calculate the intermediate result, wherein the another input has a floating point format.

11. A method for executing a machine learning model, the method comprising: calculating, by the first circuit system, a base-2 logarithm of an input to a floating-point operation in the machine learning model; calculating, by a second circuit system, an intermediate result by operating the base-2 logarithm of the input; as well as A third circuit system applies a power function with base 2 to the intermediate result to generate an output of the floating-point operation in the machine learning model.

12. The method of claim 11, wherein: The input has a floating point format, wherein calculating the base-2 logarithm of the input comprises performing a calculation on a mantissa portion of the input.

13. The method of claim 12, wherein: Performing a calculation on the mantissa portion of the input includes applying a piecewise approximation to the mantissa portion of the input.

14. The method of claim 13, wherein: Applying the piecewise approximation includes: using one or more coefficients stored in a piecewise approximation table; and The one or more coefficients are retrieved from the piecewise approximation table based on a range that includes the mantissa portion of the input.

15. The method of claim 12, wherein: Calculating the base 2 logarithm of the input further includes performing a separate calculation on an exponential portion of the input.

16. The method of claim 15, wherein: Performing the separate calculation on the exponential portion of the input includes subtracting a bias from the exponential portion of the input to calculate an unbiased exponent.

17. The method of claim 11, wherein: Computing the intermediate result includes an operation on another input having a floating point format.

18. One or more non-transitory computer-readable media storing instructions executable to perform operations for executing a machine learning model, the operations comprising: calculating, by the first circuit system, a base-2 logarithm of an input to a floating-point operation in the machine learning model; calculating, by a second circuit system, an intermediate result by operating the base-2 logarithm of the input; as well as A third circuit system applies a power function with base 2 to the intermediate result to generate an output of the floating-point operation in the machine learning model.

19. One or more non-transitory computer readable media as recited in claim 18, wherein: The input has a floating point format, and wherein computing the base-2 logarithm of the input comprises performing a calculation on a mantissa portion of the input.

20. The one or more non-transitory computer readable media of claim 19, wherein: Performing a calculation on the mantissa portion of the input includes applying a piecewise approximation to the mantissa portion of the input.

Images