Approximate calculation of activation functions with taylor series

Abstract

An activation function unit is capable of calculating an activation function calculated by Taylor series approximation. The activation function unit may include a plurality of computing elements. Each computing element may include two multipliers and an accumulator. The first multiplier may calculate an intermediate product using an activation, such as an output activation of the DNN layer. The second multiplier may calculate terms of the Taylor series of the approximate calculation activation function based on the intermediate product from the first multiplier and coefficients of the Taylor series. The accumulator may compute a partial sum of terms as an output of an activation function. The number of terms may be determined based on a predetermined accuracy of the output of the activation function. The activation function unit may process a plurality of activations. Different activations may be input into different computing elements in different clock cycles. The activation function unit can calculate activation functions with different precisions.

Description

[0001] Cross-references to related applications This application claims priority to U.S. Patent Application No. 18 / 346992 (filed July 5, 2023, entitled “APPROXIMATING ACTIVATION FUNCTIONS WITH TAYLOR SERIES”), which is incorporated herein by reference in its entirety. Technical Field

[0002] This disclosure generally relates to deep neural networks (DNNs), and more specifically to using Taylor series to approximate the activation functions in DNNs. Background Technology

[0003] DNNs are widely used in a wide variety of artificial intelligence applications, ranging from computer vision to speech recognition and natural language processing, due to their ability to achieve high accuracy. However, this high accuracy comes at the cost of significant computational cost. DNNs have extremely high computational requirements, as each inference can require hundreds of millions of MAC (multiply-accumulate) operations and the reading and writing of large amounts of data. DNN inference also requires the computation of activation functions. Therefore, techniques to improve the efficiency of DNNs are needed. Attached Figure Description

[0004] The embodiments will be readily understood from the following detailed description taken in conjunction with the accompanying drawings. For the purpose of this description, similar reference numerals denote similar structural elements. The embodiments are illustrated in the accompanying drawings by way of example and not by way of limitation.

[0005] FIG. 1 The illustration shows an example DNN based on various embodiments.

[0006] FIG. 2 The illustration shows example DNN convolutions based on various embodiments.

[0007] FIG. 3 This is a block diagram of a DNN accelerator according to various embodiments.

[0008] FIG. 4 The illustration shows example activation function units according to various embodiments.

[0009] FIG. 5 The illustration shows example operations of activation function units without pause cycles according to various embodiments.

[0010] FIG. 6 The illustration shows the calculation of the activation function based on the same precision according to various embodiments.

[0011] FIG. 7 The illustration shows example operations of activation function units with pause periods according to various embodiments.

[0012] FIG. 8 FIG. illustrates computation of activation functions based on different precisions, according to various embodiments.

[0013] FIG. 9 FIG. illustrates a processing element (PE) array, according to various embodiments.

[0014] FIG. 10 is a block diagram of a PE according to various embodiments.

[0015] FIG. 11 is a flowchart illustrating a method of computing activation functions in a neural network, according to various embodiments.

[0016] FIG. 12 is a block diagram of an example computing device, according to various embodiments. DETAILED DESCRIPTION

[0017] SUMMARY The past decade has witnessed a rapid rise of AI (artificial intelligence) based data processing, in particular, DNN based data processing. DNNs are widely used in the fields of computer vision, speech recognition, image and video processing, mainly for their ability to achieve superhuman level of accuracy. Significant improvements in DNN model size and accuracy coupled with rapid increase in computing power of execution platforms have led to adoption of DNN applications even within resource constrained mobile and edge devices with limited energy availability.

[0018] A DNN layer can include one or more deep learning operations, such as convolution, pooling, element-wise operations, linear operations, non-linear operations, and so on. The deep learning operations in a DNN can be performed on one or more internal parameters (e.g., weights) of the DNN determined during a training phase, as well as one or more activations. An activation can be a data point (also referred to as a “data element” or an “element”). An activation or a weight of a DNN layer can be an element of a tensor of the DNN layer. A tensor is a data structure with multiple elements across one or more dimensions. Example tensors include a vector as a one-dimensional tensor and a matrix as a two-dimensional tensor. Three-dimensional tensors and even higher dimensional tensors can also exist. A DNN layer can have an input tensor (also referred to as an “input feature map (IFM)”) including one or more input activations (also referred to as “input elements”), and a weight tensor including one or more weights. A weight is an element in the weight tensor. The weight tensor of a convolution can be a kernel, a filter, or a set of filters. Output data of a DNN layer can be an output tensor (also referred to as an “output feature map (OFM)”) including one or more output activations (also referred to as “output elements”).

[0019] Activation functions are a crucial part of DNNs. They determine whether a neuron should or should not be activated by calculating a weighted sum of activations and summing the biases. A key purpose of activation functions is to introduce non-linearity into the neuron's output. Given the complexity of some non-linear activation functions used in many DNNs, hardware implementations may require approximations within a certain level of accuracy.

[0020] Current available implementations of activation functions are based on lookup tables (LUTs) or DSP (Digital Signal Processor) cores. LUTs sometimes employ piecewise linear approximation (PWL). PWL approximates complex nonlinear curves using a number of linear segments. While PWL-based LUTs can simplify some of the complex computations required by nonlinear functions by using LUTs, address generation logic, and simple arithmetic blocks (e.g., subtractors, multipliers, and adders), improving the accuracy of PWL-based LUTs typically requires a larger number of linear segments and therefore a higher number of entries in the LUT. Since nonlinear approximation is part of the core DNN logic, allocating additional regions on the DNN accelerator die may not always be feasible. In such cases, it may be beneficial to trade performance for additional accuracy without adding any additional regions.

[0021] DSP cores typically require specific kernel implementations of activation functions. DSP-based implementations often require offloading tasks from the neural network processor to the DSP core, which can add overhead such as handshaking and inter-module communication. Additionally, given the nature of DSP cores, they may operate at lower clock frequencies compared to neural network processors. These limitations increase the inefficiency of computationally executing activation functions within DNN accelerators.

[0022] Embodiments of this disclosure provide activation function units for DNN accelerators capable of computing approximations of activation functions. The approximation of the activation function may be a Taylor series. Example DNN accelerators in this disclosure include one or more computation blocks. A computation block may also be referred to as a computation slice. Each computation block may be a processing unit. A computation block includes a memory, a PE array, and activation function units. The memory may store data received or generated by the computation block. The PE array may perform deep learning operations, such as convolution, element-wise operations, pooling operations, etc. The activation function unit may receive an output activation computed by the PE array and apply an activation function to the output activation. The output of the activation function unit may be used as input to deep learning operations performed by the PE array. The output of the activation function unit may be stored in the memory. The activation function may be located on an outflow path from the PE array to the memory.

[0023] In various embodiments of the present disclosure, an activation function unit can include multiple computing elements. The computing elements can operate in parallel. Different activations can be input into different computing elements in different clock cycles. The computing elements are capable of computing a polynomial that approximates the activation function (e.g., a polynomial of a Taylor series). The degree of the polynomial or the number of terms in the polynomial can be determined based on a predetermined accuracy of the activation function output. Higher accuracy can require more terms in the polynomial and can require more computing elements or more clock cycles.

[0024] An example computing element can include two multipliers and an accumulator. The first multiplier can compute intermediate products using activations such as the output activations of a DNN layer. The first multiplier can compute intermediate products for different terms in different clock cycles. For example, the first multiplier can compute the square of an activation in a first clock cycle, compute the cube of an activation in a second clock cycle, and so on. The second multiplier can compute a term in the polynomial based on the intermediate products from the first multiplier and a coefficient of the Taylor series. In an example clock cycle, the second multiplier can multiply the coefficient with an intermediate product computed by the first multiplier in a previous clock cycle. The second multiplier can pass the result of the term to the accumulator. The accumulator can compute a partial sum of the terms as the output of the activation function.

[0025] The present disclosure provides a way to compute activation functions based on Taylor series. This way has the ability to program the accuracy-performance tradeoff. The accuracy of the activation function can be modeled by controlling the number of Taylor series terms to be computed. The accuracy of the result of the activation function can be improved by adding additional computation cycles. High accuracy can be achieved with minimal performance loss, such as performance loss in a small number of clock cycles (e.g., one, two, etc.). Thus, the present disclosure provides a more advantageous technique to compute activation functions compared to currently available techniques.

[0026] For the purpose of illustration, specific numbers, materials, and configurations are set forth in order to provide a thorough understanding of illustrative implementations. It will be apparent, however, to one skilled in the art that the present disclosure can be practiced without specific details, or with only some of the described aspects. In other instances, well-known features are omitted or simplified in order not to obscure the illustrative implementations.

[0027] Further, with respect to the accompanying drawings that form a part of this disclosure, show by way of illustration various embodiments that can be practiced. It is to be understood that other embodiments can be utilized and structural or logical changes can be made without departing from the scope of the present disclosure. Therefore, the following detailed description is not to be taken in a limiting sense.

[0028] Various operations could be described as being performed in real-time or in "online" mode. However, operations described as being performed in real-time or online are not necessarily performed in real-time. Rather, such operations could be performed in batch mode or offline, e.g., using data that was collected at an earlier time.

[0029] For purposes of this disclosure, the phrase "A or B" or the phrase "A and / or B" means (A), (B), or (A and B). For purposes of this disclosure, the phrase "A, B, or C" or the phrase "A, B, and / or C" means (A), (B), (C), (A and B), (A and C), (B and C), or (A, B and C). The term "between" when used in reference to a measurement range includes the two ends of the measurement range.

[0030] This description uses the phrases "in an embodiment," or "in embodiments," each of which can refer to one or more of the same or different embodiments. The terms "comprising," "including," "having" and the like, as used in the foregoing specification, are same in meaning and are synonymous with "containing." The present disclosure can use perspective-based descriptions such as "above," "below," "top," "bottom" and "side" that can be used to describe the various features illustrated in the drawings. However, these terms are used herein for the intent of providing orientation in connection with the illustrations as shown and should not be used to imply or require a specific orientation in use or required use of the claimed subject matter. The drawings are not necessarily drawn to scale. The ordinal prefixes "first," "second," and "third," etc. are used merely to distinguish different sets of elements from each other. The ordinal prefixes are not meant to imply or require a certain spatial or chronological sequence among or between elements having the same ordinal prefix or prefixes.

[0031] In the following detailed description, various aspects of the illustrative implementations will be described using terms commonly used in the art, such as "application," "module," "region," "component," "layer," "element," "step," "operation," and the like. These terms are used herein to connote physical elements or operational actions that comprise part of or control over the specified technical effects presented by the described illustrative implementations.

[0032] The terms "substantially," "near," "approximately," "close," and "about" generally mean within + / - 20% of the target value as described herein or as otherwise known in the art. Similarly, terms such as "co-planar," "perpendicular," "orthogonal," "parallel," or any other angular indication of orientation between elements generally mean within + / - 5-20% of the target value as described herein or as otherwise known in the art.

[0033] Additionally, the terms "comprise," "comprising," "include," "including," "have," "having," or any variation thereof, are intended to cover a non-exclusive inclusion. For example, a process, method, article, or DNN accelerator that comprises a list of elements is not necessarily limited to only those elements but can include other elements not expressly listed or inherent to such process, method, article, or DNN accelerator. The term "or" is also intended to cover "and" unless explicitly stated otherwise.

[0034] The systems, methods, and devices of the present disclosure each have several novel aspects, no single one of which is solely responsible for its desirable attributes. Details of one or more implementations of the subject matter described in this specification are set forth in the accompanying description and drawings.

[0035] Example DNN FIG. 1 An example DNN 100 is illustrated in accordance with various embodiments. For purposes of illustration, FIG. 1 The DNN 100 in FIG. 1 is a CNN. In other embodiments, the DNN 100 can be other types of DNNs. The DNN 100 is trained to receive an image and output a classification of an object in the image. In FIG. 1 In the embodiment of FIG. 1, the DNN 100 receives an input image 105 that includes objects 115, 125, and 135. The DNN 100 includes a sequence of layers that includes a plurality of convolutional layers 110 (individually referred to as "convolutional layer 110"), a plurality of pooling layers 120 (individually referred to as "pooling layer 120"), and a plurality of fully connected layers 130 (individually referred to as "fully connected layer 130"). In other embodiments, the DNN 100 can include fewer, more, or different layers. In inference of the DNN 100, the layers of the DNN 100 perform tensor computations that include many tensor operations such as convolution (e.g., multiply-accumulate (MAC) operations, etc.), pooling operations, element-wise operations (e.g., element-wise addition, element-wise multiplication, etc.), other types of tensor operations, or some combination thereof.

[0036] The convolutional layers 110 summarize the presence of features in the input image 105. The convolutional layers 110 serve as feature extractors. The first layer of the DNN 100 is a convolutional layer 110. In an example, the convolutional layer 110 performs a convolution on an input tensor 140 (also referred to as IFM 140) and a filter 150. As FIG. 1As shown in the middle, IFM 140 is represented by a 7x7x3 three-dimensional (3D) matrix. IFM 140 includes 3 input channels, each of which is represented by a 7x7 two-dimensional (2D) matrix. The 7x7 2D matrix includes 7 input elements (also referred to as input points) in each row and 7 input elements in each column. Filter 150 is represented by a 3x3x3 3D matrix. Filter 150 includes 3 kernels, each of which can correspond to a different input channel of IFM 140. A kernel is a 2D matrix of weights, where the weights are arranged in columns and rows. A kernel can be smaller than an IFM. In FIG. 1 In embodiments, each kernel is represented by a 3x3 2D matrix. The 3x3 kernel includes 3 weights in each row and 3 weights in each column. The weights can be initialized and updated by backpropagation using gradient descent. The magnitude of the weights can indicate the importance of filter 150 in extracting features from IFM 140.

[0037] Convolution includes MAC operations with input elements in IFM 140 and weights in filter 150. Convolution can be standard convolution 163 or depthwise convolution 183. In standard convolution 163, the entire filter 150 is slid across IFM 140. All input channels are combined to produce output tensor 160 (also referred to as output feature map (OFM) 160). OFM 160 is represented by a 5x5 2D matrix. The 5x5 2D matrix includes 5 output elements (also referred to as output points) in each row and 5 output elements in each column. For illustration purposes, standard convolution includes one filter in embodiments. In embodiments where there are multiple filters, standard convolution can produce multiple output channels in OFM 160. FIG. 1

[0038] The multiplication applied between the kernel size patch of IFM 140 and the kernel can be a dot product. A dot product is an element-wise multiplication between the kernel size patch of IFM 140 and the corresponding kernel, which is then summed, resulting in a single value. Because it results in a single value, the operation is often referred to as a "scalar product." Using a kernel that is smaller than IFM 140 is intentional because it allows the same kernel (set of weights) to be multiplied with IFM 140 multiple times at different points on IFM 140. Specifically, the kernel is systematically applied to each overlapping portion or kernel size patch of IFM 140 from left to right and top to bottom. The result from multiplying the kernel with IFM 140 once is a single value. Because the kernel is applied multiple times to IFM 140, the multiplication results in a 2D matrix of output elements. Thus, the 2D output matrix from standard convolution 163 (i.e., OFM 160) is referred to as an OFM.

[0039] ​In depthwise convolution 183, the input channels are not combined. Instead, the MAC operation is performed on individual input channels and individual kernels, producing an output channel. For example... FIG. 1 As shown, depthwise convolution 183 produces a depthwise output tensor 180. The depthwise output tensor 180 is represented by a 5×5×3 3D matrix. The depthwise output tensor 180 includes three output channels, each represented by a 5×5 2D matrix. The 5×5 2D matrix includes five output elements in each row and five output elements in each column. Each output channel is the result of a MAC operation between the input channel of IFM 140 and the kernel of filter 150. For example, the first output channel (patterned with dots) is the result of a MAC operation between the first input channel (patterned with dots) and the first kernel (patterned with dots), the second output channel (patterned with horizontal bars) is the result of a MAC operation between the second input channel (patterned with horizontal bars) and the second kernel (patterned with horizontal bars), and the third output channel (patterned with diagonal lines) is the result of a MAC operation between the third input channel (patterned with diagonal lines) and the third kernel (patterned with diagonal lines). In this depthwise convolution, the number of input channels equals the number of output channels, and each output channel corresponds to a different input channel. The input and output channels are collectively referred to as depthwise channels. After the depthwise convolution, pointwise convolution 193 is then performed on the depthwise output tensor 180 and the 1×1×3 tensor 190 to produce OFM 160.

[0040] OFM 160 is then passed to the next layer in the sequence. In some embodiments, OFM 160 is passed through an activation function. An example activation function is the Modified Linear Activation Function (ReLU). ReLU is a computation that directly returns the value provided as input, or returns zero if the input is zero or less. Convolutional layer 110 may receive several images as input and compute the convolution of each of them with each of the kernels. This process can be repeated several times. For example, OFM 160 is passed to the next convolutional layer 110 (i.e., the convolutional layer 110 after which OFM 160 in the sequence is generated). The next convolutional layer 110 performs a convolution of OFM 160 with a new kernel and generates a new feature map. The new feature map may also be normalized and resized. The new feature map can be kernelized again by another subsequent convolutional layer 110, and so on.

[0041] In some embodiments, convolutional layer 110 has 4 hyperparameters: the number of kernels, F kernel size (e.g., a kernel has dimensions F x F x D pixels), S stride to drag a window corresponding to the kernel over the image (e.g., a stride of one means to move the window by one pixel each time), and zero padding P (e.g., add a black outline of P pixel thickness to the input image of convolutional layer 110). Convolutional layer 110 can perform various types of convolutions, such as 2-dimensional convolution, dilated or atrous convolution, spatial separable convolution, depth-wise separable convolution, transposed convolution, and so on. DNN 100 includes 16 convolutional layers 110. In other embodiments, DNN 100 can include a different number of convolutional layers.

[0042] Pooling layer 120 down-samples the feature maps generated by convolutional layers, e.g., by summarizing the presence of features in patches of the feature maps. Pooling layer 120 is placed between 2 convolutional layers 110: a preceding convolutional layer 110 (a convolutional layer 110 that precedes the pooling layer 120 in the layer sequence) and a following convolutional layer 110 (a convolutional layer 110 that follows the pooling layer 120 in the layer sequence). In some embodiments, pooling layer 120 is added after convolutional layer 110, e.g., after an activation function (e.g., ReLU, etc.) has been applied to OFM 160.

[0043] Pooling layer 120 receives the feature maps generated by the preceding convolutional layer 110 and applies a pooling operation to the feature maps. The pooling operation reduces the size of the feature maps while preserving their important characteristics. Accordingly, the pooling operation improves the efficiency of the DNN and avoids overfitting. Pooling layer 120 can perform the pooling operation by average pooling (computing the average value of each patch on the feature map), max pooling (computing the maximum value of each patch of the feature map), or a combination of both. The size of the pooling operation is smaller than the size of the feature map. In various embodiments, the pooling operation is a 2 x 2 pixel applied with a stride of 2 pixels, such that the pooling operation reduces the size of the feature map by a factor of 2, e.g., the number of pixels or values in the feature map is reduced to one quarter size. In one example, a pooling layer 120 applied to a 6 x 6 feature map produces a 3 x 3 output pooled feature map. The output of pooling layer 120 is input into the following convolutional layer 110 for further feature extraction. In some embodiments, pooling layer 120 operates on each feature map individually to create a new set of the same number of pooled feature maps.

[0044] The fully connected layer 130 is the last layer of the DNN. The fully connected layer 130 can or can not be a convolution. The fully connected layer 130 receives an input operand. The input operand defines the outputs of the convolutional layers 110 and the pooling layers 120 and includes the values of the last feature map generated by the last pooling layer 120 in the sequence. The fully connected layer 130 applies a linear combination and an activation function to the input operand and generates a vector. The vector can contain as many elements as classes: element i represents the probability that the image belongs to class i. Each element is therefore between 0 and 1 and the sum of all elements amounts to one. These probabilities are computed by the last fully connected layer 130 by using a logistic function (for binary classification) or a softmax function (for multi-class classification) as activation function.

[0045] In some embodiments, the fully connected layer 130 classifies the input image 105 and returns an operand of size N, where N is the number of classes in the image classification problem. In FIG. 1 embodiments, N is equal to 3 because there are 3 objects 115, 125 and 135 in the input image. Each element of the operand indicates the probability that the input image 105 belongs to a class. To compute the probabilities, the fully connected layer 130 multiplies each input element with a weight, sums them up and then applies an activation function (e.g., logistic if N = 2 or softmax if N > 2). This amounts to multiplying the input operand with a matrix containing the weights. In an example, the vector includes 3 probabilities: a first probability indicating that the object 115 is a tree, a second probability indicating that the object 125 is a car and a third probability indicating that the object 135 is a person. In other embodiments, in which the input image 105 includes different objects or a different number of objects, the respective values can be different.

[0046] Example convolution FIG. 2 An example DNN convolution is illustrated in accordance with various embodiments. The convolution can be a convolution in a convolutional layer of a DNN (e.g., the convolutional layer in FIG. 1 In some embodiments, the convolution is performed by a DNN accelerator that includes one or more compute tiles. An example of a DNN accelerator can be the DNN accelerator 300 in FIG. 3 In some embodiments, the convolution is performed by a DNN accelerator that includes one or more compute tiles. An example of a DNN accelerator can be the DNN accelerator 300 in FIG. 3 In some embodiments, the convolution is performed by a DNN accelerator that includes one or more compute tiles. An example of a DNN accelerator can be the DNN accelerator 300 in

[0047] In some embodiments, the convolution is performed by a DNN accelerator that includes one or more compute tiles. An example of a DNN accelerator can be the DNN accelerator 300 in FIG. 2In an embodiment, the input tensor 210 includes activations (also referred to as“input activations,”“elements,” or“input elements”) arranged in a 3D matrix. An input element is a data point in the input tensor 210. The input tensor 210 has a spatial size of where is a height of the 3D matrix (i.e., a length along the Y-axis that indicates a number of activations in a column in the 2D matrix for each input channel), is a width of the 3D matrix (i.e., a length along the X-axis that indicates a number of activations in a row in the 2D matrix for each input channel), and is a depth of the 3D matrix (i.e., a length along the Z-axis that indicates a number of input channels). For brevity and illustration purposes, the input tensor 210 has a spatial size of 7 x 7 x 3, i.e., the input tensor 210 includes three channels, and each input channel has a 7 x 7 2D matrix. Each input element in the input tensor 210 can be represented by an (X, Y, Z) coordinate. In other embodiments, the height, width, or depth of the input tensor 210 can be different.

[0048] Each filter 220 includes weights arranged in a 3D matrix. The values of the weights can be determined by training the DNN. The filter 220 has a spatial size of where is a height of the filter (i.e., a length along the Y-axis that indicates a number of weights in a column in each kernel), is a width of the filter (i.e., a length along the X-axis that indicates a number of weights in a row in each kernel), and is a depth of the filter (i.e., a length along the Z-axis that indicates a number of channels). In some embodiments, equals For brevity and illustration purposes, FIG. 2 each filter in the filter 220 has a spatial size of 3 x 3 x 3, i.e., the filter 220 includes 3 convolutional kernels with a spatial size of 3 x 3. In other embodiments, the height, width, or depth of the filter 220 can be different. The spatial size of the convolutional kernel is smaller than the spatial size of the 2D matrix for each input channel in the input tensor 210.

[0049] An activation or a weight can occupy one or more bytes in memory. The number of bytes for an activation or a weight can depend on the data format. For example, when an activation or a weight has an INT8 format, the activation occupies one byte. When an activation or a weight has an EP16 format, the activation occupies two bytes. Other data formats can be used for an activation or a weight.

[0050] In a convolution, each filter 220 slides across the input tensor 210 and generates a 2D matrix of output channels in the output tensor 230. In FIG. 2 In an embodiment, the 2D matrix has a spatial size of 5x5. The output tensor 230 includes activations (also referred to as "output activations," "elements," or "output elements") arranged in a 3D matrix. An output element is a data point in the output tensor 230. The output tensor 230 has a spatial size of where is a height of the 3D matrix (i.e., a length along the Y-axis that indicates a number of output activations in a column in the 2D matrix for each output channel), is a width of the 3D matrix (i.e., a length along the X-axis that indicates a number of output activations in a row in the 2D matrix for each output channel), and is a depth of the 3D matrix (i.e., a length along the Z-axis that indicates a number of output channels). may be equal to a number of filters 220 in the convolution. and may depend on the input tensor 210 and a height and a weight of each filter 220.

[0051] As part of the convolution, a MAC operation can be performed on a 3x3x3 sub-tensor 215 (which is highlighted in FIG. 2 with a dotted pattern) in the input tensor 210 and each filter 220. A result of the MAC operation on the sub-tensor 215 and one filter 220 is an output activation. In some embodiments (e.g., embodiments in which the convolution is an integral convolution), the output activation can include 8 bits, e.g., one byte. In other embodiments (e.g., embodiments in which the convolution is a floating point convolution), the output activation can include more than one byte. For example, the output element can include two bytes.

[0052] After the MAC operations on the sub-tensor 215 and all filters 220 are completed, a vector 235 is produced. The vector 235 is highlighted in FIG. 2 with a slash. The vector 235 includes a sequence of output activations arranged along the Z-axis. The output activations in the vector 235 have the same (x, y) coordinates, but the output activations correspond to different output channels and have different Z coordinates. A dimension of the vector 235 along the Z-axis can be equal to a total number of output channels in the output tensor 230. After the vector 235 is produced, further MAC operations are performed to produce additional vectors until the output tensor 230 is produced.

[0053] In some embodiments, the MAC operation on a 3x3x3 sub-tensor (e.g., the sub-tensor 215) and a filter 220 can be performed by multiple PEs. One or more PEs can receive input operands (e.g.,FIG. 2 input operands (e.g., input operands 217 shown in FIG. 2) and weight operands (e.g., weight operands 227 shown in FIG. 2). The input operands 217 include sequences of activations with the same (x, y) coordinates but different z coordinates. The input operands 217 include activations from each of the input channels in the input tensor 210. The weight operands 227 include sequences of weights with the same (x, y) coordinates but different z coordinates. The weight operands 227 include weights from each of the channels in the filter 220. The activations in the input operands 217 and the weights in the weight operands 227 can be fed into the PEs in turn. The PEs can receive a pair of an activation and a weight at a time and multiply the activation and the weight. The position of an activation in the input operands 217 can match the position of a weight in the weight operands 227. The activation and the weight can correspond to the same channel. FIG. 2

[0054] An activation or a weight can be a floating-point number. A floating-point number can have various data formats, such as FP32, FP16, BF16, and so on. A floating-point number can be a positive or negative number with a decimal point. A floating-point number can be represented by a bit sequence that includes one or more bits that represent the sign (e.g., positive or negative) of the floating-point number, bits that represent the exponent of the floating-point number, and bits that represent the mantissa of the floating-point number. The mantissa is the part of the floating-point number that represents the significant digits of that number. The mantissa is multiplied by a base raised to the power of the exponent to give the actual value of the floating-point number.

[0055] In some embodiments, the output activations in the output tensor 230 can be further processed based on one or more activation functions before they are stored or input into the next layer of the DNN. The processing based on one or more activation functions can be at least part of the post-processing of the convolution. In some embodiments, the post-processing can include one or more other computations, such as offset computation, bias computation, and so on. The results of the post-processing can be stored in the local memory of the compute block and used as input to the next DNN layer. In some embodiments, the input activations in the input tensor 210 can be the results of the post-processing of the previous DNN layer.

[0056] The computation of the activation function can be based on a Taylor series. In some embodiments, the Taylor series is used to approximate the computation of the activation function. The Taylor series can be an infinite sum of terms expressed in terms of the derivatives of a function at a single point. The activation function that is approximated by the Taylor series may be represented as: where, represents the input to the activation function; represents the power series, which is a real or complex number; and represents the point ​The function being evaluated The nth derivative.

[0057] The activation function can be approximated using the first t terms of a Taylor series. The accuracy of the activation function can be modulated by changing t. As t increases, the accuracy of the approximated activation function increases. The accuracy of the activation function and t can be predetermined, for example, determined before the activation function is computed. The first t terms can form a polynomial of the Taylor series, also known as a Taylor polynomial. The degree of the polynomial can be equal to... t -1. An example Taylor polynomial of degree six, including the first seven terms of a Taylor series, can be expressed as: Approximately calculating the Taylor series of the activation function involves the computation of the first t terms. In some embodiments (e.g., In the zero-valued embodiment, a term is a power of x multiplied by a coefficient that can be pre-computed. The partial sum of the first t terms is an approximate calculated output of the activation function.

[0058] Example activation functions that can be approximated by Taylor series include ReLU, Tanh activation function, Gaussian error linear unit (GELU), Sigmoid activation function, Sigmoid linear unit (SiLU), and so on. In one example, the Taylor series expansion of the Tanh activation function can be expressed as: in, B It is the Bernoulli number. The GELU activation function can be computed from the Tanh activation function: The SiLU activation function can be expressed as: in, It can be approximated using Taylor series. The following combines... FIG. 4-8 This will provide more details regarding the approximate computation of activation functions using Taylor series pairs.

[0059] Example DNN accelerator FIG. 3 This is a block diagram of an example DNN accelerator 300 according to various embodiments. The DNN accelerator 300 is capable of executing DNNs (e.g., FIG. 1 The deep learning operations and activation functions in DNN 100. For example... FIG. 3As shown, the DNN accelerator 300 includes a memory 310, a DMA (Direct Memory Access) engine 320, and a compute block 330 (referred to separately as "compute block 330"). In other embodiments, alternative configurations, different, or additional components may be included in the DNN accelerator 300. For example, the DNN accelerator 300 may include more than one memory 310 or DMA engine 320. As another example, the DNN accelerator 300 may include a single compute block 330. Furthermore, the functionality of the components belonging to the DNN accelerator 300 may be implemented by different components included in the DNN accelerator 300 or by different systems. The components of the DNN accelerator 300 may be implemented in hardware, software, firmware, or a combination thereof.

[0060] DNN accelerator 300 and FIG. 3 The pre-computation module 305 is associated with the embodiment described above. In other embodiments, the functionality of the components attributed to the pre-computation module 305 may be implemented by the DNN accelerator 300 or by different systems. The pre-computation module 305 can compute variables that will be used by the DNN accelerator 300 for DNN inference. Since the computation occurs before DNN inference, it is referred to as pre-computation. The pre-computation module 305 can perform pre-computation of activation functions in the DNN (such as activation functions approximated by Taylor series). For example, the pre-computation module 305 can determine the number of terms in the Taylor series that will be computed by the DNN accelerator 300 during DNN inference. The pre-computation module 305 can determine the number of terms to be computed based on a predetermined precision of the corresponding activation function. t The accuracy of the corresponding activation function may be based on the target accuracy of the DNN or other factors that have been determined.

[0061] In some embodiments, the pre-calculation module 305 may use acquired data indicating the correlation between t and the accuracy of the activation function. In one example, the pre-calculation module 305 may use an acquired curve showing how the accuracy of the activation function changes with t. The pre-calculation module 305 may determine the value of t based on the curve and a predetermined accuracy of the activation function.

[0062] In some embodiments, the pre-computation module 305 may also pre-compute the coefficients of the Taylor series before the DNN accelerator 300 computes the first t terms of the Taylor series. In some embodiments, the coefficients may be expressed as: Where n represents an integer from one to t. The pre-calculation module 305 can determine... The value of . In some embodiments, The value can be zero.

[0063] Memory 310 stores data associated with deep learning operations (including activation functions) performed by the DNN accelerator. In some embodiments, memory 310 can store data to be used by compute block 330 for DNN inference. For example, memory 310 can store data computed by pre-compute module 305, such as coefficients of a Taylor series. As another example, memory 310 can store weights, such as weights of a convolutional layer, determined by training a DNN. Memory 310 can also store data generated by compute block 330 from performing deep learning operations in a DNN. Example deep learning operations include convolution (also referred to as a "convolution operation"), pooling operations, element-wise operations, activation functions, other types of deep learning operations, or some combination thereof. Memory 310 can be a main memory of DNN accelerator 300. In some embodiments, memory 310 includes one or more DRAMs (dynamic random access memories).

[0064] DMA engine 320 facilitates data transfers between memory 310 and local memory of compute block 330. For example, DMA engine 320 can read data from memory 310 and write the data into local memory of compute block 330. As another example, DMA engine 320 can read data from local memory of compute block 330 and write the data into memory 310. DMA engine 320 provides DMA features that allow compute block 330 to initiate data transfers between memory 310 and local memory of compute block 330 and perform other operations while the data transfers are in progress. In some embodiments, DMA engine 320 can read a tensor from memory 310, modify the tensor in a manner optimized for compute block 330 before writing the tensor to local memory of compute block 330.

[0065] Compute block 330 can perform deep learning operations in a DNN. For example, compute block 330 can run each time a deep learning operation or a portion of a deep learning operation in a DNN layer. Compute block 330 can be capable of running various types of deep learning operations, such as convolution, pooling, element-wise operations, linear operations, non-linear operations, and so on. In an example, compute block 330 can perform a convolution, e.g., a standard convolution or a depthwise convolution. In some embodiments, compute block 330 receives an input tensor and one or more convolution kernels and performs a convolution using the input tensor and the convolution kernels. The result of the convolution can be an output tensor, which can be further computed, e.g., by compute block 330 or another compute block 330. In some embodiments, operations of a DNN layer can be run in parallel by multiple compute blocks 330. For example, multiple compute blocks 330 can each perform a portion of a workload of a convolution. Data can be shared among compute blocks 330.

[0066] In FIG. 3 Embodiments of each compute block 330 include a local memory 340, a PE array 350, and an activation function unit 360. The local memory 340, the PE array 350, and the activation function unit 360 can be implemented on the same chip. In other embodiments, alternative configurations, different, or additional components can be included in the compute block 330. Further, the functionality attributed to the components of the compute block 330 can be implemented by different components included in the compute block 330 or the DNN accelerator 300, or by a different system. The components of the compute block 330 can be implemented in hardware, software, firmware, or some combination thereof.

[0067] The local memory 340 is local to the corresponding compute block 330. In FIG. 3 In embodiments, the local memory 340 is internal to the compute block 330. In other embodiments, the local memory 340 can be external to the compute block 330. The local memory 340 can store data received by, used by, or generated by the PE array 350 and the activation function unit 360. Examples of data can include input activations, weights, output activations, coefficients of Taylor series, results of activation functions, sparse bitmaps, and the like. Data in the local memory 340 can be transferred to or from the memory 310, e.g., by the DMA engine 320. In some embodiments, data in the local memory 340 can be transferred to or from the local memory of another compute block 330.

[0068] In some embodiments, the local memory 340 includes one or more static random access memories (SDRAMs). The local memory 340 can be byte-addressable, and each memory address identifies a single byte (eight bits) of storage. In some embodiments, the local memory 340 can include memory banks. The number of data banks in the local memory 340 can be 16, 64, 128, 256, 512, 1024, 2048, or other numbers. A memory bank can include a number of memory locations. In an example, a data bank can include 8, 16, 64, or different numbers of memory locations. A memory bank or a memory location in a memory bank can have a memory address. In an example, a memory location can store a single byte, and data larger than a single byte can be stored in memory locations with consecutive memory addresses, e.g., adjacent memory locations. For example, a memory location can be able to store an integer in INT8 format, while two memory locations can be needed to store a number with 16 bits in FP16 or BF16 format. In some embodiments, 16 bits can be transferred from the local memory 340 in a single read cycle. In other embodiments, 16 bits can be transferred from the local memory 340 in multiple read cycles, such as two cycles.

[0069] The PE array 350 can include PEs arranged in columns or in columns and rows. Each PE is capable of performing a MAC operation. In some embodiments, a PE includes one or more multipliers for performing multiplications. The PE can also include one or more accumulators (“adders”) for performing accumulations. A column of PEs is referred to as a PE column. A PE column can be associated with one or more MAC channels. A MAC channel is a path for loading data into a MAC column. A MAC channel can also be referred to as a data transfer channel or a data load channel. A PE column can have multiple MAC channels. The load bandwidth of a MAC column is the set of load bandwidths of all MAC channels associated with the MAC column. With a certain number of MAC channels, data can be fed into the same number of individual PEs simultaneously. In some embodiments where a MAC column has four MAC channels for feeding activations or weights into the MAC column and each MAC channel can have a bandwidth of 16 bytes, the four MAC channels can have a total load bandwidth of 64 bytes.

[0070] In some embodiments, the PE array 350 can have the capability of depthwise convolution, standard convolution, or both. In depthwise convolution, a PE can perform a MAC operation that includes a sequence of multiplications of input operands and weight operands. Each multiplication in the sequence (also referred to as a cycle) is a multiplication of a different activation in the input operands and a different weight in the weight operands. The activations and weights in the same cycle can correspond to the same channel. The sequence of multiplications produces a product operand that includes a sequence of products. The MAC operation can also include an accumulation in which multiple product operands are accumulated to produce an output operand of the PE. The PE array 350 can output multiple output operands each time, each generated by a different PE. In standard convolution, a MAC operation can include an accumulation across channels. For example, instead of generating output operands, a PE can accumulate products across different channels to generate a single output point.

[0071] In some embodiments, the PE array 350 can perform MAC operations in quantized inference, such as MAC operations in quantized convolution. In some embodiments, a PE in the PE array 350 can receive quantized activations and quantized weights and compute a quantized MAC result. The quantized MAC result can be a quantized value in an integer format and can be the output of the PE. In some embodiments, the PE can also include a quantized multiplier that is capable of multiplying a quantization scale with the quantized MAC result and the output of the PE can be a real value in a floating point format. The PE can not include a quantized subtractor because a zero point offset is not needed for MAC operations in quantized inference.

[0072] The activation function unit 360 computes an activation function. The activation function unit 360 can receive the output of the PE array 350 as input to the activation function. The activation function unit 360 can transfer the output of the activation function to the local memory 340. The output of the activation function can be later retrieved by the PE array 350 from the local memory 340 for further computation. For example, the activation function unit 360 can receive an output tensor of a DNN layer from the PE array 350 and compute one or more activation functions on the output tensor. The result of the computation by the activation function unit 360 can be stored in the local memory 340 and later used as an input tensor for the next DNN layer. In some embodiments, the local memory 340 is associated with a load path, and an ejection path can be used for data transfer within the compute block 330. For example, data can be transferred from the local memory 340 to the PE array 350 through the load path. Data can be transferred from the PE array 350 to the local memory 340 through the ejection path. The activation function unit 360 (optionally, as well as one or more other post-processing units) can be arranged on the ejection path for processing the output of the PEs before the data is written into the local memory 340.

[0073] In some embodiments, the activation function unit 360 can compute an activation function that is computed by a Taylor series approximation, e.g., based on the pre-computation data generated by the pre-computation module 305. For example, the activation function unit 360 can compute one or more terms in the Taylor series by multiplying one or more coefficients of the Taylor series with one or more powers of the input to the activation function unit 360. The activation function unit 360 can include one or more multipliers that are capable of multiplying a coefficient with a power of the input. The input can be an activation computed by the PE array 350, such as the output activation of a convolution. The activation function unit 360 can further compute a partial sum of the one or more terms as the output of the activation function. The output of the activation function unit 360 can be written into the local memory 340. In some embodiments, the output of the activation function unit 360 can be read from the local memory 340 for future computation by the PE array 350. Certain aspects of the activation function unit 360 are described below in connection with FIG. 4-8

[0074] Example activation function unit FIG. 4 An example activation function unit 400 is illustrated in accordance with various embodiments. The activation function unit 400 can be an embodiment of the activation function unit 360 in FIG. 3 FIG. 4 As shown in FIG. 4A, the activation function unit 400 includes a plurality of compute elements 410A-410E (collectively referred to as “the plurality of compute elements 410” or “the compute elements 410”). Even though FIG. 4 ​​Five compute elements 410 are shown, but in other embodiments, the activation function unit 400 can include a different number of compute elements 410.

[0075] The compute elements 410 can operate in parallel. The operations of the compute elements 410 can be independent of each other. In some embodiments, the compute elements 410 receive different activations from the PE array 350. In one embodiment, each compute element 410 can receive a different activation from the output tensor computed by the PE array 350. In another embodiment, two or more of the compute elements 410 can process the same activation. In some embodiments, different compute elements 410 can receive an activation at different times, e.g., in different clock cycles.

[0076] The compute element 410 includes a memory 420, multipliers 430 and 440, an accumulator 450, and a register 460. In other embodiments, the compute element 410 can include different, fewer, or more components. Further, the functionality attributed to the components of the compute element 410 can be implemented by different components included in the compute element 410 or by different systems or devices.

[0077] The memory 420 stores coefficients of the Taylor series, which can be from the pre-compute module 305. Even though the memory 420 is in the compute element 410 at FIG. 4 The memory 420 can not be included in the compute element 410 or even in the activation function unit 400. In some embodiments, the memory 420 can be the memory 310 or the local memory 340.

[0078] The multiplier 430 can receive an activation from the PE array 350. In some embodiments, the multiplier 430 can compute a single activation at a time. The multiplier 430 can compute one or more powers of the activation. The multiplier 430 can compute powers of the activation sequentially. In one example, the multiplier 430 can compute a power of the activation in a computation period, e.g., a clock cycle. The exponent of the power of the activation computed in a certain period can be higher than the power of the activation computed in the previous period, e.g., by one.

[0079] The multiplier 440 multiplies the output of the multiplier 430 with a coefficient of the Taylor series from the memory 420. In one example, the multiplier 440 can multiply a power of the activation with a corresponding coefficient of the Taylor series in a certain period. The output of the multiplier 440 in the period can be a result of a term in the Taylor series. In some embodiments, the multiplier 440 can compute a term in a certain period.

[0080] Accumulator 450 receives the output of multiplier 440 and generates a partial sum of the terms in the Taylor series as a result of the activation function. In some embodiments, accumulator 450 can receive the output of multiplier 440 in a certain cycle. In the cycle, accumulator 450 can accumulate the output of multiplier 440 with a sum calculated by accumulator 450 in a previous cycle. The sum from the previous cycle can be stored in register 460. The new sum can also be stored in register 460 and further accumulated with the output of multiplier 440 in the next cycle. This process can continue until the partial sums of all required terms of the Taylor series are calculated.

[0081] FIG. 5 An example operation of activation function unit 400 without a stall cycle is illustrated in accordance with various embodiments. The operation of activation function unit 400 includes 12 clock cycles: 0-11. In the 12 clock cycles, activation function unit 400 processes six activations and calculates six results of the activation function. FIG. 5 Two tables 510 and 520 are shown. Table 510 shows the inputs of activation function unit 400 in six clock cycles: 0-5. Table 520 shows the outputs of activation function unit 400 in six clock cycles: 6-11. In other embodiments, the operation of activation function unit 400 can include different, fewer, or more cycles.

[0082] Table 510 shows six activations: xl-x6. In some embodiments, the six activations can be in the output operands calculated by PE array 350, e.g., by performing a convolution. In other embodiments, the six activations can be in the input operands of PE array 350, e.g., in the input of a convolutional layer. FIG. 5 In the illustrated embodiment, activation function unit 400 receives a new activation in each of the six cycles: activation xl is received in cycle 0, activation x2 is received in cycle 1, activation x3 is received in cycle 2, activation x4 is received in cycle 3, activation x5 is received in cycle 4, and activation x6 is received in cycle 5. Even though the six activations are shown in the illustrated embodiment, activation function unit 400 can receive more activations in future cycles. The activations are input into different computing elements 410 of activation function unit 400. The first five activations xl-x5 are input into five computing elements 410, respectively. The sixth activation x6 is input into computing element 410A. FIG. 5 In the illustrated embodiment, activation function unit 400 receives a new activation in each of the six cycles: activation xl is received in cycle 0, activation x2 is received in cycle 1, activation x3 is received in cycle 2, activation x4 is received in cycle 3, activation x5 is received in cycle 4, and activation x6 is received in cycle 5. Even though the six activations are shown in the illustrated embodiment, activation function unit 400 can receive more activations in future cycles. The activations are input into different computing elements 410 of activation function unit 400. The first five activations xl-x5 are input into five computing elements 410, respectively. The sixth activation x6 is input into computing element 410A.

[0083] Table 520 shows six outputs: y1-y6, each of which is computed by using a corresponding activation. Output y1 corresponds to activation x1, output y2 corresponds to activation x2, output y3 corresponds to activation x3, output y4 corresponds to activation x4, output y5 corresponds to activation x5, and output y6 corresponds to activation x6. The outputs y1-y6 are also generated in different cycles. Output y1 is computed in cycle 6, output y2 is computed in cycle 7, output y3 is computed in cycle 8, output y4 is computed in cycle 9, output y5 is computed in cycle 10, and output y6 is computed in cycle 11.

[0084] FIG. 6 Figure illustrates the computation of activation functions based on the same precision, according to various embodiments. The computation is performed by activation function unit 400. FIG. 6 Table 520 is a table that lists the computations performed in multipliers 430 and 440 and accumulator 450 in each of computation elements 410 in activation function unit 400. In FIG. 6 In an embodiment of Table 520, each computation element 410 computes the partial sum of the first six terms in a Taylor series. The first six terms make up a Taylor polynomial of degree five. Computation element 410A receives activation x1 in cycle 0. In each of cycles 0-4, multiplier 430 computes an intermediate product that is a power of x1. In the first cycle after the cycle in which the intermediate product is computed, multiplier 440 receives the intermediate product from multiplier 430 and multiplies the intermediate product by one of the six coefficients of the Taylor series, denoted as c1-c6 in FIG. 6 The product computed by multiplier 440 is the result of a term in the Taylor series. Accumulator 450 receives the result of the term in the second cycle after the cycle in which the intermediate product is computed. Accumulator 450 accumulates the result of the term with the output of accumulator 450 in the first cycle after the cycle in which the intermediate product is computed. Accumulator 450 outputs the result of the activation function for x1 in cycle 6.

[0085] Computation element 410B receives activation x2 in cycle 1 and outputs the result of the activation function for x2 in cycle 7. Computation element 410C receives activation x3 in cycle 2 and outputs the result of the activation function for x3 in cycle 8. Computation element 410D receives activation x4 in cycle 3 and outputs the result of the activation function for x4 in cycle 9. Computation element 410E receives activation x5 in cycle 4 and outputs the result of the activation function for x5 in cycle 10.

[0086] Computation element 410 can start processing a new activation in the cycle immediately after multiplier 430 computes the last intermediate product of an activation. As FIG. 6As shown, computing element 410A receives activation x6 in cycle 5, even though multiplier 440 and accumulator 450 are still processing activation x1. Computing element 410A outputs the result of the activation function of x6 in cycle 11. Similarly, computing element 410B receives activation x7 in cycle 6 and outputs the result of the activation function of x7 in cycle 12. Even though FIG. 6 Not shown in the diagram, but activation function unit 400 can handle more activations.

[0087] exist FIG. 6 In one embodiment, the seven results of the activation function have the same precision, and the same number of terms are computed for all seven activations. In other embodiments, different activations may be processed by the activation function unit 400 based on different precisions.

[0088] FIG. 7 The illustration shows example operations of an activation function unit 400 with a pause period according to various embodiments. FIG. 7 In the operation illustrated in the diagram, activation function unit 400 processes five activations x1-x5 and computes five results y1-y5 of the activation function. The operation of activation function unit 400 consists of 12 clock cycles: 0-11, but cycle 2 is a pause cycle because activation function unit 400 does not receive any new activations, thus producing no output in cycle 7. The presence of the pause cycle can be used to facilitate higher precision for result y2 than for other results. Higher precision requires the computation of more terms in the Taylor series and takes more clock cycles to compute result y2. The following section combines... FIG. 8 To describe more details related to the pause clock cycles used to achieve higher precision.

[0089] FIG. 7 Tables 710 and 720 are shown. Table 710 shows six clock cycles 0-6, where activation function unit 400 receives five activations x1-x5. Activation function unit 400 receives different activations in each of cycles 0, 1, and 3-5. Table 720 shows six clock cycles 6-12, where activation function unit 400 outputs five results y1-y5. Activation function unit 400 outputs different results in each of cycles 6 and 8-11. In other embodiments, the operation of activation function unit 400 may include different, fewer, or more cycles.

[0090] In some embodiments, five activations x1-x5 can be in the output operands computed by the PE array 350, e.g., by performing a convolution. The activations can be input to different compute elements 410 of the activation function unit 400. The first five activations x1-x5 are input to five compute elements 410, respectively. Each of the results y1-y6 is computed by using the corresponding activation. The output y1 corresponds to the activation x1, the output y2 corresponds to the activation x2, the output y3 corresponds to the activation x3, the output y4 corresponds to the activation x4, and the output y5 corresponds to the activation x5.

[0091] FIG. 8 The computations of the activation function based on different precisions are illustrated according to various embodiments. The computations are performed by the activation function unit 400. FIG. 8 is a table that lists the computations performed in the multipliers 430 and 440 and the accumulators 450 in each of the compute elements 410 in the activation function unit 400. In FIG. 8 In an embodiment of the table, each compute element 410 computes a partial sum of the first six terms in the Taylor series. The first six terms make up a Taylor polynomial of degree five. The compute element 410A receives the activation x1 in cycle 0. In each of cycles 0-4, the multiplier 430 computes an intermediate product that is a power of x1. In the first cycle after the cycle in which the intermediate product is computed, the multiplier 440 receives the intermediate product from the multiplier 430 and multiplies the intermediate product by one of the six coefficients of the Taylor series, denoted as c1-c6 in FIG. 8 The product computed by the multiplier 440 is the result of a term in the Taylor series. The accumulator 450 receives the result of the term in the second cycle after the cycle in which the intermediate product is computed. The accumulator 450 accumulates the result of the term with the output of the accumulator 450 in the first cycle after the cycle in which the intermediate product is computed. The accumulator 450 outputs the result of the activation function for x1 in cycle 6.

[0092] The compute elements 410B and 410C handle the activation x2, which requires higher precision, and thus more terms in the Taylor series need to be computed. In FIG. 8 In an embodiment of the table, the first 12 terms in the Taylor series are computed for the activation x2. The first 12 terms make up a Taylor polynomial of degree 11. The compute element 410B receives the activation x2 in cycle 1 and computes some of the 12 terms. The compute element 410C receives the activation x2 in cycle 2 and computes the other of the 12 terms. In cycle 8, the compute element 410C outputs the result of the activation function for x2.

[0093] The other activations x3, x4, and x5 have the same precision as activation xl, and thus are each computed by a single compute element 410 as is activation xl. Compute element 410D receives activation x3 in cycle 3 and outputs the result of the activation function for x3 in cycle 9. Compute element 410E receives activation x4 in cycle 5 and outputs the result of the activation function for x4 in cycle 10.

[0094] The compute elements 410 can begin processing a new activation in the cycle immediately after the last intermediate product for an activation is computed by the multipliers 430. As shown in FIG. 4B, compute element 410A receives activation x5 in cycle 5, even though the multipliers 440 and accumulators 450 are still processing activation xl. Compute element 410A outputs the result of the activation function for x5 in cycle 11. Similarly, compute element 410B receives activation x6 in cycle 6 and outputs the result of the activation function for x6 in cycle 12. Compute element 410C receives activation x7 in cycle 7 and outputs the result of the activation function for x7 in cycle 13. Even though the activation function unit 400 is processing more activations than in the embodiment of FIG. 4A, the activation function unit 400 is still able to process all of the activations in 16 clock cycles. FIG. 8 FIG. 8 More activations can be processed by the activation function unit 400, even though not shown in FIG. 4B.

[0095] Even though the activation function unit 400 processes the same number of activations in the embodiments of FIG. 4A and FIG. 4B, the increased precision of the result of activation x2 causes activation x2 to be repeated for two computation cycles, and FIG. 7 FIG. 8 The operation of the activation function unit 400 in FIG. 4B requires one additional cycle. FIG. 8

[0096] In embodiments where the total number of inputs is n and for each input x i at least k i additional terms are needed to achieve the desired precision, the total number of clock cycles N can then be expressed as: There can be a linear tradeoff between performance (e.g., number of clock cycles) and precision in terms of the number of terms within the Taylor series. For the same number of computation resources, k additional Taylor series terms can be computed by adding k / t cycles instead of the overhead of k additional cycles. For an input size n = 64 (i.e., 64 activations input into the activation function unit 400), this translates to approximately a 49% reduction in the number of clock cycles required for computation using the same amount of hardware resources in embodiments that require 16 additional terms to meet higher precision requirements.

[0097] Example PE array FIG. 9 A PE array is illustrated in accordance with various embodiments. The PE array 900 can be a systolic array. The PE array 900 can be a 4x4 array of PEs 910.​​​FIG. 3 An embodiment of the PE array 350 is described. The PE array 900 includes a plurality of PEs 910 (collectively referred to as "PE 910"). PE 910 is capable of performing MAC operations, including MAC operations in quantized inference. PE 910 may also be referred to as a neuron in a DNN. Each PE 910 has two input signals 950 and 960 and an output signal 970. Input signal 950 is at least a portion of the IFM to the layer. Input signal 960 is at least a portion of the filter of the layer. In some embodiments, the input signal 950 of PE 910 includes one or more input operands, and the input signal 960 includes one or more weight operands.

[0098] Each PE 910 performs a MAC operation on input signals 950 and 960 and outputs an output signal 970, which is the result of the MAC operation. Some or all of the input signals 950 and 960 and the output signal 970 can be in integer format such as INT8 or floating-point format such as FP16 or BF16. For simplicity and illustration purposes, the input and output signals of the PE 910 have the same reference numerals, but the PE 910 can receive different input signals and output different output signals. A PE 910 can also differ from another PE 910, for example, by including more, fewer, or different components.

[0099] like FIG. 9 As shown, PE 910s are interconnected, as... FIG. 9 The output signal 970 of PE 910 can be sent as an input signal to many other PE 910s (and possibly sent back to itself) via interconnections between PE 910s. In some embodiments, the output signal 970 of PE 910 can include the output signals of one or more other PE 910s through an accumulation operation of PE 910s, and generate an internal portion of the PE array.

[0100] exist FIG. 3In embodiments, the PE 910 is arranged into a column 905 (individually referred to as“column 905”). The inputs and weights can be distributed to the PE 910 based on the column 905. Each column 905 has a column buffer 920. The column buffer 920 stores data to be provided to the PE 910 in the column 905 for a short amount of time. The column buffer 920 can also store data output by the last PE 910 in the column 905. The output of the last PE 910 can be the sum of the MAC operations of all the PEs 910 in the column 905, which is a column-level internal partial sum of the PE array 900. In other embodiments, the inputs and weights can be distributed to the PE 910 based on the row in the PE array 900. The PE array 900 can include a row buffer instead of the column buffer 920. The row buffer can store the input signals for the PEs in the corresponding row, and can also store a row-level internal partial sum of the PE array 900.

[0101] In some embodiments, the column buffer 920 can be part of the local memory 340 in FIG. 3 The column buffer 920 can be associated with an upper memory hierarchy (e.g., the memory 310 in FIG. 10 The data in the column buffer 920 can be sent to the upper memory hierarchy. The column buffer 920 can receive data from the upper memory hierarchy.

[0102] FIG. 9 is a block diagram of a PE 1000 according to various embodiments. The PE 1000 can be an embodiment of the PE 910 in FIG. 3 The PE 1000 can perform MAC operations, such as MAC operations using data in integer format. The PE 1000 can be an example PE in the PE array 350 described above in connection with FIG. 10 As shown in Example method of computing activation functions The PE 1000 includes an input register file 1010 (individually referred to as“input register file 1010”), a weight register file 1020 (individually referred to as“weight register file 1020”), a multiplier 1030 (individually referred to as“multiplier 1030”), an internal adder assembly 1040, and an output register file 1050. In other embodiments, the PE 1000 can include fewer, more, or different components. For example, the PE 1000 can include multiple output register files 1050. As another example, the PE 1000 can include a single input register file 1010, weight register file 1020, or multiplier 1030. As still another example, the PE 1000 can include an adder instead of the internal adder assembly 1040.

[0103] Input register file 1010 temporarily stores active operands for MAC operations performed by PE 1000. In some embodiments, input register file 1010 can store a single active operand at a time. In other embodiments, input register file 1010 can store multiple active operands or a portion of an active operand at a time. An active operand includes multiple input elements (i.e., input elements) in an input tensor. The input elements of an active operand can be stored in input register file 1010 sequentially, so that the input elements can be processed sequentially. In some embodiments, each input element in an active operand can be from a different input channel of the input tensor. An active operand can include an input element from each of the input channels of the input tensor, and the number of input elements in an active operand can be equal to the number of input channels. The input elements in an active operand can have the same XY coordinates, which can be used as the XY coordinates of the active operand. For example, all of the input elements of an active operand can be X0Y0, X0Y1, X1Y1, etc.

[0104] Weight register file 1020 temporarily stores weight operands for MAC operations performed by PE 1000. A weight operand includes weights in a filter of a DNN layer. In some embodiments, weight register file 1020 can store a single weight operand at a time. In other embodiments, input register file 1010 can store multiple weight operands or a portion of a weight operand at a time. A weight operand can include multiple weights. The weights of a weight operand can be stored in weight register file 1020 sequentially, so that the weights can be processed sequentially. In some embodiments, for a multiplication operation involving a weight operand and an active operand, each weight in the weight operand can correspond to an input element of the active operand. The number of weights in a weight operand can be equal to the number of input elements in an active operand.

[0105] In some embodiments, weight register file 1020 can be the same as or similar to input register file 1010, e.g., have the same size, etc. PE 1000 can include multiple register files, some of which are designated as input register file 1010 for storing active operands, some of which are designated as weight register file 1020 for storing weight operands, and some of which are designated as output register file 1050 for storing output operands. In other embodiments, the register files in PE 1000 can be designated for other purposes, e.g., for storing scale operands used in element-wise addition operations, etc.

[0106] The multiplier 1030 performs a multiplication operation on an activation operand and a weight operand. The multiplier 1030 can perform a sequence of multiplication operations on a single activation operand and a single weight operand, and generate a product operand that includes a sequence of products. Each multiplication operation in the sequence includes multiplying an input element in the activation operand and a weight in the weight operand. In some embodiments, the position (or index) of the input element in the activation operand matches the position (or index) of the weight in the weight operand. For example, a first multiplication operation is multiplying a first input element in the activation operand and a first weight in the weight operand, a second multiplication operation is multiplying a second input element in the activation operand and a second weight in the weight operand, a third multiplication operation is multiplying a third input element in the activation operand and a third weight in the weight operand, and so on. The input element and the weight in the same multiplication operation can correspond to the same depthwise channel, and their product can also correspond to the same depthwise channel.

[0107] The plurality of multipliers 1030 can perform multiplication operations concurrently. These multiplication operations can be referred to as a round of multiplication operations. In a round of multiplication operations by the multipliers 1030, each of the multipliers 1030 can use a different activation operand and a different weight operand. The different activation operands or weight operands can be stored in different register files of the PE 1000. For example, a first multiplier 1030 uses a first activation operand (e.g., stored in a first input register file 1010) and a first weight operand (e.g., stored in a first weight register file 1020), while a second multiplier 1030 uses a second activation operand (e.g., stored in a second input register file 1010) and a second weight operand (e.g., stored in a second weight register file 1020), a third multiplier 1030 uses a third activation operand (e.g., stored in a third input register file 1010) and a third weight operand (e.g., stored in a third weight register file 1020), and so on. For a single multiplier 1030, the round of multiplication operations can include multiple cycles. A cycle includes a multiplication operation on an input element and a weight.

[0108] The multipliers 1030 can perform multiple rounds of multiplication operations. The multipliers 1030 can use the same weight operand but different activation operands in different rounds. For example, the multipliers 1030 perform a sequence of multiplication operations on a first activation operand stored in a first input register file in a first round, while performing on a second activation operand stored in a second input register file in a second round. In the second round, different multipliers 1030 can perform another sequence of multiplication operations using the first activation operand and different weight operands. That way, the first activation operand is reused in the second round. The first activation operand can be further reused in additional rounds, e.g., by additional multipliers 1030.

[0109] The internal adder assembly 1040 includes one or more adders, i.e., internal adders, internal to the PE 1000. The internal adder assembly 1040 can perform accumulation operations on two or more product operands from the multipliers 1030 and produce an output operand of the PE 1000. In some embodiments, the internal adders are arranged in a tile sequence. A tile includes one or more internal adders. For a first tile of the internal adder assembly 1040, the internal adders can receive product operands from two or more multipliers 1030 and generate and operate on the operands by a sequence of accumulation operations. Each accumulation operation produces a sum of two or more products, each from a different multiplier 1030. The sum operands include a sequence of sums, each of which is a result of an accumulation operation and corresponds to a depthwise channel. For another (further) tile of the internal adder assembly 1040, the internal adders in the tile receive the sum operands from a preceding tile in the sequence. Each of these quantities can be generated by a different internal adder in the preceding tile. The ratio of the number of internal adders in the tile to the number of internal adders in the following tile can be 2: 1. In some embodiments, a last tile of the internal adder assembly 1040 can include a single internal adder that produces the output operand of the PE 1000.

[0110] The output register file 1050 stores the output operand of the PE 1000. In some embodiments, the output register file 1050 can store the output operand at a time. In other embodiments, the output register file 1050 can store multiple output operands or a portion of an output operand at a time. The output operand includes multiple output elements in the IFM. The output elements of the output operand can be stored in the output register file 1050 sequentially, so that the output elements can be processed sequentially. In some embodiments, each output element in the output operand corresponds to a different depthwise channel and is an element of a different output channel of the output channels of the depthwise convolution. The number of output elements in the output operand can be equal to the number of depthwise channels of the depthwise convolution.

[0111] FIG. 11 FIG. 3 is a flowchart illustrating a method 1100 of computing an activation function in a neural network, according to various embodiments. The method 1100 can be performed by the activation function unit 360 in FIG. 11 . Although the method 1100 is described with reference to the flowchart illustrated in FIG. 11 , many other methods of computing an activation function can alternatively be used. For example, the order of execution of the steps in the method 1100 can be changed. As another example, some of the steps can be changed, deleted, or combined. Example computing device ​

[0112] The activation function unit 360 receives (1110) one or more pre-computed coefficients of an approximation of an activation function in a neural network. In some embodiments, the approximation is a Taylor series that can approximate the computation of the activation function. The one or more pre-computed coefficients include one or more coefficients of the Taylor series.

[0113] The activation function unit 360 receives (1120) an activation that is computed in a layer of the neural network. In some embodiments, the activation is an output activation of a layer of the neural network.

[0114] The activation function unit 360 computes (1130), using the activation, one or more intermediate products by a first multiplier. In some embodiments, the first multiplier computes a first intermediate product in a first clock cycle. After computing the first intermediate product, the first multiplier computes a second intermediate product in a second clock cycle based on the activation and the first intermediate product. In an example, the first intermediate product can be a square of the activation, and the second intermediate product can be a cube of the activation.

[0115] The activation function unit 360 computes (1140), based on the one or more intermediate products from the first multiplier and the one or more coefficients, one or more terms of the approximation by a second multiplier. In some embodiments, the second multiplier computes the one or more terms of the approximation in a sequence of clock cycles by using a different coefficient of the approximation in each clock cycle of the sequence. In some embodiments, the one or more terms of the approximation include one or more Taylor series terms.

[0116] The activation function unit 360 computes (1150), based on a polynomial that includes the one or more terms of the approximation, an output of the activation function by an accumulator. A degree of the polynomial is determined based on a predetermined accuracy of the output of the activation function. In some embodiments, the degree of the polynomial is equal to the number of terms minus one. In some embodiments, the activation function unit 360 provides the output of the activation function to another layer of the neural network. The other layer is after the layer in the neural network.

[0117] In some embodiments, the activation function unit 360 receives one or more other activations in one or more clock cycles that are different from the clock cycle in which the activation is received. The one or more other activations are computed in a layer of the neural network. In some embodiments, the activation function unit 360 computes another output of the activation function based on another activation that is computed in a layer of the neural network. The other output of the activation function has a different predetermined accuracy than the output of the activation function.

[0118] FIG. 12 FIG. 12is a block diagram of an example computing device 1200 in accordance with various embodiments. In some embodiments, the computing device 1200 can be used as at least part of the DNN accelerator 300. A number of components are illustrated in FIG. 12 the computing device 1200, although any one or more of these components can be omitted or duplicated as appropriate to the application. In some embodiments, some or all of the components included in the computing device 1200 can be attached to one or more motherboards. In some embodiments, some or all of the components are fabricated onto a single system on a chip (SoC) die. Additionally, in various embodiments, the computing device 1200 can not include one or more of the components illustrated in FIG. 11 the computing device 1200, but the computing device 1200 can include interface circuitry to couple to the one or more components. For example, the computing device 1200 can not include the display device 1206, but can include display device interface circuitry (e.g., a connector and driver circuitry) to which a display device 1206 can be coupled. In another set of examples, the computing device 1200 can not include the audio input device 1218 or the audio output device 1208, but can include audio input or output device interface circuitry (e.g., connectors and support circuitry) to which an audio input device 1218 or an audio output device 1208 can be coupled.

[0119] The computing device 1200 can include a processing device 1202 (e.g., one or more processing devices). The processing device 1202 processes electronic data from registers and / or memory to transform that electronic data into other electronic data that can be stored in registers and / or memory. The computing device 1200 can include memory 1204, which can itself include one or more memory devices such as volatile memory (e.g., DRAM), non-volatile memory (e.g., read-only memory (ROM)), high-bandwidth memory (HBM), flash memory, solid-state memory, and / or a hard disk drive. In some embodiments, the memory 1204 can include memory that shares a die with the processing device 1202. In some embodiments, the memory 1204 includes one or more non-transitory computer-readable media that store instructions that are executable to perform operations to compute activation functions in a DNN, such as the method 1100 described above in connection with FIG. 3 or some of the operations performed by the DNN accelerator 300 (e.g., the activation function unit 360) described above in connection with Selection example The instructions stored in the one or more non-transitory computer-readable media can be executed by the processing device 1202.

[0120] In some embodiments, the computing device 1200 can include a communication chip(s) 1212 (e.g., one or more communication chips). For example, the communication chip(s) 1212 can be configured to manage wireless communications for the computing device 1200. The term "wireless" and its derivatives can be used to describe circuits, devices, systems, methods, techniques, communications channels, etc., that can communicate data through the use of modulated electromagnetic radiation through a non-solid medium. The term does not imply that the associated devices do not contain any wires, although in some embodiments they might not.

[0121] The communication chip(s) 1212 can implement any of a number of wireless standards or protocols, including but not limited to Wi-Fi (IEEE 802.10 family standards), IEEE 802.16 standards (e.g. IEEE 802.16-2005 Revision), Long Term Evolution (LTE) project

[0122] In some embodiments, the communication chip 1212 can manage wired communications, such as electrical, optical, or any other suitable communication protocol (e.g., Ethernet). As noted above, the communication chip 1212 can include multiple communication chips. For instance, a first communication chip 1212 can be dedicated to shorter-range wireless communications, such as Wi-Fi or Bluetooth, and a second communication chip 1212 can be dedicated to longer-range wireless

[0123] The computing device 1200 can include a battery / power circuit module 1214. The battery / power circuit module 1214 can include one or more energy storage devices (e.g., batteries or capacitors) and / or circuitry for coupling components of the computing device 1200 to an energy source separate from the computing device 1200 (e.g., AC line power).

[0124] The computing device 1200 can include a display device 1206 (or corresponding interface circuitry, as discussed above). The display device 1206 can include any visual

[0125] The computing device 1200 can include an audio output device 1208 (or corresponding interface circuitry, as discussed above). The audio output device 1208 can include any device that generates an audible indicator, such as speakers, headphones, or earbuds, for example.

[0126] The computing device 1200 can include an audio input device 1218 (or corresponding interface circuitry, as discussed above). The audio input device 1218 can include any device that generates a signal representative of a sound, such as microphones, microphone arrays, or digital instruments (e.g., instruments with a musical instrument digital interface (MIDI) output).

[0127] The computing device 1200 can include a GPS device 1216 (or corresponding interface circuitry, as discussed above). The GPS device 1216 can communicate with satellite-based systems and can receive a location of the computing device 1200, as is known in the art.

[0128] The computing device 1200 can include another output device 1210 (or a corresponding interface circuitry, as discussed above). Examples of the other output device 1210 can include an audio codec, a video codec, a printer, a wired or wireless transmitter for providing information to other devices, or an additional storage device.

[0129] The computing device 1200 can include another input device 1220 (or a corresponding interface circuitry, as discussed above). Examples of the other input device 1220 can include an accelerometer, a gyroscope, a compass, an image capture device, a keyboard, a cursor control device such as a mouse, a stylus, a touchpad, a bar code reader, a quick response (QR) code reader, any sensor, or a radio frequency identification (RFID) reader.

[0130] The computing device 1200 can have any desired form factor, such as a handheld or mobile computer system (e.g., a cellular telephone, a smartphone, a mobile Internet device, a music player, a tablet computer, a laptop computer, a netbook computer, an ultrabook computer, a personal digital assistant (PDA), an ultra-mobile personal computer, etc.), a desktop computer system, a server or other networked computing component, a printer, a scanner, a monitor, a set-top box, an entertainment control unit, a vehicle control unit, a digital camera, a digital video recorder, or a wearable computer system. In some embodiments, the computing device 1200 can be any other electronic device that processes data.

[0131] ​ The following paragraphs provide various examples of the embodiments disclosed herein.

[0132] Example 1 provides a computing element for computing an activation function, the computing element comprising: a first multiplier configured to compute one or more intermediate products using an activation, the activation being computed in a layer of a neural network; a second multiplier configured to compute one or more terms of the approximation of the activation function based on the one or more intermediate products from the first multiplier and the one or more coefficients of the approximation; and an accumulator configured to compute an output of the activation function based on a polynomial comprising the one or more terms of the approximation, wherein a degree of the polynomial is determined based on a predetermined accuracy of the output of the activation function.

[0133] Example 2 provides the computing element of Example 1, wherein the approximation of the activation function is a Taylor series, and the one or more coefficients of the approximation comprise one or more coefficients of the Taylor series computed prior to the activation being computed.

[0134] Example 3 provides the computing element of example 1 or 2, further comprising a storage unit associated with the accumulator, the storage unit configured to store an intermediate sum computed by the accumulator, wherein the accumulator is configured to compute the output of the activation function by accumulating the intermediate sum with the approximate term computed by the second multiplier.

[0135] Example 4 provides the computing element of any of the preceding examples, wherein the first multiplier is configured to compute the one or more intermediate products by: computing a first intermediate product in a first clock cycle; and after computing the first intermediate product, computing a second intermediate product based on the activation and the first intermediate product in a second clock cycle.

[0136] Example 5 provides the computing element of any of the preceding examples, wherein the second multiplier is configured to compute the one or more terms of the approximation in a sequence of clock cycles, and the second multiplier is configured to use a different coefficient of the approximation in each clock cycle of the sequence.

[0137] Example 6 provides the computing element of any of the preceding examples, wherein the computing element is included in a plurality of computing elements for computing an output of the activation function using a plurality of activations, the plurality of activations computed in the layer of the neural network and including the activation, and the plurality of activations input into different computing elements of the plurality of computing elements in different clock cycles.

[0138] Example 7 provides the computing element of example 6, wherein a first output of the activation function based on a first activation of the plurality of activations has a higher predetermined accuracy than a second output of the activation function based on a second activation of the plurality of activations, and the first output of the activation function is computed by more computing elements than the second output of the activation function.

[0139] Example 8 provides a device for a deep learning operation, the device comprising: one or more processing elements configured to compute one or more activations by performing the deep learning operation in a neural network; a memory configured to store one or more coefficients of an approximation of an activation function in the neural network; and one or more computing elements configured to receive the one or more activations from the one or more processing elements and the one or more coefficients from the memory, a computing element comprising: a first multiplier configured to compute one or more intermediate products using an activation in the one or more activations; a second multiplier configured to compute one or more terms of the approximation based on the one or more intermediate products from the first multiplier and the one or more coefficients; and an accumulator configured to compute an output of the activation function based on a polynomial comprising the one or more terms of the approximation, wherein a degree of the polynomial is determined based on a predetermined accuracy of the output of the activation function.

[0140] Example 9 provides the device of Example 8, wherein the one or more processing elements are coupled to the memory by a data transfer path, and the computing element is on the data transfer path.

[0141] Example 10 provides the device of Example 8 or 9, wherein the first multiplier is configured to compute the one or more intermediate products by computing a first intermediate product in a first clock cycle; and after computing the first intermediate product, computing a second intermediate product in a second clock cycle based on the activation and the first intermediate product.

[0142] Example 11 provides the device of any of Examples 8-10, wherein the second multiplier is configured to compute the one or more terms of the approximation in a sequence of clock cycles, and the second multiplier is configured to use a different coefficient of the approximation in each clock cycle in the sequence.

[0143] Example 12 provides the device of any of Examples 8-11, wherein different activations in the one or more activations are input into different computing elements in the one or more computing elements in different clock cycles.

[0144] Example 13 provides the device of Example 12, wherein a first output of the activation function based on a first activation in the one or more activations has a higher predetermined accuracy than a second output of the activation function based on a second activation in the one or more activations, and the first output of the activation function is computed by more computing elements than the second output of the activation function.

[0145] Example 14 provides the apparatus of any of Examples 8-13, wherein the deep learning operation is in a first layer of the neural network, the output of the activation function is input to a second layer of the neural network, and the second layer is after the first layer in the neural network.

[0146] Example 15 provides a method for deep learning, comprising: receiving one or more pre-computed coefficients of an approximation of an activation function in a neural network; receiving an activation computed in a layer of the neural network; computing, by a first multiplier, one or more intermediate products using the activation; computing, by a second multiplier, one or more terms of the approximation based on the one or more intermediate products from the first multiplier and the one or more coefficients; and computing, by an accumulator, an output of the activation function based on a polynomial comprising the one or more terms of the approximation, wherein a degree of the polynomial is determined based on a predetermined accuracy of the output of the activation function.

[0147] Example 16 provides the method of Example 15, wherein computing the one or more intermediate products comprises: computing a first intermediate product in a first clock cycle; and computing, in a second clock cycle after computing the first intermediate product, a second intermediate product based on the activation and the first intermediate product.

[0148] Example 17 provides the method of Example 15 or 16, wherein computing the one or more terms of the approximation comprises computing the one or more terms of the approximation in a sequence of clock cycles by using different coefficients of the approximation in each clock cycle of the sequence.

[0149] Example 18 provides the method of any of Examples 15-17, further comprising computing another output of the activation function based on another activation computed in the layer of the neural network, wherein the other output of the activation function has a different predetermined accuracy than the output of the activation function.

[0150] Example 19 provides the method of any of Examples 15-18, further comprising receiving one or more other activations in one or more clock cycles different from a clock cycle in which the activation is received, the one or more other activations being computed in the layer of the neural network.

[0151] Example 20 provides the method of any of Examples 15-19, further comprising providing the output of the activation function to another layer of the neural network, wherein the other layer is after the layer in the neural network.

[0152] The above description of illustrated implementations of the present disclosure, including what is described in the Abstract, is not intended to be exhaustive or to limit the present disclosure to the precise forms disclosed. While specific implementations of, and examples for, the present disclosure are described herein for illustrative purposes, various equivalent modifications are possible within the scope of this present disclosure, as those skilled in the relevant art will recognize. These modifications can be made to the present disclosure in light of the above detailed description.

Claims

1. A computational element for calculating an activation function, the computational element comprising: The first multiplier is configured to use activations to compute one or more intermediate products, the activations being computed in layers of the neural network; The second multiplier is configured to compute one or more of the approximations based on the one or more intermediate products from the first multiplier and one or more coefficients of the approximation of the activation function; as well as An accumulator is configured to compute the output of the activation function based on a polynomial that includes one or more of the approximations, wherein the degree of the polynomial is determined based on a predetermined precision of the output of the activation function.

2. The computing element as claimed in claim 1, wherein, The approximation of the activation function is a Taylor series, and the one or more coefficients of the approximation include one or more coefficients of the Taylor series calculated prior to the activation being calculated.

3. The computing element as described in claim 1 or 2, further comprising: A storage unit associated with the accumulator, the storage unit being configured to store intermediate sums calculated by the accumulator.

4. The computing element as claimed in claim 3, wherein, The accumulator is configured to compute the output of the activation function by accumulating the intermediate sum with the approximate term calculated by the second multiplier.

5. The computing element as described in any one of claims 1-4, wherein, The first multiplier is configured to compute the one or more intermediate products by the following steps: Calculate the first intermediate product in the first clock cycle; as well as After calculating the first intermediate product, the second intermediate product is calculated in the second clock cycle based on the activation and the first intermediate product.

6. The computing element as claimed in any one of claims 1-5, wherein, The second multiplier is configured to compute one or more of the approximation in a sequence of clock cycles, and the second multiplier is configured to use different coefficients of the approximation in each clock cycle of the sequence.

7. The computing element as claimed in any one of claims 1-6, wherein: The computational element is included in a plurality of computational elements for computing the output of the activation function using a plurality of activations, and The plurality of activations are computed in the layers of the neural network and include the activations.

8. The computing element as claimed in claim 7, wherein, The multiple activations are input to different computing elements among the multiple computing elements at different clock cycles.

9. The computing element as claimed in claim 8, wherein, The first output of the activation function based on the first activation among the plurality of activations has a higher predetermined accuracy than the second output of the activation function based on the second activation among the plurality of activations.

10. The computing element as claimed in claim 9, wherein, The first output of the activation function is computed by more computational elements than the second output of the activation function.

11. An apparatus for deep learning operations, the apparatus comprising: One or more processing elements are configured to compute one or more activations by performing the deep learning operations in the neural network; A memory configured to store one or more coefficients of an approximation of the activation function in the neural network; as well as One or more computing elements are configured to receive the one or more activations from the one or more processing elements and to receive the one or more coefficients from the memory, the computing elements comprising: The first multiplier is configured to compute one or more intermediate products using one or more activations. A second multiplier is configured to compute one or more terms of the approximation based on the one or more intermediate products from the first multiplier and the one or more coefficients. An accumulator is configured to compute the output of the activation function based on a polynomial that includes one or more of the approximations, wherein the degree of the polynomial is determined based on a predetermined precision of the output of the activation function.

12. The device as claimed in claim 11, wherein, The one or more processing elements are coupled to the memory via a data transmission path, and the computing elements are located on the data transmission path.

13. The device as claimed in claim 11 or 12, wherein, The first multiplier is configured to compute the one or more intermediate products by the following steps: Calculate the first intermediate product in the first clock cycle; as well as After calculating the first intermediate product, the second intermediate product is calculated in the second clock cycle based on the activation and the first intermediate product.

14. The device as claimed in any one of claims 11-13, wherein, The second multiplier is configured to compute one or more of the approximations in a sequence of clock cycles.

15. The device as claimed in claim 14, wherein, The second multiplier is configured to use different coefficients of the approximation in each clock cycle of the sequence.

16. The device as claimed in any one of claims 11-15, wherein, Different activations of the one or more activations are input to different computing elements of the one or more computing elements in different clock cycles.

17. The device of claim 16, wherein: The first output of the activation function based on the first activation of the one or more activations has a higher predetermined accuracy than the second output of the activation function based on the second activation of the one or more activations. The first output of the activation function is computed by more computational elements than the second output of the activation function.

18. The device as claimed in any one of claims 11-17, wherein: The deep learning operation is located in the first layer of the neural network, and The output of the activation function is input into the second layer of the neural network.

19. The device as claimed in claim 18, wherein, The second layer is located after the first layer in the neural network.

20. A method for deep learning, comprising: Receive one or more pre-computed coefficients that approximate the activation function in the neural network; Receive activations computed in the layers of the neural network; The activation is used by the first multiplier to compute one or more intermediate products; The second multiplier calculates one or more of the approximations based on the one or more intermediate products from the first multiplier and the one or more pre-calculated coefficients; as well as The accumulator computes the output of the activation function based on a polynomial that includes one or more of the approximations. The degree of the polynomial is determined based on a predetermined precision of the output of the activation function.

21. The method of claim 20, wherein, Calculating the one or more intermediate products includes: Calculate the first intermediate product in the first clock cycle; and After calculating the first intermediate product, the second intermediate product is calculated in the second clock cycle based on the activation and the first intermediate product.

22. The method of claim 20 or 21, wherein, Calculating one or more of the approximations includes: One or more terms of the approximation are calculated in a sequence of clock cycles by using different coefficients of the approximation in each clock cycle of the sequence.

23. The method of any one of claims 20-22, further comprising: Another output of the activation function is calculated based on another activation computed in the layer of the neural network. The other output of the activation function has a predetermined precision different from the output of the activation function.

24. The method of any one of claims 20-23, further comprising: One or more other activations are received in one or more clock cycles that are different from the clock cycle in which the activation is received, and the one or more other activations are computed in the layer of the neural network.

25. The method of any one of claims 20-24, further comprising: The output of the activation function is provided to another layer of the neural network, wherein the other layer is located after the layer in the neural network.

Images