Approximation of activation functions using Taylor series

Abstract

An activation function unit can compute an activation function approximated by a Taylor series. An activation function unit may include multiple computational elements. Each computational element may include two multipliers and one accumulator. The first multiplier may compute an intermediate product using an activation, such as the output activation of a DNN layer. The second multiplier may compute terms of a Taylor series approximating the activation function based on the intermediate product from the first multiplier and the coefficients of the Taylor series. The accumulator may compute a partial sum of the terms as the output of the activation function. The number of terms may be determined based on a predetermined precision of the output of the activation function. An activation function unit can handle multiple activations. Different activations may be input to different computational elements in different clock cycles. An activation function unit may compute activation functions with different precisions.

Description

[Technical Field]

[0001] Cross-references to related applications This application claims priority to U.S. Patent Application No. 18 / 346,992, “Approximation of Activation Functions Using Taylor Series,” filed on 5 July 2023, which is incorporated herein by reference in its entirety.

[0002] Technical field This disclosure generally relates to deep neural networks (DNNs), and more specifically to approximating the activation function in a DNN using a Taylor series. [Background technology]

[0003] Due to their ability to achieve high accuracy, DNNs are widely used in a variety of artificial intelligence applications, from computer vision to speech recognition and natural language processing. However, this high accuracy comes at the cost of considerable computational expense. DNNs have extremely high computational demands, as each inference can require hundreds of millions of MAC (multiply-accumulate) operations, as well as reading and writing large amounts of data. DNN inference also requires the calculation of activation functions. Therefore, techniques are needed to improve the efficiency of DNNs. [Brief explanation of the drawing]

[0004] Embodiments will be readily understood by the following detailed description in relation to the accompanying drawings. For the sake of this description, similar reference numerals indicate similar structural elements. Embodiments are shown in the drawings of the accompanying drawings as examples, not as limitations.

[0005] [Figure 1] Exemplary DNNs in various embodiments are shown.

[0006] [Figure 2] Exemplary convolutions are shown in various embodiments.

[0007] [Figure 3] Block diagrams of DNN accelerators according to various embodiments.

[0008] [Figure 4] Exemplary activation function units according to various embodiments are shown.

[0009] [Figure 5] Exemplary operations of activation function units without stall cycles according to various embodiments are shown.

[0010] [Figure 6] Calculations of activation functions based on the same accuracy according to various embodiments are shown.

[0011] [Figure 7] Exemplary operations of activation function units with stall cycles according to various embodiments are shown.

[0012] [Figure 8] Calculations of activation functions based on different accuracies according to various embodiments are shown.

[0013] [Figure 9] An array of processing elements (PEs) according to various embodiments is shown.

[0014] [Figure 10] Block diagrams of PEs according to various embodiments.

[0015] [Figure 11] A flowchart showing a method for calculating an activation function in a neural network according to various embodiments.

[0016] [Figure 12]This is a block diagram of an exemplary computing device in various embodiments. [Modes for carrying out the invention]

[0017] Overview Over the past decade, we have seen a rapid increase in AI-based data processing, particularly DNN-based data processing. DNNs are widely used in the fields of computer vision, speech recognition, image processing, and video processing, primarily due to their ability to achieve accuracy exceeding human levels. Significant improvements in DNN model size and accuracy, coupled with the rapid increase in computing power of execution platforms, have led to the adoption of DNN applications even within resource-constrained mobile and edge devices with limited energy availability.

[0018] A DNN layer may include one or more deep learning operations, such as convolution, pooling, element-wise operations, linear operations, and nonlinear operations. Deep learning operations in a DNN may be performed on one or more internal parameters of the DNN (e.g., weights) and one or more activations, which are determined during the training phase. Activations may be data points (also called "data elements" or "elements"). Activations or weights in a DNN layer may be elements of the DNN layer's tensor. A tensor is a data structure having multiple elements across one or more dimensions. Exemplary tensors include vectors, which are one-dimensional tensors, and matrices, which are two-dimensional tensors. Three-dimensional tensors and even higher-dimensional tensors are also possible. A DNN layer may have an input tensor (also called an "input feature map, IFM") containing one or more input activations (also called "input elements"), and a weight tensor containing one or more weights. Weights are elements in the weight tensor. The weight tensor for a convolution may be a kernel, a filter, or a group of filters. The output data of a DNN layer may be an output tensor (also called an "output feature map (OFM)") that contains one or more output activations (also called "output elements").

[0019] The activation function is a crucial part of a DNN. By calculating a weighted sum activation and adding a bias, the activation function can determine whether a neuron should be activated or not. A key purpose of the activation function is to introduce nonlinearity into the neuron's output. Given the complexity of some of the nonlinear activation functions used in many DNNs, hardware implementations may require approximations within a certain level of accuracy.

[0020] Currently available activation function implementations are based on either an activation function lookup table (LUT) or a DSP (Digital Signal Processor) core. LUTs may use piecewise linear approximation (PWL). PWL is based on approximating complex nonlinear curves using several linear segments. While a PWL-based LUT approach can simplify the complex calculations required for some nonlinear functions by using the LUT, address generation logic, and simple arithmetic blocks (e.g., subtractors, multipliers, and adders) for approximation, improving the accuracy of a PWL-based LUT approach typically requires more linear segments, and therefore more entries within the LUT. Since nonlinear approximation is part of the core DNN logic, allocating additional area on the DNN accelerator die is not always feasible. In such scenarios, it may be beneficial to trade additional accuracy at the expense of performance without adding additional area.

[0021] DSP cores typically require a kernel implementation of the activation function. DSP-based implementations typically require offloading tasks from the neural network processor onto the DSP core, which can add additional overhead such as handshakes and inter-module communication. Furthermore, given the nature of DSP cores, they can run at lower clock frequencies compared to the neural network processor. These limitations add to the efficiency of the computational activation function within the DNN accelerator.

[0022] Embodiments of this disclosure provide a DNN accelerator having an activation function unit capable of computing an approximation of an activation function. The approximation of the activation function may be a Taylor series. An exemplary DNN accelerator in this disclosure comprises one or more computation blocks. A computation block may also be called a computation tile. Each computation block may be a processing unit. A computation block comprises memory, a PE array, and an activation function unit. Memory may store data received or generated by the computation block. The PE array may perform deep learning operations such as convolution, element-wise operations, and pooling operations. The activation function unit may receive output activations computed by the PE array and apply an activation function to the output activations. The output of the activation function unit may be used as input to a deep learning operation performed by the PE array. The output of the activation function unit may be stored in memory. The activation function may be on a drain path from the PE array to memory.

[0023] In various embodiments of this disclosure, an activation function unit may include multiple computation elements. The computation elements may operate in parallel. Different activations may be input to different computation elements in different clock cycles. The computation elements may compute a polynomial (e.g., a Taylor series polynomial) that approximates the activation function. The degree of the polynomial or the number of terms in the polynomial may be determined based on a predetermined precision of the activation function output. Higher precision may require more terms in the polynomial, and may require more computation elements or more clock cycles.

[0024] An exemplary computational element may include two multipliers and an accumulator. The first multiplier may compute an intermediate product using an activation, such as the output activation of a DNN layer. The first multiplier may compute intermediate products for different terms in different clock cycles. For example, the first multiplier may compute a squared activation in the first clock cycle and a cubed activation in the second clock cycle, and so on. The second multiplier may compute a polynomial term based on the intermediate product from the first multiplier and the coefficients of a Taylor series. In an exemplary clock cycle, the second multiplier may multiply the intermediate product computed by the first multiplier in the previous clock cycle by the coefficients. The second multiplier may send the result of the term to an accumulator. The accumulator may compute a partial sum of the terms as the output of the activation function.

[0025] This disclosure provides a technique for computing activation functions based on Taylor series. This technique allows for a programmable precision-performance trade-off. The precision of the activation function can be adjusted by controlling the number of Taylor series terms computed. The precision of the resulting activation function can be improved by adding additional computation cycles. High precision can be achieved with minimal performance penalties, such as a performance penalty of a very small number of clock cycles (e.g., one or two). Thus, this disclosure provides a technique for computing activation functions that is more advantageous than currently available techniques.

[0026] For explanatory purposes, certain numbers, materials, and configurations are described to provide a full understanding of the exemplary implementation. However, it will be apparent to those skilled in the art that this disclosure may be implemented without such specific details, or / or using only some of the aspects described. On the other hand, well-known features have been omitted or simplified so as not to overshadow the exemplary implementation.

[0027] Furthermore, reference is made to the accompanying drawings, which form part of this application. The drawings illustrate possible embodiments. Other embodiments may be used, and structural or logical modifications may be made without departing from the scope of this disclosure. Therefore, the following detailed description should not be construed as limiting.

[0028] Various actions may be described sequentially as multiple separate actions or operations in the manner that is most helpful in understanding the claimed subject matter. However, the order of description should not be interpreted as meaning that these actions are necessarily order-dependent. In particular, these actions do not have to be performed in the order presented. The described actions may be performed in a different order than in the described embodiments. Various additional actions may be performed, or described actions may be omitted in additional embodiments.

[0029] For the purposes of this disclosure, the phrase "A or B" or "A and / or B" means (A), (B), or (A and B). For the purposes of this disclosure, the phrase "A, B, or C" or "A, B and / or C" means (A), (B), (C), (A and B), (A and C), (B and C), or (A, B, and C). When used in relation to measurement ranges, the term "between..." includes both ends of the measurement range.

[0030] The descriptions use phrases such as “in one embodiment” or “in one embodiment,” which may refer to one or more of the same or different embodiments, respectively. Terms such as “equipped,” “included,” and “having” as used in reference to embodiments of this disclosure are synonymous. This disclosure may use descriptions based on perspective views, such as “above,” “below,” “top,” “bottom,” and “side,” to describe various features of the drawings, but these terms are for the sake of discussion only and do not imply any desired or required orientation. The accompanying drawings are not necessarily drawn to scale. Unless otherwise specified, the use of ordinal adjectives such as “first,” “second,” and “third” to describe common subjects simply indicates that different instances of similar subjects are being referred to, and is not intended to imply that the subjects described in this way must be in a given order, temporally, spatially, in rank, or in any other way.

[0031] In the following detailed description, various aspects of the exemplary implementation will be described using terminology commonly used by those skilled in the art to communicate their work to others skilled in the art.

[0032] The terms “substantially,” “near,” “almost,” “near,” and “about” generally refer to being within ±20% of the target value, as described herein or known in the art. Similarly, terms indicating the orientation of various elements, such as “coplanar,” “perpendicular,” “orthogonal,” “parallel,” or any other angle between elements, generally refer to being within ±5 to 20% of the target value, as described herein or known in the art.

[0033] Furthermore, the terms “equipped,” “containing,” “included,” “having,” “having,” or any other variation thereof are intended to encompass non-exclusive inclusion. For example, a method, process, device, or DNN accelerator that includes a set of elements is not necessarily limited to those elements alone, and may include other elements not explicitly listed or inherent in such a method, process, device, or DNN accelerator. Also, the term “or” refers to an inclusive “or” and not an exclusive “or.”

[0034] The systems, methods, and devices of this disclosure each have several inventive aspects, and no single aspect alone accounts for all the desirable attributes disclosed herein. Details of one or more implementations of the subject matter described herein are given in the following description and accompanying drawings.

[0035] Exemplary DNN Figure 1 shows exemplary DNN 100 in various embodiments. For illustrative purposes, DNN 100 in Figure 1 is a CNN. In other embodiments, DNN 100 may be other types of DNNs. DNN 100 is trained to receive an image and output the classification of objects in the image. In the embodiment of Figure 1, DNN 100 receives an input image 105 containing objects 115, 125, and 135. DNN 100 includes a set of layers including a plurality of convolutional layers 110 (each referred to as “convolutional layer 110”), a plurality of pooling layers 120 (each referred to as “pooling layer 120”), and a plurality of fully connected layers 130 (each referred to as “fully connected layer 130”). In other embodiments, DNN 100 may include fewer, more, or different layers. In DNN 100 inference, the layers of DNN 100 perform tensor computations that include many tensor operations, such as convolution (e.g., multiply-accumulate, MAC), pooling, element-wise operations (e.g., element-wise addition, element-wise multiplication), other types of tensor operations, or any combination thereof.

[0036] The convolutional layer 110 summarizes the presence of features in the input image 105. The convolutional layer 110 functions as a feature extractor. The first layer of the DNN 100 is the convolutional layer 110. In one example, the convolutional layer 110 performs convolution on the input tensor 140 (also called IFM 140) and the filter 150. As shown in Figure 1, the IFM 140 is represented by a 7x7x3 three-dimensional (3D) matrix. The IFM 140 contains three input channels, each input channel represented by a 7x7 two-dimensional (2D) matrix. The 7x7 2D matrix contains seven input elements (also called input points) in each row and seven input elements in each column. The filter 150 is represented by a 3x3x3 three-dimensional matrix. The filter 150 contains three kernels, each of which may correspond to a different input channel of the IFM 140. A kernel is a 2D matrix of weights, where the weights are arranged in columns and rows. A kernel can be smaller than an IFM. In the embodiment of Figure 1, each kernel is represented by a 3x3 2D matrix. The 3x3 kernel contains three weights in each row and three weights in each column. The weights can be initialized and updated by backpropagation using gradient descent. The magnitude of the weights can indicate the importance of the filter 150 in extracting features from the IFM 140.

[0037] The convolution involves a MAC operation between the input elements in IFM 140 and the weights in filter 150. The convolution may be a standard convolution 163 or a depth-dependent convolution 183. In a standard convolution 163, the entire filter 150 slides across IFM 140. All input channels are combined to produce an output tensor 160 (also called an output feature map (OFM) 160). The OFM 160 is represented by a 5x5 2D matrix. The 5x5 2D matrix contains five output elements (also called output points) in each row and five output elements in each column. For illustrative purposes, the standard convolution includes one filter in the embodiment shown in Figure 1. In embodiments with multiple filters, the standard convolution may produce multiple output channels in OFM 160.

[0038] The multiplication applied between a kernel-sized patch and a kernel in IFM 140 may be a dot product. The dot product is an element-wise multiplication between a kernel-sized patch and the corresponding kernel in IFM 140, which is then summed, always resulting in a single value. Because it gives a single value, this operation is often called a "scalar product." Using a kernel smaller than IFM 140 is intentional, as it allows the same kernel (set of weights) to be multiplied by IFM 140 multiple times at different points on IFM 140. Specifically, the kernel is applied systematically from left to right and top to bottom to each overlapping portion or kernel-sized patch of IFM 140. The result of multiplying a kernel by IFM 141 once is a single value. When a kernel is applied to IFM 140 multiple times, the result of the multiplication is a 2D matrix of output elements. Thus, the 2D output matrix from a standard convolution (i.e., OFM 160) is called OFM.

[0039] In the depth-based convolution 183, the input channels are not combined. Rather, MAC operations are performed on each individual input channel and each individual kernel to generate output channels. As shown in Figure 1, the depth-based convolution 183 generates the depth-based output tensor 180. The depth-based output tensor 180 is represented by a 5x5x3 3D ​​matrix. The depth-based output tensor 180 contains three output channels, each represented by a 5x5 2D matrix. The 5x5 2D matrix contains five output elements in each row and five output elements in each column. Each output channel is the result of MAC operations on the input channels of IFM 140 and the kernels of filter 150. For example, the first output channel (patterned with dots) is the result of a MAC operation between the first input channel (patterned with dots) and the first kernel (patterned with dots), the second output channel (patterned with horizontal lines) is the result of a MAC operation between the second input channel (patterned with horizontal lines) and the second kernel (patterned with horizontal lines), and the third output channel (patterned with diagonal stripes) is the result of a MAC operation between the third input channel (patterned with diagonal stripes) and the third kernel (patterned with diagonal stripes). In such depth-based convolution, the number of input channels is equal to the number of output channels, and each output channel corresponds to a different input channel. The input and output channels are collectively called depth-based channels. After the depth-based convolution, a point-based convolution 193 is performed on the depth-based output tensor 180 and the 1x1x3 tensor 190 to produce OFM 160.

[0040] OFM 160 is then passed to the next layer in the sequence. In some embodiments, OFM 160 is passed through an activation function. An exemplary activation function is a rectified linear activation function (ReLU). ReLU is a computation that either directly returns the value provided as input, or returns zero if the input is less than or equal to zero. The convolutional layer 110 may receive several images as input and compute the convolution of each of them with each kernel. This process can be repeated several times. For example, OFM 160 is passed to a subsequent convolutional layer 110 (i.e., a convolutional layer 110 following the convolutional layer 110 that generates OFM 160 in the sequence). The subsequent convolutional layer 110 performs a convolution on OFM 160 using the new kernel and generates a new feature map. The new feature map may also be normalized and resized. The new feature map may be kernel-processed again by further subsequent convolutional layers 110, and so on.

[0041] In some embodiments, the convolutional layer 110 has four hyperparameters, namely, the number of kernels, a kernel of size F (for example, the kernel has dimensions of F × F × D pixels), S steps in which the window corresponding to the kernel is dragged on the image (for example, a step of 1 means moving the window by 1 pixel at a time), and zero padding P (for example, adding a black contour of P pixels thickness to the input image of the convolutional layer 110). The convolutional layer 110 can perform various types of convolution, such as 2D convolution, dilation or atlas convolution, spatially separable convolution, depth-separable convolution, and transpose convolution. The DNN 100 includes 16 convolutional layers 110. In other embodiments, the DNN 100 may include a different number of convolutional layers.

[0042] The pooling layer 120 downsamples the feature map generated by the convolutional layer, for example, by summarizing the presence of features within a patch of the feature map. The pooling layer 120 is placed between two convolutional layers 110, namely, a preceding convolutional layer 110 (a convolutional layer 110 that precedes the pooling layer 120 in the layer sequence) and a succeeding convolutional layer 110 (a convolutional layer 110 that follows the pooling layer 120 in the layer sequence). In some embodiments, the pooling layer 120 is added after the convolutional layer 110, for example, after an activation function (e.g., ReLU) has been applied to OFM 160.

[0043] The pooling layer 120 receives the feature map generated by the preceding convolutional layer 110 and applies a pooling operation to the feature map. The pooling operation reduces the size of the feature map while preserving important characteristics. Thus, the pooling operation improves the efficiency of the DNN and avoids overfitting. The pooling layer 120 can perform the pooling operation through mean pooling (calculating the average value for each patch on the feature map), max pooling (calculating the maximum value for each patch on the feature map), or a combination of both. The size of the pooling operation is smaller than the size of the feature map. In various embodiments, the pooling operation is 2x2 pixels applied with a stride of 2 pixels, thereby reducing the size of the feature map by half, for example, the number of pixels or values ​​in the feature map is reduced to a quarter of its original size. In one example, a pooling layer 120 applied to a 6x6 feature map yields a 3x3 output pooled feature map. The output of the pooling layer 120 is input to the subsequent convolutional layer 110 for further feature extraction. In some embodiments, the pooling layer 120 operates separately for each feature map to create a new set of the same number of pooled feature maps.

[0044] The fully connected layer 130 is the last layer of the DNN. The fully connected layer 130 may or may not be a convolutional layer. The fully connected layer 130 accepts an input operand. The input operand defines the outputs of the convolutional layer 110 and the pooling layer 120 and includes the value of the last feature map generated by the last pooling layer 120 in the sequence. The fully connected layer 130 applies a linear combination and activation function to the input operand to generate a vector. The vector can contain as many elements as there are classes, where element i represents the probability that the image belongs to class i. Thus, each element is between 0 and 1, and the sum of all is 1. These probabilities are calculated by the last fully connected layer 130 using the logistic function (binary classification) or the softmax function (multi-class classification) as the activation function.

[0045] In some embodiments, the fully connected layer 130 classifies the input image 105 and returns an operand of size N, where N is the number of classes in the image classification problem. In the embodiment of Figure 1, N is equal to 3 because there are three objects 115, 125, and 135 in the input image. Each element of the operand represents the probability that the input image 105 belongs to a class. To calculate the probabilities, the fully connected layer 130 multiplies each input element by a weight, sums them up, and then applies an activation function (e.g., logistic for N=2, softmax for N>2). This is equivalent to multiplying the input operands by a matrix containing the weights. In one example, the vector contains three probabilities: a first probability that object 115 is a tree, a second probability that object 125 is a car, and a third probability that object 135 is a person. In other embodiments, where the input image 105 contains different objects or different numbers of objects, the individual values ​​may differ.

[0046] Exemplary convolution Figure 2 shows an exemplary convolution according to various embodiments. The convolution may be a convolution in a convolutional layer of a DNN, such as the convolutional layer 110 in FIG. 1. The convolution can be performed on an input tensor 210 and filters 220 (individually referred to as "filter 220"). The result of the convolution is an output tensor 230. In some embodiments, the convolution is performed by a DNN accelerator that includes one or more compute tiles. An example of a DNN accelerator may be the DNN accelerator 300 in FIG. 3. An example of a compute tile may be a compute block such as the compute block 330 in FIG. 3.

[0047] In the embodiment of FIG. 2, the input tensor 210 includes activations (also referred to as "input activations", "elements", or "input elements") arranged in a 3D matrix. The input elements are data points in the input tensor 210. The input tensor 210 has a spatial size of H in ×W in ×C in where H in is the height of the 3D matrix (i.e., the length along the Y-axis indicating the number of activations in a column within the 2D matrix for each input channel), W in is the width of the 3D matrix (i.e., the length along the X-axis indicating the number of activations in a row within the 2D matrix for each input channel), and C in is the depth of the 3D matrix (i.e., the length along the Z-axis indicating the number of input channels). For simplicity of illustration, the input tensor 210 has a spatial size of 7×7×3, i.e., the input tensor 210 includes three input channels, and each input channel has a 7×7 2D matrix. Each input element within the input tensor 210 can be represented by (X, Y, Z) coordinates. In other embodiments, the height, width, or depth of the input tensor 210 may be different.

[0048] Each filter 220 includes weights arranged in a 3D matrix. The weight values can be determined through training the DNN. The filter 220 has a spatial size of H f ×W f ×C fIt has H f is the height of the filter (i.e., the length along the Y-axis, which indicates the number of weights in a column within each kernel), and W f is the width of the filter (i.e., the length along the X-axis, which indicates the number of weights in each row within each kernel), and C f is the filter depth (i.e., the length along the Z-axis, which indicates the number of channels). In some embodiments, C f is C in It is equal to . For simplicity and illustration, each filter 220 in Figure 2 has a spatial size of 3 × 3 × 3, that is, filter 220 contains three convolution kernels with a spatial size of 3 × 3. In other embodiments, the height, width, or depth of filter 220 may be different. The spatial size of the convolution kernel is smaller than the spatial size of the 2D matrix of each input channel in the input tensor 210.

[0049] Activation or weights can occupy one or more bytes in memory. The number of bytes for activation or weights may depend on the data format. For example, if activation or weights have an INT8 format, activation takes up one byte. If activation or weights have an FP16 format, activation or weight takes up two bytes. Other data formats may be used for activation or weights.

[0050] In the convolution, each filter 220 slides across the input tensor 210 to generate a 2D matrix for the output channels in the output tensor 230. In the embodiment of Figure 2, the 2D matrix has a spatial size of 5 × 5. The output tensor 230 contains activations (also called “output activations,” “elements,” or “output elements”) arranged in a 3D matrix. Output activations are data points in the output tensor 230. The output tensor 230 has a spatial size H out ×W out ×C out It has H out is the height of the 3D matrix (i.e., the length along the Y axis, which indicates the number of output activations in the column within the 2D matrix for each output channel), and Wout is the width of the 3D matrix (i.e., the length along the X-axis, which indicates the number of output activations in a row within the 2D matrix for each output channel), and C out This is the depth of the 3D matrix (i.e., the length along the Z-axis, which indicates the number of output channels). out This may be equal to the number of filters 220 in the convolution. out and W out This may depend on the input tensor 210 and the heights and weights of each filter 220.

[0051] As part of the convolution, MAC operations can be performed on the input tensor 210 and the 3x3x3 subtensors 215 within each filter 220 (highlighted with a dot pattern in Figure 2). The result of the MAC operations on the subtensors 215 and one filter 220 is the output activation. In some embodiments (for example, embodiments where the convolution is an integer convolution), the output activation may contain 8 bits, e.g., 1 byte. In other embodiments (for example, embodiments where the convolution is a floating-point convolution), the output activation may contain 2 or more bytes. For example, the output element may contain 2 bytes. After the MAC operations on the subtensors 215 and all filters 220 are completed, a vector 235 is generated. Vector 235 is highlighted with diagonal lines in Figure 2. Vector 235 contains a sequence of output activations arranged along the Z axis. Output activations in vector 235 have the same (x,y) coordinates, but the output activations correspond to different output channels and have different Z coordinates. The dimension of vector 235 along the Z-axis may be equal to the total number of output channels in output tensor 230.

[0052] After vector 235 is generated, further MAC operations are performed to generate additional vectors, and this continues until output tensor 230 is generated.

[0053] In some embodiments, MAC operations on a 3×3×3 subtensor (e.g., subtensor 215) and filter 220 can be performed by multiple PEs. One or more PEs may receive input operands (e.g., input operand 217 shown in Figure 2) and weight operands (e.g., weight operand 227 shown in Figure 2). Input operand 217 includes a sequence of activations having the same (x,y) coordinates but different z coordinates. Input operand 217 includes activations from each input channel in input tensor 210. Weight operand 227 includes a sequence of weights having the same (x,y) coordinates but different z coordinates. Weight operand 227 includes weights from each channel in filter 220. The activations in input operand 217 and the weights in weight operand 227 may be supplied to the PEs sequentially. The PEs may receive activation-weight pairs at a time and multiply the activations by the weights. The location of the activation in input operand 217 may coincide with the location of the weight in weight operand 227. The activation and weight may correspond to the same channel.

[0054] The activation or weight may be a floating-point number. Floating-point numbers can have various data formats, such as FP32, FP16, and BF16. A floating-point number can be a positive or negative number with a decimal point. A floating-point number can be represented by a sequence of bits that includes one or more bits representing the sign of the floating-point number (e.g., positive or negative), bits representing the exponent of the floating-point number, and bits representing the mantissa of the floating-point number. The mantissa is part of the floating-point number that represents the significant figures of the floating-point number. The mantissa is multiplied by the exponent of the base to give the actual value of the floating-point number.

[0055] In some embodiments, the output activations in the output tensor 230 may be further processed based on one or more activation functions before being stored or input to the next layer of the DNN. The processing based on one or more activation functions may be at least part of the post-processing of the convolution. In some embodiments, the post-processing may include one or more other calculations such as offset calculation and bias calculation. The results of the post-processing may be stored in the local memory of the calculation block and used as input to the next DNN layer. In some embodiments, the input activations in the input tensor 210 may be the result of post-processing of the previous DNN layer.

[0056] The calculation of the activation function may be based on a Taylor series. In some embodiments, a Taylor series is used to approximate the activation function. A Taylor series can be an infinite sum of terms that are expressed as terms of the derivative of the function at a single point. The activation function f(x) approximated by the Taylor series is

number

[0057] The activation can be approximated using the first t terms of the Taylor series. The precision of the activation function can be adjusted by changing t. As t increases, the precision of the approximated activation function increases. The precision of the activation function and t may be predetermined, for example, determined before the activation function is computed. The first t terms can constitute a polynomial of the Taylor series, also called the Taylor polynomial. The degree of the polynomial may be equal to t-1. An exemplary Taylor polynomial with degree 6 and containing the first seven terms of the Taylor series can be expressed as follows:

number

[0058] Exemplary activation functions that can be approximated by Taylor series include ReLU, Tanh activation function, Gaussian error linear unit (GELU), sigmoid activation function, and sigmoid linear unit (SiLU). For example, the Taylor series expansion of the Tanh activation function can be expressed as follows:

number

number

number

[0059] Exemplary DNN accelerator Figure 3 is a block diagram of an exemplary DNN accelerator 300 in various embodiments. The DNN accelerator 300 can perform deep learning operations and activation functions in a DNN, for example, DNN 100 in Figure 1. As shown in Figure 3, the DNN accelerator 300 includes memory 310, a DMA (Direct Memory Access) engine 320, and a compute block 330 (each referred to as “Compute Block 330”). In other embodiments, alternative configurations, different or additional components may be included in the DNN accelerator 300. For example, the DNN accelerator 300 may include two or more memories 310 or DMA engines 320. As another example, the DNN accelerator 300 may include a single compute block 330. Furthermore, functions attributed to the components of the DNN accelerator 300 may be achieved by different components included in the DNN accelerator 300 or by different systems. The components of the DNN accelerator 300 may be implemented as hardware, software, firmware, or any combination thereof.

[0060] The DNN accelerator 300 is associated with a pre-computation module 305 in the embodiment shown in Figure 3. In other embodiments, functions attributed to components of the pre-computation module 305 may be achieved by the DNN accelerator 300 or by a different system. The pre-computation module 305 may compute variables used by the DNN accelerator 300 for DNN inference. This computation is called pre-computation because it is performed before DNN inference. The pre-computation module 305 can perform pre-computation for activation functions in the DNN, such as activation functions approximated by Taylor series. For example, the pre-computation module 305 may determine the number of terms in the Taylor series computed by the DNN accelerator 300 during DNN inference. The pre-computation module 305 may determine the number of terms (t) to be computed based on a predetermined precision of the corresponding activation function. The precision of the corresponding activation function may be determined based on the target precision of the DNN or other factors.

[0061] In some embodiments, the pre-calculation module 305 can obtain data showing the correlation between t and the accuracy of the activation function. In one example, the pre-calculation module 305 may obtain a curve showing how the accuracy of the activation function changes as t changes. The pre-calculation module 305 may determine a value of t based on that curve and a predetermined accuracy of the activation function.

[0062] In some embodiments, the pre-calculation module 305 may also pre-calculate the coefficients of the Taylor series before the DNN accelerator 300 computes the first t term of the Taylor series. In some embodiments, the coefficients are

number

[0063] Memory 310 stores data related to deep learning operations (including activation functions) performed by the DNN accelerator. In some embodiments, memory 310 may store data used by the compute block 330 for DNN inference. For example, memory 310 can store data computed by the pre-computation module 305, such as the coefficients of a Taylor series. As another example, memory 310 may store weights, such as the weights of convolutional layers determined by training the DNN. Memory 310 can also store data generated by the compute block 330 from performing deep learning operations in the DNN. Exemplary deep learning operations include convolution (also called “convolution operations”), pooling operations, element-wise operations, activation functions, other types of deep learning operations, or any combination thereof. Memory 310 may be the main memory of the DNN accelerator 300. In some embodiments, memory 310 includes one or more DRAMs (Dynamic Random Access Memory).

[0064] The DMA engine 320 facilitates data transfer between memory 310 and the local memory of the compute block 330. For example, the DMA engine 320 can read data from memory 310 and write data to the local memory of the compute block 330. As another example, the DMA engine 320 can read data from the local memory of the compute block 330 and write data to memory 310. The DMA engine 320 provides DMA functionality that allows the compute block 330 to initiate a data transfer between memory 310 and its local memory and to perform other operations while the data transfer is taking place. In some embodiments, the DMA engine 320 may read a tensor from memory 310 and modify the tensor in a manner optimized for the compute block 330 before writing the tensor to the local memory of the compute block 330.

[0065] The computation block 330 can perform deep learning operations in a DNN. For example, the computation block 330 can perform deep learning operations in a DNN layer, or a portion of deep learning operations, at once. The computation block 330 can perform various types of deep learning operations, such as convolution, pooling, element-wise operations, linear operations, and nonlinear operations. In one example, the computation block 330 can perform a convolution, such as a standard convolution or a depth-wise convolution. In some embodiments, the computation block 330 receives an input tensor and one or more convolution kernels and performs a convolution using the input tensor and convolution kernels. The result of the convolution may be an output tensor, which can be further computed, for example, by the computation block 330 or another computation block 330. In some embodiments, the operations of a DNN layer may be performed in parallel by multiple computation blocks 330. For example, multiple computation blocks 330 may each perform a portion of the workload for the convolution. Data may be shared among the computation blocks 330.

[0066] In the embodiment shown in Figure 3, each compute block 330 includes a local memory 340, a PE array 350, and an activation function unit 360. The local memory 340, PE array 350, and activation function unit 360 may be implemented on the same chip. In other embodiments, alternative configurations, different or additional components may be included in the compute block 330. Furthermore, functions attributed to the components of the compute block 330 may be achieved by different components included in the compute block 330 or the DNN accelerator 300, or by different systems. The components of the compute block 330 may be implemented in hardware, software, firmware, or any combination thereof.

[0067] Local memory 340 is local to the corresponding compute block 330. In the embodiment of Figure 3, local memory 340 is located inside compute block 330. In other embodiments, local memory 340 may be located outside compute block 330. Local memory 340 stores data received, used, or generated by the PE array 350 and activation function unit 360. Examples of data may include input activations, weights, output activations, coefficients of Taylor series, results of activation functions, sparsity bitmaps, etc. Data in local memory 340 may be transferred to or from memory 310, for example, through the DMA engine 320. In some embodiments, data in local memory 340 may be transferred to or from the local memory of another compute block 330.

[0068] In some embodiments, local memory 340 includes one or more static random access memories (SRAM). Local memory 340 may be byte-addressable, with each memory address identifying a single byte (8 bits) of storage. In some embodiments, local memory 340 may include memory banks. The number of data banks in local memory 340 may be 16, 64, 128, 256, 512, 1024, 2048, or other numbers. A memory bank may include multiple storage units. In one example, a data bank may include 8, 16, 64, or a different number of storage units. A memory bank or a storage unit within a memory bank may have a memory address. In one example, a storage unit may store a single byte, and data larger than a single byte may be stored in storage units with contiguous memory addresses, i.e., adjacent storage units. For example, one storage unit may store an integer in INT8 format, whereas two storage units may be required to store a number in FP16 or BF16 format, which have 16 bits. In some embodiments, 16 bits may be transferred from local memory 340 in a single read cycle. In other embodiments, 16 bits may be transferred from local memory 340 in multiple read cycles, such as two cycles.

[0069] The PE array 350 may include PEs arranged in columns, or in columns and rows. Each PE can perform MAC operations. In some embodiments, a PE includes one or more multipliers for performing multiplication. A PE may also include one or more accumulators ("adders") for performing accumulation. A column of PEs is called a PE column. A PE column may be associated with one or more MAC lanes. A MAC lane is a path for loading data into a MAC column. A MAC lane may also be called a data transmission lane or data load lane. A PE column may have multiple MAC lanes. The load bandwidth of a MAC column is the sum of the load bandwidths of all MAC lanes associated with the MAC column. Using a certain number of MAC lanes, data may be fed simultaneously to the same number of independent PEs. In some embodiments, where a MAC column has four MAC lanes for feeding activation or weights into the MAC column, and each MAC lane may have a bandwidth of 16 bytes, the four MAC lanes may have a total load bandwidth of 64 bytes.

[0070] In some embodiments, the PE array 350 may be capable of depth-based convolution, standard convolution, or both. In depth-based convolution, the PE may perform a MAC operation which includes a sequence of multiplications for the input operands and weight operands. Each multiplication in the sequence (also called a cycle) is a multiplication of a different activation in the input operand with a different weight in the weight operand. The activations and weights in the same cycle may correspond to the same channel. The sequence of multiplications produces a product operand which includes a sequence of products. The MAC operation may also include a cumulative operation in which multiple product operands are accumulated to produce an output operand of the PE. The PE array 350 may output multiple output operands at once, each of which is produced by a different PE. In standard convolution, the MAC operation may include a cumulative operation across channels. For example, the PE may accumulate products across different channels to produce a single output point, as opposed to producing an output operand.

[0071] In some embodiments, the PE array 350 may perform MAC operations in quantized inference, such as MAC operations in quantized convolution. In some embodiments, the PEs within the PE array 350 may receive quantized activations and quantized weights and compute a quantized MAC result. The quantized MAC result may be a quantized value in integer format and may be the output of the PE. In some embodiments, the PE may also include a quantization multiplier that can multiply the quantized MAC result by a quantization scale, and the output of the PE may be a real number in floating-point format. The PE may not include a quantization subtractor because zero-point offsets are not required for MAC operations in quantized inference.

[0072] The activation function unit 360 computes an activation function. The activation function unit 360 may receive the output of the PE array 350 as input to the activation function. The activation function unit 360 may send the output of the activation function to local memory 340. The output of the activation function may be retrieved later by the PE array 350 from local memory 340 for further computation. For example, the activation function unit 360 may receive the output tensor of a DNN layer from the PE array 350 and compute one or more activation functions on the output tensor. The results of the computation by the activation function unit 360 are stored in local memory 340 and may later be used as the input tensor for the next DNN layer. In some embodiments, local memory 340 may be associated with a load path, and a drain path may be used for data transfer within the computation block 330. For example, data may be transferred from local memory 340 to the PE array 350 via the load path. Data may be transferred from the PE array 350 to local memory 340 via the drain path. The activation function unit 360 (and optionally one or more other post-processing units) may be placed on the drain path to process the output of the PE array before the data is written to local memory 340.

[0073] In some embodiments, the activation function unit 360 may compute an activation function approximated by a Taylor series based on pre-computed data generated, for example, by the pre-computation module 305. For example, the activation function unit 360 may compute one or more terms in a Taylor series by multiplying one or more coefficients of the Taylor series by one or more powers of the input to the activation function unit 360. The activation function unit 360 may include one or more multipliers that can multiply coefficients by powers of the input. The input may be an activation computed by the PE array 350, such as an output activation of a convolution. The activation function unit 360 may further compute a partial sum of one or more terms as the output of the activation function. The output of the activation function unit 360 may be written to local memory 340. In some embodiments, the output of the activation function unit 360 may be read from local memory 340 for future computations by the PE array 350. Certain aspects of the activation function unit 360 are described below in relation to Figures 4 to 8.

[0074] Exemplary activation function unit Figure 4 shows exemplary activation function units 400 in various embodiments. The activation function unit 400 may be an embodiment of the activation function unit 360 in Figure 3. As shown in Figure 4, the activation function unit 400 includes a plurality of computational elements 410A to 410E (collectively referred to as “computational elements 410” or “computational elements 410”). While Figure 4 shows five computational elements 410, the activation function unit 400 may include a different number of computational elements 410 in other embodiments.

[0075] The computation elements 410 may operate in parallel. The operation of the computation elements 410 may be independent of each other. In some embodiments, the computation elements 410 receive different activations from the PE array 350. In some embodiments, each computation element 410 may receive different activations in the output tensor computed by the PE array 350. In other embodiments, two or more of the computation elements 410 may process the same activation. In some embodiments, different computation elements 410 may receive activations at different times, for example, in different clock cycles.

[0076] The computing element 410 includes memory 420, multipliers 430 and 440, accumulator 450, and register 460. In other embodiments, the computing element 410 may include different, fewer, or more components. Furthermore, functions attributed to the components of the computing element 410 may be achieved by different components included in the computing element 410, or by different systems or devices.

[0077] Memory 420 stores the coefficients of a Taylor series, which may be from the pre-calculation module 305. Although memory 420 is located within the computation element 410 in Figure 4, memory 420 may not be included in the computation element 410, or even in the activation function unit 400. In some embodiments, memory 420 may be memory 310 or local memory 340.

[0078] The multiplier 430 may receive activations from the PE array 350. In some embodiments, the multiplier 430 may compute a single activation at a time. The multiplier 430 may compute one or more powers of activations. The multiplier 430 may compute activation powers sequentially. In one example, the multiplier 430 may compute activation powers in a computation cycle, for example, a clock cycle. The exponent of the activation power computed in one cycle may be higher than the activation power computed in the previous cycle, for example, by 1.

[0079] The multiplier 440 multiplies the output of the multiplier 430 by the coefficients of the Taylor series from memory 420. In one example, the multiplier 440 multiplies the activation power by the corresponding coefficient of the Taylor series within a cycle. The output of the multiplier 440 in that cycle may be the result of a term in the Taylor series. In some embodiments, the multiplier 440 may compute a term within a cycle.

[0080] The accumulator 450 receives the output of the multiplier 440 and generates partial sums of the terms of the Taylor series as a result of the activation function. In some embodiments, the accumulator 450 may receive the output of the multiplier 440 in a cycle. In this cycle, the accumulator 450 may accumulate the output of the multiplier 440 with the sum calculated by the accumulator 450 in the previous cycle. The sum from the previous cycle may be stored in register 460. The new sum may also be stored in register 460 and further accumulated with the output of the multiplier 440 in the next cycle. This process can continue until partial sums of all the required terms of the Taylor series have been calculated.

[0081] Figure 5 shows exemplary operation of the activation function unit 400 without stall cycles in various embodiments. The operation of the activation function unit 400 includes 12 clock cycles 0 to 11. In these 12 clock cycles, the activation function unit 400 processes 6 activations and calculates 6 results of the activation function. Figure 5 shows two tables 510 and 520. Table 510 shows the inputs to the activation function unit 400 in 6 clock cycles 0 to 5. Table 520 shows the outputs to the activation function unit 400 in 6 clock cycles 6 to 11. In other embodiments, the operation of the activation function unit 400 may include different, fewer, or more cycles.

[0082] Table 510 shows six activations x1 to x6. In some embodiments, these six activations may be within an output operand computed by the PE array 350, for example, by performing a convolution. In the embodiment of Figure 5, the activation function unit 400 receives a new activation in each of the six cycles. Activation x1 is received in cycle 0, activation x2 in cycle 1, activation x3 in cycle 2, activation x4 in cycle 3, activation x5 in cycle 4, and activation x6 in cycle 5. Although not shown in Figure 5, the activation function unit 400 may receive more activations in future cycles. The activations are input to different computation elements 410 of the activation function unit 400. The first five activations x1 to x5 are input to five computation elements 410, respectively. The sixth activation x6 is input to computation element 410A.

[0083] Table 520 shows six outputs y1 to y6, each calculated using the corresponding activation. Output y1 corresponds to activation x1, output y2 to activation x2, output y3 to activation x3, output y4 to activation x4, output y5 to activation x5, and output y6 to activation x6. Outputs y1 to y6 are also generated in different cycles. Output y1 is calculated in cycle 6, output y2 in cycle 7, output y3 in cycle 8, output y4 in cycle 9, output y5 in cycle 10, and output y6 in cycle 11.

[0084] Figure 6 shows the calculation of an activation function based on the same precision in various embodiments. This calculation is performed by the activation function unit 400. Figure 6 is a table listing the calculations performed in the multipliers 430 and 440 and the accumulator 450 within each calculation element 410 in the activation function unit 400. In the embodiment of Figure 6, each calculation element 410 calculates the partial sum of the first six terms in the Taylor series. The first six terms constitute a fifth-degree Taylor polynomial. Calculation element 410A receives activation x1 in cycle 0. In each of cycles 0-4, the multiplier 430 calculates an intermediate product which is a power of x1. In the first cycle immediately following the cycle in which the intermediate product is calculated, the multiplier 440 receives the intermediate product from multiplier 430 and multiplies the intermediate product by one of the six coefficients of the Taylor series, represented as c1-c6 in Figure 6. The product calculated by the multiplier 440 is the result of a term in the Taylor series. The accumulator 450 receives the result of the term in the second cycle following the cycle in which the intermediate product is calculated. The accumulator 450 accumulates the result of the term with the output of the accumulator 450 in the first cycle immediately following the cycle in which the intermediate product is calculated. The accumulator 450 outputs the result of the activation function for x1 in cycle 6.

[0085] Computation element 410B receives activation x2 in cycle 1 and outputs the result of the activation function for x2 in cycle 7. Computation element 410C receives activation x3 in cycle 2 and outputs the result of the activation function for x3 in cycle 8. Computation element 410D receives activation x4 in cycle 3 and outputs the result of the activation function for x4 in cycle 9. Computation element 410E receives activation x5 in cycle 4 and outputs the result of the activation function for x5 in cycle 10.

[0086] The computation element 410 may begin processing a new activation in the cycle immediately following the calculation of the last intermediate product for the activation by the multiplier 430. As shown in Figure 6, computation element 410A receives activation x6 in cycle 5, even though the multiplier 440 and accumulator 450 are still processing activation x1. Computation element 410A outputs the result of the activation function for x6 in cycle 11. Similarly, computation element 410B receives activation x7 in cycle 6 and outputs the result of the activation function for x7 in cycle 12. Although not shown in Figure 6, the activation function unit 400 may process more activations.

[0087] In the embodiment shown in Figure 6, the seven results of the activation function have the same precision, and the same number of terms are calculated for all seven activations. In other embodiments, different activations may be processed by the activation function unit 400 based on different precisions.

[0088] Figure 7 shows exemplary operation of an activation function unit 400 with a stall cycle in various embodiments. In the operation shown in Figure 7, the activation function unit 400 processes five activations x1 to x5 and computes five results y1 to y5 of the activation function. The operation of the activation function unit 400 includes 12 clock cycles 0 to 11, but cycle 2 is a stall cycle because the activation function unit 400 does not receive any new activations and does not produce an output in cycle 7. The presence of the stall cycle may be to facilitate a higher precision of result y2 than other results. Higher precision requires computing more terms in the Taylor series, and therefore requires more clock cycles to compute result y2. Further details regarding stalling clock cycles to achieve higher precision will be discussed later in relation to Figure 8.

[0089] Figure 7 shows two tables 710 and 720. Table 710 shows six clock cycles 0 to 6, during which the activation function unit 400 receives five activations x1 to x5. The activation function unit 400 receives different activations in cycles 0, 1, and 3 to 5, respectively. Table 720 shows six clock cycles 6 to 12, during which the activation function unit 400 outputs five results y1 to y5. The activation function unit 400 outputs different results in cycles 6 and 8 to 11, respectively. In other embodiments, the operation of the activation function unit 400 may include different, fewer, or more cycles.

[0090] In some embodiments, the five activations x1 to x5 may be within the output operands computed by the PE array 350, for example, by performing a convolution. The activations may be input to different computation elements 410 of the activation function unit 400. The first five activations x1 to x5 are each input to the five computation elements 410. Each of the results y1 to y6 is computed by using the corresponding activation. Output y1 corresponds to activation x1, output y2 corresponds to activation x2, output y3 corresponds to activation x3, output y4 corresponds to activation x4, and output y5 corresponds to activation x5.

[0091] Figure 8 shows the calculation of activation functions based on different accuracies in various embodiments. This calculation is performed by the activation function unit 400. Figure 8 is a table listing the calculations performed in the multipliers 430 and 440 and the accumulator 450 within each calculation element 410 in the activation function unit 400. In the embodiment of Figure 8, each calculation element 410 calculates the partial sum of the first six terms in the Taylor series. The first six terms constitute a fifth-degree Taylor polynomial. Calculation element 410A receives activation x1 in cycle 0. In each of cycles 0-4, the multiplier 430 calculates an intermediate product which is a power of x1. In the first cycle immediately following the cycle in which the intermediate product is calculated, the multiplier 440 receives the intermediate product from multiplier 430 and multiplies it by one of the six coefficients of the Taylor series, represented as c1-c6 in Figure 8. The product calculated by multiplier 440 is the result of a term in the Taylor series. The accumulator 450 receives the result of the term in the second cycle following the cycle in which the intermediate product is calculated. The accumulator 450 accumulates the result of the term with the output of the accumulator 450 in the first cycle immediately following the cycle in which the intermediate product is calculated. The accumulator 450 outputs the result of the activation function for x1 in cycle 6.

[0092] Computation elements 410B and 410C process activation x2, which requires higher precision, and therefore more terms in the Taylor series need to be computed. In the embodiment shown in Figure 8, the first 12 terms in the Taylor series are computed for activation x2. The first 12 terms constitute a Taylor polynomial of degree 11. Computation element 410B receives activation x2 in cycle 1 and computes some of the 12 terms. Computation element 410C receives activation x2 in cycle 2 and computes the remaining terms of the 12 terms. In cycle 8, computation element 410C outputs the result of the activation function for x2.

[0093] The other activations x3, x4, and x5 have the same precision as activation x1 and are therefore each computed by a single computation element 410, just like activation x1. Computation element 410D receives activation x3 in cycle 3 and outputs the result of the activation function for x3 in cycle 9. Computation element 410E receives activation x4 in cycle 5 and outputs the result of the activation function for x4 in cycle 10.

[0094] Computation element 410 may begin processing a new activation in the cycle immediately following the calculation of the final intermediate product for activation by multiplier 430. As shown in Figure 8, computation element 410A receives activation x5 in cycle 5, even though multiplier 440 and accumulator 450 are still processing activation x1. Computation element 410A outputs the result of the activation function for x5 in cycle 11. Similarly, computation element 410B receives activation x6 in cycle 6 and outputs the result of the activation function for x6 in cycle 12. Computation element 410C receives activation x7 in cycle 7 and outputs the result of the activation function for x7 in cycle 13. Although not shown in Figure 8, activation function unit 400 may process more activations.

[0095] In Figures 7 and 8, the activation function unit 400 processes the same number of activations, but for the additional precision of the results for activation x2, activation x2 is iterated sequentially over two computation cycles, so the operation of the activation function unit 400 in Figure 8 requires one extra cycle.

[0096] In some embodiments, for example, the total number of inputs is n, and the input x i At least k i If an additional term is required for the desired precision, the total number of clock cycles N can be expressed as follows:

number

[0097] Exemplary PE array Figure 9 shows PE arrays according to various embodiments. PE array 900 may be an embodiment of PE array 350 in Figure 3. PE array 900 includes a plurality of PEs 910 (each referred to as "PE 910"). PE 910 can perform MAC operations, including MAC operations in quantized inference. PE 910 may also be called neurons in a DNN. Each PE 910 has two input signals 950 and 960 and an output signal 970. Input signal 950 is at least part of the IFM to the layer. Input signal 960 is at least part of the filter to the layer. In some embodiments, input signal 950 of PE 910 includes one or more input operands, and input signal 960 includes one or more weight operands.

[0098] Each PE 910 performs a MAC operation on input signals 950 and 960 and outputs an output signal 970, which is the result of the MAC operation. Some or all of the input signals 950 and 960 and the output signal 970 may be in integer format such as INT8, or floating-point format such as FP16 or BF16. For simplicity and illustrative purposes, the input and output signals of all PE 910s have the same reference sign, however, PE 910s may receive different input signals from each other and output different output signals from each other. Also, PE 910s may differ from other PE 910s, for example, by containing more, fewer, or different components.

[0099] As shown in Figure 9, the PEs 910 are connected to one another, as indicated by the dashed arrows in Figure 9. The output signal 970 of a PE 910 can be transmitted as an input signal to many other PEs 910 through the interconnection between the PEs 910 (and potentially returned to itself). In some embodiments, the output signal 970 of a PE 910 may incorporate the output signals of one or more other PEs 910 through the cumulative operation of the PE 910 to generate an internal partial sum of the PE array.

[0100] In the embodiment shown in Figure 9, the PEs 910 are located in column 905 (each referred to as “column 905”). The inputs and weights of the layer may be distributed to the PEs 910 based on column 905. Each column 905 has a column buffer 920. The column buffer 920 temporarily stores the data provided to the PEs 910 in column 905. The column buffer 920 may also store the data output by the last PE 910 in column 905. The output of the last PE 910 may be the sum of the MAC operations of all PEs 910 in column 905, which is the column-level internal partial sum of the PE array 900. In other embodiments, the inputs and weights may be distributed to the PEs 910 based on rows in the PE array 900. The PE array 900 may include row buffers instead of column buffers 920. The row buffers may store the input signals of the PEs in the corresponding rows, or they may store the row-level internal partial sums of the PE array 900.

[0101] In some embodiments, the column buffer 920 may be part of the local memory 340 in Figure 3. The column buffer 920 may be associated with a higher memory hierarchy, for example, memory 310 in Figure 3. Data in the column buffer 920 may be sent to the higher memory hierarchy. The column buffer 920 can receive data from the higher memory hierarchy.

[0102] Figure 10 is a block diagram of PE 1000 in various embodiments. PE 1000 may be an embodiment of PE 910 in Figure 9. PE 1000 can perform MAC operations, for example, MAC operations using integer format data. PE 1000 may be an exemplary PE within the PE array 350 described above in relation to Figure 3. As shown in Figure 10, PE 1000 includes an input register file 1010 (individually referred to as "input register file 1010"), a weight register file 1020 (individually referred to as "weight register file 1020"), a multiplier 1030 (individually referred to as "multiplier 1030"), an internal adder assembly 1040, and an output register file 1050. In other embodiments, PE 1000 may include fewer, more, or different components. For example, PE 1000 may include multiple output register files 1050. As another example, PE 1000 may include a single input register file 1010, a weight register file 1020, or a multiplier 1030. As yet another example, PE 1000 may include an adder instead of the internal adder assembly 1040.

[0103] The input register file 1010 temporarily stores the activation operands for MAC operations by PE 1000. In some embodiments, the input register file 1010 may store a single activation operand at a time. In other embodiments, the input register file 1010 may store multiple activation operands or a portion of activation operands at a time. An activation operand contains multiple input elements (i.e., input elements) in an input tensor. The input elements of an activation operand may be stored sequentially in the input register file 1010 so that the input elements can be processed sequentially. In some embodiments, each input element in an activation operand may come from a different input channel of the input tensor. An activation operand may contain input elements from each of the input channels of the input tensor, and the number of input elements in an activation operand may be equal to the number of input channels. The input elements in an activation operand may have the same XY coordinates, which may be used as the XY coordinates of the activation operand. For example, all input elements of the activation operand may be X0Y0, X0Y1, X1Y1, etc.

[0104] The weight register file 1020 temporarily stores weight operands for MAC operations by PE 1000. The weight operands include weights in the DNN layer's filter. In some embodiments, the weight register file 1020 may store a single weight operand at a time. In other embodiments, the input register file 1010 may store multiple weight operands or a portion of weight operands at a time. A weight operand may contain multiple weights. The weights of a weight operand may be stored sequentially in the weight register file 1020, so that the weights can be processed sequentially. In some embodiments, for multiplication operations involving weight operands and activation operands, each weight in the weight operand may correspond to an input element of the activation operand. The number of weights in the weight operand may be equal to the number of input elements in the activation operand.

[0105] In some embodiments, the weight register file 1020 may be the same as or similar to the input register file 1010, for example, having the same size. The PE 1000 may contain multiple register files, some of which are designated as the input register file 1010 for storing activation operands, some as the weight register file 1020 for storing weight operands, and some as the output register file 1050 for storing output operands. In other embodiments, register files within the PE 1000 may be designated for other purposes, for example, for storing scale operands used in element-wise addition operations.

[0106] The multiplier 1030 performs multiplication operations on the activation operand and the weight operand. The multiplier 1030 may perform a sequence of multiplication operations on a single activation operand and a single weight operand to produce a product operand containing a sequence of products. Each multiplication operation in the sequence involves multiplying the input element in the activation operand by the weight in the weight operand. In some embodiments, the position (or index) of the input element in the activation operand coincides with the position (or index) of the weight in the weight operand. For example, the first multiplication operation is the multiplication of the first input element in the activation operand by the first weight in the weight operand, the second multiplication operation is the multiplication of the second input element in the activation operand by the second weight in the weight operand, the third multiplication operation is the multiplication of the third input element in the activation operand by the third weight in the weight operand, and so on. The input elements and weights in the same multiplication operation may correspond to channels of the same depth, and their product may also correspond to channels of the same depth.

[0107] Multiple multipliers 1030 can perform multiplication operations simultaneously. These multiplication operations are sometimes referred to as a round of multiplication. In a round of multiplication by multipliers 1030, each multiplier 1030 may use different activation operands and different weight operands. Different activation operands or weight operands may be stored in different register files of PE 1000. For example, the first multiplier 1030 uses a first activation operand (for example, stored in the first input register file 1010) and a first weight operand (for example, stored in the first weight register file 1020), the second multiplier 1030 uses a second activation operand (for example, stored in the second input register file 1010) and a second weight operand (for example, stored in the second weight register file 1020), the third multiplier 1030 uses a third activation operand (for example, stored in the third input register file 1010) and a third weight operand (for example, stored in the third weight register file 1020), and so on. For each multiplier 1030, a round of multiplication operations may include multiple cycles. Each cycle includes multiplication operations between the input elements and the weights.

[0108] The multiplier 1030 can perform multiplication operations over multiple rounds. The multiplier 1030 may use the same weight operands, but different activation operands, in different rounds. For example, the multiplier 1030 performs a sequence of multiplication operations on a first activation operand stored in a first input register file in the first round, and on a second activation operand stored in a second input register file in the second round. In the second round, a different multiplier 1030 may perform another sequence of multiplication operations using the first activation operand and different weight operands. In this way, the first activation operand is reused in the second round. The first activation operand may be reused further in additional rounds, for example, by an additional multiplier 1030.

[0109] The internal adder assembly 1040 includes one or more adders, i.e., internal adders, within the PE 1000. The internal adder assembly 1040 can perform cumulative operations on two or more product operands from the multiplier 1030 to produce an output operand of the PE 1000. In some embodiments, the internal adders are arranged in a series of hierarchies. Each hierarchy includes one or more internal adders. For the first hierarchy of the internal adder assembly 1040, the internal adder can receive product operands from two or more multipliers 1030 and produce a sum operand through a series of cumulative operations. Each cumulative operation produces the sum of two or more products, each from a different multiplier 1030. The sum operand is the result of each cumulative operation and includes a sequence of sums corresponding to channels at each depth. For other hierarchies (one or more) of the internal adder assembly 1040, the internal adder in one hierarchy receives a sum operand from a preceding hierarchy in the sequence. Each of these numbers may be generated by a different internal adder in a preceding hierarchy. The ratio of the number of internal adders in one hierarchy to the number of internal adders in a subsequent hierarchy may be 2:1. In some embodiments, the final hierarchy of the internal adder assembly 1040 may include a single internal adder that generates the output operand of PE 1000.

[0110] The output register file 1050 stores the output operands of PE 1000. In some embodiments, the output register file 1050 may store an output operand at a time. In other embodiments, the output register file 1050 may store multiple output operands or a portion of output operands at a time. An output operand includes multiple output elements in the IFM. The output elements of an output operand may be stored sequentially in the output register file 1050, and as a result, the output elements may be processed sequentially. In some embodiments, each output element in an output operand corresponds to a channel at a different depth and is an element of a different output channel of the output channels of the convolution at each depth. The number of output elements in an output operand may be equal to the number of channels at each depth of the convolution at each depth.

[0111] An illustrative method for calculating the activation function Figure 11 is a flowchart of method 1100 for calculating an activation function in a neural network according to various embodiments. Method 1100 may be performed by the activation function unit 360 in Figure 3. Although method 1100 is described with reference to the flowchart shown in Figure 11, many other methods for calculating the activation function may be used as alternatives. For example, the order in which the steps in Figure 11 are performed may be changed. As another example, some of the steps may be modified, deleted, or combined.

[0112] The activation function unit 360 receives one or more pre-calculated coefficients of an approximation of the activation function in the neural network (1110). In some embodiments, the approximation is a Taylor series that can approximate the activation function. The one or more pre-calculated coefficients include one or more coefficients of the Taylor series.

[0113] The activation function unit 360 receives the activations computed in the layers of the neural network (1120). In some embodiments, the activations are the output activations of that layer of the neural network.

[0114] The activation function unit 360 uses the activation to calculate one or more intermediate products (1130) by the first multiplier. In some embodiments, the first multiplier calculates a first intermediate product in a first clock cycle. After calculating the first intermediate product, the first multiplier calculates a second intermediate product in a second clock cycle based on the activation and the first intermediate product. In one example, the first intermediate product may be the square of the activation and the second intermediate product may be the cube of the activation.

[0115] The activation function unit 360 calculates one or more terms of the approximation by a second multiplier based on the one or more intermediate products from the first multiplier and the one or more coefficients (1140). In some embodiments, the second multiplier calculates one or more terms of the approximation in a sequence of clock cycles by using different coefficients of the approximation in each clock cycle in the sequence of clock cycles. In some embodiments, the one or more terms of the approximation include one or more Taylor series terms.

[0116] The activation function unit 360 calculates the output of the activation function based on a polynomial containing the one or more of the approximate terms (1150) by an accumulator. The degree of the polynomial is determined based on a predetermined precision of the output of the activation function. In some embodiments, the degree of the polynomial is equal to the number of terms minus 1. In some embodiments, the activation function unit 360 provides the output of the activation function to another layer of the neural network, which is located after the previous layer in the neural network.

[0117] In some embodiments, the activation function unit 360 receives one or more other activations in one or more clock cycles different from the clock cycle in which the activation is received. The one or more other activations are computed in the layer of the neural network. In some embodiments, the activation function unit 360 computes another output of the activation function based on the other activations computed in the layer of the neural network. The other output of the activation function has a predetermined precision different from the output of the activation function.

[0118] Exemplary computing device Figure 12 is a block diagram of exemplary computing device 1200 according to various embodiments. In some embodiments, computing device 1200 can be used as at least part of a DNN accelerator 300. Figure 12 shows several components included in computing device 1200, but any one or more of these components may be omitted or duplicated as appropriate depending on the application. In some embodiments, some or all of the components included in computing device 1200 may be mounted on one or more motherboards. In some embodiments, some or all of these components are manufactured on a single system-on-chip (SoC) die. Furthermore, in various embodiments, computing device 1200 may not include one or more of the components shown in Figure 12, but computing device 1200 may include interface circuits for coupling to such one or more components. For example, computing device 1200 may not include a display device 1206, but may include a display device interface circuit (e.g., connector and driver circuit) to which the display device 1206 can be coupled. In another set of examples, the computing device 1200 may not include an audio input device 1218 or an audio output device 1208, but may include an audio input or output device interface circuit (e.g., a connector and support circuit) to which the audio input device 1218 or the audio output device 1208 can be coupled.

[0119] The computing device 1200 may include a processing device 1202 (for example, one or more processing devices). The processing device 1202 processes electronic data from registers and / or memory and converts that electronic data into other electronic data that can be stored in registers and / or memory. The computing device 1200 may also include a memory 1204, which itself may include one or more memory devices such as volatile memory (e.g., DRAM), non-volatile memory (e.g., read-only memory (ROM)), high-bandwidth memory (HBM), flash memory, solid-state memory, and / or a hard drive. In some embodiments, the memory 1204 may include memory that shares a die with the processing device 1202. In some embodiments, the memory 1204 includes one or more non-temporary computer-readable media that store executable instructions for performing operations to compute activation functions in a DNN. The operations are, for example, several operations performed by the method 1100 described above in relation to Figure 11, or by the DNN accelerator 300 (e.g., activation unit 360) described above in relation to Figure 3. Instructions stored in one or more non-temporary computer-readable media may be executed by the processing device 1202.

[0120] In some embodiments, the computing device 1200 may include a communication chip 1212 (for example, one or more communication chips). For example, the communication chip 1212 may be configured to manage wireless communication for transferring data to and from the computing device 1200. The term “wireless” and its derivatives may be used to describe circuits, devices, systems, methods, techniques, communication channels, etc., that can communicate data through the use of modulated electromagnetic radiation through a non-solid medium. The term does not imply that the devices in question do not contain wires, although in some embodiments they may not contain wires.

[0121] The 1212 communication chip may implement any of several wireless standards or protocols, including, but not limited to, Institute of Electrical and Electronics Engineers (IEEE) standards, including Wi-Fi (IEEE 802.10 family), IEEE 802.16 standards (e.g., IEEE 802.16-2005 amendment), and the Long-Term Evolution (LTE) project (e.g., the Advanced LTE project, the Ultra-Mobile Broadband (UMB) project (also known as "3GPP2")), including any amendments, updates, and / or revisions. IEEE 802.16-compatible broadband radio access (BWA) networks are commonly referred to as WiMAX networks, an acronym representing global interoperability for microwave access, and a certification mark for products that pass conformity and interoperability tests for the IEEE 802.16 standards. The communication chip 1212 may operate in accordance with Global System for Mobile Communications (GSM®), General-Purpose Packet Radio Service (GPRS), Universal Mobile Telecommunications System (UMTS), High-Speed ​​Packet Access (HSPA), Evolved HSPA (E-HSPA), or LTE networks. The communication chip 1212 may operate in accordance with GSM® Evolution Enhanced Data (EDGE), GSM EDGE Radio Access Network (GERAN), Universal Terrestrial Radio Access Network (UTRAN), or Evolved UTRAN (E-UTRAN). The communication chip 1212 may operate in accordance with Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Digital Enhanced Cordless Telecommunications (DECT), Evolution Data Optimization (EV-DO), and their derivatives, as well as any other radio protocols designated as 3G, 4G, 5G, and later. In other embodiments, the communication chip 1212 may operate in accordance with other radio protocols. The computing device 1200 may include an antenna 1222 for facilitating wireless communication and / or for receiving other wireless communications (such as AM or FM radio transmissions).

[0122] In some embodiments, the communication chip 1212 may manage wired communications, such as electrical, optical, or any other suitable communication protocol (e.g., Ethernet®). As described above, the communication chip 1212 may comprise multiple communication chips. For example, the first communication chip 1212 may be dedicated to short-range wireless communications such as Wi-Fi or Bluetooth®, and the second communication chip 1212 may be dedicated to long-range wireless communications such as Global Positioning System (GPS), EDGE, GPRS, CDMA, WiMAX, LTE, EV-DO, or others. In some embodiments, the first communication chip 1212 may be dedicated to wireless communications, and the second communication chip 1212 may be dedicated to wired communications.

[0123] The computing device 1200 may include a battery / power supply circuit 1214. The battery / power supply circuit 1214 may include one or more energy storage devices (e.g., batteries or capacitors) and / or circuits for coupling components of the computing device 1200 to an energy source separate from the computing device 1200 (e.g., AC line power).

[0124] The computing device 1200 may include a display device 1206 (or a corresponding interface circuit as described above). The display device 1206 may include any visual indicator, such as a head-up display, computer monitor, projector, touchscreen display, liquid crystal display (LCD), light-emitting diode display, or flat panel display.

[0125] The computing device 1200 may include an audio output device 1208 (or a corresponding interface circuit as described above). The audio output device 1208 may include any device that generates an audible indicator, such as a speaker, headset, or earphones.

[0126] The computing device 1200 may include an audio input device 1218 (or a corresponding interface circuit as described above). The audio input device 1218 may include any device that generates a signal representing sound, such as a microphone, a microphone array, or a digital instrument (for example, an instrument with a Musical Instrument Digital Interface (MIDI) output).

[0127] The computing device 1200 may include a GPS device 1216 (or a corresponding interface circuit as described above). The GPS device 1216 may communicate with a satellite-based system and receive the position of the computing device 1200, as is known in the art.

[0128] The computing device 1200 may include another output device 1210 (or a corresponding interface circuit as described above). Examples of other output devices 1210 may include an audio codec, a video codec, a printer, a wired or wireless transmitter for providing information to other devices, or additional storage devices.

[0129] The computing device 1200 may include another input device 1220 (or a corresponding interface circuit as described above). Examples of other input devices 1220 may include an accelerometer, gyroscope, compass, image acquisition device, cursor control device such as a keyboard or mouse, stylus, touchpad, barcode reader, quick response (QR) code reader, any sensor, or radio frequency identification (RFID) reader.

[0130] The computing device 1200 may have any desired form factor, such as a handheld or mobile computer system (e.g., a mobile phone, smartphone, mobile internet device, music player, tablet computer, laptop computer, netbook computer, ultrabook computer, personal digital assistant (PDA), ultramobile personal computer, etc.), a desktop computer system, a server or other network-connected computing component, a printer, scanner, monitor, set-top box, entertainment control unit, vehicle control unit, digital camera, digital video recorder, or wearable computer system. In some embodiments, the computing device 1200 may be any other electronic device that processes data.

[0131] Selected Examples The following paragraphs provide various examples of embodiments disclosed herein.

[0132] Embodiment 1 provides a computational element for calculating an activation function, the computational element comprising: a first multiplier configured to calculate one or more intermediate products using activations calculated in layers of a neural network; a second multiplier configured to calculate one or more terms of an approximation of the activation function based on the one or more intermediate products from the first multiplier and one or more coefficients of the approximation; and an accumulator configured to calculate the output of the activation function based on a polynomial containing the one or more terms of the approximation, the order of which is determined based on a predetermined precision of the output of the activation function.

[0133] Example 2 provides a calculation element of Example 1, wherein the approximation of the activation function is a Taylor series, and one or more coefficients of the approximation include one or more coefficients of the Taylor series calculated before the activation is calculated.

[0134] Embodiment 3 provides a computation element of Embodiment 1 or 2, further comprising a storage unit associated with the accumulator, wherein the storage unit is configured to store an intermediate sum calculated by the accumulator, and the accumulator is configured to calculate the output of the activation function by accumulating the intermediate sum with the approximate term calculated by the second multiplier.

[0135] Embodiment 4 provides a calculation element of any of the above embodiments, wherein the first multiplier is configured to calculate one or more intermediate products by: calculating a first intermediate product in a first clock cycle; and, after calculating the first intermediate product, calculating a second intermediate product in a second clock cycle based on the activation and the first intermediate product.

[0136] Embodiment 5 provides a calculation element of any of the above embodiments, wherein the second multiplier is configured to calculate one or more terms of the approximation in a sequence of clock cycles, and the second multiplier is configured to use different coefficients of the approximation in each clock cycle in the sequence.

[0137] Embodiment 6 provides a computation element of any of the above embodiments, wherein the computation element comprises multiple computation elements for calculating the output of the activation function using multiple activations, the multiple activations being computed in the layer of the neural network, and the multiple activations being input to different of the multiple computation elements in different clock cycles.

[0138] Example 7 provides the computational elements of Example 6, wherein the first output of the activation function based on the first activation among the plurality of activations has a predetermined accuracy higher than the second output of the activation function based on the second activation among the plurality of activations, and the first output of the activation function is calculated by more computational elements than the second output of the activation function.

[0139] Embodiment 8 provides a device for a deep learning operation, the device comprising: one or more processing elements configured to compute one or more activations by performing the deep learning operation in a neural network; a memory configured to store one or more coefficients of an approximation of an activation function in the neural network; and one or more computation elements configured to receive the one or more activations from the one or more processing elements and receive the one or more coefficients from the memory, the computation element comprising: a first multiplier configured to compute one or more intermediate products using one of the one or more activations; a second multiplier configured to compute one or more terms of the approximation based on the one or more intermediate products from the first multiplier and the one or more coefficients; and an accumulator configured to compute the output of the activation function based on a polynomial containing the one or more terms of the approximation, the order of which is determined based on a predetermined precision of the output of the activation function.

[0140] Example 9 provides the apparatus of Example 8, wherein one or more processing elements are coupled to the memory via a data transfer path, and the computing elements are located on the data transfer path.

[0141] Example 10 provides the apparatus of Example 8 or 9, wherein the first multiplier is configured to calculate one or more intermediate products by calculating a first intermediate product in a first clock cycle and, after calculating the first intermediate product, calculating a second intermediate product in a second clock cycle based on the activation and the first intermediate product.

[0142] Example 11 provides an apparatus of any one of Examples 8 to 10, wherein the second multiplier is configured to compute one or more terms of the approximation in a sequence of clock cycles, and the second multiplier is configured to use different coefficients of the approximation in each clock cycle in the sequence.

[0143] Example 12 provides an apparatus of any one of Examples 8 to 11, wherein one or more different activations are input to one or more different computational elements in different clock cycles.

[0144] Example 13 provides the apparatus of Example 12, wherein the first output of the activation function based on a first activation among the one or more activations has a predetermined accuracy higher than the second output of the activation function based on a second activation among the one or more activations, and the first output of the activation function is calculated by more computational elements than the second output of the activation function.

[0145] Example 14 provides one of the devices described in Examples 8 to 13, wherein the deep learning operation is located in the first layer of the neural network, the output of the activation function is input to the second layer of the neural network, and the second layer is located after the first layer in the neural network.

[0146] Example 15 provides a method for deep learning, comprising the steps of: receiving one or more pre-calculated coefficients of an approximation of an activation function in a neural network; receiving activations computed in layers of the neural network; computing one or more intermediate products using the activations by a first multiplier; computing one or more terms of the approximation by a second multiplier based on the one or more intermediate products from the first multiplier and the one or more coefficients; and computing the output of the activation function by an accumulator based on a polynomial containing the one or more terms of the approximation, the order of which is determined based on a predetermined precision of the output of the activation function.

[0147] Example 16 provides the method of Example 15, wherein calculating one or more intermediate products includes calculating a first intermediate product in a first clock cycle and, after calculating the first intermediate product, calculating a second intermediate product in a second clock cycle based on the activation and the first intermediate product.

[0148] Example 17 provides the method of Example 15 or 16, wherein calculating the one or more terms of the approximation involves calculating the one or more terms of the approximation in a sequence of clock cycles by using different coefficients of the approximation in each clock cycle in the sequence of clock cycles.

[0149] Example 18 further comprises calculating another output of the activation function based on another activation computed in the layer of the neural network, wherein the other output of the activation function has a predetermined precision different from the output of the activation function, providing one of the methods of Examples 15 to 17.

[0150] Example 19 further comprises receiving one or more other activations in one or more clock cycles different from the clock cycle in which the activation is received, wherein the one or more other activations are computed in the layer of the neural network, providing any one of the methods of Examples 15 to 18.

[0151] Example 20 further includes providing the output of the activation function to another layer of the neural network, the other layer being located after the previous layer in the neural network, providing any one of the methods of Examples 15 to 19.

[0152] The above description of illustrated implementations of the Disclosure, including those described in the Abstract, is not intended to be exhaustive or to limit the Disclosure to the exact form disclosed. While specific implementations of the Disclosure and examples of the Disclosure are described herein for illustrative purposes, various equivalent modifications are possible within the scope of the Disclosure, as will be apparent to those skilled in the art. These modifications may be made to the Disclosure in light of the above detailed description.

Claims

1. A computational element for calculating an activation function, wherein the computational element is: A first multiplier configured to compute one or more intermediate products using activations computed in layers of a neural network; A second multiplier configured to calculate one or more terms of the approximation of the activation function based on one or more intermediate products from the first multiplier and one or more coefficients of the approximation; An accumulator configured to calculate the output of the activation function based on a polynomial containing one or more of the aforementioned approximations, The polynomial has such that the degree of the polynomial is determined based on a predetermined precision of the output of the activation function. calculation element.

2. The calculation element according to claim 1, wherein the approximation of the activation function is a Taylor series, and one or more coefficients of the approximation include one or more coefficients of the Taylor series calculated before the activation is calculated.

3. The system further comprises a storage unit associated with the accumulator, the storage unit being configured to store the intermediate sum calculated by the accumulator. The calculation element according to claim 1 or 2.

4. The calculation element according to claim 3, wherein the accumulator is configured to calculate the output of the activation function by accumulating the intermediate sum with the approximate term calculated by the second multiplier.

5. The first multiplier calculates the first intermediate product in the first clock cycle; After calculating the first intermediate product, a second intermediate product is calculated in the second clock cycle based on the activation and the first intermediate product, The calculation element according to claim 1, configured to calculate one or more intermediate products.

6. The calculation element according to claim 1, wherein the second multiplier is configured to calculate one or more terms of the approximation in a sequence of clock cycles, and the second multiplier is configured to use different coefficients of the approximation in each clock cycle in the sequence.

7. The aforementioned computational element includes multiple computational elements for calculating the output of the activation function using multiple activations, The plurality of activations are computed in the layer of the neural network, and include the activations, The calculation element according to claim 1.

8. The computation element according to claim 7, wherein the multiple activations are input to different of the multiple computation elements in different clock cycles.

9. The calculation element according to claim 8, wherein the first output of the activation function based on the first activation among the plurality of activations has a predetermined accuracy higher than the second output of the activation function based on the second activation among the plurality of activations.

10. The computational element according to claim 9, wherein the first output of the activation function is calculated by more computational elements than the second output of the activation function.

11. A device for deep learning operations, the device being: One or more processing elements configured to compute one or more activations by performing the deep learning operation in a neural network; A memory configured to store one or more coefficients of an approximation of the activation function in the neural network; One or more computation elements configured to receive one or more activations from the one or more processing elements and to receive one or more coefficients from the memory, It has the following calculation elements: A first multiplier configured to calculate one or more intermediate products using one or more activations from the aforementioned one or more activations; A second multiplier configured to calculate one or more terms of the approximation based on the one or more intermediate products and one or more coefficients from the first multiplier; An accumulator configured to calculate the output of the activation function based on a polynomial containing one or more of the aforementioned approximations, The polynomial has such that the degree of the polynomial is determined based on a predetermined precision of the output of the activation function. Device.

12. The apparatus according to claim 11, wherein one or more processing elements are coupled to the memory via a data transfer path, and the calculation element is located on the data transfer path.

13. The first multiplier is: Calculate the first intermediate product in the first clock cycle; After calculating the first intermediate product, a second intermediate product is calculated in the second clock cycle based on the activation and the first intermediate product, The apparatus according to claim 11 or 12, configured to calculate one or more intermediate products.

14. The apparatus according to claim 11, wherein the second multiplier is configured to compute the one or more terms of the approximation in a sequence of clock cycles.

15. The apparatus according to claim 14, wherein the second multiplier is configured to use different coefficients of the approximation in each clock cycle in the sequence.

16. The apparatus according to claim 11, wherein different of the one or more activations are input to different of the one or more computation elements in different clock cycles.

17. The first output of the activation function based on the first activation among the one or more activations has a predetermined accuracy that is higher than the second output of the activation function based on the second activation among the one or more activations. The first output of the activation function is calculated by more computational elements than the second output of the activation function. The apparatus according to claim 16.

18. The deep learning operation described above occurs in the first layer of the neural network, The output of the activation function is input to the second layer of the neural network. The apparatus according to claim 11.

19. The apparatus according to claim 18, wherein the second layer is located after the first layer in the neural network.

20. A method for deep learning: A step of receiving one or more pre-calculated coefficients for approximating the activation function in a neural network; The step of receiving the computationally calculated activation in the layer of the neural network; The first multiplier calculates one or more intermediate products using the activation; A step of calculating one or more terms of the approximation using a second multiplier based on the one or more intermediate products from the first multiplier and the one or more pre-calculated coefficients; The step of using an accumulator to calculate the output of the activation function based on a polynomial containing one or more of the approximate terms, The degree of the polynomial is determined based on a predetermined precision of the output of the activation function. method.

21. Calculating the one or more intermediate products mentioned above is: In the first clock cycle, the first intermediate product is calculated. The process includes calculating the first intermediate product, and then calculating a second intermediate product in a second clock cycle based on the activation and the first intermediate product. The method according to claim 20.

22. Calculating one or more of the aforementioned approximations is: This includes calculating one or more terms of the approximation in a sequence of clock cycles by using different coefficients of the approximation in each clock cycle in the sequence of clock cycles. The method according to claim 20 or 21.

23. The further includes calculating another output of the activation function based on another activation calculated in the layer of the neural network, The other output of the activation function has a predetermined precision different from the output of the activation function. The method according to claim 20.

24. The method according to claim 20, further comprising receiving one or more other activations in one or more clock cycles different from the clock cycle in which the activation is received, wherein the one or more other activations are computed in the layer of the neural network.

25. The method according to claim 20, further comprising providing the output of the activation function to another layer of the neural network, wherein the other layer is located after the layer in the neural network.

Images