Activation function approximation based on input range partitioning

Abstract

An activation function in a neural network can be approximated. An input range, such as a range of input data elements of the activation function (e.g., data elements having a floating point format), can be partitioned into a sequence of segments, for example, based on an exponent of the input data elements. For a first segment, a linear function can be used to approximate the activation function. Sub-segments within a second segment can each map to the first segment. An approximated output of the activation function for the first segment can be reused as an approximated output of the activation function for each of the sub-segments. For a third segment, a lookup table can be used to approximate the activation function, the lookup table associating a mantissa bit of an input element to an approximated output element of the activation function. For a fourth segment, an approximated output of the activation function is a predetermined integer.

Description

Cross-reference to related applications

[0001] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 639,989, filed April 29, 2024, entitled “Periodic Function Approximation Using Piecewise Linear Activation Functions in Deep Neural Network Accelerators,” which is incorporated herein by reference in its entirety. Technical Field

[0002] In general, this disclosure relates to deep neural networks (DNNs), and more specifically, to approximations of activation functions in DNNs based on the partitioning of the input range of the activation function. Background Technology

[0003] DNNs are widely used in various artificial intelligence applications, from computer vision to speech recognition and natural language processing, due to their ability to achieve high accuracy. However, this high accuracy comes at the cost of significant computational cost. DNNs have extremely high computational requirements, as each inference may require hundreds of millions of multiply-accumulate (MAC) operations and large amounts of data to read and write. DNN inference also requires the computation of activation functions. Therefore, techniques to improve the efficiency of DNNs are needed. Attached Figure Description

[0004] The embodiments will be readily understood from the following detailed description taken in conjunction with the accompanying drawings. For ease of description, the same reference numerals denote the same structural elements. In the figures, embodiments are shown by way of example rather than limitation.

[0005] Figure 1 Example DNNs based on various implementation schemes are shown.

[0006] Figure 2 It is a block diagram of a DNN system based on various implementation schemes.

[0007] Figure 3 This is a block diagram of the DNN module according to various implementation schemes.

[0008] Figure 4 Example input range partitioning for activation functions according to various implementation schemes is shown.

[0009] Figure 5 Another example of input range partitioning for activation functions based on various implementation schemes is shown.

[0010] Figure 6 Example activation function units according to various implementation schemes are shown.

[0011] Figure 7 Pipelines for approximate activation functions according to various implementation schemes are shown.

[0012] Figure 8A The curves of the activation functions are shown for example representations of various implementation schemes.

[0013] Figure 8B A linear approximation of the first part of the curve according to various implementation schemes is shown.

[0014] Figure 8C Example linear curves based on various implementation schemes are shown.

[0015] Figure 8D Example saturation curves according to various implementation schemes are shown.

[0016] Figure 9 The generation of lookup table (LUT) addresses for linear approximations of activation functions according to various implementation schemes is illustrated.

[0017] Figure 10 Approximations of the activation function based on output lookup according to various implementation schemes are shown.

[0018] Figure 11 The generation of approximate LUT addresses for finding the output of the activation function is illustrated according to various implementation schemes.

[0019] Figure 12 An array of post-processing elements (PPEs) according to various embodiments is shown.

[0020] Figure 13 This is a flowchart illustrating methods for activation functions in approximate DNNs according to various implementation schemes.

[0021] Figure 14 It is a block diagram of an example computing device based on various implementation schemes. Detailed Implementation

[0022] The past decade has witnessed rapid growth in AI (artificial intelligence)-based data processing, particularly DNN-based processing. DNNs are widely used in computer vision, speech recognition, image and video processing, primarily because they can achieve accuracy exceeding human levels. Significant improvements in DNN model size and accuracy, along with the rapid increase in the computing power of execution platforms, have led to the adoption of DNN applications even in resource-constrained mobile and edge devices with limited energy availability.

[0023] Activation functions are a crucial part of DNNs. They determine whether a neuron should be activated by calculating a weighted sum of activations and adding a bias. A key purpose of activation functions is to introduce nonlinearity into the neuron's output. Given the complexity of some nonlinear activation functions used in many DNNs, hardware implementations may need to approximate them within a certain level of accuracy. Piecewise linear (PWL) approximation is one method for approximating complex nonlinear activation functions. PWL approximations are typically based on approximating complex nonlinear curves using several linear segments. Each linear segment can be represented using a slope and an intercept. The full range of the nonlinear activation function can be divided into smaller regions, such that each region can be approximated using a linear function. These regions can have variable ranges, but performing a linear function (even if there can be a larger number of linear functions) can be more efficient than performing the nonlinear activation function itself. The slope and intercept of the linear segments can be stored in a LUT. Increased accuracy typically requires more entries in the LUT. LUT address generation logic is typically used to generate the address of the slope and truncate it with the LUT corresponding to the linear segment containing the input.

[0024] For a general-purpose, flexible, and future-proven DNN accelerator, it is important to enable the accelerator to be programmable for various types of activation functions as needed. Many currently available solutions for approximating activation functions are based on kernel-based implementations using digital signal processors (DSPs). Another variation of the activation function implementation is based on Taylor series approximation. However, these currently available solutions lack flexibility.

[0025] Approximating periodic functions presents the additional challenge of approximating the entire input range. Most activations have a finite range, meaning they need to be approximated, and the outputs of these activations saturate with a higher exponent in both the positive and negative ranges. Due to this saturation with a higher exponent in both the positive and negative ranges, the number of LUT entries used for PWL approximation can be relatively small. This can be problematic when approximating the full range, potentially requiring a large number of LUT entries, which may not always be available within DNN accelerators.

[0026] Another challenge arises when approximating a periodic function across the full range in floating-point formats such as FP16 or FP32, particularly with higher exponents, where approximating the curve can be more difficult due to the lower precision of these exponents. For example, when approximating the full FP16 range of a sine (SIN) function, the PWL approximation cannot be used to approximate the SIN curve within a reasonable unit of minimum precision (ULP) for higher exponents.

[0027] Implementations of this disclosure can improve at least some of the challenges and problems mentioned above by providing systems and methods for approximating activation functions in DNNs based on the partitioning of the input range of the activation function. An example input range can be a range that includes all possible data elements (e.g., activations) to be input into the activation function to perform the activation function. Input data elements can be activations generated in DNN layers (such as convolutional layers). The input range can be partitioned into segments (“input segments”), and the activation function can be approximated in different ways for different input segments.

[0028] In various embodiments of this disclosure, a DNN system including a DNN module and a DNN accelerator can be used to perform a DNN. The DNN module can divide the input range of the activation function into input segments. The DNN module can also determine an approximation method for the input segments. The DNN module can determine the boundaries of the input segments or approximation methods based on the estimated accuracy of the activation function approximation. Different approximation methods can be determined and applied to different input segments. Examples of approximation methods can include linear approximation, range-folding approximation, output lookup approximation, saturation approximation, etc. When a linear approximation is used for an input segment, one or more linear functions can be used to approximate the activation function. The DNN module can program the LUT to store parameters of the linear function (e.g., intercept, range, etc.). After receiving input data elements from the input segments, the DNN accelerator (e.g., the PPE in the DNN accelerator) can identify LUT entries for the input data elements and approximate the activation function using the linear function parameters stored in the LUT entries. When the range-folding approximation is used for an input segment, after receiving the input data elements in the input segment, the PPE can map the input data elements to data elements in another input segment using a linear approximation, so that the linear approximation result of the data elements in the other input segment can be used as an approximate output of the activation function of the received data elements.

[0029] When the output lookup approximation is used for the input segment, the DNN module can program a LUT or another LUT to store predetermined values ​​as approximate outputs of the activation functions for the input segment. After receiving input data elements from the input segment, the PPE can identify entries in the LUT and obtain the values ​​of the input data elements from the LUT entries. Different input data elements can correspond to different values ​​in the LUT. The PPE can identify LUT entries for input data elements by generating LUT addresses based on the bits of the input data elements and using the LUT addresses to identify the LUTs (for linear approximation or output lookup approximation). When the saturation approximation is used for the input segment, the DNN module can program a saturation table to store fixed values, such as fixed integers. The DNN accelerator can obtain fixed values ​​from the saturation table and output these fixed values ​​as approximate outputs of the activation functions for all input data elements in the input segment. Other approximation methods can also be used.

[0030] This disclosure allows for scaling periodic functions by applying a scaling factor so that the period of the periodic function is a multiple of 2, utilizing floating-point formats (e.g., lm×2). e This property facilitates the partitioning of the input range based on the exponent. It also facilitates range folding. For example, the higher accuracy of a lower exponent in floating-point format can be used to approximate a single period (T) of a periodic function, such as the first period within the input range, using the PWL approximation. Inputs greater than T can be remapped to the first period using the periodicity of the activation function. For higher exponents, range folding may not provide the desired accuracy because the weights associated with the lower mantissa bits can be significant. Assuming T is a power of 2, the least significant bit of the mantissa can be used to store the function output in a programmable LUT. For exponents higher than a certain value, all floating-point values ​​can correspond to integers. For these exponents, the intervals between successive values ​​make the input a multiple of T, and the output will be fixed. Since the boundaries of the input segments and the approximation method can be programmable, this disclosure can better optimize the accuracy of activation function approximations compared to currently available activation function approximation methods.

[0031] This disclosure provides a hardware-software co-design method to approximate activation functions of periodic functions covering the entire range within the hardware (e.g., the full FP16, BF16, or FP32 range). Unlike currently available activation function approximation methods that cannot approximate the full range of periodic functions due to limited LUT entries and the nature of periodic functions, the method in this disclosure can approximate the full range of periodic functions without compromising accuracy. The method in this disclosure not only provides highly accurate approximations of periodic functions (e.g., sine, cosine, trigonometric wave functions, periodic rectified linear unit (ReLU) functions, etc.), but also facilitates the execution of position encoding steps within stable diffusion and other converter networks.

[0032] For illustrative purposes, specific figures, materials, and configurations have been set forth to provide a thorough understanding of the illustrative embodiments. However, it will be apparent to those skilled in the art that this disclosure may be practiced without specific details, and / or may be practiced using only some of the aspects described. In other instances, well-known features have been omitted or simplified so as not to obscure the illustrative embodiments.

[0033] Furthermore, reference is made to the accompanying drawings, which form part of this disclosure, in which feasible embodiments are illustrated. It should be understood that other embodiments may be utilized, and structural or logical changes may be made, without departing from the scope of this disclosure. Therefore, the following detailed description should not be considered limiting.

[0034] Various operations can be described sequentially as multiple discrete actions or operations in a manner most conducive to understanding the claimed subject matter. However, the order of description should not be construed as implying that these operations must be sequentially related. In particular, these operations may not be performed in the order presented. The described operations may be performed in a different order than the described embodiments. Various additional operations may be performed, or in additional embodiments, the described operations may be omitted.

[0035] For the purposes of this disclosure, the phrase "A or B" or the phrase "A and / or B" means (A), (B), or (A and B). For the purposes of this disclosure, the phrase "A, B, or C" or the phrase "A, B, and / or C" means (A), (B), (C), (A and B), (A and C), (B and C), or (A, B, and C). When used to refer to a measurement range, the term "between" includes the ends of the measurement range.

[0036] The specification uses the phrases "in one embodiment" or "in an embodiment," which may each refer to one or more of the same or different embodiments. The terms "comprising," "including," "having," etc., used with respect to embodiments of this disclosure are synonymous. This disclosure may use perspective-based descriptions (such as "above," "below," "top," "bottom," and "side") to interpret various features of the drawings; however, these terms are merely for ease of discussion and do not imply an expected or desired orientation. The drawings are not necessarily drawn to scale. Unless otherwise stated, the use of ordinal adjectives such as "first," "second," and "third" to describe common objects merely indicates different instances of similar objects being referenced and is not intended to imply that the objects so described must be in a given sequence in time, space, hierarchy, or any other way.

[0037] In the following detailed description, various aspects of the illustrative embodiments will be described using terminology commonly employed by those skilled in the art in order to convey the substance of the work to others skilled in the art.

[0038] The terms “substantially,” “nearly,” “approximately,” “close to,” and “about” generally refer to a target value as described herein or known in the art within + / - 20%. Similarly, terms indicating the orientation of various elements, such as “coplanar,” “perpendicular,” “orthogonal,” “parallel,” or any other angle between elements, generally refer to a target value as described herein or known in the art within + / - 5-20%.

[0039] Furthermore, the terms "comprise," "comprising," "include," "including," "have," "having," or any other variations thereof are intended to cover non-exclusive inclusion. For example, a method, process, apparatus, or DNN accelerator that includes a list of elements is not necessarily limited to those elements, but may include other elements not expressly listed or inherent to the method, process, apparatus, or DNN accelerator. Additionally, the term "or" refers to an inclusive "or," not an exclusive "or."

[0040] The systems, methods, and apparatuses disclosed herein each have several innovative aspects, and no single aspect is solely responsible for all the desired properties disclosed herein. Details of one or more embodiments of the subject matter described herein are set forth in the following description and accompanying drawings.

[0041] Figure 1 Example DNN 100 according to various implementation schemes is shown. For illustrative purposes, Figure 1 DNN 100 in this example is a convolutional DNN, such as a Convolutional Neural Network (CNN). In other implementations, DNN 100 can be other types of DNNs. DNN 100 is trained to receive an image and output a classification of objects in the image. Figure 1 In one implementation, DNN 100 receives an input image 105 including objects 115, 125, and 135. DNN 100 includes a sequence of layers comprising multiple convolutional layers 110 (referred to as "convolutional layer 110"), multiple pooling layers 120 (referred to as "pooling layer 120"), and multiple fully connected layers 130 (referred to as "fully connected layer 130"). In other implementations, DNN 100 may include fewer, more, or different layers. In inference, the layers of DNN 100 perform tensor computations, which include a variety of tensor operations such as convolution (e.g., multiply-accumulate (MAC) operation, pooling operation, element-wise operations (e.g., element-wise addition, element-wise multiplication, etc.), other types of tensor operations, or some combination thereof.

[0042] Convolutional layer 110 summarizes the presence of features in input image 105. Convolutional layer 110 acts as a feature extractor. The first layer of DNN 100 is convolutional layer 110. In the example, convolutional layer 110 performs convolution on input tensor 140 (also called input feature map (IFM) 140) and filter 150. Figure 1As shown, the IFM 140 is represented by a 7x7x3 three-dimensional (3D) matrix. The IFM 140 includes three input channels, each represented by a 7x7 two-dimensional (2D) matrix. The 7x7 2D matrix includes seven input elements (also called input points) in each row and seven input elements in each column. The filter 150 is represented by a 3x3x3 3D ​​matrix. The filter 150 includes three kernels, each kernel corresponding to a different input channel of the IFM 140. The kernel is a 2D matrix of weights, where the weights are arranged in columns and rows. The kernel can be smaller than the IFM. Figure 1 In this implementation, each kernel is represented by a 3x3 2D matrix. The 3x3 kernel consists of 3 weights in each row and 3 weights in each column. The weights can be initialized and updated using backpropagation with gradient descent. The magnitude of the weights indicates the importance of filter 150 in extracting features from IFM 140.

[0043] Convolution involves a MAC operation with the input elements in IFM 140 and the weights in filter 150. The convolution can be a standard convolution 163 or a depthwise convolution 183. In a standard convolution 163, the entire filter 150 slides through IFM 140. All input channels are combined to produce an output tensor 160 (also called OFM 160). OFM 160 is represented by a 5x5 2D matrix. The 5x5 2D matrix includes 5 output elements (also called output points) in each row and 5 output elements in each column. For illustrative purposes, in Figure 1 In one implementation, the standard convolution includes a filter. In an implementation with multiple filters, the standard convolution can produce multiple output channels in the OFM 160.

[0044] The multiplication applied between a kernel-sized region of IFM 140 and a kernel can be a dot product. A dot product is an element-wise multiplication between a kernel-sized region of IFM 140 and its corresponding kernel, which is then summed, always producing a single value. Because it produces a single value, this operation is often called a "scalar product." Using kernels smaller than IFM 140 is intentional because it allows the same kernel (the set of weights) to be multiplied by IFM 140 multiple times at different points on IFM 140. Specifically, kernels are systematically applied from left to right and top to bottom to each overlapping portion or kernel-sized region of IFM 140. Multiplying a kernel by IFM 140 once results in a single value. When a kernel is applied multiple times to IFM 140, the result of the multiplication is a 2D matrix of output elements. Therefore, the 2D output matrix from a standard convolution 163 (i.e., OFM 160) is called OFM.

[0045] In depthwise convolution 183, the input channels are not combined. Instead, MAC operations are performed on separate input channels and separate kernels to produce the output channels. For example... Figure 1 As shown, depthwise convolution 183 produces a depthwise output tensor 180. The depthwise output tensor 180 is represented by a 5×5×3 3D matrix. The depthwise output tensor 180 includes three output channels, each represented by a 5x5 2D matrix. The 5x5 2D matrix includes 5 output elements in each row and 5 output elements in each column. Each output channel is the result of a MAC operation between the input channel of IFM 140 and the kernel of filter 150. For example, the first output channel (patterned with dots) is the result of a MAC operation between the first input channel (patterned with dots) and the first kernel (patterned with dots), the second output channel (patterned with horizontal stripes) is the result of a MAC operation between the second input channel (patterned with horizontal stripes) and the second kernel (patterned with horizontal stripes), and the third output channel (patterned with diagonal stripes) is the result of a MAC operation between the third input channel (patterned with diagonal stripes) and the third kernel (patterned with diagonal stripes). In this depthwise convolution, the number of input channels equals the number of output channels, and each output channel corresponds to a different input channel. The input and output channels are collectively referred to as depthwise channels. After the depthwise convolution, a pointwise convolution 193 is then performed on the depthwise output tensor 180 and the 1x1x3 tensor 190 to produce OFM 160.

[0046] OFM 160 is then passed to the next layer in the sequence. In some implementations, OFM 160 is passed through an activation function. An example activation function is ReLU. ReLU is a computation that directly returns the value provided as input, or returns zero if the input is zero or less. Convolutional layer 110 can receive several images as input and compute the convolution of each of them with each of the kernels in the kernel. This process can be repeated several times. For example, OFM 160 is passed to a subsequent convolutional layer 110 (i.e., a convolutional layer 110 after OFM 160 is generated in the sequence). The subsequent convolutional layer 110 performs convolution on OFM 160 with the new kernel and generates a new feature map. The new feature map can also be normalized and adjusted. The new feature map can be convolved again by further subsequent convolutional layers 110, and so on.

[0047] In some implementations, convolutional layer 110 has four hyperparameters: the number of kernels, the size of F kernels (e.g., the kernel size is FxFxD pixels), the stride S of dragging the window corresponding to the kernel on the image (e.g., a stride of 1 means moving the window one pixel at a time), and zero padding P (e.g., adding a black outline of P pixel thickness to the input image of convolutional layer 110). Convolutional layer 110 can perform various types of convolution, such as 2D convolution, dilated or attenuated convolution, spatially separable convolution, depthwise separable convolution, transposed convolution, etc. DNN 100 includes 16 convolutional layers 110. In other implementations, DNN 100 may include a different number of convolutional layers.

[0048] Pooling layer 120 downsamples the feature map generated by the convolutional layer, for example, by summarizing the presence of features in regions of the feature map. Pooling layer 120 is placed between two convolutional layers 110: the preceding convolutional layer 110 (the convolutional layer 110 before pooling layer 120 in the layer sequence) and the following convolutional layer 110 (the convolutional layer 110 after pooling layer 120 in the layer sequence). In some implementations, pooling layer 120 is added after convolutional layer 110, for example, after an activation function (e.g., ReLU, etc.) has been applied to OFM 160.

[0049] Pooling layer 120 receives feature maps generated by previous convolutional layer 110 and applies pooling operations to them. Pooling operations reduce the size of the feature maps while preserving their important characteristics. Therefore, pooling operations improve the efficiency of the DNN and avoid overlearning. Pooling layer 120 can perform pooling operations using average pooling (calculating the average value for each region on the feature map), max pooling (calculating the maximum value for each region on the feature map), or a combination of both. The size of the pooling operation is smaller than the size of the feature map. In various implementations, the pooling operation is applied with a stride of two pixels at 2×2 pixels, such that the pooling operation reduces the size of the feature map by a factor of 2, for example, reducing the number of pixels or values ​​in the feature map to one-quarter of its size. In the example, pooling layer 120 applied to a 6x6 feature map results in a 3x3 output pooled feature map. The output of pooling layer 120 is fed into subsequent convolutional layers 110 for further feature extraction. In some implementations, pooling layer 120 operates on each feature map separately to create a set of the same number of new pooled feature maps.

[0050] Fully connected layer 130 may be the last layer of a DNN. Fully connected layer 130 may be convolutional or non-convolutional. Fully connected layer 130 may be a linear layer. In some implementations, fully connected layer 130 receives input operands. The input operands define the outputs of convolutional layer 110 and pooling layer 120, and include the values ​​of the last feature map generated by the last pooling layer 120 in the sequence. In some implementations, fully connected layer 130 applies a linear combination and activation function to the input operands and generates a vector. The vector may contain as many elements as there are classes: element i represents the probability that an image belongs to class i. Therefore, each element is between 0 and 1, and the sum of all elements is 1. These probabilities are calculated by the last fully connected layer 130 using a logistic function (for binary classification) or a SoftMax function (for multi-class classification) as the activation function.

[0051] In some implementations, the fully connected layer 130 classifies the input image 105 and returns an operand of size N, where N is the number of classes in the image classification problem. Figure 1 In this implementation, N equals 3 because there are 3 objects 115, 125, and 135 in the input image. Each element of the operand indicates the probability that the input image 105 belongs to a category. To calculate the probabilities, the fully connected layer 130 multiplies each input element by a weight, sums the results, and then applies an activation function (e.g., logical if N=2, and SoftMax if N>2). This is equivalent to multiplying the input operand by a matrix containing the weights. In this example, the vector contains 3 probabilities: a first probability indicating that object 115 is a tree, a second probability indicating that object 125 is a car, and a third probability indicating that object 135 is a person. In other implementations where the input image 105 includes different objects or different numbers of objects, the individual values ​​may differ.

[0052] Figure 2 This is a block diagram of a DNN system 200 according to various implementation schemes. The entire DNN system 200 or a portion thereof can be located on one or more computing devices (such as...). Figure 14 Implemented in computing device 1400. DNN system 200 can generate and execute DNNs, such as... Figure 1 DNN 100 in the example. Figure 2As shown, the DNN system 200 includes a DNN module 201 and a DNN accelerator 202. In other embodiments, the DNN system 200 may include alternative configurations, different components, or additional components. For example, the DNN system 200 may include multiple DNN modules or multiple DNN accelerators. Furthermore, the functionality of components belonging to the DNN system 200 may be implemented by different components or different systems included in the DNN system 200. In some embodiments, the DNN module 201 and the DNN accelerator 202 may include different types of processing units. The DNN module 201 and the DNN accelerator 202 may be implemented on the same chip or on separate chips.

[0053] DNN module 201 facilitates the generation and deployment of DNNs. In some implementations, DNN module 201 can generate and train DNNs. For example, DNN module 201 can define the hierarchical architecture of the DNN. DNN module 201 can also determine the intrinsic parameters of the DNN through the DNN training process. DNN module 201 can also determine one or more hyperparameters that define how the DNN is trained. Example hyperparameters are sparsity rates that define the sparsity level of one or more deep learning tensors of the DNN. DNN module 201 can also, for example, compress the DNN during or after training.

[0054] DNN module 201 can deploy trained, compressed, or validated DNNs for deep learning applications. In some implementations, DNN module 201 can distribute trained, compressed, or validated DNNs to devices or systems that can use the DNNs to perform tasks trained on their DNNs (e.g., image classification, motion planning, etc.). In other implementations, DNN module 201 can use DNN accelerator 202 to facilitate DNN deployment. For example, DNN module 201 can receive data from a device or system coupled to DNN system 200 and input the received data (or data generated by DNN module 201, for example, based on the received data) into the DNN. DNN module 201 can generate instructions (e.g., configuration files) to control the operation of DNN accelerator 202 during DNN execution. DNN module 201 can receive the output of the DNN from DNN accelerator 202. DNN module 201 can send the output of the DNN (or the result of processing the output of the DNN by DNN module 201) to a device or system.

[0055] DNN module 201 can control the execution process of a trained, compressed, or validated DNN. For example, DNN module 201 can facilitate the approximation of activation functions in the DNN, including nonlinear activation functions. In some implementations, DNN module 201 can determine the input range of the activation function, for example, based on the accuracy of the input data. DNN module 201 can also divide the input range into multiple segments and determine an approximation method for each segment. In some implementations, DNN module 201 can divide the input range or determine the approximation method based on the estimated accuracy of the activation function approximation. The following is in conjunction with... Figure 3 Provides certain aspects of the DNN module 201.

[0056] DNN accelerator 202 executes the DNN provided by DNN module 201. For example, DNN accelerator 202 can perform DNN execution, for instance, by running neural network operations within the DNN, for training the DNN or for performing tasks using a trained / compressed / validated DNN. DNN accelerator 202 may also be referred to as an AI accelerator, neural processing unit, or neural engine. Figure 2 As shown, the DNN accelerator 202 includes a memory 210, a DMA (Direct Memory Access) engine 220, and a data processing unit (DPU) 230 (referred to separately as "DPU 230"). In other embodiments, the DNN accelerator 202 may include alternative configurations, different components, or additional components. For example, the DNN accelerator 202 may include more than one memory 210 or DMA engine 220. As another example, the DNN accelerator 202 may include a single DPU 230. Furthermore, the functionality of the components attributed to the DNN accelerator 202 may be implemented by different components included in the DNN accelerator 202 or by different systems. The components of the DNN accelerator 202 may be implemented in hardware, software, firmware, or some combination thereof.

[0057] Memory 210 stores data associated with neural network operations performed by the DNN accelerator. In some embodiments, memory 210 may store data to be used by DPU 230 for DNN execution. For example, memory 210 may store weights determined by training the DNN, such as the weights of convolutional layers. As another example, memory 210 may store the inputs or outputs of the DNN. Memory 210 may also store data generated by DPU 230 by performing neural network operations in the DNN. Example neural network operations include convolution (also known as a “convolution operation”), pooling operations, element-wise operations, activation functions, other types of neural network operations, or some combination thereof. Memory 210 may be the main memory of DNN accelerator 202. In some embodiments, memory 210 includes one or more dynamic random access memories (DRAM).

[0058] DMA engine 220 facilitates data transfer between memory 210 and the local memory of DPU 230. For example, DMA engine 220 can read data from memory 210 and write data to the local memory of DPU 230. As another example, DMA engine 220 can read data from the local memory of DPU 230 and write data to memory 210. DMA engine 220 provides DMA features that allow DPU 230 to initiate data transfers between memory 210 and the local memory of DPU 230, and to perform other operations during the data transfer. In some implementations, DMA engine 220 can read tensors from memory 210 and modify the tensors in a manner optimized for DPU 230 before writing the tensors to the local memory of DPU 230.

[0059] DPU 230 can perform neural network operations within a DNN. For example, DPU 230 can run neural network operations one at a time within a DNN layer, or run a portion of a neural network operation. DPU 230 can be capable of running various types of neural network operations, such as activation functions, convolutions, pooling, element-wise operations, linear operations, non-linear operations, etc. In the example, DPU 230 can perform convolutions, such as standard convolutions or depthwise convolutions. In some implementations, DPU 230 receives an input tensor and one or more convolution kernels, and performs convolutions using the input tensor and the convolution kernels. The result of the convolution can be an output tensor, which can be further computed, for example, by DPU 230 or another DPU 230. In some implementations, the operations of the DNN layer can be run in parallel by multiple DPUs 230. For example, multiple DPUs 230 can each perform a portion of the workload for convolution. Data can be shared among DPUs 230. DPU 230 can also be referred to as a computation tile or tile. In some implementations, one or more DPUs 230 can constitute a processor. DPU 230 or its components may be or include a circuit system.

[0060] exist Figure 2In one implementation, each DPU 230 includes local memory 240, processing elements (PEs) 250 (referred to as "PE 250"), and PPE 260 (referred to as "PPE 260"). Some or all of the components of the DPU 230 may be implemented on the same chip. In other implementations, the DPU 230 may include alternative configurations, different components, or additional components. For example, the DPU 230 may include additional modules for loading data from local memory 240 into PE 250, or for draining data from PPE 260 into local memory 240. Furthermore, the functionality of components belonging to the DPU 230 may be implemented by different components included in the DPU 230 or a different system, a different DPU 230, or another component of the DNN accelerator 202. The components of the DPU 230 may be implemented in hardware, software, firmware, or some combination thereof.

[0061] Local memory 240 is local to the corresponding DPU 230. Figure 2 In one implementation, local memory 240 is internal to DPU 230. In other implementations, local memory 240 may be external to DPU 230. Data in local memory 240 may be transferred to or from memory 210, for example, via DMA engine 220. In some implementations, data in local memory 240 may be transferred to or from the local memory of another DPU 230. Local memory 240 may store data received, used, or generated by PE 250 and PPE 260. Examples of data may include input activations, weights, output activations, sparse bitmaps, etc. In some implementations, a data delivery unit (e.g., an input delivery unit) may read input data for neural network operations from local memory 240 and load the input data into PE 250 or PPE 260. A data delivery unit (e.g., an output delivery unit) may discharge output data for neural network operations from PE 250 or PPE 260 and write the output data into local memory 240.

[0062] In some implementations, local memory 240 includes one or more static random access memories (SRAMs). Local memory 240 may be byte-addressable, and each memory address identifies a single byte (eight bits) stored. In some implementations, local memory 240 may contain a memory bank. The number of databases in local memory 240 may be 16, 64, 128, 256, 512, 1024, 2048, or other numbers. A memory bank may include multiple memory cells. In an example, a database may include 8, 16, 64, or different numbers of memory cells. A memory bank or a memory cell within a memory bank may have a memory address. In an example, a memory cell may store a single byte, and data larger than a single byte may be stored in memory cells with contiguous memory addresses (i.e., adjacent memory cells). For example, a memory cell may store an integer in INT8 format, while two memory cells may be needed to store a 16-bit number in FP16 or BF16 format. In some implementations, 16 bits may be transferred from local memory 240 in a single read cycle. In other implementations, 16 bits can be transferred from local memory 240 in multiple read cycles (e.g., two cycles).

[0063] PEs 250 can be arranged in a PE array, where PEs 250 are arranged in columns or columns and rows. Each PE 250 may include at least one MAC unit capable of performing MAC operations. PEs 250 may also include data storage units, such as register files for storing data processed or computed by the MAC units. In the example, each MAC unit may be associated with an input register file for storing input activations, a weight register file for storing weights, and an output register file for storing output activations.

[0064] In some implementations (e.g., the implementation where the DPU 230 performs convolutional layers), computation in the MAC unit can be a MAC operation on activation operands and weight operands. Activation operands are activation tensors that can include one or more activations from the input tensor of the convolution. Different activations can be in different input channels. Weight operands are weight tensors that can include one or more weights from the filters of the convolution. The values ​​of the weights are determined by training the DNN. The weights in the weight operands can be in different input channels.

[0065] In some implementations, a MAC unit includes one or more multipliers for performing multiplication. A MAC unit may also include one or more accumulators (“adders”) for performing accumulation. The column of a MAC unit is called a MAC column. A MAC column may be associated with one or more MAC channels. A MAC channel is a path used to load data into a MAC column. A MAC channel may also be referred to as a data transfer channel or a data loading channel. A MAC column may have multiple MAC channels. The load bandwidth of a MAC column is the aggregation of the load bandwidths of all MAC channels associated with the MAC column. Using a certain number of MAC channels, data can be fed simultaneously into the same number of independent MAC units. In some implementations where a MAC column has four MAC channels for feeding activations or weights into the MAC column, and each MAC channel may have 16 bytes of bandwidth, the four MAC channels may have a total load bandwidth of 64 bytes.

[0066] In some implementations, the PE 250 may be capable of performing depthwise convolution, standard convolution, or both. In depthwise convolution, the MAC unit may perform a MAC operation, which comprises a sequence of multiplications of input operands and weight operands. Each multiplication (also called a loop) in the sequence is a multiplication of different activations in the input operands with different weights in the weight operands. Activations and weights in the same loop may correspond to the same channel. The sequence of multiplications produces a product operand that includes the sequence of products. The MAC operation may also include accumulation, where multiple product operands are accumulated to produce the output operand of the MAC unit. The PE 250 may output multiple output operands at once, each generated by a different MAC unit. In standard convolution, the MAC operation may include accumulation across channels. For example, instead of generating output operands, the MAC unit may accumulate products across different channels to generate a single output point.

[0067] In some implementations, PE 250 may include sparsity acceleration logic to facilitate sparsity acceleration. In some implementations, the sparsity module accelerates computation in one or more PE 250s based on the sparsity of activations or the sparsity of weights. The sparsity module may include storage units storing sparsity tensors (such as sparsity bitmaps). The sparsity tensor may be an activation sparsity tensor, a weight sparsity tensor, or a combination of both. The sparsity module may use the sparsity tensor to identify which data elements of the dense tensor correspond to data elements of the sparse tensor. Each identified data element of the dense tensor and its corresponding data element of the sparse tensor may constitute an activation-weight pair for a MAC operation. For example, the identified data element of the dense tensor will be multiplied by its corresponding data element of the sparse tensor in a MAC operation. The sparsity module may select one or more data elements of the dense tensor based on one or more sparsity elements of the sparse tensor corresponding to one or more non-zero value data elements of the dense format of the sparse tensor. The sparsity module may forward the identified activation-weight pairs to the MAC unit. Other data elements of the dense tensor will be skipped and not computed by the MAC unit to accelerate computation in the PE250, since these data elements will not contribute to the result of the MAC operation.

[0068] PPE 260 can receive the output of PE 250 and perform additional neural network operations on the output of PE 250. PPE 260 can be an array, where PPE 260 can be arranged by column or by a combination of columns and rows. An example of an array is... Figure 12 The PPE array 1200 in the example. Figure 2 As shown, each PPE 260 includes a bias unit 270, a scaling unit 275, an activation function unit 280, and an output conversion module 290. In other embodiments, the PPE 260 may include fewer, more, or different components.

[0069] Bias unit 270 can apply a bias to data elements. For example, bias unit 270 can accumulate each data element with a bias value. The bias value can be an internal parameter of the DNN and can be determined by training the DNN. The bias value can be positive or negative. Scaling unit 275 can apply scaling to data elements. In some embodiments, scaling unit 275 can multiply each data element with a scaling value. The scaling value can be an internal parameter of the DNN and can be determined by training the DNN. The scaling value can be positive or negative. For certain DNN layers that do not require bias or scaling operations, bias unit 270 or scaling unit 275 can be bypassed. In some embodiments, bias unit 270 or scaling unit 275 can process data elements output from PE 250 or from another component of PPE 260.

[0070] Activation function unit 280 can execute activation functions, including nonlinear activation functions, within a DNN. The input to the activation function can be a tensor comprising multiple input data elements. The tensor can be the output tensor of a DNN layer. In some implementations, the data elements to be input into the activation function can be within the range of the input to the activation function. Activation function unit 280 can use linear functions or other types of functions to compute an approximate output of the nonlinear activation function. In some implementations, the input range of the activation function can be divided into input segments. Activation function unit 280 can approximate the activation function in different ways for different input segments.

[0071] For example, activation function unit 280 can perform a linear approximation of an activation function on one or more input segments. For the linear approximation, activation function unit 280 can use data stored in a programmable LUT. A programmable LUT can be included in activation function unit 280. The data stored in the programmable LUT can be determined by DNN module 201. The LUT can include entries, each storing the intercept and slope of the linear function. Activation function unit 280 can determine the LUT address of the input data element and obtain the intercept and slope of the linear function from the entry at the LUT address. Activation function unit 280 can also compute an approximate output of the activation function by applying the linear function to the input data element using the intercept and slope. Activation function unit 280 can use more than one linear function for the input segment.

[0072] In some implementations, before activation function unit 280 begins linear approximation, it can map input data elements in one input segment to another input data element in a different input segment. Activation function unit 280 can reuse the approximate output of the activation function of the other input data element as the approximate output of the activation function of the input data element. In some implementations, the programmable LUT may include entries storing predetermined values. For one or more input segments, activation function unit 280 can obtain predetermined values ​​from LUT entries as approximate outputs of the activation functions of the input data elements. Activation function unit 280 can identify LUT entries using LUT addresses determined based on the input data elements.

[0073] In some implementations, one or more input segments can be saturation segments. For a saturation segment, activation function unit 280 can use a fixed value as an approximate output of the activation function for all input data elements falling within the saturation segment. The fixed value of the saturation segment can be stored in a saturation table, which can also be programmable. The following is in conjunction with... Figure 6 Provides certain aspects of the activation function unit 280.

[0074] Data conversion unit 285 can convert data output from PE 250 or data output from activation function unit 280 (e.g., an approximate output of an activation function). In some embodiments, data conversion unit 285 can convert the precision of the data to different precisions. For example, data conversion unit 285 can convert FP16 or BF16 data to FP32 data and vice versa. In some embodiments, data conversion unit 285 can be bypassed for certain DNN layers. In some embodiments, the output of data conversion unit 285 can be input to bias unit 270, scaling unit 275, or activation function unit 280 for further processing. In other embodiments, the output of data conversion unit 285 can be the output of PPE 260 and written into local memory 240.

[0075] In some implementations, PPE 260 (or the data delivery unit associated with PPE 260) may send the output of the activation function to local memory 240. In the example, an approximate output of the activation function may later be retrieved from local memory 240 by PE 250 for further computation. For example, PPE 260 may receive the output tensor of the DNN layer from PE 250 and compute one or more activation functions on the output tensor. The computation result of PPE 260 may be stored in local memory 240 and later used as the input tensor for the next DNN layer to be executed by PE 250.

[0076] Figure 3 This is a block diagram of the DNN module 201 according to various implementation schemes. Figure 3 In one implementation, the DNN module 201 includes an interface module 310, a training module 320, a compression module 330, a validation module 340, a compiler 350, an activation function module 360, and a data storage 370. In other implementations, the DNN module 201 may include alternative configurations, different components, or additional components. Furthermore, the functionality of the components belonging to the DNN module 201 may be implemented by different components, modules, or systems included in the DNN module 201.

[0077] Interface module 310 facilitates communication between DNN module 201 and other modules or systems. For example, interface module 310 establishes communication between DNN module 201 and an external database to receive data that can be used to train the DNN or input into the DNN to perform AI tasks. As another example, interface module 310 supports DNN module 201 in distributing the DNN to other systems, such as computing devices configured to apply the DNN to perform tasks.

[0078] Training module 320 trains the DNN using a training dataset. Training module 320 forms the training dataset. In an implementation where training module 320 trains the DNN to recognize objects in an image, the training dataset includes training images and training labels. The training labels describe the ground truth classification of objects in the training images. In some implementations, each label in the training dataset corresponds to an object in the training image. In some implementations, a portion of the training dataset may be used to initially train the DNN, and the remainder of the training dataset may be reserved as a validation subset used by validation module 340 to validate the performance of the trained DNN. The portion of the training dataset other than the tuning subset and validation subset can be used to train the DNN.

[0079] The training module 320 also determines the hyperparameters used to train the DNN. Hyperparameters are variables that specify the DNN training process. Hyperparameters are different from parameters internal to the DNN (e.g., filter weights). In some implementations, hyperparameters include variables that determine the architecture of the DNN, such as the number of hidden layers. Hyperparameters also include variables that determine how the DNN is trained, such as batch size, number of epochs, etc. The batch size defines the number of training samples to work on before updating the parameters of the DNN. The batch size is the same as or less than the number of samples in the training dataset. The training dataset can be divided into one or more batches. The number of epochs defines how many times the entire training dataset is passed forward and backward through the entire network. The number of epochs defines the number of times the deep learning algorithm works on the entire training dataset. An epoch means that each training sample in the training dataset has a chance to update the parameters internal to the DNN. An epoch can include one or more batches. The number of epochs can be 1, 5, 10, 50, 100, 500, 1000, or even larger.

[0080] The training module 320 defines the architecture of the DNN based on some hyperparameters, for example. The DNN architecture includes an input layer, an output layer, and multiple hidden layers. The input layer of the DNN can include tensors (e.g., multidimensional arrays) specifying attributes of the input image, such as the height, width, and depth of the input image (e.g., specifying the number of bits of color in the pixels of the input image). The output layer includes labels for the objects in the input layer. Hidden layers are layers between the input and output layers. Hidden layers include one or more convolutional layers and one or more other types of layers, such as pooling layers, fully connected layers, normalization layers, SoftMax, or logistic layers. The convolutional layers of the DNN abstract the input image into feature maps represented by tensors, which specify the feature map height, width, and channels (e.g., red, green, and blue images include 2 channels). Pooling layers are used to reduce the spatial volume of the input image after convolution. They are used between two convolutional layers. Fully connected layers involve weights, biases, and neurons. They connect neurons in one layer to neurons in another layer. They are used to classify images between different categories through training.

[0081] In defining the architecture of the DNN, the training module 320 also adds activation functions to the hidden or output layers. The activation function of a layer transforms the weighted sum of the layer's inputs into the layer's output. Activation functions can be, for example, ReLU, tangent, sine, or other types of activation functions.

[0082] After defining the architecture of the DNN in training module 320, training module 320 feeds the training dataset into the DNN. The training dataset includes multiple training samples. Examples of training samples include objects in images and their ground truth labels. Training module 320 modifies the parameters internal to the DNN (“DNN internal parameters”) to minimize the error between the labels of the training objects generated by the DNN and the ground truth labels of the objects. Internal parameters include the weights of the filters in the convolutional layers of the DNN. In some implementations, training module 320 uses a cost function to minimize the error.

[0083] The training module 320 can train the DNN for a predetermined number of periods. The number of periods is a hyperparameter defining how many times the deep learning algorithm works across the entire training dataset. One period means that each sample in the training dataset has a chance to update the internal parameters of the DNN. After the training module 320 has completed the predetermined number of periods, it can stop updating the parameters in the DNN. The DNN with the updated parameters is called the trained DNN.

[0084] Compression module 330 compresses the DNN. For example, compression module 330 can add pruning operations to the DNN layers to reduce computational complexity or memory usage. The pruning operation can trim the weight tensors of the DNN layer by changing one or more non-zero value weights of the layer to zero. The modification can be performed before, during, or after training. Weights can be pruned during training, during inference, or a combination of both. Compression module 330 can determine the sparsity rate of the DNN layer. The sparsity rate can be the ratio of the number of zero-value weights in the layer to the total number of weights. Compression module 330 can perform pruning operations until the sparsity rate of the DNN layer meets a target sparsity rate, such as 10%, 20%, 20%, 30%, 50%, etc.

[0085] In some implementations, the compression module 330 may select one or more layers in the DNN and modify each selected layer with a pruning operation. For example, the compression module 330 may select computationally complex layers, such as layers with large filters. For a pruning operation on a layer or a type of layer, the compression module 330 may determine a weight threshold that will not cause the DNN to lose accuracy beyond the accuracy loss constraint. The pruning operation may modify weights with absolute values ​​above the weight threshold to zero, while leaving other weights unchanged. Weight pruning can reduce memory storage because zero-value weights do not need to be stored. Furthermore, the number of operations in a layer can be reduced because the computation of zero-value weights can be skipped without affecting the layer's output. In some implementations, the compression module 330 may also measure the energy savings, final DNN accuracy, or layer-by-layer sparsity resulting from the pruning operation.

[0086] After compressing the DNN, the compression module 330 can fine-tune the DNN, for example, through a retraining process. The compression module 330 can fine-tune the DNN after the weights have been pruned. In some implementations, the fine-tuning process is a retraining or further training process. For example, after pruning the weights in the DNN, the compression module 330 can further train the DNN by feeding a training dataset into the DNN. The values ​​of the unpruned weights in the DNN can be modified based on the output of the DNN and the ground truth labels of the training samples in the training dataset. In some implementations, the values ​​of the pruned weights (i.e., zero) do not change during the fine-tuning process. For example, the compression module 330 can place a mask over the pruned weight block, and the mask can prevent the values ​​in the pruned weight block from being changed during the fine-tuning process. In other implementations, the values ​​of all weights (including the pruned weights) can be changed during the fine-tuning process. After one or more cycles of retraining and weight modification by the compression module 330, the compression module 330 can perform a new pruning process, for example, by selecting a weight block and pruning the selected weight block. In some implementations, the weight pruning process can be repeated multiple times before the fine-tuning process is complete.

[0087] In some implementations, the number of epochs in the fine-tuning process can differ from the number of epochs in the training process that determines the pre-pruned weights. For example, the fine-tuning process can have fewer epochs than the training process. In the examples, the number of epochs in the fine-tuning process can be relatively small, such as 2, 2, 3, 5, etc.

[0088] Validation module 340 validates the accuracy of the trained or compressed DNN. In some implementations, validation module 340 feeds samples from a validation dataset into the trained DNN and uses the DNN's output to determine model accuracy. In some implementations, the validation dataset may consist of some or all of the samples from the training dataset. Additionally or alternatively, the validation dataset includes additional samples beyond those in the training set. In some implementations, validation module 340 may determine an accuracy score that measures the precision, recall, or a combination of precision and recall of the DNN. Validation module 340 may determine the accuracy score using the following metrics: Precision = TP / (TP+FP) and Recall = TP / (TP+FN), where precision can be how many of the total number of predictions the DNN correctly made (TP or true positives) out of its predicted total (TP+FP or false positives), and recall can be how many of the total number of objects with the attribute in question correctly made (TP). The F-score (F-score = 2 * PR / (P + R)) unifies precision and recall into a single metric.

[0089] The validation module 340 can compare the accuracy score with a threshold score. In an example where the validation module 340 determines that the accuracy score of the augmented model is less than the threshold score, the validation module 340 instructs the training module 320 to retrain the DNN. In one implementation, the training module 320 can iteratively retrain the DNN until a stopping condition occurs, such as an accuracy measurement indicating that the DNN may be sufficiently accurate, or multiple training epochs have already occurred.

[0090] Compiler 350 compiles information from the DNN into executable instructions that can be executed, for example, by DNN accelerator 202, to perform neural network operations within the DNN. In some implementations, compiler 350 may generate a graph representing the DNN. This graph may include nodes and edges. Nodes may represent specific neural network operations in the DNN. Edges may connect two nodes and represent a connection between two corresponding neural network operations. In an example, an edge may encode a tensor flowing from one neural network operation to another. The tensor may be the output tensor of the first neural network operation and the input tensor of the second neural network operation. An edge may encode one or more properties of the tensor, such as size, shape, storage format, etc. Compiler 350 may use the graph to generate an executable DNN. For example, the compiler may generate computer program instructions (e.g., compiler descriptors) for executing the DNN. The instructions may be stored in registers associated with components of DNN accelerator 202.

[0091] The compiler 350 can also generate configuration descriptors specifying the operating modes of components in the DNN accelerator 202, such as one or more configuration descriptors specifying whether the DPU 230 operates in intensive or sparse mode. In intensive mode, the DPU 230 can execute the DNN layer without any sparsity acceleration. For example, the DPU 230 can perform computations on zero-value data elements without skipping them. In sparse mode, the DPU 230 can accelerate the execution of the DNN layer based on sparsity by skipping computations on zero-value data elements. The compiler 350 can determine whether to accelerate the layer based on weight sparsity, activation sparsity, or both. The compiler 350 can select a sparse mode for a layer from a set of sparse modes, including, for example, a combined sparse mode, an activation sparse mode, a weighted sparse mode, and a dense mode. In the combined sparse mode, the layer is accelerated based on both weight sparsity and activation sparsity. In the activation sparse mode, the layer is accelerated based on activation sparsity but not on weight sparsity. In the weighted sparse mode, the layer is accelerated based on weight sparsity but not on activation sparsity. In the dense mode, the layer is not accelerated based on sparsity. In some implementations (e.g., where the layer is executed by multiple DPUs 230), the compiler 350 can determine the sparse mode for all DPUs 230 executing the layer. Configuration descriptors can be provided to the DPUs 230, for example, through interface module 310.

[0092] In some implementations, compiler 350 can generate configuration descriptors that configure the operations of DPU 230 for approximating activation functions. For example, compiler 350 can generate one or more configuration descriptors specifying whether activation function units in DPU 230 operate in linear function mode or bypass mode. Compiler 350 can also program LUTs or saturation tables in DPU 230. For example, compiler 350 can program LUTs or saturation tables using data generated by activation function module 360. In some implementations, compiler 350 can reprogram LUTs or saturation tables during DNN execution. For example, after executing an activation function, compiler 350 can reprogram the LUT or saturation table to execute the next activation function.

[0093] Activation function module 360 ​​facilitates the approximation of activation functions in the DNN. In some implementations, activation function module 360 ​​can identify activation functions in the DNN, for example, based on a graph generated by compiler 350. For each activation function, activation function module 360 ​​can determine the input range of the activation function. In some implementations, activation function module 360 ​​can determine the precision of the input data for the activation function and determine the input range based on that precision. The input data can be computed in DNN layers, such as convolutional layers, fully connected layers, pooling layers, etc. In the example, the input data can be the result of convolution in the DNN.

[0094] The activation function module 360 ​​can also divide the input range into input segments. In some implementations (e.g., implementations where the precision is floating-point precision), the activation function module 360 ​​can divide the input range based on the exponent. Each input segment can correspond to a specific range of exponents. In the example, the first input segment can correspond to exponents from 0 to 16, the second input segment can correspond to exponents from 17 to 21, the third input segment can correspond to exponents from 22 to 25, and the fourth input segment can correspond to exponents greater than 25.

[0095] The activation function module 360 ​​can also determine the approximation method for each segment. In the example, the activation function module 360 ​​can determine that a linear approximation can be used for the first input segment, a range-folding approximation can be used for the second input segment, an output lookup approximation can be used for the third input segment, and a saturation approximation can be used for the fourth input segment. Using the linear approximation, one or more linear functions can be used to approximate the activation function. Using the range-folding approximation, the second input segment can be divided into sub-segments, each of which can be mapped to the first input segment, and the linear function used for the first input segment can be used for the sub-segments within the second input segment.

[0096] Activation function module 360 ​​can determine one or more linear functions that can approximate the activation function within a target accuracy. In some implementations, activation function module 360 ​​can determine the ULP, which is also referred to as a cell. The ULP can be used as the target accuracy for the first input segment. The ULP indicates the interval between two consecutive floating-point numbers. When ULP is 1, ULP can be a value represented by the least significant bit (i.e., the rightmost bit). ULP may be different for different exponents. For example, a ULP with an exponent of 15 can be approximately 0.0009765625, and a ULP with an exponent of 23 can be approximately 0.25. In the example, activation function module 360 ​​can determine that ULP = 1 for the first input segment of a periodic activation function.

[0097] In some implementations, the activation function module 360 ​​may determine a linear function for the first input element. In other implementations, the activation function module 360 ​​may determine multiple linear functions for the first input segment. Different linear functions can be used to approximate the activation function for different parts of the input segment. The linear function can be represented as... y=ax+b ,in a Represents the slope of a linear function. b Represents the intercept of a linear function. x This represents the input to a linear function, and y This represents the output of a linear function. The activation function module 360 ​​can determine the parameters of the linear function (e.g., intercept and slope). The compiler 350 can use these parameters as configuration parameters to program the LUT. The intercept and slope of the linear function can be stored as a single entry in the LUT.

[0098] In some implementations, the activation function module 360 ​​can determine whether the size of the LUT meets the LUT's area budget. When the LUT size exceeds the area budget, the activation function module 360 ​​can modify one or more target accuracies of one or more input segments within the input range based on the area budget, and redetermine the linear function for one or more input segments. In some implementations, the activation function module 360 ​​can also select a data precision for the LUT. The parameters of the linear function stored in the LUT will have the selected data precision. The activation function module 360 ​​can select the LUT's data precision based on the input or output of the activation function.

[0099] For output lookup approximation, activation function module 360 ​​can determine values ​​that can be used as approximate outputs of activation functions. Activation function module 360 ​​can provide values ​​to compiler 350 to program the LUT, which can be the same LUT storing the parameters of a linear function. Each value can take one entry from the LUT. Activation function module 360 ​​can determine multiple values ​​for the input segment using output lookup approximation. For saturation approximation, activation function module 360 ​​can determine a single value for the entire input segment. This value can be used as an approximate output of the activation function for all input data elements falling within the input segment. This value is called the saturation value. Activation function module 360 ​​can provide the saturation value to compiler 350 for programming the saturation table in PPE 260.

[0100] Data storage 370 stores data that is received, generated, used, or otherwise associated with DNN module 201. For example, data storage 370 stores datasets used by training module 320 and validation module 340. Data storage 370 may also store data generated by training module 320 and validation module 340, such as hyperparameters used to train the DNN, intrinsic parameters of the trained DNN (e.g., weights), data used for sparsity acceleration (e.g., sparse bitmaps), etc. Data storage 370 may store configuration parameters generated by activation function module 360. Figure 3 In one implementation, data storage 370 is a component of DNN module 201. In other implementations, data storage 370 may be external to DNN module 201 and communicate with DNN module 201 via a network.

[0101] Figure 4 The diagram illustrates a partition of an example input range 400 for an activation function according to various implementations. The input range of the activation function can depend on the precision of the input data elements. For illustrative purposes, the input range 400 can be used for FP32. In some implementations, the input range 400 is the range of all data elements with FP32 precision. The actual input data elements for the activation function used in a partial DNN or a DNN layer can be some, not all, of the data elements. Figure 4 As shown, the input range 400 is divided into four segments: segment 410, segment 420, segment 430, and segment 440. Each of the four segments is an input segment, that is, a portion of the input range 400. In some implementations, the input range 400 is divided based on the exponent of the data elements. For example, segment 410 includes data elements with exponents from 0 to 16, segment 420 includes data elements with exponents from 16 to 21, segment 430 includes data elements with exponents from 21 to 25, and segment 440 includes data elements with exponents greater than 25.

[0102] Figure 5 Another example partitioning of the input range 500 for activation functions according to various implementations is shown. For illustrative purposes, the input range 500 can be used for FP16. In some implementations, the input range 500 is the range of all data elements with FP16 accuracy. The actual input data elements used for the activation function of a partial DNN or DNN layer can be some, not all, of the data elements. Figure 5 As shown, the input range 500 is divided into two segments: segment 510 and segment 520. Each of the two segments is an input segment, that is, a portion of the input range 500. In some implementations, the input range 500 is divided based on the exponent of the data elements. For example, segment 510 includes data elements with exponents from 0 to 16, and segment 520 includes data elements with exponents greater than 16.

[0103] The input range of 400 or 500 can be divided into different numbers of segments, or it can be divided into segments with... Figure 4 and 5 The examples shown illustrate different ways of partitioning the input range. In some implementations, the partitioning of the input range into 400 or 500 can be handled by a DNN module (e.g., Figure 2 The DNN module 201 in the DNN module executes the function. The number of segments or the range of each segment in the input range can be determined based on the estimated accuracy approximated by the activation function of the segment. For example, when the estimated accuracy is below a threshold accuracy, the input range can be divided in different ways to increase the accuracy of the approximation of the activation function.

[0104] Figure 6 Example activation function unit 600 according to various embodiments is shown. Activation function unit 600 can, for example, perform the activation function in the DNN by means of an approximate activation function. Activation function unit 600 can be... Figure 2 An example of activation function unit 280 in [the example]. Figure 6 As shown, the activation function unit 600 includes an approximate control unit 610 and a calculation unit 620 (referred to as "calculation unit 620"). In other embodiments, the activation function unit 600 may include fewer or more components or different components. Furthermore, the functionality of the components attributed to the activation function unit 600 may be implemented by different components or different modules included in the activation function unit 600.

[0105] The approximate control unit 610 can control the operation of the calculation unit 620 used to calculate the output data elements of the activation function. For example... Figure 6As shown, the approximation control unit 610 includes a range module 615, an address module 630, a LUT 640, a saturation module 650, and a saturation table 660. In other embodiments, alternative configurations, different components, or additional components may be included in the approximation control unit 610. Furthermore, the functionality of the components attributed to the approximation control unit 610 may be implemented by different components or different modules, devices, or systems included in the approximation control unit 610.

[0106] exist Figure 6 In one implementation, the approximate control unit 610 receives a configuration signal 601 and a data signal 602. The configuration signal 601 can be received via a configuration bus. The configuration signal 601 may include one or more configuration descriptors, which may be provided by the DNN module 201. The configuration signal 601 can be sent to the activation function unit 600 via the configuration bus. The configuration signal 601 can program multiple activation function units (e.g., some or even all activation function units in a PPE array) in parallel.

[0107] Data signal 602 can be received via a data port (such as a data input port). Data signal 602 may include one or more input data elements of an activation function. In some embodiments, configuration signal 601 may trigger one or more operating cycles of approximation control unit 610 and calculation unit 620 for calculating an approximate output of the activation function using the input data elements in data signal 602. Input data elements may have a precision. In some embodiments, the activation function unit (e.g., range module 615 in approximation control unit 610) may convert the precision to different precisions. For example, approximation control unit 610 may convert FP32 data to FP16 data and vice versa. The precision may correspond to the input range of the activation function. The input range may be divided into a sequence of input segments, such as a first input segment, a second input segment, a third input segment, and a fourth input segment. Each input data element may fall into one of the input segments.

[0108] In an example operating loop for processing input data elements in data signal 602, range module 615 in approximation control unit 610 can identify input segments that include the input data elements. In some embodiments, range module 615 can compare the input data element with the minimum or maximum value of the input segment. Range module 615 can determine that the input data element falls within the input segment based on determining that the input data element is not greater than the maximum value or not less than the minimum value. In other embodiments, range module 615 can compare the exponent of the input data element with the minimum or maximum exponent of the input segment. Range module 615 can determine that the input data element falls within the input segment based on determining that the exponent of the input data element is not greater than the maximum exponent or not less than the minimum exponent. In some embodiments, range module 615 can examine multiple input segments of the activation function until an input segment including the input data element is found.

[0109] In an embodiment where the range module 615 determines that the input data element falls within a first input segment of the input range, the range module 615 can also identify a sub-segment within the first input segment into which the input data element falls. After identifying the sub-segment, the range module 615 can trigger the address module 630 to determine a LUT address, which indicates the address of a LUT entry in LUT 640. The LUT entry can store the intercept and slope of a linear function used to approximate the activation function of the sub-segment. In some embodiments, the address module 630 can determine the address based on one or more bits in the input data element. The address module 630 or the computation unit 620 can use this address to extract the intercept and slope from LUT 640. For example, LUT 640 can include multiple entries. Entries can correspond to different linear functions. Entries can have a specific address and can include the intercept and slope of the corresponding linear function. In some embodiments, the intercept and slope can be in a different data format than the input data element. For example, the input data element can be in FP32 data format, while the intercept and slope can be in FP16 or BF16 data format. In the example, an entry may include 32 bits, an intercept of 16 bits, and a slope of 16 bits. In some implementations, the total number of entries in the LUT 640 may be a power of 2, such as 2, 4, 8, 16, 32, 64, 128, etc.

[0110] In an implementation where the range module 615 determines that an input data element falls into the second input segment, the range module 615 can perform a range folding operation, wherein the range module 615 can map the input data element to a data element in the first input segment. Using the SIN function (e.g., Figure 8A Taking the SIN function shown as an example, the steps in the range folding operation can be represented as sin( x Xπ) = sin(( x -2)Xπ). In some implementations,x You can use floating-point mantissa m Sum of Indices e express: x = l. m X 2 e The range folding operation can be represented as sin( x Xπ) = sin(( x -2)Xπ)=sin((l). m X 2 e -1.0 X 2 e )Xπ)=sin((0) m X2 e )X π)=sin((1 ) m’ X 2 e’ )Xπ), where m’ and e’ These are the normalized mantissa and exponent, respectively. The range module 615 can perform this step once or multiple times to map the input data elements to the first input segment. For example, for the example SIN function, the range module 615 can repeat this step until... e’ The input data falls within the exponent range of the first input segment. In some implementations, the range module 615 may use additional mantissa bits when the input data element has higher precision. In an example where the input data element has FP32 precision and is converted to an FP16 value, the range module 615 may use the FP32 mantissa to obtain additional precision compared to the FP16 mantissa input.

[0111] In an embodiment where the range module 615 determines that the input data element falls into the third input segment, the range module 615 may trigger the address module 630 to determine the LUT address, which indicates the address of the LUT entry in the LUT 640. The LUT entry may store a value that is an approximate output of the activation function to be used as the input data element. In some embodiments, the address module 630 may determine the address based on one or more bits (e.g., one or more mantissa bits) in the input data element. The address module 630 or the calculation unit 620 may use the address to extract a value from the LUT 640. This value may be output from the activation function unit 600 as an approximate output of the activation function of the input data element.

[0112] In some implementations (e.g., where the fourth input segment is a saturation segment, i.e., a segment where the activation function can be approximated as a fixed value), after the range module 615 determines that an input data element falls into the fourth input segment, the range module 615 can trigger the saturation module 650 to identify a saturation value from the saturation table 660 based on the input data element. The saturation value will be used as an approximate output of the activation function for all input elements falling into the saturation segment. The saturation value can be a predetermined integer. In some implementations, the saturation table 660 can support multiple saturation segments. In some implementations, each saturation segment can have a separate entry in the saturation table 660. Each entry can include bits indicating the lower threshold (e.g., minimum) of the saturation segment, bits indicating the higher threshold (e.g., maximum) of the saturation segment, and the saturation value. In an example, the entry can include 48 bits: 16 bits for the lower threshold, 16 bits for the upper threshold, and 16 bits for the saturation value. The saturation module 650 can extract a fixed output from the saturation table 660 and provide the fixed output to the computation unit 620.

[0113] Each computation unit 620 includes a multiplier 670 and an adder 680. In other embodiments, computation unit 620 may include different, fewer, or more components. For example, computation unit 620 may include multiple multipliers or multiple accumulators. Computation unit 620 may have various operating modes. For example, computation unit 620 may have a linear function mode and a bypass mode. The operating mode of computation unit 620 may be configured by approximation control unit 610. For example, address module 630 may configure computation unit 620 to operate in linear function mode, while saturation module 650 may configure computation unit 620 to operate in bypass mode. In some embodiments, computation unit 620 may be in the same operating mode within the same operating cycle. In other embodiments, computation unit 620 may be in different operating modes within the same operating cycle.

[0114] In some implementations of the linear function mode (e.g., where data signal 602 is an input data element falling within a sub-segment of the first input segment), computation unit 620 can compute the output of the linear function as an approximate output of the activation function. In the example computation loop, multiplier 670 can receive an increment value and a slope value from LUT 640. The increment value can be the difference between the input data element of the activation function and the initial value of the sub-segment. The slope value can be the slope of the linear function. Multiplier 670 can compute the product of the increment value and the slope value. Adder 680 receives the product from multiplier 670 and accumulates the product with the intercept value to compute an approximate output data element of the activation function. The output of adder 680 can be emitted from computation unit 620 as a data element in the output signal 603 of the activation function through a data port (e.g., a data output port).

[0115] In other implementations of the linear function mode (e.g., where data signal 602 is an input data element falling into the third input segment), computation unit 620 may receive a value from LUT 640. Computation unit 620 may use this value as the intercept and zero as the slope. The output of adder 680 is the value itself. The output of adder 680 may be sent from computation unit 620 as a data element in the output signal 603 of the activation function via a data port (e.g., a data output port). In bypass mode, computation unit 620 may not perform any computation. Instead, computation unit 620 may send the saturation value received from saturation table 660 as a data element in output signal 603.

[0116] Figure 7 Pipeline 700 with approximate activation functions according to various embodiments is shown. Pipeline 700 may be constructed from PPE (such as...) Figure 2 PPE 260 in PPE) or PPE arrays (such as Figure 12 The activation function module in the PPE array 1200 is executed. Figure 7 In one implementation, pipeline 700 includes five loops: loop 710, loop 720, loop 730, loop 740, and loop 750. In other implementations, pipeline 700 may include fewer, more, or different loops. Pipeline 700 facilitates input in various formats, such as FP32, FP16, BF16, etc.

[0117] In loop 710, an FP32 input is received. The FP32 input may include input data elements in FP32 format. The input data elements are converted to FP16 data elements. In other embodiments, the input data elements may be converted to data elements of different precision, such as BF16. The FP16 data elements may be input to multiplexer 715. Multiplexer 715 may also receive another input, which can be generated by renormalizing the FP16 data elements, for example, when the input data elements fall into the second input segment of the activation function. Renormalization can map or convert the FP16 data elements into data elements falling into the first input segment of the activation function. The renormalized data elements are input to multiplexer 715. Multiplexer 715 can select one of the two inputs as its output. The output of multiplexer 715 is then provided to LUT configuration table 713 or saturation table 717. For example, in an implementation where the input data element or FP16 data element falls into the first, second, or third input segment of the activation function, the LUT configuration table 713 is used to determine the address of the LUT entry. Furthermore, the input increment can be determined, which may be the increment of the FP16 data element from the beginning of a sub-segment within the first input segment.

[0118] In an implementation where the input data element or FP16 data element falls into the fourth input segment of the activation function, the saturation table 717 can be used to obtain a predetermined value (“saturation value”). The saturation value is forwarded to loop 740.

[0119] In loop 720, the addresses determined by LUT configuration table 713 are used to extract entries from LUT 725. In some implementations (e.g., where input data elements or FP16 data elements fall into the first or second input segment of the activation function), entries may include the intercept and slope values ​​of the linear function. Both the intercept and slope values ​​can be in FP16 or BF16 format. In other implementations (e.g., where input data elements or FP16 data elements fall into the third input segment of the activation function), entries may include a single value. The intercept and slope of the linear segment can be extracted from the LUT based on the address. Input increments can be normalized in loop 720.

[0120] In loop 730, the intercept and slope values ​​are converted to FP32 format. Multiplier 735 multiplies the normalized increment by the slope value. The output of multiplier 735, the product of the normalized increment and slope value, can also be in FP32 format. The output of multiplier 735 is forwarded to loop 740.

[0121] In loop 740, accumulator 745 accumulates the intercept value and the output of multiplier 735 to calculate a sum. The sum is rounded. In an implementation where the FP32 input data element falls into the saturation segment, the data format of the saturation value is changed to FP32. The rounded sum or FP32 saturation value is forwarded to loop 750.

[0122] In cycle 750, the rounded sum or FP32 saturation value from the PPE array output is used as an approximate output of the nonlinear activation function. The approximate output of the nonlinear activation function can be an FP32 data element.

[0123] The various implementations described above involve using linear functions to approximate both the exponent and mantissa of the output data elements of the activation function. In other implementations, the mantissa of the output data elements of the activation function can be approximated, for example, by using a reciprocal function, a square root reciprocal function, etc., while the exponent of the output data elements can be determined using a LUT configuration table.

[0124] Figure 8A Curve 800 of the activation function is shown as an example representation according to various implementation schemes. For illustrative purposes, Figure 8A Curve 800 in the figure represents the sine (SIN) curve of the SIN function, which can be expressed as sin( z ),in zThis represents the input data element. In some implementations, the SIN function can be rescaled for the entire input range. In the example, rescaling can be represented as... x=z / π. Rescaling can be performed by the scaling unit in the PPE. With rescaling, the SIN function can be expressed as sin( x Xπ), which is derived from Figure 8A The periodic curve shown represents the sin(sin) function. x Xπ can be approximated by the activation function module in PPE.

[0125] exist Figure 8A In this implementation, curve 800 is divided into three segments corresponding to the input range of the activation function: segment 810, segment 820, and segment 830, which can be the first input segment, second input segment, and third input segment of the activation function, respectively. Segment 820 can have a larger range than segment 810. Figure 8A In the example shown, segment 810 includes one cycle (“loop”), while segment 820 includes eight cycles. In other examples, segment 810 or 820 may include fewer or more cycles.

[0126] The activation function can be approximated using different methods for segments 810, 820, and 830. In the example, the first part of curve 800 (i.e., the part of segment 810) can be approximated using one or more linear curves. The following combines... Figure 8B and 8C Describe some aspects of approximating segment 810. The second part of curve 800 (i.e., the part corresponding to segment 820) can be approximated by mapping the various parts of segment 820 onto segment 810. For example, a single period within segment 820 can be mapped to segment 810, and the activation function can be approximated using one or more linear curves used to approximate the first part of curve 800. The approximate output of the SIN function of segment 810 can be reused as the approximate output of the period within segment 820.

[0127] The third portion of curve 800 (i.e., the portion corresponding to segment 830) can be approximated using a LUT. The LUT can store the approximate output of the SIN function for segment 830. LUT entries (“LUT entries”) can be identified based on their addresses, which are determined based on the input data elements within segment 830. The LUT entries storing the approximate output of the SIN function for the input data elements can be read out.

[0128] Figure 8B A linear approximation of the first portion of curve 800 according to various embodiments is shown. Figure 8BIn the example shown, the first part of curve 800 is approximated by seven linear curves 810 A-510G representing seven linear functions. Segment 810 can be divided into seven subsegments, each corresponding to a different one of the seven linear curves 810 A-510G. These subsegments are called linear segments. For input data elements falling into a specific linear segment, the approximate output of the SIN function is the output of the corresponding linear function. The output of the linear function can be computed using the linear function and the intercept and slope of the input data elements. The intercept and slope of the linear function can be stored in a LUT. The entries of the LUT can include the intercept and slope of the linear function.

[0129] Figure 8C Example linear curves according to various implementation schemes are shown. Linear curves can be used to approximate activation functions for linear segments. Linear segments have... x 0 arrive x 1 . x 0 Scope x 0 It can represent the beginning of a linear segment (e.g., the minimum value in the linear segment), and x 1 This can indicate the end of a linear segment. A linear segment can be part of segment 810. A linear function can be represented as y=s( x - x 0 )+y i ,in y This represents the output of a linear function. s The slope (also called the "multiplier") of a linear function. x y represents the input of a linear function. i This represents the intercept (also known as the "offset") of a linear function.

[0130] although Figure 8A Not shown, but curve 800 may have a fourth portion corresponding to the fourth input segment of the activation function. The fourth input segment may include one or more saturation segments, for which the activation function can be approximated with a fixed value. Figure 8D Example saturation curves according to various implementations are shown. Saturation curves can be used to approximate activation functions for saturation segments. Saturation segments have a range from... x 2 arrive x 3 Scope x 2 This can represent the beginning of a saturation segment (e.g., the minimum value within a saturation segment), and x 3This can represent the end of a saturation range (e.g., the maximum value within the saturation range). For all input data elements falling within the saturation range, the output of the activation function can be a fixed value (such as...). y s The values ​​are approximate, despite differences in the input data elements. In some implementations, the fixed values ​​can be integers. These fixed values ​​can be referred to as saturation values ​​and can be stored in a saturation table.

[0131] Figure 9 The generation of LUT addresses for linear approximations of activation functions according to various embodiments is illustrated. LUT address generation can be performed by address module 900, which can be... Figure 6 An example of address module 630 in [the document]. For illustrative purposes, in [the document]... Figure 9 In one implementation, address module 630 generates LUT address 902 for a linear segment including input data element 905. Address module 900 includes precision reduction module 910, LUT configuration table 920, and accumulator 930. In other implementations, address module 900 may include fewer, more, or different components.

[0132] For illustrative purposes, input data element 905 has 32 bits: bits 0-31. Input data element 905 can be in FP32 data format. In other embodiments, input data element 905 can have a different number of bits and can be in different data formats. The most significant bit of input data element 905 (i.e., bit 31) can be the sign bit and encodes the sign of input data element 905. The next eight bits (i.e., bits 30-23) can be the exponent bits and encode the exponent of input data element 905. The other bits (i.e., bits 22-0) can be the mantissa bits and encode the mantissa of input data element 905.

[0133] The precision reduction module 910 converts the exponent bits of the input data element 905 from FP32 exponent bits to FP16 exponent bits. This conversion can be represented as: exponent (FP16) = exponent (FP32) - 112 In an implementation where input data element 905 is in FP16 data format, the precision reduction module 910 may not be required. The FP16 exponent bit calculated by the precision reduction module and the sign bit of input data element 905 are input into LUT configuration table 920 to determine the base address 901 of the linear segment including input data element 905. LUT configuration table 920 may contain one or more LUT configuration entries corresponding to one or more linear segments of an activation function. The number of linear segments associated with LUT configuration table 920 can be expressed as... I=2 n ,in nIt can be equal to 0, 1, 2, 3, 4, etc. The LUT configuration table 920 can be indexed based on the sign and exponent fields in FP16 format. In some implementations, each LUT configuration entry can have 16 bits, of which 6 bits are encoded. n Furthermore, the 10-bit encoding corresponds to the base address of the linear segment. LUT configuration table 920 can be a register.

[0134] The base address 901 can be extracted from the LUT configuration table 920 based on the sign bit and the FP16 exponent bit. The offset can be determined based on a predetermined number of the most significant bits in the mantissa of the input data element 905. Figure 9 The predetermined number is three. In other embodiments, the predetermined number can be other numbers. Accumulator 930 accumulates the base address 901 with the offset and calculates the LUT address 902, which includes the linear segment of input data element 905. LUT address 902 can be used to extract the entry in LUT 960, that is, the entry at LUT address 902. LUT 960 can be... Figure 6 An example of at least a portion of LUT 640. This entry may include the slope and intercept of the linear segment.

[0135] Figure 10 Approximations of the activation function based on output lookup according to various implementations are shown. The approximation can be performed using a LUT1010. For illustrative purposes, Figure 10 The input data element 1005 of the activation function is shown, which is an FP16 data element. Input data element 1005 includes a sign bit, five exponent bits, and ten mantissa bits. One or more mantissa bits can be used for approximation. Figure 10 In the implementation scheme, the two least significant mantissa bits are used for approximation. There are four different combinations of the least significant mantissa bits, listed in Table 1001. These four different combinations can correspond to the four values ​​listed in Table 1002. The values ​​in Table 1002 can be an approximate output of the activation function, and one value can be selected for each input data element based on the two least significant mantissa bits. Table 1002 can be part of LUT 1010. LUT 1010 can be... Figure 6 An example of LUT 640 is shown. In some implementations, the output lookup approximation can be used for the third input segment of the activation function.

[0136] Figure 11 The generation of an approximate LUT address for finding the output of an activation function, according to various implementation schemes, is illustrated. The LUT address generation can be performed by address module 1100, which can be... Figure 6 An example of address module 630 is provided. For illustrative purposes, address module 1130 is... Figure 10The input data element 1005 generates the LUT address. The address module 900 includes a LUT configuration table 1110 and an accumulator 1120. In other embodiments, the address module 900 may include fewer, more, or different components.

[0137] The sign and exponent bits of input data element 1005 are input into LUT configuration table 1110 to determine base address 1101. LUT configuration table 1110 may include one or more LUT configuration entries. LUT configuration table 920 may be indexed based on the sign and exponent fields in FP16 format. In some implementations, each LUT configuration entry may have 16 bits, of which 6 bits are encoded. n Furthermore, the 10-bit encoding corresponds to the base address of the linear segment. LUT configuration table 920 can be a register.

[0138] The base address 1101 can be extracted from the LUT configuration table 1110 based on the sign bit and the FP16 exponent bit. The offset can be determined based on the two least significant bits in the mantissa of the input data element 1005. The accumulator 930 can accumulate the base address 901 with the offset and calculate the LUT address 1102 of the input data element 1005. The LUT address 1102 can be used to extract entries from the LUT (e.g., LUT 1010). The entries can be located at LUT address 1102.

[0139] Figure 12 A PPE array 1200 according to various embodiments is shown. The PPE array 1200 may contain [various components]. Figure 2 An example of an array of PPE 260 is provided. The PPE array 1200 also includes a PPE 1210 (referred to as "PPE 1210") and an approximation control unit 1220 (referred to as "approximation control unit 1220"). The approximation control unit 1220 can be used as configuration logic for the PPE array 1200. The PPE 1210 can be on one or more compute data paths of the PPE array 1200. In other embodiments, the PPE array 1200 may include alternative configurations, different components, or additional components. For example, the PPE array 1200 may include different numbers of PPE 1210s or approximation control units 1220s. Furthermore, the functionality of the components attributed to the PPE array 1200 may be implemented by different components or different modules, devices, or systems included in the PPE array 1200. The PPE array 1200 may include... Figure 2 An example of an array of PPE 260.

[0140] Each PPE 1210 can receive input data and compute output data to be used as the output of an activation function. The output data can be an approximation of the output of a nonlinear activation function. In some implementations, the PPE may include one or more computation units and one or more register files. The computation units can be configured to execute linear functions, including linear functions used to approximate nonlinear activation functions. The computation units may include one or more multipliers and one or more accumulators. The register files in the PPE 1210 can be used to store data input to the PPE 1210, such as input data elements of the nonlinear activation function, the slope and intercept of the linear function approximating the nonlinear activation function, etc. The register files in the PPE 1210, or separate register files, can store data computed by the PPE 1210. The data can be output data elements of the nonlinear activation function.

[0141] The approximation control unit 1220 controls the operation of the PPE 1210 for approximating the activation function. An example of the approximation control unit 1220 is... Figure 6 The approximation control unit 610 is located within the approximation control unit 1220. In some embodiments, the approximation control unit 1220 can be configured to operate one or more PPEs 1210 associated with it. Figure 12 In the implementation scheme, each approximate control unit 1220 and Figure 12 The eight PPEs 1210 are associated with each other. In other embodiments, the approximation control unit 1220 may be associated with a different number of PPEs 1210. The approximation control unit 1220 may determine the operating mode of the PPEs 1210. For example, the approximation control unit 1220 may trigger the PPEs 1210 to operate in a linear function mode or a bypass mode. The approximation control unit 1220 may determine the operating mode of the PPEs 1210 based on the input data elements received by the approximation control unit 1220.

[0142] In an implementation where the approximation control unit 1220 selects the linear function mode, the approximation control unit 1220 can provide the PPE 1210 with the data required for executing the linear function, such as input data elements, increment values ​​of the input data elements, the slope of the linear function, the intercept of the linear function, other data, or some combination thereof. In an implementation where the approximation control unit 1220 selects the bypass mode, the approximation control unit 1220 can provide a fixed value to the PPE 1210 and cause the PPE 1210 to output a fixed value as the output data element of the activation function. Figure 12In one implementation, each approximation control unit 1220 is associated with eight PPEs 1210. In other implementations, the approximation control unit 1220 may control operation in fewer or more PPEs. In this example, each PPE 1210 may include its own approximation control unit 1220. The approximation control unit 1220 may be part of an activation function module in each PPE.

[0143] Figure 13 This is a flowchart illustrating method 1300 of an approximate DNN activation function according to various implementation schemes. Method 1300 can be derived from... Figure 2 The DNN system 200 or one or more of its components are used to perform the operation. (See reference...) Figure 13 The flowchart shown describes method 1300, but many other methods for approximating activation functions can be used alternatively. For example, the method can be changed... Figure 13 The execution order of the steps in the diagram. As another example, some steps can be changed, eliminated, or combined.

[0144] The DNN system 200 divides the input range of the input data elements of the activation function into multiple segments. In some implementations, the input range is the range of input data elements of the activation function. In some implementations, the input data elements of the activation function have a floating-point format, such as FP32.

[0145] The DNN system 200 applies a linear function 1320 to the input data elements in the first of multiple segments. In some implementations, the parameters of the linear function are stored in the entries of the LUT. The parameters can indicate the slope and intercept of the linear function.

[0146] The DNN system 200 computes an approximate output of the activation function of the first segment by applying a linear function to the input data elements in the first segment. In some implementations, the DNN system 200 reads one or more parameters of the linear function from a lookup table. The DNN system 200 computes an approximate output of the activation function of the first segment based on the one or more parameters. In some implementations, the activation function is a periodic function. The first segment is the period of the periodic function.

[0147] The DNN system 200 identifies sub-segments within the second segment out of more than 1340 segments. In some implementations, the exponent of input data elements falling into the second segment is greater than the exponent of input data elements falling into the first segment.

[0148] The DNN system 200 reuses the approximate output of the activation function of the first segment (1350) as the approximate output of the activation function of the sub-segment. In some implementations, the activation function is a periodic function. The first segment is a period of the periodic function. In some implementations, the sub-segment within the second segment is another period of the periodic function.

[0149] In some implementations, the DNN system 200 obtains an approximate output of the activation function for an input data element falling into a third segment of multiple segments from a lookup table. In some implementations, the DNN system 200 uses one or more bits of the mantissa of the input data element to determine an address in the lookup table. The DNN system 200 extracts an approximate output of the activation function for the input data element falling into the third segment from the determined address. In some implementations, the one or more bits are one or more least significant bits of the mantissa. In some implementations, the exponent of the input data element falling into the third segment is greater than the exponent of the input data elements in the first or second segments.

[0150] In some implementations, the DNN system 200 uses a predetermined integer as an approximate output of the activation function for the input data elements in the fourth segment of a plurality of segments. In some implementations, the exponent of the input data elements falling into the fourth segment is greater than the exponent of the input data elements falling into the third segment.

[0151] Figure 14 This is a block diagram of an example computing device 1400 according to various implementation schemes. In some implementation schemes, the computing device 1400 can be used as... Figure 2 At least a portion of the DNN system 200, such as the DNN module 201 in the DNN system 200. Multiple components in Figure 14 The components are shown as included in computing device 1400, but any one or more of these components may be omitted or copied to suit the application. In some embodiments, some or all of the components included in computing device 1400 may be attached to one or more motherboards. In some embodiments, some or all of these components are manufactured onto a single system-on-a-chip (SoC) die. Additionally, in various embodiments, computing device 1400 may not include... Figure 14 The computing device 1400 may include one or more components as shown, but may include interface circuitry for coupling to one or more components. For example, the computing device 1400 may not include display device 1406, but may include display device interface circuitry (e.g., connector and driver circuitry) to which display device 1406 may be coupled. In another set of examples, the computing device 1400 may not include audio input device 1418 or audio output device 1408, but may include audio input or output device interface circuitry (e.g., connector and support circuitry) to which audio input device 1418 or audio output device 1408 may be coupled.

[0152] Computing device 1400 may include processing device 1402 (e.g., one or more processing devices). Processing device 1402 processes electronic data from registers and / or memory to convert the electronic data into other electronic data that can be stored in registers and / or memory. Computing device 1400 may include memory 1404, which itself may include one or more memory devices, such as volatile memory (e.g., DRAM), non-volatile memory (e.g., read-only memory (ROM)), high-bandwidth memory (HBM), flash memory, solid-state memory, and / or hard disk drives). In some embodiments, memory 1404 may include memory sharing a die with processing device 1402. In some embodiments, memory 1404 includes one or more non-transitory computer-readable media storing instructions executable to perform operations for executing activation functions in a DNN, for example, as described above. Figure 13 Method 1300 described or combined with the above Figure 2 The described DNN system 200 (e.g., DNN module 201 or PPE 260) performs some operations. Instructions stored in one or more non-transitory computer-readable media can be executed by processing device 1402.

[0153] In some embodiments, computing device 1400 may include communication chip 1412 (e.g., one or more communication chips). For example, communication chip 1412 may be configured to manage wireless communication for transmitting data to and from computing device 1400. The term "wireless" and its derivatives can be used to describe circuits, devices, systems, methods, techniques, communication channels, etc., that can transmit data via a non-solid medium using modulated electromagnetic radiation. This term does not imply that the associated devices do not contain any wires, although they may not in some embodiments.

[0154] The 1412 communication chip can implement any of a variety of wireless standards or protocols, including but not limited to Institute of Electrical and Electronics Engineers (IEEE) standards, including Wi-Fi (IEEE 802.10 series), IEEE 802.16 standards (e.g., IEEE 802.16-2005 revision), Long Term Evolution (LTE) projects, and any revisions, updates, and / or amendments (e.g., Advanced LTE project, Ultra Mobile Broadband (UMB) project (also known as "3GPP2"), etc.). Broadband Wireless Access (BWA) networks compliant with IEEE 802.16 are often referred to as WiMAX networks. WiMAX is an acronym for Global Microwave Access Interoperability, a certification mark for products that have passed conformance and interoperability testing of the IEEE 802.16 standard. The 1412 communication chip can operate according to Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Universal Mobile Telecommunications System (UMTS), High Speed ​​Packet Access (HSPA), Evolved HSPA (E-HSPA), or LTE networks. Communication chip 1412 may operate according to Enhanced Data GSM Evolution (EDGE), GSM EDGE Radio Access Network (GERAN), Universal Terrestrial Radio Access Network (UTRAN), or Evolved UTRAN (E-UTRAN). Communication chip 1412 may operate according to Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Digital Enhanced Cordless Telecommunications (DECT), Evolved Data Optimization (EV-DO) and its derivatives, as well as any other wireless protocol designated as 3G, 4G, 5G, and above. In other embodiments, communication chip 1412 may operate according to other wireless protocols. Computing device 1400 may include antenna 1422 to facilitate wireless communication and / or receive other wireless communications (such as AM or FM radio transmissions).

[0155] In some implementations, communication chip 1412 can manage wired communications, such as electrical, optical, or any other suitable communication protocol (e.g., Ethernet). As described above, communication chip 1412 may include multiple communication chips. For example, a first communication chip 1412 may be dedicated to short-range wireless communications such as Wi-Fi or Bluetooth, and a second communication chip 1412 may be dedicated to longer-range wireless communications such as Global Positioning System (GPS), EDGE, GPRS, CDMA, WiMAX, LTE, EV-DO, or others. In some implementations, the first communication chip 1412 may be dedicated to wireless communications, and the second communication chip 1412 may be dedicated to wired communications.

[0156] The computing device 1400 may include a battery / power circuitry system 1414. The battery / power circuitry system 1414 may include one or more energy storage devices (e.g., batteries or capacitors) and / or circuitry for coupling components of the computing device 1400 to an energy source (e.g., AC line power) separate from the computing device 1400.

[0157] The computing device 1400 may include a display device 1406 (or a corresponding interface circuitry system, as described above). The display device 1406 may include any visual indicator, such as a head-up display, computer monitor, projector, touchscreen display, liquid crystal display (LCD), light-emitting diode display, or flat panel display.

[0158] The computing device 1400 may include an audio output device 1408 (or a corresponding interface circuitry, as described above). The audio output device 1408 may include any device that generates audible indicators, such as a speaker, headphones, or earphones.

[0159] The computing device 1400 may include an audio input device 1418 (or a corresponding interface circuitry system, as described above). The audio input device 1418 may include any device that generates a signal representing sound, such as a microphone, microphone array, or digital instrument (e.g., an instrument with a Musical Instrument Digital Interface (MIDI) output).

[0160] The computing device 1400 may include a GPS device 1416 (or a corresponding interface circuitry system, as described above). The GPS device 1416 may communicate with a satellite-based system and may receive the location of the computing device 1400, as is known in the art.

[0161] The computing device 1400 may include another output device 1410 (or a corresponding interface circuitry system, as described above). Examples of other output devices 1410 may include audio codecs, video codecs, printers, wired or wireless transmitters for providing information to other devices, or additional storage devices.

[0162] The computing device 1400 may include another input device 1420 (or a corresponding interface circuitry system, as described above). Examples of other input devices 1420 may include an accelerometer, a gyroscope, a compass, an image capture device, a keyboard, a cursor control device (such as a mouse, stylus, or touchpad), a barcode reader, a quick-response (QR) code reader, any sensor, or a radio frequency identification (RFID) reader.

[0163] The computing device 1400 can have any desired form factor, such as a handheld or mobile computer system (e.g., a cellular phone, smartphone, mobile internet device, music player, tablet computer, laptop computer, netbook computer, ultrabook computer, personal digital assistant (PDA), super mobile personal computer, etc.), desktop computer system, server or other networked computing component, printer, scanner, monitor, set-top box, entertainment control unit, vehicle control unit, digital camera, digital video recorder, or wearable computer system. In some embodiments, the computing device 1400 can be any other electronic device that processes data.

[0164] The following paragraphs provide various examples of the implementation schemes disclosed herein.

[0165] Example 1 provides a method for performing an activation function in a neural network, the method comprising: dividing an input range of input data elements of the activation function into multiple segments of the input data elements; applying a linear function to the input data elements in a first segment of the multiple segments; computing an approximate output of the activation function for the first segment based on the application of the linear function to the input data elements in the first segment; identifying sub-segments within a second segment of the multiple segments; mapping the sub-segments of the second segment to the first segment; and reusing the approximate output of the activation function for the first segment as an approximate output of the activation function for the sub-segments within the second segment.

[0166] Example 2 provides the method of Example 1, wherein calculating the approximate output of the activation function for the first segment includes reading one or more parameters of the linear function from a lookup table; and calculating the approximate output of the activation function for the first segment from the one or more parameters.

[0167] Example 3 provides the method described in Example 1 or 2, wherein the input data elements of the activation function have a floating-point format.

[0168] Example 4 provides the method of any one of Examples 1-3, wherein the activation function is a periodic function, and the first segment is the period of the periodic function.

[0169] Example 5 provides the method described in Example 4, wherein the sub-segment within the second segment is another period of the periodic function.

[0170] Example 6 provides the method of any one of Examples 1-5, further comprising obtaining from a lookup table an approximate output of the activation function for input data elements falling into a third segment of the plurality of segments.

[0171] Example 7 provides the method described in Example 6, wherein obtaining the approximate output of the activation function for the input data element falling into the third segment comprises: using one or more bits of the mantissa of the input data element to determine an address in the lookup table; and extracting the approximate output of the activation function for the input data element falling into the third segment from the determined address.

[0172] Example 8 provides the method described in Example 7, wherein the one or more bits are one or more least significant bits of the mantissa.

[0173] Example 9 provides the method described in Example 7 or 8, wherein the exponent of the input data element falling into the third segment is greater than the exponent of the input data element falling into the first segment or the second segment.

[0174] Example 10 provides the method of any one of Examples 6-9, further comprising using a predetermined integer as an approximate output of the activation function for input data elements within a fourth of the plurality of segments, wherein the input data elements of the activation function have a floating-point format.

[0175] Example 11 provides an apparatus comprising: a data port configured to receive input data elements of an activation function in a neural network, the range of which is divided into a first segment and a second segment comprising a plurality of sub-segments, the sub-segments within the second segment being mapped to the first segment; a lookup table configured to store parameters of linear functions that approximate at least a portion of the first segment; and a computation unit comprising a multiplier and an accumulator, the computation unit being configured to: receive the parameters from the lookup table and compute an approximate output of the activation function for the first segment based on the parameters, wherein the approximate output of the activation function for the first segment is reused as an approximate output of the activation function for the sub-segments within the second segment.

[0176] Example 12 provides the apparatus of Example 11, further comprising an address module configured to determine the address of the parameter in the lookup table based on an input data element falling into the first segment, wherein the parameter is extracted from the determined address.

[0177] Example 13 provides the apparatus described in Example 12, wherein the address module is further configured to convert another input data element falling within the sub-segment of the second segment into the input data element falling within the first segment.

[0178] Example 14 provides an apparatus as described in any one of Examples 11-13, wherein the input data elements of the activation function have a floating-point format.

[0179] Example 15 provides an apparatus as described in any one of Examples 11-14, wherein the activation function is a periodic function, and the first segment is a period of the periodic function, and the sub-segment within the second segment is another period of the periodic function.

[0180] Example 16 provides an apparatus as described in any one of Examples 11-15, wherein the lookup table is further configured to store values ​​of the activation function for the third segment of the range of the input data elements to be used as an approximate output of the activation function.

[0181] Example 17 provides the apparatus described in Example 16, further comprising an address module configured to determine an address in the lookup table based on one or more bits of the mantissa of the input data element falling in the third segment, wherein the computation unit is configured to output a value received from the lookup table based on the determined address, the value being an approximate output of the activation function for the input data element.

[0182] Example 18 provides the apparatus described in Example 17, wherein the one or more bits are one or more least significant bits of the mantissa.

[0183] Example 19 provides the apparatus described in Example 17 or 18, wherein the exponent of the input data element falling into the third segment is greater than the exponent of the input data element falling into the first segment or the second segment.

[0184] Example 20 provides an apparatus as described in any one of Examples 16-19, further comprising a saturation table configured to store predetermined integers as an approximate output of the activation function for input data elements falling within the range of the input data elements of the activation function, wherein the input data elements of the activation function have a floating-point format.

[0185] Example 21 provides an apparatus including a processing element configured to compute data elements by performing neural network operations in a neural network, the data elements being used as input data elements for an activation function in the neural network; and a PPE including a data port configured to receive the input data elements of the activation function from the processing element, the range of the input data elements of the activation function being divided into a first segment and a second segment comprising a plurality of sub-segments, the sub-segments within the second segment being mapped to the first segment; a lookup table configured to store parameters of a linear function approximating at least a portion of the first segment; and a computation unit including a multiplier and an accumulator, the computation unit being configured to receive the parameters from the lookup table and compute an approximate output of the activation function for the first segment from the parameters, wherein the approximate output of the activation function for the first segment is reused as an approximate output of the activation function for the sub-segments within the second segment.

[0186] Example 22 provides the apparatus described in Example 21, wherein the PPE further includes an address module configured to determine the address of the parameter in the lookup table based on an input data element falling into the first segment, wherein the parameter is extracted from the determined address.

[0187] Example 23 provides the apparatus described in Example 22, wherein the address module is further configured to convert another input data element falling in the sub-segment within the second segment into the input data element falling in the first segment.

[0188] Example 24 provides an apparatus as described in any one of Examples 21-23, wherein the lookup table is further configured to store values ​​of the activation function for the third segment of the range of the input data elements to be used as an approximate output of the activation function.

[0189] Example 25 provides the apparatus described in Example 24, wherein the PPE further includes a saturation table configured to store predetermined integers as an approximate output of the activation function for input data elements falling within the range of the input data elements of the activation function in a fourth segment, wherein the input data elements of the activation function have a floating-point format.

[0190] The following paragraphs provide additional examples of the implementation schemes disclosed herein.

[0191] Example 1 provides a method for performing an activation function in a neural network, the method comprising: dividing an input range of input data elements of the activation function into a plurality of segments of the input data elements; computing an approximate output of the activation function for a first segment by applying a linear function to a first segment of the plurality of segments; identifying subsegments within a second segment of the plurality of segments; mapping the subsegments of the second segment to the first segment; and reusing the approximate output of the activation function for the first segment as an approximate output of the activation function for the subsegments within the second segment.

[0192] Example 2 provides the method of Example 1, wherein calculating the approximate output of the activation function for the first segment includes reading one or more parameters of the linear function from a lookup table; and calculating the approximate output of the activation function for the first segment from the one or more parameters.

[0193] Example 3 provides the method described in Example 1 or 2, wherein the input data elements of the activation function have a floating-point format.

[0194] Example 4 provides the method of any one of Examples 1-3, wherein the activation function is a periodic function, and the first segment is the period of the periodic function.

[0195] Example 5 provides the method described in Example 4, wherein the sub-segment within the second segment is another period of the periodic function.

[0196] Example 6 provides the method of any one of Examples 1-5, further comprising obtaining from a lookup table an approximate output of the activation function for input data elements falling into a third segment of the plurality of segments.

[0197] Example 7 provides the method described in Example 6, wherein obtaining the approximate output of the activation function for the input data element falling into the third segment comprises: using one or more bits of the mantissa of the input data element to determine an address in the lookup table; and extracting the approximate output of the activation function for the input data element falling into the third segment from the determined address.

[0198] Example 8 provides the method described in Example 7, wherein the one or more bits are one or more least significant bits of the mantissa.

[0199] Example 9 provides the method described in Example 7 or 8, wherein the exponent of the input data element falling into the third segment is greater than the exponent of the input data element falling into the first segment or the second segment.

[0200] Example 10 provides the method of any one of Examples 6-9, further comprising using a predetermined integer as an approximate output of the activation function for input data elements within a fourth of the plurality of segments, wherein the input data elements of the activation function have a floating-point format.

[0201] Example 11 provides one or more non-transitory computer-readable media storing instructions executable to perform operations for performing an activation function in a neural network, the operations including: dividing an input range of input data elements of the activation function into a plurality of segments of the input data elements; computing an approximate output of the activation function for a first segment by applying a linear function to a first segment of the plurality of segments; identifying subsegments within a second segment of the plurality of segments; mapping the subsegments of the second segment to the first segment; and reusing the approximate output of the activation function for the first segment as an approximate output of the activation function for the subsegments within the second segment.

[0202] Example 12 provides one or more non-transitory computer-readable media as described in Example 11, wherein computing the approximate output of the activation function for the first segment includes: reading one or more parameters of the linear function from a lookup table; and computing the approximate output of the activation function for the first segment from the one or more parameters.

[0203] Example 13 provides one or more non-transitory computer-readable media as described in Example 11 or 12, wherein the input data elements of the activation function have a floating-point format.

[0204] Example 14 provides one or more non-transitory computer-readable media as described in any one of Examples 11-13, wherein the activation function is a periodic function, and the first segment is the period of the periodic function.

[0205] Example 15 provides one or more non-transitory computer-readable media as described in Example 14, wherein the sub-segment within the second paragraph is another period of the periodic function.

[0206] Example 16 provides one or more non-transitory computer-readable media as described in any one of Examples 11-15, wherein the operation further includes obtaining from a lookup table an approximate output of the activation function for an input data element falling into a third segment of the plurality of segments.

[0207] Example 17 provides one or more non-transitory computer-readable media as described in Example 16, wherein obtaining the approximate output of the activation function for the input data element falling within the third segment comprises: using one or more bits of the mantissa of the input data element to determine an address in the lookup table; and extracting the approximate output of the activation function for the input data element falling within the third segment from the determined address.

[0208] Example 18 provides one or more non-transitory computer-readable media as described in Example 17, wherein the one or more bits are one or more least significant bits of the mantissa.

[0209] Example 19 provides one or more non-transitory computer-readable media as described in Example 17 or 18, wherein the exponent of the input data element falling in the third paragraph is greater than the exponent of the input data element falling in the first or second paragraph.

[0210] Example 20 provides one or more non-transitory computer-readable media as described in any one of Examples 16-19, wherein the operation further includes using a predetermined integer as an approximate output of the activation function for input data elements within a fourth of the plurality of segments, wherein the input data elements of the activation function have a floating-point format.

[0211] Example 21 provides an apparatus comprising: a computer processor for executing computer program instructions; and a non-transitory computer-readable storage medium for storing the computer program instructions, which are executable by the computer processor to perform operations for performing an activation function in a neural network, the operations including: dividing an input range of input data elements of the activation function into a plurality of segments of the input data elements; computing an approximate output of the activation function for a first segment by applying a linear function to a first segment of the plurality of segments; identifying sub-segments within a second segment of the plurality of segments; mapping the sub-segments of the second segment to the first segment; and reusing the approximate output of the activation function for the first segment as an approximate output of the activation function for the sub-segments within the second segment.

[0212] Example 22 provides the apparatus described in Example 21, wherein calculating the approximate output of the activation function for the first segment includes: reading one or more parameters of the linear function from a lookup table; and calculating the approximate output of the activation function for the first segment from the one or more parameters.

[0213] Example 23 provides the apparatus described in Example 21 or 22, wherein the activation function is a periodic function, and the first segment is a period of the periodic function, and the sub-segment within the second segment is another period of the periodic function.

[0214] Example 24 provides an apparatus as described in any one of Examples 21-23, wherein the operation further includes: determining an address in the lookup table using one or more bits of the mantissa of an input data element falling into a third segment of the plurality of segments; and extracting from the determined address the approximate output of the activation function for the input data element falling into the third segment.

[0215] Example 25 provides an apparatus as described in any one of Examples 21-24, wherein the operation further includes: using a predetermined integer as an approximate output of the activation function for input data elements within a fourth of the plurality of segments, wherein the input data elements of the activation function have a floating-point format.

[0216] The above description of the embodiments shown in this disclosure, including the content described in the abstract, is not intended to be exhaustive or to limit this disclosure to the precise forms disclosed. While specific embodiments and examples of this disclosure have been described herein for illustrative purposes, various equivalent modifications are possible within the scope of this disclosure, as will be recognized by those skilled in the art. These modifications can be made to this disclosure based on the above detailed description.

Claims

1. A method for performing an activation function in a neural network, the method comprising: The input range of the input data elements of the activation function is divided into multiple segments of the input data elements; Apply a linear function to the input data elements in the first of the plurality of segments; An approximate output of the activation function for the first segment is calculated by applying a linear function to the input data elements in the first segment; Identify a sub-segment within the second segment of the plurality of segments; as well as The approximate output of the activation function for the first segment is reused as the approximate output of the activation function for the subsegments within the second segment.

2. The method according to claim 1, wherein, Calculating the approximate output of the activation function for the first segment includes: Read one or more parameters of the linear function from the lookup table; and The approximate output of the activation function for the first segment is calculated based on the one or more parameters.

3. The method according to claim 1, wherein, The input data elements of the activation function are in floating-point format.

4. The method according to claim 1, wherein, The activation function is a periodic function, and the first segment is the period of the periodic function.

5. The method according to claim 4, wherein, The sub-segment within the second segment is another period of the periodic function.

6. The method according to claim 1, further comprising: Obtain an approximate output of the activation function for the input data element falling into the third segment of the plurality of segments from the lookup table.

7. The method according to claim 6, wherein, Obtaining the approximate output of the activation function for the input data elements falling within the third segment includes: The address in the lookup table is determined using one or more bits of the mantissa of the input data element; and Extract the approximate output of the activation function for the input data element falling into the third segment from the determined address.

8. The method according to claim 7, wherein, The one or more bits are one or more least significant bits of the mantissa.

9. The method according to claim 7, wherein, The index of the input data element falling into the third segment is greater than the index of the input data element falling into the first segment or the second segment.

10. The method of claim 6, further comprising: A predetermined integer is used as an approximate output of the activation function for the input data elements within the fourth segment of the plurality of segments. The input data elements of the activation function are in floating-point format.

11. An apparatus comprising: A data port is configured to receive input data elements of an activation function in a neural network, wherein the range of the input data elements of the activation function is divided into a first segment and a second segment comprising multiple sub-segments, and sub-segments within the second segment are mapped to the first segment; A lookup table, configured to store parameters of a linear function that approximates at least a portion of the first segment; as well as A computation unit, comprising a multiplier and an accumulator, is configured as follows: Receive the parameters from the lookup table, and Calculate the approximate output of the activation function for the first segment based on the parameters. The approximate output of the activation function for the first segment is reused as the approximate output of the activation function for the sub-segment within the second segment.

12. The apparatus of claim 11, further comprising: The address module is configured to determine the address of the parameter in the lookup table based on the input data element falling into the first segment. Specifically, the parameters are extracted from the determined address.

13. The apparatus according to claim 12, wherein, The address module is further configured to convert another input data element falling into the sub-segment within the second segment into the input data element falling into the first segment.

14. The apparatus according to any one of claims 11-13, wherein, The input data elements of the activation function are in floating-point format.

15. The apparatus according to any one of claims 11-14, wherein, The activation function is a periodic function, and the first segment is a period of the periodic function, and the sub-segment within the second segment is another period of the periodic function.

16. The apparatus according to any one of claims 11-15, wherein, The lookup table is further configured to store the approximate output value of the activation function for the third segment of the range of the input data elements to be used for the activation function.

17. The apparatus of claim 16, further comprising: The address module is configured to determine the address in the lookup table based on one or more bits of the mantissa of the input data element falling within the third segment. The computing unit is configured to output a value received from the lookup table based on a determined address, the value being an approximate output of the activation function for the input data element.

18. The apparatus according to claim 17, wherein, The one or more bits are the one or more least significant bits of the mantissa.

19. The apparatus according to claim 17 or 18, wherein, The index of the input data element falling into the third segment is greater than the index of the input data element falling into the first segment or the second segment.

20. The apparatus according to any one of claims 16-19, further comprising: 、 A saturation table is configured to store predetermined integers as approximate outputs of the activation function for input data elements falling within the range of the input data elements of the activation function in a fourth segment, wherein the input data elements of the activation function have a floating-point format.

21. An apparatus comprising: A processing element configured to compute data elements by performing neural network operations in a neural network, the data elements being used as input data elements for an activation function in the neural network; as well as Post-processing elements, including: A data port is configured to receive the input data element of the activation function from the processing element, wherein the range of the input data element of the activation function is divided into a first segment and a second segment comprising multiple sub-segments, and sub-segments within the second segment are mapped to the first segment. A lookup table, configured to store parameters of a linear function that approximates at least a portion of the first segment, and A computation unit, comprising a multiplier and an accumulator, is configured to receive the parameters from the lookup table and, based on the parameters, compute an approximate output of the activation function for the first segment. The approximate output of the activation function for the first segment is reused as the approximate output of the activation function for the sub-segment within the second segment.

22. The apparatus according to claim 21, wherein, The post-processing element further includes: The address module is configured to determine the address of the parameter in the lookup table based on the input data element falling into the first segment. Specifically, the parameters are extracted from the determined address.

23. The apparatus according to claim 22, wherein, The address module is further configured to convert another input data element falling into the sub-segment within the second segment into the input data element falling into the first segment.

24. The apparatus according to any one of claims 21-23, wherein, The lookup table is also configured to store the approximate output value of the activation function for the third segment of the range of the input data elements to be used for the activation function.

25. The apparatus according to claim 24, wherein, The post-processing element further includes: A saturation table is configured to store predetermined integers as approximate outputs of the activation function for input data elements falling within the range of the input data elements of the activation function in a fourth segment, wherein the input data elements of the activation function have a floating-point format.

Images