Deep neural network accelerator with multifunctional data processing unit

Abstract

A deep neural network (DNN) accelerator may include a data processing unit (DPU) with different types of processing elements (PEs). Some PEs may be multifunctional and include components for performing different types of neural network operations. For instance, a first PE in the DPU may include a multiply-accumulate (MAC) unit coupled with a multiplexer. The multiplexer may select data from an activation and a weight and a multiplier in the MAC unit may compute a product of the selected data with the activation. The first PE may be used for performing computations in convolutions and layer normalization operations. A second PE in the DPU may include a MAC unit and a comparator. The comparator may compute a maximum or minimum value from a plurality of data elements. The second PE may be used for computations in convolutions and SoftMax operations. The DPU may include other types of PEs.

Description

DEEP NEURAL NETWORK ACCELERATOR WITH MULTIFUNCTIONAL DATA PROCESSING UNITTechnical Field

[0001] This disclosure relates generally to neural networks (also referred to as "deep neural networks" or "DNN"), and more specifically, DNN accelerators with multifunctional data processing units (DPUs).Background

[0002] DNNs are used extensively for a variety of artificial intelligence (Al) applications ranging from computer vision to speech recognition and natural language processing due to their ability to achieve high accuracy. Transformer models, such as transformer-based large language models (LLMs), are a type of DNNs that has attracted lots of attention due to their abilities to achieve general-purpose language generation and understanding. Transformer models usually include layer normalization operations and SoftMax operations. Layer normalization operations and SoftMax operations may occur repeatedly in the encoder stacks, decoder stacks, or attention blocks of a transformer model.Brief Description of the Drawings

[0003] Embodiments will be readily understood by the following detailed description in conjunction with the accompanying drawings. To facilitate this description, like reference numerals designate like structural elements. Embodiments are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings.

[0004] FIG. 1 illustrates an example DNN, in accordance with various embodiments.

[0005] FIG. 2 illustrates an example convolution, in accordance with various embodiments.

[0006] FIG. 3 illustrates an example channel-wise reduction operation, in accordance with various embodiments.

[0007] FIG. 4 illustrates an example elementwise operation, in accordance with various embodiments.

[0008] FIG. 5 is a block diagram of a DNN system, in accordance with various embodiments.

[0009] FIG. 6 is a block diagram of a DNN module, in accordance with various embodiments.

[0010] FIG. 7 illustrates an example processing element (PE) with a multiply-accumulate (MAC) unit, in accordance with various embodiments.

[0011] FIG. 8 illustrates an example multifunctional PE, in accordance with various embodiments.

[0012] FIG. 9 illustrates another example multifunctional PE, in accordance with various embodiments.

[0013] FIG. 10 illustrates a data processing cell, in accordance with various embodiments.

[0014] FIG. 11 illustrates an example sparse cell array, in accordance with various embodiments.

[0015] FIG. 12 illustrates example PE columns in a multifunctional DPU, in accordance with various embodiments.

[0016] FIG. 13 illustrates an example comparator tree in a multifunctional DPU, in accordance with various embodiments.

[0017] FIG. 14 illustrates a maximum operation on integer data elements, in accordance with various embodiments.

[0018] FIG. 15 illustrates a maximum operation on floating-point data elements, in accordance with various embodiments.

[0019] FIG. 16 illustrates an example elementwise multiplication on quantized data, in accordance with various embodiments.

[0020] FIG. 17 illustrates an example elementwise multiplication on non-quantized data, in accordance with various embodiments.

[0021] FIG. 18 illustrates a method of executing a layer normalization operation, in accordance with various embodiments.

[0022] FIG. 19 illustrates a method of executing a SoftMax operation, in accordance with various embodiments.

[0023] FIG. 20 is a block diagram of an example computing device, in accordance with various embodiments.Detailed DescriptionOverview

[0024] The last decade has witnessed a rapid rise in Al based data processing, particularly based on DNNs. DNNs are widely used in the domains of computer vision, speechrecognition, innage, and video processing mainly due to their ability to achieve beyond human-level accuracy. The significant improvements in DNN model size and accuracy coupled with the rapid increase in computing power of execution platforms have led to the adoption of DNN applications even within resource constrained mobile and edge devices that have limited energy availability.

[0025] A DNN layer may include one or more deep learning operations (also referred to as "neural network operations"), such as convolution, layer normalization operation, SoftMax operation, elementwise operation, pooling operation, linear operation, nonlinear operation, and so on. Input or output data of deep learning operations may be arranged in data structures called tensors. A tensor is a data structure having multiple elements across one or more dimensions. Example tensors include a vector (which is a one-dimensional (ID) tensor) and a matrix (which is a two-dimensional (2D) tensor). There can also be three-dimensional (3D) tensors, four-dimensional (4D) tensors, and even higher dimensional tensors. A dimension of a tensor may correspond to an axis, e.g., an axis in a coordinate system. The size of a dimension may correspond to or equal to the number of data points along the axis. The dimensions and sizes of the dimensions of a tensor may define the shape of the tensor. The shape of a tensor may be represented by a sequence of numbers, where every number indicates the size of a different dimension. The order of the numbers may indicate the order of the dimension, i.e., the order of the axes, in the tensor. A DNN layer may receive one or more input tensors and compute an output tensor from the one or more input tensors. In some examples, an input tensor or output tensor may be a 3D tensor, in which a dimension is a channel dimension, and the othertwo dimensions are spatial dimensions. For a convolution layer, the input tensors include an activation tensor (also referred to as "input feature map (IFM)") including one or more activations (also referred to as "input elements") and a weight tensor. The weight tensor may be a kernel (a 2D weight tensor), a filter (a 3D weight tensor), or a group of filters (a 4D weight tensor).

[0026] Many currently available DNN accelerators are optimized to enhance processing speed and efficiency for convolutions. For instance, many DNN systems include DPUs that can execute MAC operations in convolution and support acceleration of MAC operations based on activation and weight sparsity. Not all neural network operations can be executed by these DPUs. For instance, some neural network operations are implemented as software layers on digital signal processor (DSP) cores in DNN accelerators. An example DSP is theStreaming Hybrid Architecture Vector Engine (SHAVE) processor. However, as many DSP cores run at a lower frequency and provide fewer compute resources, the efficiency of these neural network operations can be impacted.

[0027] For example, reduction operations (such as channel-wise sum computation, variance computation, mean computation, maximum value computation, minimum value computation, etc.) and elementwise operations are usually executed serially on the DSP core. The reduction operations and elementwise operations are present in layer normalization and SoftMax, which are becoming important for emerging DNN models such as transformers and graph neural networks. Also, dynamic quantization, which is often used for online performance optimization, also includes reduction operations and elementwise operations. However, these operations, when implemented on DSPs, can become a performance bottleneck and a major showstopper for DNN efficiency. Furthermore, running the operations on DSP cores can incur programming overhead as custom kernels need to be written and optimized on these DSP cores. The programming overhead can further reduce efficiency.

[0028] Embodiments of the present disclosure may improve on at least some of the challenges and issues described above by providing DNN accelerators with multifunctional DPUs. An example multifunctional DPU includes one or more multifunctional PEs. A multifunctional PE can perform MAC operations in convolutions as well as other types of computations present in other types of neural network operations. The computations include channel-wise reduction computations, elementwise computations, and other types of computations in emerging DNN models such as transformer models and graph neural networks. The multifunctional DPU may also support multi-precision computations on various data types, such as quantized data, non-quantized data, and so on. By mapping all these computations on a high-performance compute array (e.g., the multifunctional DPU), the present disclosure can improve the efficiency of executing DNN models including transformer models and graph neural networks.

[0029] In various embodiments of the present disclosure, a DNN accelerator may include one or more DPUs, each of which may include PEs of different types. Some PEs may be multifunctional and include components for performing computations in different types of neural network operations. Examples of these computations include channel-wise sum computation, variance computation, mean computation, maximum value computation,minimum value computation, other types of computations, or some combination thereof. In an example, a first PE in the DPU may include a multiply-accumulate (MAC) unit coupled with a multiplexer. The multiplexer may select data from an activation and a weight and a multiplier in the MAC unit may compute a product of the selected data with the activation. The first PE may be used for performing computations in convolutions and layer normalization operations. A second PE in the DPU may include a MAC unit and a comparator. The comparator may compute a maximum or minimum value from a plurality of data elements. The second PE may be used for computations in convolutions and SoftMax operations. The DPU may include other types of PEs.

[0030] In some embodiments, the DPU includes a PE array, where PEs are arranged in one or more columns and one or more rows. In an example, PEs of the same type may be arranged in the same column or the same row, while PEs of different types may be arranged in different columns or different rows. Such a structure may facilitate loading data into the DPU or draining data from the DPU at a row or column level. The DNN accelerator may be coupled to a DNN module. The DNN module may facilitate executions of neural network operations by the DNN accelerator. In some embodiments, the DNN module may identify neural network operations in DNNs and assign workloads of executing the identified operations to PEs in the DPU. As an example, the DNN module may assign square or mean computations in layer normalization operations to the first PE or other PEs having the same or similar architecture as the first PE. As another example, the DNN module may assign maximum or minimum computations in SoftMax operations to the second PE or other PEs having the same or similar architecture as the second PE. As yet another example, the DNN module may assign MAC operations in convolutions to PEs with MAC units. The DNN module may also program layer descriptors (e.g., configurable registers) with the corresponding operator type. For instance, the DNN module may generate configuration parameters that configure PEs in the DPU to perform the corresponding neural network operation. In some embodiments, the DNN module may generate a configuration parameter for the multiplexer in the first PE for configuring the multiplexer to select the weight as opposed to the activation (or select the activation as opposed to the weight). In some embodiments (e.g., embodiments where comparator may have a maximum mode for outputting maximum values and a minimum mode for outputting minimum values), theDNN module may generate another configuration parameter for the comparator in the second PE for configuring the operation mode of the comparator.

[0031] For purposes of explanation, specific numbers, materials and configurations are set forth in order to provide a thorough understanding of the illustrative implementations. However, it will be apparent to one skilled in the art that the present disclosure may be practiced without the specific details or / and that the present disclosure may be practiced with only some of the described aspects. In other instances, well known features are omitted or simplified in order not to obscure the illustrative implementations.

[0032] Further, references are made to the accompanying drawings that form a part hereof, and in which is shown, by way of illustration, embodiments that may be practiced. It is to be understood that other embodiments may be utilized, and structural or logical changes may be made without departing from the scope of the present disclosure. Therefore, the following detailed description is not to be taken in a limiting sense.

[0033] Various operations may be described as multiple discrete actions or operations in turn, in a manner that is most helpful in understanding the claimed subject matter. However, the order of description should not be construed as to imply that these operations are necessarily order dependent. In particular, these operations may not be performed in the order of presentation. Operations described may be performed in a different order from the described embodiment. Various additional operations may be performed or described operations may be omitted in additional embodiments.

[0034] For the purposes of the present disclosure, the phrase "A or B" or the phrase "A and / or B" means (A), (B), or (A and B). For the purposes of the present disclosure, the phrase "A, B, or C" or the phrase "A, B, and / or C" means (A), (B), (C), (A and B), (A and C), (B and C), or (A, B, and C). The term "between," when used with reference to measurement ranges, is inclusive of the ends of the measurement ranges.

[0035] The description uses the phrases "in an embodiment" or "in embodiments," which may each refer to one or more of the same or different embodiments. The terms "comprising," "including," "having," and the like, as used with respect to embodiments of the present disclosure, are synonymous. The disclosure may use perspective-based descriptions such as "above," "below," "top," "bottom," and "side" to explain various features of the drawings, but these terms are simply for ease of discussion, and do not imply a desired or required orientation. The accompanying drawings are not necessarily drawn toscale. Unless otherwise specified, the use of the ordinal adjectives "first," "second," and "third," etc., to describe a common object, merely indicates that different instances of like objects are being referred to and are not intended to imply that the objects so described must be in a given sequence, either temporally, spatially, in ranking or in any other manner.

[0036] In the following detailed description, various aspects of the illustrative implementations will be described using terms commonly employed by those skilled in the art to convey the substance of their work to others skilled in the art.

[0037] The terms "substantially," "close," "approximately," "near," and "about," generally refer to being within + / - 20% of a target value as described herein or as known in the art.Similarly, terms indicating orientation of various elements, e.g., "coplanar," "perpendicular," "orthogonal," "parallel," or any other angle between the elements, generally refer to being within + / - 5-20% of a target value as described herein or as known in the art.

[0038] In addition, the terms "comprise," "comprising," "include," "including," "have," "having" or any other variation thereof, are intended to cover a non-exclusive inclusion. For example, a method, process, device, or DNN accelerator that comprises a list of elements is not necessarily limited to only those elements but may include other elements not expressly listed or inherent to such method, process, device, or DNN accelerators. Also, the term "or" refers to an inclusive "or" and not to an exclusive "or."

[0039] The systems, methods and devices of this disclosure each have several innovative aspects, no single one of which is solely responsible for all desirable attributes disclosed herein. Details of one or more implementations of the subject matter described in this specification are set forth in the description below and the accompanying drawings.Example DNN

[0040] FIG. 1 illustrates an example DNN 100, in accordance with various embodiments. The DNN 100 may be executed by a DNN accelerator, e.g., the DNN accelerator 302 in FIG. 3. For the purpose of illustration, the DNN 100 includes a sequence of layers comprising a plurality of convolutional layers 110 (individually referred to as "convolutional layer 110"), a plurality of pooling layers 120 (individually referred to as "pooling layer 120"), and a plurality of fully- connected layers 130 (individually referred to as "fully-connected layer 130"). In other embodiments, the DNN 100 may include fewer, more, or different layers. For instance, the DNN 100 may include layer normalization layers, SoftMax layers, and so on. In an execution of the DNN 100, the layers of the DNN 100 execute tensor computation that includes manytensor operations, such as matrix multiplications, convolutions (e.g., multiply-accumulate (MAC) operations, etc.), pooling operations, elementwise operations (e.g., elementwise addition, elementwise multiplication, etc.), other types of tensor operations, or some combination thereof.

[0041] The convolutional layers 110 summarize the presence of features in inputs to the DNN 100. The convolutional layers 110 function as feature extractors. The first layer of the DNN 100 is a convolutional layer 110. In an example, a convolutional layer 110 performs a convolution on an input tensor 140 (also referred to as IFM 140) and a filter 150. As shown in FIG. 1, the IFM 140 is represented by a 7x7x3 three-dimensional (3D) matrix. The IFM 140 includes 3 input channels, each of which is represented by a 7x7 two-dimensional (2D) matrix. The 7x7 2D matrix includes 7 input elements (also referred to as input points) in each row and 7 input elements in each column. The filter 150 is represented by a 3x3x3 3D matrix. The filter 150 includes 3 kernels, each of which may correspond to a different input channel of the IFM 140. A kernel is a 2D matrix of weights, where the weights are arranged in columns and rows. A kernel can be smaller than the IFM. In the embodiments of FIG. 1, each kernel is represented by a 3x3 2D matrix. The 3x3 kernel includes 3 weights in each row and 3 weights in each column. Weights can be initialized and updated by backpropagation using gradient descent. The magnitudes of the weights can indicate importance of the filter 150 in extracting features from the IFM 140.

[0042] The convolution includes MAC operations with the input elements in the IFM 140 and the weights in the filter 150. The convolution may be a standard convolution 163 or a depthwise convolution 183. In the standard convolution 163, the whole filter 150 slides across the IFM 140. All the input channels are combined to produce an output tensor 160 (also referred to as OFM 160). The OFM 160 is represented by a 5x5 2D matrix. The 5x5 2D matrix includes 5 output elements (also referred to as output points) in each row and 5 output elements in each column. For the purpose of illustration, the standard convolution includes one filter in the embodiments of FIG. 1. In embodiments where there are multiple filters, the standard convolution may produce multiple output channels in the OFM 160.

[0043] The multiplication applied between a kernel-sized patch of the IFM 140 and a kernel may be a dot product. A dot product is the elementwise multiplication between the kernelsized patch of the IFM 140 and the corresponding kernel, which is then summed, always resulting in a single value. Because it results in a single value, the operation is often referredto as the "scalar product." Using a kernel smaller than the IFM 140 is intentional as it allows the same kernel (set of weights) to be multiplied by the IFM 140 multiple times at different points on the IFM 140. Specifically, the kernel is applied systematically to each overlapping part or kernel-sized patch of the IFM 140, left to right, top to bottom. The result from multiplying the kernel with the IFM 140 one time is a single value. As the kernel is applied multiple times to the IFM 140, the multiplication result is a 2D matrix of output elements. As such, the 2D output matrix (i.e., the OFM 160) from the standard convolution 163 is referred to as an OFM.

[0044] In the depthwise convolution 183, the input channels are not combined. Rather, MAC operations are performed on an individual input channel and an individual kernel and produce an output channel. As shown in FIG. 1, the depthwise convolution 183 produces a depthwise output tensor 180. The depthwise output tensor 180 is represented by a 5x5x3 3D matrix. The depthwise output tensor 180 includes 3 output channels, each of which is represented by a 5x5 2D matrix. The 5x5 2D matrix includes 5 output elements in each row and 5 output elements in each column. Each output channel is a result of MAC operations of an input channel of the IFM 140 and a kernel of the filter 150. For instance, the first output channel (patterned with dots) is a result of MAC operations of the first input channel (patterned with dots) and the first kernel (patterned with dots), the second output channel (patterned with horizontal strips) is a result of MAC operations of the second input channel (patterned with horizontal strips) and the second kernel (patterned with horizontal strips), and the third output channel (patterned with diagonal stripes) is a result of MAC operations of the third input channel (patterned with diagonal stripes) and the third kernel (patterned with diagonal stripes). In such a depthwise convolution, the number of input channels equals the number of output channels, and each output channel corresponds to a different input channel. The input channels and output channels are referred to collectively as depthwise channels. After the depthwise convolution, a pointwise convolution 193 is then performed on the depthwise output tensor 180 and a 1x1x3 tensor 190 to produce the OFM 160.

[0045] The OFM 160 is then passed to the next layer in the sequence. In some embodiments, the OFM 160 is passed through an activation function. An example activation function is rectified linear unit (ReLU). ReLU is a calculation that returns the value provided as input directly, or the value zero if the input is zero or less. The convolutional layer 110may receive several images as input and calculate the convolution of each of them with each of the kernels. This process can be repeated several times. For instance, the OFM 160 is passed to the subsequent convolutional layer 110 (i.e., the convolutional layer 110 following the convolutional layer 110 generating the OFM 160 in the sequence). The subsequent convolutional layers 110 perform a convolution on the OFM 160 with new kernels and generate a new feature map. The new feature map may also be normalized and resized. The new feature map can be kernelled again by a further subsequent convolutional layer 110, and so on.

[0046] In some embodiments, a convolutional layer 110 has four hyperparameters: the number of kernels, the size F kernels (e.g., a kernel is of dimensions FxFxD pixels), the S step with which the window corresponding to the kernel is dragged on the image (e.g., a step of one means moving the window one pixel at a time), and the zero-padding P (e.g., adding a black contour of P pixels thickness to the input image of the convolutional layer 110). The convolutional layers 110 may perform various types of convolutions, such as 2-dimensional convolution, dilated or atrous convolution, spatial separable convolution, depthwise separable convolution, transposed convolution, and so on. The DNN 100 includes 16 convolutional layers 110. In other embodiments, the DNN 100 may include a different number of convolutional layers.

[0047] The pooling layers 120 down-sample feature maps generated by the convolutional layers, e.g., by summarizing the presence of features in the patches of the feature maps. A pooling layer 120 is placed between two convolution layers 110: a preceding convolutional layer 110 (the convolution layer 110 preceding the pooling layer 120 in the sequence of layers) and a subsequent convolutional layer 110 (the convolution layer 110 subsequent to the pooling layer 120 in the sequence of layers). In some embodiments, a pooling layer 120 is added after a convolutional layer 110, e.g., after an activation function (e.g., ReLU, etc.) has been applied to the OFM 160.

[0048] A pooling layer 120 receives feature maps generated by the preceding convolution layer 110 and applies a pooling operation to the feature maps. The pooling operation reduces the size of the feature maps while preserving their important characteristics. Accordingly, the pooling operation improves the efficiency of the DNN and avoids over- learning. The pooling layers 120 may perform the pooling operation through average pooling (calculating the average value for each patch on the feature map), max pooling(calculating the maximum value for each patch of the feature map), or a combination of both. The size of the pooling operation is smaller than the size of the feature maps. In various embodiments, the pooling operation is 2x2 pixels applied with a stride of two pixels, so that the pooling operation reduces the size of a feature map by a factor of 2, e.g., the number of pixels or values in the feature map is reduced to one quarter the size. In an example, a pooling layer 120 applied to a feature map of 6x6 results in an output pooled feature map of 3x3. The output of the pooling layer 120 is inputted into the subsequent convolution layer 110 for further feature extraction. In some embodiments, the pooling layer 120 operates upon each feature map separately to create a new set of the same number of pooled feature maps.

[0049] The fully-connected layers 130 are the last layers of the DNN. The fully-connected layers 130 may be convolutional or not. The fully-connected layers 130 receive an input operand. The input operand defines the output of the convolutional layers 110 and pooling layers 120 and includes the values of the last feature map generated by the last pooling layer 120 in the sequence. The fully-connected layers 130 apply a linear combination and an activation function to the input operand and generate a vector. The vector may contain as many elements as there are classes: element i represents the probability that the image belongs to class i. Each element is therefore between 0 and 1, and the sum of all is worth one. These probabilities are calculated by the last fully-connected layer 130 by using a logistic function (binary classification) or a SoftMax function (multi-class classification) as an activation function. In some embodiments, the fully-connected layers 130 multiply each input element by weight, make the sum, and then apply an activation function (e.g., logistic if N=2, SoftMax if N>2). This is equivalent to multiplying the input operand by the matrix containing the weights.

[0050] FIG. 2 illustrates an example convolution, in accordance with various embodiments. The convolution may be a deep learning operation in a convolutional layer of a DNN, e.g., a convolutional layer 110 in FIG. 1. The convolution can be executed on an activation tensor 210 and filters 220 (individually referred to as "filter 220"). A filter, a portion of a filter, or a combination of multiple filters may be referred to as a weight tensor of the convolution. The result of the convolution is an output tensor 230. In some embodiments, the convolution is performed by a DNN accelerator. An example of the DNN accelerator may be the DNNaccelerator 302 in FIG. 3. For instance, the convolution may be performed by the DPU 530 in the DNN accelerator 302.

[0051] In the embodiments of FIG. 2, the activation tensor 210 includes activations (also referred to as "input activations," "elements," or "input elements") arranged in a 3D matrix. The activation tensor 210 may also be referred to as an input tensor of the convolution. An input element is a data point in the activation tensor 210. The activation tensor 210 has a spatial size Hinx Winx Cin, where Hinis the height of the 3D matrix (i.e., the length along the Y axis, which indicates the number of activations in a column in the 3D matrix of each input channel), Winis the width of the 3D matrix (i.e., the length along the X axis, which indicates the number of activations in a row in the 2D matrix of each input channel), and Cinis the depth of the 3D matrix (i.e., the length along the Z axis, which indicates the number of input channels). For the purpose of simplicity and illustration, the activation tensor 210 has a spatial size of 7x7x3, i.e., the activation tensor 210 includes three input channels and each input channel has a 7x72D matrix. Each input element in the activation tensor 210 may be represented by a (X, Y, Z) coordinate. In other embodiments, the height, width, or depth of the activation tensor 210 may be different.

[0052] Each filter 220 includes weights arranged in a 3D matrix. The values of the weights may be determined through training the DNN. A filter 220 has a spatial size Hf x Wf x Cf, is the height of the filter (i.e., the length along the Y axis, which indicates the number of weights in a column in each kernel), Wf is the width of the filter (i.e., the length along the X axis, which indicates the number of weights in a row in each kernel), and is the depth of the filter (i.e., the length along the Z axis, which indicates the number of channels). In some embodiments, equals Cin. For purpose of simplicity and illustration, each filter 220 in FIG. 2 has a spatial size of 2x3x3, i.e., the filter 220 includes 2 convolutional kernels with a spatial size of 2x3. In other embodiments, the height, width, or depth of the filter 220 may be different. The spatial size of the convolutional kernels is smaller than the spatial size of the 2D matrix of each input channel in the activation tensor 210.

[0053] An activation or weight may take one or more bytes in a memory. The number of bytes for an activation or weight may depend on the data format. For example, when the activation or weight has an INT8 format, the activation takes one byte. When the activationor weight has a FP16 format, the activation or weight takes two bytes. Other data formats may be used for activations or weights.

[0054] In the convolution, each filter 220 slides across the activation tensor 210 and generates a 2D matrix for an output channel in the output tensor 230. In the embodiments of FIG. 2, the 2D matrix has a spatial size of 5x5. The output tensor 230 includes activations (also referred to as "output activations," "elements," or "output element") arranged in a 3D matrix. An output activation is a data point in the output tensor 230. The output tensor 230 has a spatial size HoutX WoutCout, where Houtis the height of the 3D matrix (i.e., the length along the Y axis, which indicates the number of output activations in a column in the 2D matrix of each output channel), Woutis the width of the 3D matrix (i.e., the length along the X axis, which indicates the number of output activations in a row in the 2D matrix of each output channel), and Coutis the depth of the 3D matrix (i.e., the length along the Z axis, which indicates the number of output channels). Coutmay equal the number of filters 220 in the convolution. Houtand Woutmay depend on the heights and weights of the activation tensor 210 and each filter 220.

[0055] As a part of the convolution, MAC operations can be performed on a 2x3x3 subtensor 215 (which is highlighted with a dotted pattern in FIG. 2) in the activation tensor 210 and each filter 220. The result of the MAC operations on the subtensor 215 and one filter 220 is an output activation. In some embodiments (e.g., embodiments where the convolution is an integral convolution), an output activation may include 8 bits, e.g., one byte. In other embodiments (e.g., embodiments where the convolution is a floating-point convolution), an output activation may include more than one byte. For instance, an output element may include two bytes.

[0056] After the MAC operations on the subtensor 215 and all the filters 220 are finished, a vector 235 is produced. The vector 235 is highlighted with slashes in FIG. 2. The vector 235 includes a sequence of output activations, which are arranged along the Z axis. The output activations in the vector 235 have the same (X, Y) coordinate, but the output activations correspond to different output channels and have different Z coordinates. The dimension of the vector 235 along the Z axis may equal the total number of output channels in the output tensor 230. After the vector 235 is produced, further MAC operations are performed to produce additional vectors till the output tensor 230 is produced.

[0057] In some embodiments, the MAC operations on a 2x3x3 subtensor (e.g., the subtensor 215) and a filter 220 may be performed by a plurality of MAC units. One or more MAC units may receive an input operand (e.g., an activation operand 217 shown in FIG. 2) and a weight operand (e.g., the weight operand 227 shown in FIG. 2). The activation operand 217 includes a sequence of activations having the same (x, y) coordinate but different z coordinates. The activation operand 217 includes an activation from each of the input channels in the activation tensor 210. The weight operand 227 includes a sequence of weights having the same (x, y) coordinate but different z coordinates. The weight operand 227 includes a weight from each of the channels in the filter 220. Activations in the activation operand 217 and weights in the weight operand 227 may be sequentially fed into a MAC unit. The MAC unit may receive an activation and a weight ("an activation-weight pair") at a time and multiple the activation and the weight. The position of the activation in the activation operand 217 may match the position of the weight in the weight operand 227. The activation and weight may correspond to the same channel.

[0058] Activations or weights may be floating-point numbers. Floating-point numbers may have various data formats, such as FP32, FP16, BF16, and so on. A floating-point number may be a positive or negative number with a decimal point. A floating-point number may be represented by a sequence of bits that includes one or more bits representing the sign of the floating-point number (e.g., positive or negative), bits representing an exponent of the floating-point number, and bits representing a mantissa of the floating-point number. The mantissa is the part of a floating-point number that represents the significant digits of that number. The mantissa is multiplied by the base raised to the exponent to give the actual value of the floating-point number.

[0059] In some embodiments, the output activations in the output tensor 230 may be further processed based on one or more activation functions before they are stored or inputted into the next layer of the DNN. The processing based on the one or more activation functions may be at least part of the post processing of the convolution. In some embodiments, the post processing may include one or more other computations, such as offset computation, bias computation, scale computation, and so on. The results of the post processing may be stored in a local memory of the DPU and be used as input to the next DNN layer. In some embodiments, the input activations in the activation tensor 210 may be results of post processing of the previous DNN layer.

[0060] FIG. 3 illustrates an example channel-wise reduction operation, in accordance with various embodiments. Channel-wise reduction operations may reduce the number of channels in tensors. Channel-wise reduction operations may be performed by multifunctional DPUs. For the purpose of illustration, the channel-wise reduction operation in FIG. 3 has an input tensor 310 and an output tensor 320. The input tensor 310 is a 3D tensor with a spatial size of H X W X C . The two dimensions along the X- and Y-axes, respectively, may be the spatial dimensions. An activation may have a 2D spatial position, which may be represented by the (X,Y) coordinate of the activation. The spatial position may indicate the spatial location of the activation in the input tensor 310. There may be C activations having the same spatial position, which constitute a vector in the input tensor 310. The C activations are in the C channel, respectively. The dimension along the Z axis is the channel dimension. The number of channels in the input tensor 310 may be the spatial size of the input tensor 310 in the channel dimension. The output tensor 320 is a 3D tensor in an X-Y plane. The output tensor 320 may be considered to have one channel, i.e., the channel-wise reduction operation reduces the number of channels from C to one.

[0061] Examples of the channel-wise reduction operation may include channel-wise variance computation, channel-wise maximum computation, channel-wise minimum computation, other types of channel-wise reduction operation, and so on. In an example where the channel-wise reduction operation in FIG. 3 is a channel-wise variance computation, each activation in the output tensor 320 may be a variance denoted as:is the variance and an output activation in the output tensor 320, i denotes the X coordinate, j denotes the Y coordinate, j denotes a vector including the input activations having the same spatial position ofIJ-Lis the mean value of input activations that have the same spatial position but are in different channels, which may be computed by using an algorithm denoted asThe length of the vector Xt (i.e., the number of activations in the vector) may equal the number of channels C.

[0062] In an example where the channel-wise reduction operation in FIG. 3 is a channelwise maximum computation, each vector j in the input tensor 310 may be replaced with the maximum value in the vector to produce the output tensor 320. In an example where the channel-wise reduction operation in FIG. 3 is a channel-wise minimum computation,each vector xtin the input tensor 310 may be replaced with the minimum value in the vector to produce the output tensor 320. In other examples, the channel-wise reduction operation in FIG. 3 may be other types of channel-wise reduction operations.

[0063] FIG. 4 illustrates an example elementwise operation, in accordance with various embodiments. Elementwise operations may change values in tensors but may make no changes to tensor size. For the purpose of illustration, the elementwise operation in FIG. 4 has an input tensor 410 and another input tensor 420. The two input tensors 410 and 420 are both 3D tensors have the same spatial size H X W X C. The elementwise operation may include H X W X C rounds of computations. The computations may be of the same type, such as multiplication, addition, subtraction, division, etc. Each computation may be performed on an activation in the input tensor 410 and a corresponding activation in the input tensor 420. The two activations may have the same 3D location, i.e., the same (X,Y,Z) coordinate. In some embodiments, one of the input tensors 410 and 420 may be a single data element or a smaller tensor (e.g., a vector, a matrix, or a smaller 3D tensor). The single data element or the smaller tensor may be duplicated to create a tensor of the same spatial size H X W X C as the other tensor. The output tensor of the elementwise operation, even though not shown in FIG. 4, may also have the spatial size H X W X C. Each output activation in the output tensor may be the result of a single round of computation in the elementwise operation.

[0064] Elementwise operations may be performed by multifunctional DPUs. In some embodiments, the elementwise operation in FIG. 4 may be allocated to multiple DPUs for parallel processing. In an example, the input tensor 410 may be divided to subtensors 415 and 417, and the input tensor420 may be divided to subtensors 425 and 427. The subtensor 415 and the subtensor 417 may have the same spatial size and have the same location in the corresponding input tensor. Similarly, the subtensor 425 and the subtensor 427 may have the same spatial size and have the same location in the corresponding input tensor. A portion of the elementwise operation may be performed by a first DPU on the subtensors 415 and 425. Another portion of the elementwise operation may be performed by a second DPU on the subtensors 417 and 427. The two DPUs may perform the computation at the same time. In other example, the workload for executing the elementwise operation may be divided differently.Example DNN System with Multifunctional DPU

[0065] FIG. 5 is a block diagram of a DNN system 500, in accordance with various embodiments. The whole DNN system 500 or a part of the DNN system 500 may be implemented in one or more computing devices, such as the computing device 2000 in FIG. 20. The DNN system 500 can generate and execute DNNs, such as convolution-based models, transformer-based models, graph neural networks, and so on. As shown in FIG. 5, the DNN system 500 includes a DNN module 501 and a DNN accelerator 502.

[0066] In other embodiments, alternative configurations, different or additional components may be included in the DNN system 500. For example, the DNN system 500 may include multiple DNN modules or multiple DNN accelerators. As another example, the DNN system 500 may include one or more DSPs. Further, functionality attributed to a component of the DNN system 500 may be accomplished by a different component included in the DNN system 500 or a different system. In some embodiments, the DNN module 501 and DNN accelerator 502 may include different types of processing units. In an example, the DNN module 501 may be implemented by one or more central processing units (CPUs). The DNN accelerator 502 may also be referred to as an Al accelerator or an Al processor. The DNN module 501 and DNN accelerator 502 may be implemented in the same chip or separate chips.

[0067] The DNN module 501 facilitates generation and deployment of DNNs. In some embodiments, the DNN module 501 may generate and train DNNs. For instance, the DNN module 501 can define the layered architecture of a DNN. The DNN module 501 can also determine the internal parameters of the DNN through a DNN training process. The DNN module 501 may also determine one or more hyperparameters that define how the DNN is trained. An example hyperparameter is a sparsity ratio that defines the sparsity level of one or more deep learning tensors for the DNN.

[0068] The DNN module 501 may also compress DNNs, e.g., during or after training. In some embodiments, the DNN module 501 may prune weights in one or more layers of a DNN by changing nonzero valued weight to zeros. The DNN module 501 may prune weights based on a target weight sparsity ratio. A weight sparsity ratio may be the ratio of the number of zero-valued weights to the total number of weights. In an example where the DNN module 501 prunes weight during DNN training, the DNN module 501 may prune weight of a layer to achieve a target sparsity ratio after one or more epochs. The DNN module 501 may prevent the pruned weights from changing values during the rest of the training process.Y1Alternatively, the DNN module 501 may allow the pruned weights to change values so that a pruned, zero-valued weight may have a nonzero value after further training. The DNN module 501 may prune weights of the layer again after one or more additional epochs.

[0069] The DNN module 501 may deploy trained, compressed, or validated DNNs for use in deep learning applications. In some embodiments, the DNN module 501 may distribute trained, compressed, or validated DNNs to devices or systems which may use the DNNs to perform tasks (e.g., image classification, motion planning, language generation, etc.) for which the DNNs were trained. In other embodiments, the DNN module 501 may facilitate deployment of the DNNs using the DNN accelerator 502. For instance, the DNN module 501 may receive data from a device or system coupled with the DNN system 500 and input the received data (or data generated by the DNN module 501, e.g., based on the received data) into a DNN. The DNN module 501 may generate instructions (e.g., configuration files) that control the operation of the DNN accelerator 502 during the DNN execution. The DNN module 501 may receive an output of the DNN from the DNN accelerator 502. The DNN module 501 may transmit the output of the DNN (or a result of processing the output of the DNN by the DNN module 501) to the device or system.

[0070] The DNN module 501 may facilitate execution of DNNs for training and deployment purposes. In some embodiments, the DNN module 501 facilitates mapping neural network operations to multifunctional DPUs in the DNN accelerator 302. In some embodiments, the DNN module 501 may identify one or more neural network operations in a DNN model and generate one or more configuration parameters, such as configuration descriptors, for an identified neural network operation. The configuration parameters may encode information indicating allocation of the workload executing the neural network operation to PEs in the DPUs. The configuration parameters may also encode information controlling operation modes of the PEs. The PEs in the DNN accelerator 502 may perform computations in the neural network operation in accordance with the configuration parameters. Certain aspects of the DNN module 501 are provided below in conjunction with FIG. 6.

[0071] The DNN accelerator 502 executes DNNs provided by the DNN module 501. For instance, the DNN accelerator 502 can execute a DNN by running deep learning operations in the DNN. The process of carrying out a deep learning operation is also referred to as a process of executing the deep learning operation or performing the deep learning operation. The execution of the DNN may be for training the DNN or for using the DNN toperform Al tasks. In some embodiments, the DNN accelerator 502 includes components designed for optimal efficiency in running convolution-based DNNs. As shown in FIG. 5, the DNN accelerator 502 includes a memory 510, a DMA (direct memory access) engine 520, and DPUs 530 (individually referred to as "DPU 530"). In other embodiments, alternative configurations, different or additional components may be included in the DNN accelerator 502. For example, the DNN accelerator 502 may include more than one memory 510 or DMA engine 520. As another example, the DNN accelerator 502 may include a single DPU 530. Further, functionality attributed to a component of the DNN accelerator 502 may be accomplished by a different component included in the DNN accelerator 502 or by a different system. A component of the DNN accelerator 502 may be implemented in hardware, software, firmware, or some combination thereof.

[0072] The memory 510 stores data associated with deep learning operations performed by the DNN accelerator 502. In some embodiments, the memory 510 may store data to be used by the DPUs 530 for DNN execution. The memory 510 may store weights, such as weights of convolutional layers, which are determined by training DNNs. The memory 510 may further store inputs to DNN layers or outputs of DNN layers, such as data generated by the DPUs 530 from performing neural network operations in DNNs. Example neural network operations include convolutional operations (e.g., standard convolution, depthwise convolution, deconvolution, etc.), layer normalization operations, SoftMax operations, elementwise operations (e.g., elementwise addition, elementwise subtract, elementwise multiplication, elementwise division, etc.) matrix multiplication operations, pooling operations, activation functions, other types of deep learning operations, or some combination thereof. The memory 510 may be a main memory of the DNN accelerator 502. In some embodiments, the memory 510 includes one or more dynamic random-access memories (DRAMs).

[0073] The DMA engine 520 facilitates data transfer between the memory 510 and local memories of the DPUs 530. For example, the DMA engine 520 can read data from the memory 510 and write data into a local memory of a DPU 530. As another example, the DMA engine 520 can read data from a local memory of a DPU 530and write data into the memory 510. The DMA engine 520 provides a DMA feature that allows the DPU 530 to initiate data transfer between the memory 510 and the local memories of the DPUs 530 and to perform other operations while the data transfer is being conducted. In someembodiments, the DMA engine 520 may read tensors from the memory 510, modify the tensors in a way that is optimized for the DPI! 530 before it writes the tensors into the local memories of the DPUs 530.

[0074] The DPUs 530 can perform deep learning operations in DNNs. For instance, a DPU 530 may execute a DNN layer by running one or more neural network operations in the DNN layer. A DPU 530 may execute a layer, or a portion of a layer, at a time. At least one of the DPUs 530 is multifunctional. The multifunctional DPU can perform computations in various types of neural network operations, including convolutions, layer normalization operations, SoftMax operations, elementwise operations, and so on. In some embodiments, one or more DPUs 530 may receive one or more input tensors of a neural network operation in a DNN layer and compute an output tensor of the neural network operation using the one or more input tensors. The output tensor may be further processed, e.g., by the one or more DPUs 530 or one or more other DPU 530, in the next DNN layer. In some embodiments, a neural network operation may be performed by multiple DPUs 530 in parallel. For instance, multiple DPUs 530 may each perform a portion of a workload for a convolution, layer normalization operation, or SoftMax operation. Data may be shared between the DPUs 530. A DPU 530 may also be referred to as a compute tile or compute block. In some embodiments, each DPU 530 may be a processing unit.

[0075] In the embodiments of FIG. 5, each DPU 530 includes a local memory 540, a sparsity mode module 550, a load module 560, a sparse cell array 570, a post processing engine (PPE) 580, and a drain module 590. Some or all the components of the DPU 530 can be implemented on the same chip. In other embodiments, alternative configurations, different or additional components may be included in the DPU 530. Further, functionality attributed to a component of the DPU 530 may be accomplished by a different component included in the DPU 530, a different DPU 530, another component of the DNN accelerator 502, or a different system. A component of the DPU 530 may be implemented in hardware, software, firmware, or some combination thereof.

[0076] The local memory 540 is local to the corresponding DPU 530. In the embodiments of FIG. 5, the local memory 540 is inside the DPU 530. In other embodiments, the local memory 540 may be outside the DPU 530. Data in the local memory 540 may be transferred to or from the memory 510, e.g., through the DMA engine 520. In some embodiments, data in the local memory 540 may be transferred to or from the local memory of another DPU530. The local memory 540 may store data received, used, or generated by the sparsity mode module 550, the load module 560, the sparse cell array 570, or the drain module 590. Examples of the data may include input activations, weights, output activations, sparsity bitmaps, and so on.

[0077] In some embodiments, the local memory 540 may store dense tensors (e.g., dense activation tensors, dense weight tensors, etc.), sparse tensors (e.g., sparse activation tensors, sparse weight tensors, etc.), and so on. A dense tensor may be a tensor from which zero-valued elements (if any) are not removed. A dense tensor may be converted to a sparse tensor by removing one or more zero-valued elements in the dense tensor. A sparse tensor may also be referred to as a compressed tensor or packed tensor. The process of converting a dense tensor to a sparse tensor may be referred to as sparsity encoding. Sparsity encoding may also generate a sparsity tensor. Each element in the sparsity tensor may correspond to a different element in the dense tensor and indicate whether the element in the dense tensor is zero or not. The sparsity tensor may indicate positions of elements of the sparse tensor in the dense tensor. The sparsity tensor may be a sparsity bitmap, each element of which is a bit. A sparse tensor may be converted to a dense tensor through a densifying process, in which one or more zeros may be added to the sparse tensor based on the sparsity tensor.

[0078] In some embodiments, the local memory 540 includes one or more static randomaccess memories (SRAMs). The local memory 540 may be byte-addressable, and each memory address identifies a single byte (eight bits) of storage. In some embodiments, the local memory 540 may include memory banks. The number of data banks in the local memory 540 may be 16, 64, 128, 556, 512, 1024, 5048, or other numbers. A memory bank may include a plurality of storage units. In an example, a data bank may include 8, 16, 64, or a different number of storage units. A memory bank or a storage unit in a memory bank may have a memory address. In an example, a storage unit may store a single byte, and data larger than a single byte may be stored in storage units with consecutive memory addresses, i.e., adjacent storage units. For instance, a storage unit can store an integer number in the INT8 format, versus two storage units may be needed to store a number in the FP16 or BF16 format, which has 16 bits. In some embodiments, 16 bits can be transferred from the local memory 540 in a single read cycle. In other embodiments, 16 bits can be transferred from the local memory 540 in multiple read cycles, such as two cycles.

[0079] The sparsity mode module 550 determines sparsity modes in which the DPU 530 operates to execute DNN layers. For instance, the sparsity mode module 550 may determine whether to accelerate a layer based on weight sparsity, activation sparsity, or both. The sparsity mode module 550 select the sparsity mode for a layer from a group of sparsity modes that includes, for example, combined sparsity mode in which the layer is accelerated based on both weight sparsity and activation sparsity, activation sparsity mode in which the layer is accelerated based on activation sparsity but not based on weight sparsity, weight sparsity mode in which the layer is accelerated based on weight sparsity but not based on activation sparsity, and a dense mode in which the layer is not accelerated based on sparsity. In some embodiments (e.g., embodiments where a layer is executed by multiple DPUs 530), the sparsity mode module 550 may determine the sparsity mode for all the DPUs 530 that executes the layer. In some embodiments, the sparsity mode module 550 may receive configuration parameters from the DNN module 501. A configuration parameter may correspond to a layer and indicate whether to accelerate the layer based on weight sparsity. The sparsity mode module 550 may determine the sparsity mode of the layer based on the configuration parameter.

[0080] The load module 560 loads data from the local memory 540 to the sparse cell array 570. The load module 560 may read tensors from the local memory 540. The tensors may include sparse or dense activation tensors, sparse or dense weight tensors, activation sparsity tensors, weight sparsity tensors, and so on. In some embodiments, the load module 560 may load data based on the sparsity mode determined by the sparsity mode module 550. The load module 560 may select different data to transmit to the sparse cell array 570 in different sparsity modes. For instance, the load module 560 may transmit an activation sparsity tensor and a weight sparsity tensor of a layer to the sparse cell array 570 in the combined sparsity mode, while transmit the activation sparsity tensor but not the weight sparsity tensor to the sparse cell array 570 in the activation sparsity mode and transmit the weight sparsity tensor but not the activation sparsity tensor to the sparse cell array 570 in the weight sparsity mode. In the dense mode, the load module 560 does not transmit either the activation sparsity tensor or the weight sparsity tensor to the sparse cell array 570.

[0081] In some embodiments, the load module 560 may process (e.g., densify) data stored in the local memory 540 before providing the data to the sparse cell array 570. In an example, the load module 560, while operating in the weight sparsity mode, may densifysparse activation tensors to generate dense activation tensors based on corresponding activation sparsity tensors. For instance, the load module 560 may add one or more zeros into a sparse activation tensor based on an activation sparsity tensor associated with the sparse activation tensor to generate the dense activation tensor. The dense activation tensor includes one or more elements than the sparse activation tensor. The additional element(s) are zero-valued. The load module 560 may identify one or more elements in the activation sparsity tensor that correspond to the zero-valued element(s), determine the position of each of the zero-valued element(s) in the dense activation tensor, and insert the zero-valued element(s) into the sparse activation tensor based on the determined positions. After the densification, the load module 560 may transmit the dense activation tensors to the sparse cell array 570. The load module 560 may also transmit corresponding sparse weight tensors and weight sparsity tensors to the sparse cell array 570. Activation sparsity tensor of the dense activation tensors may not be loaded to the sparse cell array 570. In another example, the load module 560, while operating in the activation sparsity mode, may densify sparse weight tensors to generate dense weight tensors based on corresponding weight sparsity tensors by inserting zeros into sparse weight tensors. The densification of sparse weight tensors may be similar to the densification of sparse activation tensors described above. After the densification, the load module 560 may transmit the dense weight tensors to the sparse cell array 570. The load module 560 may also transmit corresponding sparse activation tensors and activation sparsity tensors to the sparse cell array 570. Weight sparsity tensor of the dense weight tensors may not be loaded to the sparse cell array 570. In yet another example, the load module 560, while operating in the dense mode, may densify both sparse weight tensors and sparse activation tensors. The load module 560 may generate the input tensor and weight tensor of the layer and transmit the tensors to the sparse cell array 570 for executing the layer without sparsity acceleration.

[0082] The sparse cell array 570 is the compute component of the DPU 530. In some embodiments, the sparse cell array 570 may be a compute array including one or more data processing cells. The data processing cells may be arranged in one or more rows and one or more columns in the sparse cell array 570. Each data processing cell may include a PE array in which PEs are arranged in columns and rows. In some embodiments, one or more PEs in the sparse cell array 570 may be multifunctional PEs. A multifunctional PE may includecomponents that can perform computations in two or more types of neural network operations. In an example, a multifunctional PE may be able to perform one or more types of computations in addition to MAC operations. The one or more types of computations may include minimum or maximum computations, square computations, mean computations, elementwise computations, quantization, dequantization, other types of computations, or some combination thereof.

[0083] In some embodiments, the sparse cell array 570 may include different types of PEs. A particular type of PEs in the sparse cell array 570 may be designed to perform one or more particular types of computations. For instance, the sparse cell array 570 may include a type of PEs dedicated to performing MAC operations, a type of PEs that can perform minimum or maximum computations, a type of PEs that can perform square or mean computations, a type of PEs that can perform elementwise computations, or a type of PEs that can perform quantization or dequantization computations. PEs of the same type may be arranged adjacent to each other. In an example architecture of the sparse cell array 570, PEs of the same type may be arranged in the same column(s) or the same row(s). This architecture may facilitate efficiently loading data into the PEs or draining data from the PEs and the input signals and output signals of the PEs of the same type may be for the same neural network operation, while the input signals and output signals of PEs of different types may be for different neural network operations.

[0084] A PE may include one or more MAC units that can perform MAC operations. In some embodiments, all the PEs in the sparse cell array 570 may include MAC units. In other embodiments, one or more PEs in the sparse cell array 570 may include no MAC units. In some embodiments, the sparse cell array 570 may include sparsity acceleration logic for facilitating sparsity acceleration of neural network operations, including convolutions. For instance, each data processing cell in the sparse cell array 570 may include one or more sparsity modules. In an example, each MAC column or each MAC row may have a corresponding sparsity module that accelerates MAC operations in the MAC column or MAC row. In some embodiments, a sparsity module accelerates computations in the sparse cell array 570 based on sparsity in activations, sparsity in weights, or both. The sparsity module may include a storage unit that stores a sparsity tensor, which may be loaded to the storage unit by the load module 560. The sparsity tensor may be an activation sparsity tensor, a weight sparsity tensor, or a combined sparsity tensor.

[0085] An activation sparsity tensor may be the sparsity tensor of an activation tensor and has the same number of elements as the activation tensor. An element in the activation sparsity tensor may indicate whether the corresponding element in the activation tensor is zero or not. For instance, a zero-valued in the activation sparsity tensor may indicate that the corresponding element in the activation tensor is zero. A one-valued in the activation sparsity tensor may indicate that the corresponding element in the activation tensor is nonzero. A weight sparsity tensor may be the sparsity tensor of a weight tensor and has the same number of elements as the weight tensor. An element in the weight sparsity tensor may indicate whether the corresponding element in the weight tensor is zero or not. For instance, a zero-valued in the weight sparsity tensor may indicate that the corresponding element in the weight tensor is zero. A one-valued in the weight sparsity tensor may indicate that the corresponding element in the weight tensor is nonzero. The sparsity module may generate a combined sparsity tensor using an activation sparsity tensor and a weight sparsity tensor. For instance, the sparsity module may multiply an element of the activation sparsity tensor with a corresponding element of the weight sparsity tensor to compute an element of the combined sparsity tensor. The positions of the three elements in their corresponding sparsity tensors may match. In some embodiments, each element in a sparsity tensor may be a bit, and the sparsity tensor may be referred to as a sparsity bitmap.

[0086] The sparsity module may use the sparsity tensor to identify activations and weights to be used in MAC operations by the MAC units. In an embodiment where the sparse cell array 570 operates in the combined sparsity mode, the sparsity module may identify activations and weights that correspond to nonzero valued elements of a combined sparsity tensor. In an embodiment where the sparse cell array 570 operates in the activation sparsity mode, the sparsity module may identify activations and weights that correspond to nonzero valued elements of an activation sparsity tensor. In an embodiment where the sparse cell array 570 operates in the weight sparsity mode, the sparsity module may identify activations and weights that correspond to nonzero valued elements of a weight sparsity tensor. The sparsity module may be bypassed in the dense mode as no sparsity acceleration would be conducted. Certain aspects of the sparse cell array 570 and PEs in the sparse cell array 570 are described below in conjunction with FIGS. 1-Yl .

[0087] The PPE 580 processes outputs of the sparse cell array 570. In some embodiments, the PPE 580 computes activation functions. The PPE 580 may receive outputs of the sparsecell array 570 as inputs to the activation functions. The PPE 580 may transmit the outputs of the activation functions to the local memory 340. The outputs of the activation functions may be retrieved later by the sparse cell array 570 from the local memory 340 for further computation. For instance, the PPE 580 may receive an output tensor of a DNN layer from the sparse cell array 570 and computes one or more activation functions on the output tensor. The results of the computation by the PPE 580 may be stored in the local memory 340 and later used as input tensor of the next DNN layer.

[0088] In addition or alternative to activation functions, the PPE 580 may perform other types of post processing on outputs of the sparse cell array 570. For instance, the PPE 580 may perform post processing for neural network operations, such as layer normalization operations, SoftMax operations, and so on. In some embodiments, the PPE 580 may perform a biasing operation by applying a bias on an output of the sparse cell array 570. The biasing operation may include accumulating each output element computed by the sparse cell array 570 with the bias value. The bias value may be positive or negative. Additionally or alternatively, the PPE 580 may perform a scaling operation by applying a scale on an output of the sparse cell array 570. The scaling operation may include computing multiplying output element computed by the sparse cell array 570 with the scale value, in some embodiments, the PPE 580 may include an array of post processing elements, in which the post processing elements may be arranged in one or more columns and one or more rows.

[0089] The drain module 590 drains data from the sparse cell array 570 or the PPE 580 and writes the data to the local memory 540. The data may be outputs of computations performed by the PEs in the sparse cell array 570 or by the PPE 580. In some embodiments, the drain module 590 may drain data on a cell level. For each data processing cell, the drain module 590 may drain outputs of PEs in the data processing cell based on a row index or column index of each PE. For instance, the drain module 590 may use a sequence of cycles to drain data from a data processing cell. The drain module 590 may drain the output of some of the PEs in each cycle.

[0090] In some embodiments, the drain module 590 may determine whether to drain the output of a PE based on the column index of the PE or based on the row index of the PE. In the activation sparsity mode, the drain module 590 may drain the output of a different PE column in each cycle. The sequence of cycles may start with the first PE column (e.g., the PE column on the left side of the sparse cell array 570) and end with the last PE column (e.g.,the PE column on the right side of the sparse cell array 570). In the weight sparsity mode, the drain module 590 may drain the output of a different MAC row in each cycle. The sequence of cycles may start with the first PE row (e.g., the PE row at the top of the sparse cell array 570) and end with the last PE row (e.g., the PE column at the bottom of the sparse cell array 570). In other embodiments, the drain module 590 may determine whether to drain the output of an PE based on the row index of the PE when the load module operates in the activation sparsity mode versus based on the column index of the PE when the load module operates in the weight sparsity mode.

[0091] The drain module 590 may also include sparsity encoding logic that can convert outputs of the sparse cell array 570 from a dense format to a sparse format. For instance, the drain module 590 may be implemented with one or more sparsity encoders. A sparsity encoder converts dense data to compressed data based on sparsity in the dense data. For instance, the sparsity encoder may remove zeros in an activation tensor computed by the sparse cell array 570 to convert the activation tensor to a compressed activation tensor. The sparsity encoder may also generate sparsity tensors, including activation sparsity tensors.

[0092] In some embodiments, the data drained from the sparse cell array 570 may be at least part of an output tensor (e.g., the output tensor 230 in FIG. 2) of a deep learning operation. The sparsity encoder may generate a compressed version of the output tensor. The sparsity encoder may identify every zero-valued activation in the output tensor and remove these activations from the output tensor to generate a compressed activation tensor (aka "sparse activation tensor"). The sparsity encoder may also generate one or more sparsity tensors for the output tensor. A sparsity tensor may correspond to a portion of the output tensor (e.g., the vector 235 in FIG. 2). The sparsity tensor may include sparsity elements (e.g., bits), each of which corresponds to a different activation in the vector and indicates whether the corresponding activation is zeroed or not.

[0093] The drain module 590 may write the compressed activation tensor and the one or more sparsity tensors into the local memory 540. The sparse activation tensor and the one or more sparsity tensors may be further loaded to the memory 510, e.g., through the DMA engine 520. Additionally or alternatively, the sparse activation tensor and the one or more sparsity tensors may be loaded by the load module 560 to the DPU for further computation, e.g., for performing a deep learning operation in the next layer.

[0094] FIG. 6 is a block diagram of a DNN module 600, in accordance with various embodiments. The DNN module 600 facilitates execution of various types of neural networks operations by multifunctional DPUs, such as the DPU 530. The DNN module 600 may be an embodiment of the DNN module 301 in FIG. 3. As shown in FIG. 6, the DNN module 600 includes an interface module 610, a training module 620, a compressing module 630, a validating module 640, a mapping module 650, and a datastore 660. In other embodiments, alternative configurations, different or additional components may be included in the DNN module 600. Further, functionality attributed to a component of the DNN module 600 may be accomplished by a different component included in the DNN module 600 or a different module or system.

[0095] The interface module 610 facilitates communications of the DNN module 600 with other modules or systems. For example, the interface module 610 establishes communications between the DNN module 600 with an external database to receive data that can be used to train DNNs or input into DNNs to perform tasks. As another example, the interface module 610 supports the DNN module 600 to distribute DNNs to other systems, e.g., computing devices configured to apply DNNs to perform tasks.

[0096] The training module 620 trains DNNs by using a training dataset. The training module 620 forms the training dataset. In an embodiment where the training module 620 trains an DNN to recognize objects in images, the training dataset includes training images and training labels. The training labels describe ground-truth classifications of objects in the training images. In some embodiments, each label in the training dataset corresponds to an object in a training image. In some embodiments, a part of the training dataset may be used to initially train the DNN, and the rest of the training dataset may be held back as a validation subset used by the validating module 640 to validate performance of a trained DNN. The portion of the training dataset not including the tuning subset and the validation subset may be used to train the DNN.

[0097] The training module 620 also determines hyperparameters for training the DNN. Hyperparameters are variables specifying the DNN training process. Hyperparameters are different from parameters inside the DNN (e.g., weights). In some embodiments, hyperparameters include variables determining the architecture of the DNN, such as number of hidden layers, etc. Hyperparameters also include variables which determine how the DNN is trained, such as batch size, number of epochs, etc. A batch size defines thenumber of training samples to work through before updating the parameters of the DNN. The batch size is the same as or smallerthan the number of samples in the training dataset. The training dataset can be divided into one or more batches. The number of epochs defines how many times the entire training dataset is passed forward and backwards through the entire network. The number of epochs defines the number of times that the deep learning algorithm works through the entire training dataset. One epoch means that each training sample in the training dataset has had an opportunity to update the parameters inside the DNN. An epoch may include one or more batches. The number of epochs may be 1, 5, 10, 50, 100, 500, 1000, or even larger.

[0098] The training module 620 defines the architecture of the DNN, e.g., based on some of the hyperparameters. The architecture of the DNN includes an input layer, an output layer, and a plurality of hidden layers. The input layer of an DNN may include tensors (e.g., a multidimensional array) specifying attributes of the input image, such as the height of the input image, the width of the input image, and the depth of the input image (e.g., the number of bits specifying the color of a pixel in the input image). The output layer includes labels of objects in the input layer. The hidden layers are layers between the input layer and output layer. The hidden layers include one or more convolutional layers and one or more other types of layers, such as pooling layers, fully-connected layers, normalization layers, SoftMax or logistic layers, and so on. The convolutional layers of the DNN abstract the input image to a feature map that is represented by a tensor specifying the feature map height, the feature map width, and the feature map channels (e.g., red, green, blue images include 3 channels). A pooling layer is used to reduce the spatial volume of input image after convolution. It is used between two convolution layers. A fully-connected layer involves weights, biases, and neurons. It connects neurons in one layer to neurons in another layer. It is used to classify images between different categories by training.

[0099] In the process of defining the architecture of the DNN, the training module 620 also adds an activation function to a hidden layer or the output layer. An activation function of a layer transforms the weighted sum of the input of the layer to an output of the layer. The activation function may be, for example, a ReLU activation function, a tangent activation function, or other types of activation functions.

[0100] After the training module 620 defines the architecture of the DNN, the training module 620 inputs a training dataset into the DNN. The training dataset includes a pluralityof training samples. An example of a training sample includes an object in an image and a ground-truth label of the object. The training module 620 modifies the parameters inside the DNN ("internal parameters of the DNN") to minimize the error between labels of the training objects that are generated by the DNN and the ground-truth labels of the objects. The internal parameters include weights of filters in the convolutional layers of the DNN. In some embodiments, the training module 620 uses a cost function to minimize the error.

[0101] The training module 620 may train the DNN for a predetermined number of epochs. The number of epochs is a hyperparameter that defines the number of times that the deep learning algorithm will work through the entire training dataset. One epoch means that each sample in the training dataset has had an opportunity to update internal parameters of the DNN. After the training module 620 finishes the predetermined number of epochs, the training module 620 may stop updating the parameters in the DNN. The DNN having the updated parameters is referred to as a trained DNN.

[0102] The compressing module 630 compresses DNNs. For instance, the compressing module 630 may add pruning operations to DNN layers to reduce computational complexity or memory usage. A pruning operation may prune weight tensors of a DNN layer by changing one or more nonzero valued weights of the layer to zeros. The modification may be done before, during, or after training. Weights may be pruned during training, during inference, or a combination of both. The compressing module 630 may determine a sparsity ratio for a DNN layer. The sparsity ratio may be a ratio of the number of zero-valued weight to the total number of weights in the layer. The compressing module 630 may perform the pruning operation till the sparsity ratio of the DNN layer meets a target sparsity ration, such as 10%, 20%, 30%, 60%, 50%, and so on.

[0103] In some embodiments, the compressing module 630 may select one or more layers in a DNN and modify each selected layer with a pruning operation. For instance, the compressing module 630 may select computationally complex layers, such as layers with large filters. For a pruning operation of a layer or of a type of layer, the compressing module 630 may determine a weight threshold that would not cause a loss of the accuracy of the DNN to exceed an accuracy loss constraint. A pruning operation may modify weights having absolute values above the weight threshold to zeros and leave the other weights unchanged. The weight pruning can reduce memory storage as zero-valued weights may not be stored. Also, the number of operations in the layer can be reduced as computations onzero-valued weights can be skipped without impacting the output of the layer. In some embodiments, the compressing module 630 may also measure energy saving, final DNN accuracy, or layer-wise sparsity caused by pruning operations.

[0104] After compressing a DNN, the compressing module 630 may fine tune the DNN, e.g., through a retraining process. The compressing module 630 may fine tunes DNNs after weights are pruned. In some embodiments, the fine-tuning process is a retraining or further training process. For instance, after weights in a DNN are pruned, the compressing module 630 may furthertrain the DNN by inputting a training dataset into the DNN. The values of the unpruned weights in the DNN may be modified based on outputs of the DNN and ground-truth labels of the training samples in the training dataset. In some embodiments, the values of the pruned weights (i.e., zero) are not changed during the fine-tuning process. For instance, the compressing module 630 may place a mask over a pruned weight block and the mask can prevent values in the pruned weight blocks from being changed during the fine-tuning process. In other embodiments, the values of all weights, including the pruned weights, may be changed during the fine-tuning process. After one or more cycles of retraining and weight changing by the compressing module 630, the compressing module 630 may perform a new pruning process, e.g., by selecting weight blocks and pruning the selected weight blocks. In some embodiments, the weight pruning process may be repeated multiple times before the fine-tuning process is done. In some embodiments, the number of epochs in the fine-tuning process may be different from the number of epochs in the training process in which the pre-pruning values of the weights are determined. For instance, the fine-tuning process may have less epochs than the training process. In an example, the number of epochs in the fine-tuning process may be relatively small, such as 2, 3, 6, 5, and so on.

[0105] The validating module 640 verifies accuracy of trained or compressed DNNs. In some embodiments, the validating module 640 inputs samples in a validation dataset into a trained DNN and uses the outputs of the DNN to determine the model accuracy. In some embodiments, a validation dataset may be formed of some or all the samples in the training dataset. Additionally or alternatively, the validation dataset includes additional samples, other than those in the training sets. In some embodiments, the validating module 640 may determine an accuracy score measuring the precision, recall, or a combination of precision and recall of the DNN. The validating module 640 may use the following metrics todetermine the accuracy score: Precision = TP / (TP + FP) and Recall = TP / (TP + FN), where precision may be how many the DNN correctly predicted (TP or true positives) out of the total it predicted (TP + FP or false positives), and recall may be how many the DNN correctly predicted (TP) out of the total number of objects that did have the property in question (TP + FN or false negatives). The F-score (F-score = 2 * PR / (P + R)) unifies precision and recall into a single measure.

[0106] The validating module 640 may compare the accuracy score with a threshold score. In an example where the validating module 640 determines that the accuracy score of the DNN is less than the threshold score, the validating module 640 instructs the training module 620 to re-train the DNN. In one embodiment, the training module 620 may iteratively re-train the DNN until the occurrence of a stopping condition, such as the accuracy measurement indication that the DNN may be sufficiently accurate, or a number of training rounds having taken place.

[0107] The mapping module 650 maps workload of executing neural network operations to multinational DPUs. The mapping module 650 may identify a neural network operation in a DNN model. The mapping module 650 may further identify computations in the neural network operation. The mapping module 650 may then map the identified computations to PEs in a multifunctional DPI). In some embodiments, the mapping module 650 may map computations in multiple neural networks operations (which may be neural network operations of different types) to the multifunctional DPU for the multifunctional DPU to perform the neural network operations in parallel, e.g., the multifunctional DPU may perform computations in multiple neural network operations at the same time.

[0108] To map a computation in a neural network operation to the multifunctional DPU, the mapping module 650 may identify the PEs in the multifunctional DPU that can perform the computation. The mapping module 650 may assign the workload of the computation to the identified PEs. In an example, the mapping module 650 may map a maximum or minimum value computation to a comparator tree in the DPU. The comparator tree may include comparators which are located at various PEs in the DPU. The DPU may include one or more PE columns (or rows) where PEs with comparators are located. In another example, the mapping module 650 may map a square or mean computation to one or more PE columns (or rows) in the DPU that include PEs with multiplexers (MUXs) coupled with multipliers. Inyet another example, the mapping module 650 may map a square or mean computation to one or more PE columns (or rows) in the DPU that include PEs with scale multipliers.

[0109] In some embodiments, the mapping module 650 may map a single neural network operation or a single computation in a neural network operation to a single DPU. In other embodiments, the mapping module 650 may map a single neural network operation or a single computation in a neural network operation to multiple DPUs, e.g., based on the size of the input tensor, output tensor, the number of PEs that can perform the computation in the DPU, other factors, or some combination thereof. In some embodiments, the mapping module 650 may generate configuration parameters that configure operations of PEs in the DPU. The mapping module 650 may provide configuration parameters to be used by MUXs in PEs.

[0110] For example, the mapping module 650 may generate a configuration parameter that configures MUXs coupled with multipliers to select activations and provide the activations to the multipliers for computing squares of the activations. The mapping module 650 may generate a different configuration parameter that configures the MUXs to select weights and provide the weights to the multipliers for performing MAC operations. As another example, the mapping module 650 may generate a configuration parameter that configures MUXs coupled with comparators to make the comparators output maximum values for a SoftMax operation. For computations of minimum values, the mapping module 650 may generate a configuration parameter that configures the MUXs to make the comparators output minimum values. As another example, the mapping module 650 may generate a configuration parameter that configures MUXs in PEs, each of which includes both a MAC unit and a comparator, to make the MAC unit active for performing MAC operations in convolutions. The mapping module 650 may generate a different configuration parameter that configures the MUXs in the PEs to make the comparators active for performing maximum or minimum value computations. In some embodiments, the mapping module 650 may predetermine configuration parameters, meaning the mapping module 650 may determine a configuration parameter before the execution of the neural network operation is started, or even before the execution of the DNNs is started. Configuration parameters may be determined offline.

[0111] In some embodiments, the mapping module 650 may determine a type of a neural network operation in a DNN. For instance, the mapping module 650 may identify one ormore computations in the neural network operation. The mapping module 650 may select PEs from the DPU based on the determined type of the neural network operation, e.g., based on the identified computation(s). The mapping module 650 may also generate one or more configuration parameters based on the determined type of the neural network operation, e.g., based on the identified computation(s).

[0112] In some embodiments, the mapping module 650 may provide instruction to the load module 560 to load the right data into the PEs to which a computation is mapped to. The mapping module may also provide instruction to the drain module 590 to drain the outputs of the PEs as the result of the computation. For instance, the mapping module 650 may instruct the load module 560 to load a fixed value of one to multipliers in MAC units in PEs for the PEs to compute means.

[0113] The datastore 660 stores data received, generated, used, or otherwise associated with the DNN module 600. For example, the datastore 660 stores the datasets used by the training module 620 and validating module 640. The datastore 660 may also store data generated by the training module 620 and validating module 640, such as the hyperparameters for training DNNs, internal parameters of trained DNNs (e.g., weights, etc.), data for sparsity acceleration (e.g., sparsity bitmap, etc.), and so on. The datastore 660 may store configuration parameters generated by the mapping module 650. In the embodiment of FIG. 6, the datastore 660 is a component of the DNN module 600. In other embodiments, the datastore 660 may be external to the DNN module 600 and communicate with the DNN module 600 through a network.

[0114] FIG. 7 illustrates an example PE 700 with a MAC unit, in accordance with various embodiments. The PE 700 includes a MAC unit 710, an activation register file 720, a weight register file 730, and an output register file 740. The MAC unit 710 includes a multiplier 750, an adder 760, and an accumulator 770. In other embodiments, the PE 700 may include different, fewer, or more components. For instance, the PE 700 may include more than one MAC unit 710, activation register file 720, weight register file 730, or output register file 740.

[0115] The MAC unit 710 performs MAC operations on data in the activation register file 720 and weight register file 730. Even though the MAC unit 710 in FIG. 7 includes a single multiplier and a single adder, the MAC unit 710 may include multiple multipliers or adders. The multiplier 750 performs multiply operations on activations in the activation register file720 and weights in the weight register file 730. In a single multiplication operation, the multiplier 750 may compute a product of an activation and a weight. The multiplier 750 may perform multiple multiplication operations to complete an activation operand and a weight operand. The adder 760 may compute a sum of the products computed by the multiplier 750 to compute an internal partial sum of the MAC unit 710. The accumulator 770 may compute a sum of the internal partial sum of the MAC unit 710 with one or more internal partial sums of one or more other MAC units to compute an external partial sum. For instance, the accumulator 770 may compute a sum of all the internal partial sums computed by all the MAC units 710 in a PE column to compute a column-level partial sum. In some embodiments, the accumulator 770 may compute an output activation of a convolution.

[0116] In some embodiments (e.g., embodiments where the DPU 530 executes a convolutional layer), a computation in the MAC unit 710 may be an MAC operation on an activation operand and a weight operand. The activation operand may be an activation tensor that may include one or more activations in the input tensor of the convolution. Different activations may be in different input channels. The weight operand may be a weight tensor that may include one or more weights in the filter of the convolution. The values of the weights are determined through training the DNN. The weights in the weight operand may be in different input channels.

[0117] In some embodiments, the PE 700 may be arranged in a column that includes one or more other PEs. The PE column may also be referred to as a MAC column. The MAC column may be associated with one or more MAC lanes. A MAC lane is a path for loading data e.g., by the load module 560, into an MAC column. A MAC lane may also be referred to as a data transmission lane or data loading lane. A MAC column may have multiple MAC lanes. The loading bandwidth of the MAC column is an aggregation of the loading bandwidths of all the MAC lanes associated with the MAC column. With a certain number of MAC lanes, data can be fed into the same number of independent MAC units simultaneously. In some embodiments where an MAC column has four MAC lanes for feeding activations or weights into the MAC column and each MAC lane may have a bandwidth of 16 bytes, the four MAC lanes can have a total loading bandwidth of 64 bytes.

[0118] In some embodiments, the MAC unit 710 can perform MAC operations in depthwise convolution, standard convolution, or both. In a depthwise convolution, the MAC unit 710 may perform an MAC operation that includes a sequence of multiplications for an inputoperand and a weight operand. Each multiplication in the sequence (also referred to as a cycle) is a multiplication of a different activation in the input operand with a different weight in the weight operand. The activation and weight in the same cycle may correspond to the same channel. The sequence of multiplication produces a product operand that includes a sequence of products. The MAC operation may also include accumulations in which multiple product operands are accumulated to produce an output operand of the MAC unit 710. The MAC unit 710 may output a output operand at a time. In a standard convolution, MAC operations may include accumulations across the channels. For instance, as opposed to generating an output operand, the MAC unit 710 may accumulate products across different channels to generate a single output point.

[0119] In some embodiments, the MAC unit 710 may perform MAC operations in quantized deep learning operations, such as MAC operations in a quantized convolution. In some embodiments, the MAC unit 710 may receive quantized activation and quantized weights and compute a quantized MAC result. The quantized MAC result may be a quantized value in an integer format. In some embodiments, the MAC unit 710 may also include a quantization multiplier that can multiply a quantization scale with the quantized MAC result, and the output of the MAC unit 710 may be a real value in a floating-point format. The MAC unit 710 may include no quantization subtractors as zero point offsetting is not needed for the MAC operations in quantized deep learning operations.

[0120] The activation register file 720 temporarily stores activations received by the PE 700. The activations may be in an input tensor of a neural network operation. The activations may be loaded into the activation register file 720 by the load module 560. In some embodiments, the activation register file 720 may receive an activation operand in a single data load cycle. The activation operand may be a vector including a sequence of activations. The activations may have the same spatial position but in different input channels.

[0121] The weight register file 730 temporarily stores weights received by the PE 700. The weights may be in a weight tensor of a neural network operation. The weights may be loaded into the weight register file 730 by the load module 560. In some embodiments, the weight register file 730 may receive a weight operand in a single data load cycle. The weight operand may correspond to an activation operand received by the activation register file 720. The weight operand may be a vector including a sequence of weights. The weights may have the same spatial position but in different input channels.

[0122] The output register file 740 temporarily stores data computed by the MAC unit 710. In some embodiments, the output register file 740 may temporarily store internal partial sums, which may be provided to the accumulator 770 to compute external partial sums. The external partial sums may also be stored in the output register file 740. For the purpose of illustration, the PE 700 in FIG. 7 is dedicated to performing MAC operations. A PE may be multifunctional and can perform other types of operations.

[0123] FIG. 8 illustrates an example multifunctional PE 800, in accordance with various embodiments. The multifunctional PE 800 can perform multiple types of computations in neural network operations, such as square computation, mean computation, and MAC operation. The multifunctional PE 800 includes a MAC unit 810, an activation register file 820, a weight register file 830, and an output register file 840. The MAC unit 810 includes a multiplier 850, an adder 860, and an accumulator 870. Additionally, the multifunctional PE 800 includes a MUX 880 that is coupled to the activation register file 820, the weight register file 830, and the multiplier 850.

[0124] In other embodiments, the multifunctional PE 800 may include different, fewer, or more components. For instance, the multifunctional PE 800 may include more than one MAC unit 810, activation register file 820, weight register file 830, output register file 840, or MUX 880. In some embodiments, the MAC unit 810 may be the same or similar as the MAC unit 810. The activation register file 820 may be the same or similar as the activation register file 720. The weight register file 830 may be the same or similar as the weight register file 730. The output register file 840 may be the same or similar as the output register file 740.

[0125] The MUX 880 may receive multiple data elements and select one of data elements. The selected data element may be transmitted to the multiplier 850 for computation. In an example, the MUX 880 may receive an activation from the activation register file 820 and receive a weight from the weight register file 830. The MUX 880 may select either the activation or the weight for being processed later by the multiplier 850. The MUX 880 may facilitate multiple operation modes of the multifunctional PE 800, such as a square computation mode, a mean computation mode, and a MAC operation mode. For instance, when the MUX 880 selects the activation, the multiplier 850 may compute a square of the activation as the multiplier 850 also receives the activations directly from the activation register file 820. When the MUX 880 selects the weight, the multiplier 850 may compute aproduct of the weight and the activation received directly from the activation register file 820, which can be part of a MAC operation.

[0126] In some embodiments, the weight may have a fixed value. The fixed value may be one. The output of the multiplier 850 may be the value of the activation. The adder 860 or the accumulator 870 can compute a channel-wise sum of activations, which may then be used to compute a channel-wise mean of activations. In some embodiments, the fixed value may be input into the multiplier 850 as a configuration parameter, as opposed to a weight from the weight register file 830. The configuration parameter may be provided by a DNN module, such as the DNN module 301.

[0127] FIG. 9 illustrates another example multifunctional PE, in accordance with various embodiments. The multifunctional PE 900 can perform multiple types of computations in neural network operations, such as minimum value computation, maximum value computation, and MAC operation. The multifunctional PE 900 includes a MAC unit 910, a comparator 915, an activation register file 920, a weight register file 930, and an output register file 940. The MAC unit 910 includes a multiplier 950, an adder 960, and an accumulator 970. Additionally, the multifunctional PE 900 includes a MUX 980 that is coupled to the MAC unit 910 and the comparator 915.

[0128] In other embodiments, the multifunctional PE 900 may include different, fewer, or more components. For instance, the multifunctional PE 900 may include more than one MAC unit 910, comparator 915, activation register file 920, weight register file 930, output register file 940, or MUX 980. In some embodiments, the MAC unit 910 may be the same or similar as the MAC unit 910. The activation register file 920 may be the same or similar as the activation register file 720. The weight register file 930 may be the same or similar as the weight register file 730. The output register file 940 may be the same or similar as the output register file 740.

[0129] The comparator 915 may receive data elements from the activation register file 920 and compare the data elements. In some embodiments, the comparator 915 may perform a channel-wise comparison. The data elements may be activations in different channels. The activations may have the same spatial position, such as the same (x,y) coordinate. In some embodiments, the comparator 915 may output the maximum value of the data elements. In other embodiments, the comparator 915 may output the minimum value of the data elements.

[0130] The MUX 980 may control the operation mode of the multifunctional PE 900. In some embodiments, the MUX 980 may receive a configuration parameter 905. In embodiments where the configuration parameter 905 indicates that the operation mode of the multifunctional PE 900 is a MAC mode, the MUX 980 may enable the MAC unit 910, and the MAC unit 910 may perform MAC operations while the comparator 915 may be inactive. In embodiments where the configuration parameter 905 indicates that the operation mode of the multifunctional PE 900 is a comparison mode, the MUX 980 may enable the comparator 915, and the comparator 915 may perform comparisons while the MAC unit 910 may be inactive.

[0131] In some embodiments, the MUX 980 may also control the operation mode of the comparator 915 based on the configuration parameter 905 or another configuration parameter. For instance, when the configuration parameter may indicate that the operation mode of the comparator 915 is a maximum mode, the MUX 980 may enable the comparator 915 to the output maximum value of inputs. When the configuration parameter may indicate that the operation mode of the comparator 915 is a minimum mode, the MUX 980 may enable the comparator 915 to the output minimum value of inputs. In other embodiments, the operation mode of the comparator 915 may be fixed and not configurable. For instance, the comparator 915 can select maximum values but cannot select minimum values. The DPU may include one or more other comparators for performing minimum value computations.

[0132] By having multifunctional PE in the DPU, the DPU can perform various types of deep learning operations with the existing load and drain logic, which may be designed or optimized for convolutions. This can reduce the overhead required to implement functions for performing other types of deep learning operations. The multifunctional PEs 800 and 900 in FIGS. 8 and 9 are examples. Multifunctional PEs may have different components or architecture. In an example, a multifunctional PE may include a MAC unit, the MUX 880 in FIG. 8, the comparator 915 in FIG. 9, and the MUX 980 in FIG. 9 so that the multifunctional PE can perform square computation, mean computation, maximum computation, and minimum computation on top of MAC operations. As another example, a multifunctional PE may include the MUX 880 in FIG. 8, the comparator 915 in FIG. 9, and the MUX 980 in FIG. 9 for performing square computation, mean computation, maximum computation, and minimum computation. But the multifunctional PE may include no MAC unit and cannotperform MAC operations. In other example, multifunctional PEs may be able to perform other types of computations and may have different components for performing those computations.

[0133] FIG. 10 illustrates an example data processing cell 1000, in accordance with various embodiments. The data processing cell 1000 may be in a sparse cell array, e.g., the sparse cell array 570 in FIG. 3. The data processing cell 1000 can accelerate neural network operations based on sparsity in data, such as sparsity in activation or sparsity in weight. The data processing cell 1000 is also referred to as a sparse cell. The data processing cell 1000 includes 16 PEs 1010 (individually referred to as "PE 1010") arranged in four rows and four columns, 16 weight register files 1020 (individually referred to as "weight register file 1020"), 16 activation register files 1030 (individually referred to as "activation register file 1030"), four row buffers 1040 (individually referred to as "row buffer 1040"), and sparsity modules 1060 (individually referred to as "sparsity module 1060"). In other embodiments, the data processing cell 1000 may include fewer, more, or different components. For example, the data processing cell 1000 may include a different number of PEs 1010, weight register files 1020, activation register files 1030, row buffers 1040, or sparsity modules 1060. As another example, the data processing cell 1000 may include column buffers in lieu of or in addition to the row buffers 1040.

[0134] The PEs 1010 are configured to perform computations in various neural network operations. Examples of the computations include MAC operations, square computation, mean computation, variance computation, maximum computation, minimum computation, and so on. Examples of the PEs 1010 include the PEs 700, 800, and 900 in FIGS. 7-9. In some embodiments, some or all the PEs 1010 may operate simultaneously to process multiple operands in one cycle. In some embodiments, the PEs 1010 may have different architectures. For instance, one or more PEs may have the same or similar architecture as the PE 700 in FIG. 7, one or more other PEs may have the same or similar architecture as the PE 800 in FIG. 8, and one or more other PEs may have the same or similar architecture as the PE 900 in FIG. 9. The data processing cell 1000 may include other types of PEs. PEs with the same architecture may be arranged in the same column or the same row of the data processing cell 1000. Different types of PEs may be arranged in different columns or rows.

[0135] In some embodiments, components of the same type in multiple PEs 1010 may form a tree. For example, the adders or accumulators in some or all the PEs 1010 may constituteat least part of an adder tree in the data processing cell 1000. The adder tree includes a plurality of adder tiers. The first tier may receive outputs of a plurality of MAC units. The number of adders in the first tier may be half of the number of the MAC units, and each adder may accumulate the outputs of two MAC units. The second tier may receive outputs of adders in the first tier. The number of adders in the second tier may be half of the number of adders in the first tier, and each adder in the second tier may accumulate the outputs of two adders in the first tier. The adder tree may include one or more other tiers. The last tier may include a single adder that accumulates outputs of adders in the second last tier to compute a partial sum of the data processing cell 1000.

[0136] As another example, the comparators in one or more PEs 1010 may form a comparator tree in the data processing cell 1000. The comparator tree includes a plurality of comparator tiers. The first tier may include comparators that receive data elements from the activation register files 1030. Each comparator in the first tier may compare two data elements at a time and output the maximum or minimum value. The second tier may include one or more comparators, each of which may receive two maximum or minimum values selected by two comparators in the first tier and output the maximum or minimum value of the two maximum or minimum values. The comparator tree may include one or more other tiers. The number of comparators in a tier may be half of the number of comparators in the previous tier and double the number of comparators in the next tier. Certain aspects of comparator trees are provided below in conjunction with FIGS. 12-14.

[0137] The weight register files 1020 store weights to be processed in MAC operations. In the embodiments of FIG. 10, four weight register files 1020 are grouped into a storage set that stores data to be used by a column of PEs 1010. There are four storage sets corresponding to the four columns of PEs 1010. In some embodiments, a weight register file 1020 may correspond to a PE 1010 and store data to be processed by the MAC unit. In some embodiments, all the 16 weight register files 1020 constitute a weight storage unit.Examples of the weight register files 1020 include the weight register file 730 in FIG. 7, the weight register file 830 in FIG. 8, and the weight register file 930 in FIG. 9.

[0138] The activation register files 1030 stores activations to be processed in MAC operations. In the embodiments of FIG. 10, four activation register files 1030 are grouped into a storage set that stores data to be used by a row of PEs 1010. There are four storage sets corresponding to the four rows of PEs 1010. In some embodiments, an activationregister file 1030 may correspond to a PE 1010 and store data to be processed by the MAC unit. In some embodiments, all the 16 activation register files 1030 constitute an activation storage unit. The row buffers 1040 store outputs of the PEs 1010. Each row buffer 1040 may drain outputs of a single row of PEs 1010. Examples of the activation register files 1030 include the activation register file 720 in FIG. 7 , the activation register file 820 in FIG. 8, and the activation register file 920 in FIG. 9.

[0139] The sparsity module 1060 facilitates dynamic sparsity-based acceleration in the data processing cell 1000. In some embodiments, the sparsity module 1060 may accelerate various computations based on sparsity in data, such as MAC operations, square computation, mean computation, variance computation, maximum computation, minimum computation, and so on. In the embodiments of FIG. 10, each sparsity module 1060 includes a sparsity tensor storage unit 1065 and a control logic 1067. The sparsity tensor storage unit 1065 stores combined sparsity tensors. A combined sparsity tensor stored in the sparsity tensor storage unit 1065 may correspond to an activation tensor and a weight tensor. A nonzero element in the combined sparsity tensor may correspond to a nonzero activationweight pair that includes a nonzero activation and a nonzero weight. The position of the nonzero activation in the activation tensor may match the position of the nonzero weight in the weight tensor. The product of the nonzero activation and nonzero weight would be nonzero.

[0140] The control logic 1067 may control transmission of activations and weights stored from the weight register files 1020 and the activation register files 1030 to the PEs 1010 based on sparsity tensors. For instance, the control logic 1067 may select a subset of the weights stored in the weight register files 1020 and select a subset of activations stored in the activation register files 1030 based on a combined sparsity tensor. The selected weights and activations constitute nonzero activation-weight pairs. The control logic 1067 may transmit the selected weights and activations to the PEs 1010 for performing MAC operations. The other weights stored in the weight register files 1020 and the other activations stored in the activation register files 1030 are skipped from computation. In the embodiments of FIG. 10, each sparsity module 1060 controls sparsity acceleration in a respective PE 1010. As the sparsity acceleration is either based on both weight sparsity and activation sparsity, 16 sparsity modules 1060 are used for acceleration computations in the16 PEs 1010.

[0141] As shown in FIG. 10, the data processing cell 1000 is associated with MUXs 1003, 1004, 1005, and 1006. In other embodiments, the data processing cell 1000 may be associated with a different number of MUXs or other devices. The MUX 1003 facilitates loading weights, e.g., from the local memory 340, into the weight register files 1020. The MUX 1004 facilitates loading activations, e.g., from the local memory 340, into the activation register files 1030. The MUX 1005 facilitates loading sparsity tensors into the sparsity tensor storage unit 1065. The MUX 1006 may be a drain MUX that can facilitate draining outputs of the PEs 1010, e.g., to the local memory 340.

[0142] FIG. 11 illustrates a sparse cell array 1100, in accordance with various embodiments. The sparse cell array 1100 may be an example of the sparse cell array 570 in FIG. 3. In FIG. 11, the sparse cell array 1100 includes data processing cells 1110 (individually referred to as "data processing cell 1110") arranged in four columns and four rows, an activation memory 1120, and a weight memory 1130. In other embodiments, the sparse cell array 1100 may include fewer, more, or different components. For instance, the sparse cell array 1100 may include a different number of columns, rows, or data processing cells 1110.

[0143] Each data processing cell 1110 may perform sparsity accelerated neural network operations. The data processing cells 1110 may facilitate dynamic sparsity mode. For instance, the sparsity modes of a data processing cell 1110 may be dynamically changed between a combined sparsity mode, an activation sparsity mode, a weight sparsity mode, and a dense mode. An embodiment of a data processing cell 1110 may be the data processing cell 1000 in FIG. 10. At least one of the data processing cells 1110 may include multifunctional PEs, such as the multifunctional PEs described above.

[0144] In some embodiments, data processing cells 1110 including the same type of multifunctional PEs may be arranged in the same column or the same row. The multifunctional PEs may be aligned to constitute a multifunctional PE column (or row) in the sparse cell array 1100. For instance, within each of the four data processing cells 1110 in the same column (or row) of the sparse cell array 1100, the multifunctional PEs of the same type may be arranged in the same column (or row). The sparse cell array 1100 may have a multifunctional PE column (or row) including 16 such multifunctional PEs. In some embodiments, the sparse cell array 1100 may include more than one multifunctional PE column (or row). Different columns (or rows) may include multifunctional PE of different types.

[0145] The activation memory 1120 stores activations, such as activations in input tensors of deep learning operations. Activations may be loaded from the activation memory 1120 to data processing cells 1110. The weight memory 1130 stores weights, such as weights in filters of deep learning operations. Weights may be loaded from the weight memory 1130 to data processing cells 1110. The activation memory 1120 or weight memory 1130 may be a buffer. In some embodiments, the activation memory 1120 or weight memory 1130 may be part of a local memory, such as the local memory 540. In some embodiments, the sparse cell array 1100 may include a dense data memory and a sparse data memory in lieu of the activation memory 1120 and weight memory 1130. The dense data memory may store dense tensors, e.g., dense tensors generated by the load module 360. The sparse data memory may store sparse tensors.

[0146] FIG. 12 illustrates example PE columns 1210, 1220, 1230, and 1240 in a multifunctional DPU, in accordance with various embodiments. For the purpose of illustration and simplicity, FIG. 12 does not show all components in the multifunctional DPU. The multifunctional DPU may include one or more other columns. Also, a column may include a different number of PEs from the number shown in FIG. 12. The multifunctional DPU in FIG. 12 may be an example of the DPU 530 in FIG. 3 or the DPU including the sparse cell array 1100 in FIG. 11.

[0147] The PE column 1210 includes a plurality of PEs 1215 (individually referred to as "PE 1215"). Each PE 1215 may perform various computations including MAC operations and maximum value computations. In some embodiments, a PE 1215 may also perform minimum value computations. An example of the PEs 1215 is the PE 900 in FIG. 9. The PE column 1220 includes a plurality of PEs 1225 (individually referred to as "PE 1225"). Each PE 1225 may perform various computations including MAC operations and minimum value computations. In some embodiments, a PE 1225 may also perform maximum value computations. An example of the PEs 1225 is the PE 900 in FIG. 9.

[0148] The PE column 1230 includes a plurality of PEs 1235 (individually referred to as "PE 1235"). Each PE 1235 may perform various computations including MAC operations and square computations. The adder in each PE 1235 may be inactive when the PE 1235 performs square computations. In some embodiments, a PE 1235 may also perform variance computations. An example of the PEs 1235 is the PE 800 in FIG. 8. The PE column 1240 includes a plurality of PEs 1245 (individually referred to as "PE 1245"). Each PE 1245may perform various computations including MAC operations and sum computations. The sum computations may be channel-wise sum computation. When a PE 1245 performs sum computations, the multiplier may receive a first data element (e.g., an activation) and a second data element with a fixed value of one so that the multiplier may output the value of first data element. The adder may compute a sum of the first data element and one or more other data elements. In some embodiments, a PE 1245 may also perform mean computations. An example of the PEs 1245 is the PE 800 in FIG. 8.

[0149] In the embodiments of FIG. 12, each of the PE columns 1210, 1220, 1230, and 1240 has a unique type of PEs. And all the PEs in the same PE column have the same architecture. This can facilitate loading data for a particular computation into the right PEs. In some embodiments, the PE columns 1210, 1220, 1230, and 1240 may include the same number of PEs. In other embodiments, different PE columns may have different numbers of PEs. Even though FIG. 12 shows four PE columns, the multifunctional DPU may include multiple PE column 1210, 1220, 1230, or 1240. Also, the multifunctional DPU may include PE columns with other types of PEs. For instance, the multifunctional DPU may include one or more PE columns with PEs dedicated to MAC operations, such as the PE 700 in FIG. 7.

[0150] FIG. 13 illustrates an example comparator tree 1300 in a multifunctional DPU, in accordance with various embodiments. The multifunctional DPU in FIG. 13 may be an example of the DPU 530 in FIG. 3 or the sparse cell array 1100 in FIG. 11. For the purpose of illustration and simplicity, FIG. 12 does not show all components in the multifunctional DPU. The comparator tree 1300 includes comparators 1310 (individually referred to as "comparator 1310") arranged in three tiers 1301, 1302, and 1303. The first tier 1301 includes four comparators 1310. The second tier 1302 includes two comparators 1310. The third tier 1303 includes one comparator 1310. In some embodiments, all the comparators 1310 in the comparator tree 1300 are in a single PE in the multifunctional DPU. In other embodiments, the comparators 1310 in the comparator tree 1300 may be distributed to multiple PEs in the multifunctional DPU.

[0151] In some embodiments, the comparator tree 1300 receives a vector 1305 that includes eight data elements. In some embodiments, the eight data elements are activations in eight different channels. The activations may have the same spatial position. Each comparator 1310 in the first tier 1301 receives two of the eight data elements and performs a comparison on the two data elements. The result of the comparison may be the maximumvalue or minimum value of the two data elements. Each comparator 1310 in the tier 1302 may receive the outputs of two comparators 1310 in the first tier 1301 and perform a comparison on the outputs. Each comparator 1310 in the tier 1302 may output a value, which may be received by the comparator 1310 in the tier 1303. The comparator 1310 in the tier 1303 may perform another comparison and output the maximum or minimum value of the eight data elements. The maximum or minimum value of the vector 1305 may be received by an additional comparator 1310 in the comparator tree 1300. The additional comparator 1310 may compare the output of the comparator tree 1300 (e.g., the maximum or minimum value of the vector 1305) with the output of another comparator tree (not shown in FIG. 13). The output of the other comparator tree may be the maximum or minimum value of another vector. The other vector may include data elements in different channels from the eight data elements in the vector 1305 but having the same spatial position as the eight data elements in the vector 1305.

[0152] Each comparator 1310 is coupled to a MUX 1320. In some embodiments, a gate 1330 may be arranged between the comparator 1310 and the MUX 1320. The gate 1330 may be an OR gate. The MUX 1320 (or the combination of the MUX 1320 and the gate 1330) can control whether the comparator 1310 outputs maximum values or minimum values based on a configuration parameter 1306. The configuration parameter 1306 may be determined by the DNN module 301 based on what type of computations are included in a neural network operation performed by the multifunctional DPU.

[0153] FIG. 14 illustrates a maximum value computation on integer data elements, in accordance with various embodiments. The integer data elements may have an integer data type, such as INT8 (INT stands for integer), UINT8 (U stands for unsigned), and so on. An integer data element may be stored as a byte in memory, including register files. The maximum value computation is performed a comparator tree 1400. The comparator tree 1400 includes comparators 1410. The comparators 1410 are not coupled with MUXs. The comparators 1410 may be fixed to output maximum values. In some embodiments, all the comparators 1410 in the comparator tree 1400 are in a single PE in the DPU. In other embodiments, the comparators 1410 in the comparator tree 1400 may be distributed to multiple PEs in the DPU.

[0154] The comparator tree 1400 is associated with an activation register file 1420 and an output register file 1430. The activation register file 1420 may be an example of theactivation register files 1030 in FIG. 10. The comparator tree 1400 receives a vector 1405 from the activation register file 1420. For the purpose of illustration, the vector 1405 includes eight data elements, which may be activations in eight different channels. In other embodiments, the vector 1405 may include a different number of data elements. The comparator tree 1400 selects the maximum value in the vector 1405 and transmits the maximum value to the output register file 1430. The maximum value computation may be a channel-wise reduction computation. The comparators 1410 may be 16-bit comparators.

[0155] In some embodiments, the comparator tree 1400 may operate in either a dense mode or sparse mode. In a dense mode, the vector 1405 may include data that has not been compressed based on sparsity. The vector 1405 may include one or more zeros. In the sparse mode, the vector 1405 may include data that has been compressed based on sparsity. The vector 1405 does not include any zeros. In the sparse mode, the efficiency of the maximum value computation can be higher as zeros may be avoided from the comparisons and less comparisons may be needed for the same input data of the maximum value computation.

[0156] FIG. 15 illustrates a maximum value computation on floating-point data elements, in accordance with various embodiments. Forthe purpose of illustration, the maximum value computation is performed on a vector 1505 including eight bytes. Each floating-point data element may be stored as two bytes in memory, including register files. The floating-point data elements may have floating-point data types, such as FP16 or BF16. In other embodiments, the floating-point data elements may have different data types, such as FP32, and so on.

[0157] The vector 1505 includes four floating-point data elements, which are input into a comparator tree 1500. Even though the vector 1505 includes the same number of bytes as the vector 1405 in FIG. 14, the vector 1505 has less data elements as the data type is different. The comparator tree 1500 includes comparators 1510, which may be 16-bit comparators. A part of the comparator tree 1500 is used as each data element has two bytes, which is different from the embodiments of FIG. 14 where each data element has one byte. Some comparators 1510 are inactive during the maximum value computation. The inactive comparators 1510 are represented by dashed lines.

[0158] Multifunctional DPUs can perform various neural network operations on data of various data types and data precisions. The multifunctional DPUs can also perform dynamicquantization operations, which can change data types and precisions of data dynamically. In an example, a multifunctional DPU (e.g., the DPU 530) may convert FP32 data to FP16 data and store the FP16 data in memory to save memory space and bandwidth. For certain neural network operations, the data type may need to be converted back before or during the neural network operation or after the neural network operation is done.

[0159] In some embodiments, a multifunctional DPU can perform dynamic quantization by dynamically determining the optimal precision for a DNN layer, which may be identified by the DNN module 301. One of the steps required for performing dynamic quantization is to compute the tensor-wide minimum and maximum values of the input activations. Dynamic quantization may be facilitated by comparators in multifunctional DPUs, such as the comparator trees 1300, 1400, and 1500 in FIGS. 13-15.

[0160] Even though not shown in FIGS. 13-15, an input tensor may be reshaped before the maximum or minimum value computations are performed. For instance, the maximum or minimum value computations may be performed across a single dimension (e.g., the channel dimension). To acquire the tensor-wide minimum and maximum values, the input tensor may be reshaped (e.g., permuted) on each round so that each dimension of the input tensor may be reduced. The result of the maximum or minimum value computations may be a single maximum value of the input tensor and a single minimum value of the input tensor. The final output is a two-element tensor. In an example, the reshaping process may include changing the shape of the input tensor from H x W x C to H x W x 2 (e.g., through channel-wise reduction), then to 2 x H x IV, further to 2 x H x 2 (e.g., through width-wise reduction), further to 2 X 2 X H, and then to 2 X 2 X 2 (e.g., through height-wise reduction).

[0161] FIG. 16 illustrates an example elementwise multiplication on quantized data, in accordance with various embodiments. The elementwise multiplication may be performed by a DPU on two tensors including quantized data elements. The DPU may include a plurality of PEs, an example of which is a PE 1600. The PE 1600 includes a MAC unit 1610 with multipliers and adders. The PE 1600 also includes a MUX 1620, an adder 1630, an accumulator 1640, a scale multiplier 1650, and another MUX 1660. In other embodiments, the PE 1600 may include different, fewer, or more components.

[0162] Quantized data may be integer data or floating-point data with lower precision compared with the original data. Quantized data types such as 8-bit integers have two keyquantization parameters: scale and zero point. To convert the quantized number to a real number, the zero point is subtracted, and the result is multiplied by the scale, which may be denoted as: real_value = quantized_value — zero_point>) X scale.

[0163] The scale multiplier 1650 in FIG. 16 may be used for applying scales on quantized values for dequantization and may be referred to as scale multipliers. In some embodiments, the scale multiplier 1650, e.g., for performing other computations that includes multiplication. The scale multiplier 1650 is coupled with the two MUXs 1620 and 1660. The MUX 1620 may select data from the MAC unit 1610 or from the scale multiplier 1650. The MUX 1660 may select data from the accumulator 1640 or a scale value 1605.

[0164] In some embodiments, when the DPU reads the first tensor, it stores the value of the element to be multiplied in the accumulator 1640. When the second tensor is being read, the accumulator 1640 may be used as storage. The output of a multiplier 1615 in the MAC unit 1610 may be routed to the scale multiplier 1650. The scale multiplier 1650 may receive two inputs: the incoming activation and the scale value 1605 for the tensor. Forthe elementwise multiplication, the second input may replace the scale value with the stored value from the accumulator 1640. The value stored in the accumulator 1640 may be the corresponding element from the first tensor. This allows the scale multiplier 1650 to have visibility of the same element from the first and second tensor and then perform the multiplication operation on the right elements.

[0165] With such an approach for elementwise multiplication, the DPU can perform the elementwise multiplication operation at a small increment hardware cost while delivering a substantial performance increase relative to prior implementations. This can improve the performance and efficiency of the DNN accelerator.

[0166] FIG. 17 illustrates an example elementwise multiplication on non-quantized data, in accordance with various embodiments. The non-quantized data may have floating-point data types. The elementwise multiplication may be performed by a DPU on two tensors including quantized data elements. The DPU may include a plurality of PEs, an example of which is a PE 1700. The PE 1700 includes a MAC unit 1710, an adder 1720, a MUX 1730, an accumulator 1740, another MUX 1750, and another MUX 1760. In other embodiments, the PE 1700 may include different, fewer, or more components. Different from the PE 1600 in FIG. 16, the 1700 that has no scale multiplier.

[0167] The MAC unit 1710 may be capable of MAC operations on floating-point data, such as FP16 data, BF16 data, and so on. For the elementwise multiplication, the first tensor may be loaded to the accumulator 1740 in the first phase. The accumulator 1740 may be used as a data storage unit. The multiplication of data elements in the two tensors may be performed using the multipliers in the MAC unit 1710. In the second phase, the data (e.g., activations in the first tensor) in the accumulator 1740 may be transmitted from the accumulator 1740 to the MAC unit 1710 through the MUX 1760. One or more multipliers in the MAC unit 1710 may perform the elementwise multiplication.

[0168] The MUX 1730 may receive a configuration parameter 1703 that configures an operation mode of the PE 1700, such as a bypass mode. The bypass mode may be enabled with the first tensor but may be disabled with the second tensor. The MUX 1750 may receive a value of zero in the second phase of the elementwise multiplication. In addition to the data from the accumulator 1740, the MUX 1760 may also receive weights. The MUX 1760 may select a data element from an activation from the accumulator 1740 and a weight. The weight may be received from a weight register file. The weight may also be floating-point data. When the MUX 1760 selects weights, the MAC unit 1710 may perform MAC operations.

[0169] When zero point have a value of zero, elementwise multiplication and addition with dequantization may be respectively denoted as: real_valueAX real_valueB— (quantize _valueAX quantize _valueB) X ( scaleAX scaleB, real_valueA+ real_valueB= (quantize _valueAX scaled + (quantize _valueBX caleB) .

[0170] When performing elementwise addition of two integer tensors, this dequantization to a real number may need be performed prior to adding the elements. For the addition operation, the scale value needs to be applied during the operation and cannot be applied to the final result as the tensors may have different scale values. For elementwise multiplication, the scale value can be applied after the initial multiply. In some embodiments, the combined scale (i.e., scaleAX scaleB) can be applied in the pose processing engine, which may have dedicated logic to apply scale and bias and is typically used to apply the scale for convolution and non-elementwise operations.

[0171] In some embodiments, an elementwise multiplication may be performed on a tensor and a single data element, such as a constant. The constant can be replicated and then loaded into the DPU including the PE 1600 or the PE 1700 in place of the second tensor. The DPU including the PE 1600 or the PE 1700 may also perform elementwise division operation in the same or similar manner as elementwise multiplication. For instance, an elementwise divisional denoted as A / B may be performed as an elementwise multiplication denoted as A*(l / B), where A represents an element in the first tensor, and B represents an element in the second tensor or a constant. Elementwise division may be included in a layer normalization operation.

[0172] Elementwise subtraction may be performed as an elementwise addition after inverting the sign of the elements in the second tensor. In some embodiments, elementwise addition may be implemented as a cascading sequence of single tensor operations where each tensor may be scaled by a predetermined scale factor. For performing a two-tensor elementwise addition, this operation may be performed twice one after another with two different scale factors and use the existing accumulator (e.g., the accumulator 1640 or 1740) to add the values. In case of elementwise subtraction, the sign of the scale for the second tensor may be inverted and elementwise addition may then be performed to compute the output of the elementwise subtraction.Executing Neural Network Operations Using Multifunctional DPU

[0173] FIG. 18 illustrates a method 1800 of executing a layer normalization operation, in accordance with various embodiments. The layer normalization operation is split into a sequence of steps 1801-1809. In the embodiments of FIG. 18, the layer normalization operation is performed by one or more multifunctional DPUs, such as the DPU 530.

[0174] In step 1801, mean computation is performed by the multifunctional DPU(s). The mean computation may be denoted as .xy= ~xXz=i ^xyz > where Axyzdenotes a data element in the input tensor, x may be the positional index of the data element in one of the spatial dimensions, y may be the positional index of the data element in the other one of the spatial dimensions, z may be the positional index of the data element in the channel dimension, and p.xydenotes the output of the mean computation, which may be a 2D matrix. The mean computation may be channel-wise reduction operation. The mean computation may be performed by PEs like the PE 800. In some embodiments, the sumcomputation in the mean computation may be performed as a MAC operation in which the values of the weights are one.

[0175] In step 1802, data replication is performed. The output data of the step 1801 jj.xymay be converted to ixyzin step 1802 by replicating every data element over z output points. Step 1802 may include DMA replication. .xyzmay be a 3D tensor. In some embodiments, the data replication may be performed using z storage element pointers pointing to the same storage element. The data replication may require no data movement. The data replication may not require the data elements to be rewritten into the memory.

[0176] In step 1803, elementwise subtraction is performed by the multifunctional DPU(s). The elementwise subtraction may be denoted as Dxyz= Axyz— ixyz.

[0177] In step 1804, variance computation is performed by the multifunctional DPU(s). The square computation may be part of a variance computation, which may be denoted as = T,z=i ^2xyz- The variance computation may be computed by PEs like the PE 800 in FIG. 8.

[0178] In step 1805, post processing is performed by the multifunctional DPU(s). The post processing may be performed by one or more PPEs in the multifunctional DPU(s). The post processing may be denoted as Mxy= .1In some embodiments, e X Z may be l—X (72xy+CXZ) used as a bias, and i may be used as a scale.

[0179] In step 1806, data replication is performed. The output data of the step 1805 Mxymay be converted to Mxyzin step 1806 by replicating every data element over z output points. Step 1806 may include DMA replication. Mxyzmay be a 3D tensor. In some embodiments, the data replication may be performed using z storage element pointers pointing to the same storage element. The data replication may require no data movement. The data replication may not require the data elements to be rewritten into the memory.

[0180] In step 1807, elementwise multiplication is performed by the multifunctional DPU(s). The element multiplication may be denoted as A'xyzx

[0181] In step 1808, post processing is performed by the multifunctional DPU(s). The post processing may be denoted as A"xyzmay be used as a bias.

[0182] In step 1809, elementwise multiplication is performed by the multifunctional DPU(s). The elementwise multiplication may be denoted as LNxyz— A"xyzX yz. LNxyzmay be the output of the layer normalization operation. In some embodiments, steps 1807-1809 may be in the same DPU workload. When biasing in step 1808 happens before the elementwise multiplication in step 1809, — is needed.Zz

[0183] FIG. 19 illustrates a method 1900 of executing a SoftMax operation, in accordance with various embodiments. In some embodiments, the SoftMax operation may be denoted as:where e stands for exponent, max standards for maximum, i and j are positional indices that indicate positions of data elements in the input tensor.

[0184] In some embodiments, the SoftMax operation may be split into multiple steps. For instance, the first step is to calculate the channel-wise max across a column for each (i, / ) vector. The second step is to subtract the maximum value from each data element. The third step is to calculate the exponent of this subtracted value. The fourth step is to take the channel-wise sum of the exponent values. The fifth step is to normalize each exponent value by multiplying the scale which is evaluated from the sum of the exponent values.

[0185] In the embodiments of FIG. 19, the SoftMax operation is performed by one or more multifunctional DPUs, such as the DPU 530. The method 1900 includes steps 1901-1907. In step 1901, maximum computation is performed by the multifunctional DPU(s). The maximum computation may be denoted as Mxy= Vxy(max (zlxy)), where Axyzdenotes a data element in the input tensor, x may be the positional index of the data element in one of the spatial dimensions, y may be the positional index of the data element in the other one of the spatial dimensions, and z may be the positional index of the data element in the channel dimension. The maximum computation may be a channel-wise reduction operation. Step 1901 may be needed for the normalizations of the data elements before the exponent function call to compute smoothed softmax.

[0186] In step 1902, the output data of the step 1901 is replicated. The output data of the step 1901 may be a tensor denoted as Mxy. Mxymay be converted to Mxyzin step 1902 by replicating every data element overz output points. Step 1902 may include DMA replication. Mxyzmay be a 3D tensor. In some embodiments, the data replication may be performed using z storage element pointers pointing to the same storage element. The data replication may require no data movement. The data replication may not require the data elements to be rewritten into the memory.

[0187] In step 1903, elementwise subtraction is performed by the multifunctional DPU(s). the element wise subtraction may be denoted as Axyz— Mxyz.

[0188] In step 1904, postprocessing is performed. The postprocessing may include applying an exponent function on the output of step 1903. The post processing may be denoted as Nxyz=where Nxyzrepresents the output of step 1904. In some embodiments, the multifunctional DPU(s) may operate on certain numbers on z dimension, such as multiples of 16. Padding may be performed to align the input. The padding may include adding zeros into the input. The exponent of zero is one. The unnecessary ones in Nxyzmay be filtered out before step 1905 is performed.

[0189] In step 1905, sum computation is performed by the multifunctional DPU(s). The sum computation may be denoted as'nsome embodiments, the sum computation may be performed as a MAC operation in which the values of the weights are one.

[0190] In step 1906, postprocessing is performed. The postprocessing may be performed by one or more PPEs in the multifunctional DPU(s). The postprocessing may be denoted as R = Vxv(-= — ). xyy.z-iNzJ

[0191] In step 1907, elementwise multiplication is performed by the multifunctional DPU(s). The elementwise multifunctional may be denoted as Sxyz= NxyzX Rxyz. Rxyzmay be generated from Rxythrough data duplication.Example Computing Device

[0192] FIG. 20 is a block diagram of an example computing device 2000, in accordance with various embodiments. In some embodiments, the computing device 2000 can be used as at least part of the DNN system 500. A number of components are illustrated in FIG. 20 as included in the computing device 2000, but any one or more of these components may be omitted or duplicated, as suitable for the application. In some embodiments, some or all ofthe components included in the computing device 2000 may be attached to one or more motherboards. In some embodiments, some or all of these components are fabricated onto a single system on a chip (SoC) die. Additionally, in various embodiments, the computing device 2000 may not include one or more of the components illustrated in FIG. 20, but the computing device 2000 may include interface circuitry for coupling to the one or more components. For example, the computing device 2000 may not include a display device 2006, but may include display device interface circuitry (e.g., a connector and driver circuitry) to which a display device 2006 may be coupled. In another set of examples, the computing device 2000 may not include an audio input device 2018 or an audio output device 2008 but may include audio input or output device interface circuitry (e.g., connectors and supporting circuitry) to which an audio input device 2018 or audio output device 2008 may be coupled.

[0193] The computing device 2000 may include a processing device 2002 (e.g., one or more processing devices). The processing device 2002 processes electronic data from registers and / or memory to transform that electronic data into other electronic data that may be stored in registers and / or memory. The computing device 2000 may include a memory 2004, which may itself include one or more memory devices such as volatile memory (e.g., DRAM), nonvolatile memory (e.g., read-only memory (ROM)), high bandwidth memory (HBM), flash memory, solid state memory, and / or a hard drive. In some embodiments, the memory 2004 may include memory that shares a die with the processing device 2002. In some embodiments, the memory 2004 includes one or more non-transitory computer- readable media storing instructions executable to perform methods for executing neural network operations (e.g., the method 1800 described in conjunction with FIG. 18 and the method 1900 described in conjunction with FIG. 19) or some operations performed by the DNN system 500 (e.g., operations performed by the DNN module 501, operations performed by the DPU 530, etc.). The instructions stored in the one or more non-transitory computer-readable media may be executed by the processing device 2002.

[0194] In some embodiments, the computing device 2000 may include a communication chip 2012 (e.g., one or more communication chips). For example, the communication chip 2012 may be configured for managing wireless communications for the transfer of data to and from the computing device 2000. The term "wireless" and its derivatives may be used to describe circuits, devices, systems, methods, techniques, communications channels, etc.,that may communicate data through the use of modulated electromagnetic radiation through a nonsolid medium. The term does not imply that the associated devices do not contain any wires, although in some embodiments they might not.

[0195] The communication chip 2012 may implement any of a number of wireless standards or protocols, including but not limited to Institute for Electrical and Electronic Engineers (IEEE) standards including Wi-Fi (IEEE 802.10 family), IEEE 802.16 standards (e.g., IEEE 802.16-2005 Amendment), Long-Term Evolution (LTE) project along with any amendments, updates, and / or revisions (e.g., advanced LTE project, ultramobile broadband (UMB) project (also referred to as "3GPP2"), etc.). IEEE 802.16 compatible Broadband Wireless Access (BWA) networks are generally referred to as WiMAX networks, an acronym that stands for worldwide interoperability for microwave access, which is a certification mark for products that pass conformity and interoperability tests for the IEEE 802.16 standards. The communication chip 2012 may operate in accordance with a Global System for Mobile Communication (GSM), General Packet Radio Service (GPRS), Universal Mobile Telecommunications System (UMTS), High Speed Packet Access (HSPA), Evolved HSPA (E- HSPA), or LTE network. The communication chip 2012 may operate in accordance with Enhanced Data for GSM Evolution (EDGE), GSM EDGE Radio Access Network (GERAN), Universal Terrestrial Radio Access Network (UTRAN), or Evolved UTRAN (E-UTRAN). The communication chip 2012 may operate in accordance with Code-division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Digital Enhanced CordlessTelecommunications (DECT), Evolution-Data Optimized (EV-DO), and derivatives thereof, as well as any other wireless protocols that are designated as 3G, 4G, 5G, and beyond. The communication chip 2012 may operate in accordance with other wireless protocols in other embodiments. The computing device 2000 may include an antenna 2022 to facilitate wireless communications and / or to receive other wireless communications (such as AM or FM radio transmissions).

[0196] In some embodiments, the communication chip 2012 may manage wired communications, such as electrical, optical, or any other suitable communication protocols (e.g., the Ethernet). As noted above, the communication chip 2012 may include multiple communication chips. For instance, a first communication chip 2012 may be dedicated to shorter-range wireless communications such as Wi-Fi or Bluetooth, and a second communication chip 2012 may be dedicated to longer-range wireless communications suchas global positioning system (GPS), EDGE, GPRS, CDMA, WiMAX, LTE, EV-DO, or others. In some embodiments, a first communication chip 2012 may be dedicated to wireless communications, and a second communication chip 2012 may be dedicated to wired communications.

[0197] The computing device 2000 may include battery / power circuitry 2014. The battery / power circuitry 2014 may include one or more energy storage devices (e.g., batteries or capacitors) and / or circuitry for coupling components of the computing device 2000 to an energy source separate from the computing device 2000 (e.g., AC line power).

[0198] The computing device 2000 may include a display device 2006 (or corresponding interface circuitry, as discussed above). The display device 2006 may include any visual indicators, such as a heads-up display, a computer monitor, a projector, a touchscreen display, a liquid crystal display (LCD), a light-emitting diode display, or a flat panel display, for example.

[0199] The computing device 2000 may include an audio output device 2008 (or corresponding interface circuitry, as discussed above). The audio output device 2008 may include any device that generates an audible indicator, such as speakers, headsets, or earbuds, for example.

[0200] The computing device 2000 may include an audio input device 2018 (or corresponding interface circuitry, as discussed above). The audio input device 2018 may include any device that generates a signal representative of a sound, such as microphones, microphone arrays, or digital instruments (e.g., instruments having a musical instrument digital interface (MIDI) output).

[0201] The computing device 2000 may include a GPS device 2016 (or corresponding interface circuitry, as discussed above). The GPS device 2016 may be in communication with a satellite-based system and may receive a location of the computing device 2000, as known in the art.

[0202] The computing device 2000 may include another output device 2010 (or corresponding interface circuitry, as discussed above). Examples of the other output device 2010 may include an audio codec, a video codec, a printer, a wired or wireless transmitter for providing information to other devices, or an additional storage device.

[0203] The computing device 2000 may include another input device 2020 (or corresponding interface circuitry, as discussed above). Examples of the other input device2020 may include an accelerometer, a gyroscope, a compass, an image capture device, a keyboard, a cursor control device such as a mouse, a stylus, a touchpad, a bar code reader, a Quick Response (QR) code reader, any sensor, or a radio frequency identification (RFID) reader.

[0204] The computing device 2000 may have any desired form factor, such as a handheld or mobile computer system (e.g., a cell phone, a smart phone, a mobile internet device, a music player, a tablet computer, a laptop computer, a netbook computer, an ultrabook computer, a personal digital assistant (PDA), an ultramobile personal computer, etc.), a desktop computer system, a server or other networked computing component, a printer, a scanner, a monitor, a set-top box, an entertainment control unit, a vehicle control unit, a digital camera, a digital video recorder, or a wearable computer system. In some embodiments, the computing device 2000 may be any other electronic device that processes data.Select Examples

[0205] The following paragraphs provide various examples of the embodiments disclosed herein.

[0206] Example 1 provides a PE, including a multiplexer to select, from a plurality of inputs, data for a neural network operation performed by the processing element, the plurality of inputs including an activation and a weight, in which the multiplexer is to select different data for different neural network operations performed by the PE; and a multiplier to compute a product of the activation and the selected data.

[0207] Example 2 provides the PE of example 1, in which the PE further includes an adder, and the adder is to compute a sum of the product and an additional product computed in the PE.

[0208] Example 3 provides the PE of example 2, in which the neural network operation is an operation in a layer normalization operation of a neural network, the selected data is the weight, and the weight has a predetermined value of one.

[0209] Example 4 provides the PE of example 3, in which the additional product is computed from an additional activation, and the additional activation and the activations are in different channels of the layer normalization operation.

[0210] Example 5 provides the PE of any one of examples 2-4, in which the PE further includes an accumulator to accumulate the sum and a sum computed by another PE.

[0211] Example 6 provides the PE of any one of examples 1-4, in which the neural network operation is an operation in a layer normalization operation of a neural network, and the selected data is the activation.

[0212] Example 7 provides the PE of any one of examples 1-5, in which the neural network operation is a multiply-accumulate operation in a convolution of a neural network, the selected data is the weight, and a value of the weight is determined by training the neural network.

[0213] Example 8 provides the PE of any one of examples 1-7, in which the PE further includes a register file to store the activation, and the multiplexer and the multiplier are to receive the activation from the register file.

[0214] Example 9 provides the PE of example 8, in which the PE further includes an additional register file to store the weight, and the multiplexer is to receive the weight from the additional register file.

[0215] Example 10 provides the PE of any one of examples 1-9, in which the PE is in a DPUs, the DPUs further includes a PE, and the PE includes a first register file to store another activation, a second register file to store another weight, and a multiplier to receive the another activation from the first register file, receive the another weight from the second register file, and compute a product of the another activation and the another weight.

[0216] Example 11 provides a PE, including a comparator to perform a comparison on data elements and to generate a first output indicating a result of the comparison; a MAC unit including a multiplier and an adder, the MAC to perform a MAC operation and to generate a second output indicating a result of the MAC operation; and a multiplexer coupled to the comparator and the MAC unit, the multiplexer to output the first output or the second output as an output of the PE.

[0217] Example 12 provides the PE of example 11, in which the data elements are in different channels of a neural network operation, and the first output is a maximum value or minimum value of the data elements.

[0218] Example 13 provides the PE of example 12, in which the data elements have the same spatial position in the different channels.

[0219] Example 14 provides the PE of any one of examples 11-13, in which the comparator has a maximum mode and a minimum mode, the first output is a maximum value of thedata elements in the maximum mode, and the first output is a minimum value of the data elements in the minimum mode.

[0220] Example 15 provides the PE of example 14, further including an additional multiplexer coupled to the comparator, the additional multiplexer to select one of the maximum mode and the minimum mode based on a configuration parameter.

[0221] Example 16 provides the PE of any one of examples 11-15, in which the comparator is a first comparator in a comparator tree including a plurality of comparators, a second comparator in the comparator tree is to compute a third output indicating a result of an additional comparison, and a third comparator in the comparator tree is to compute a fourth output indicating a result of comparing the first output with the third output.

[0222] Example 17 provides the PE of example 16, in which the additional comparison is performed by the second comparator on additional data elements, and the data element of the data element and the additional data elements are in different channels of a neural network operation.

[0223] Example 18 provides the PE of example 17, in which the data elements and the additional data elements have the same spatial position in the different channels.

[0224] Example 19 provides the PE of any one of examples 16-18, in which the plurality of comparators is implemented in a plurality of PEs.

[0225] Example 20 provides the PE of any one of examples 11-19, in which the PE is in a DPUs, the DPUs further includes a PE, and the PE includes a first register file to store another activation, a second register file to store another weight, and a multiplier to receive the another activation from the first register file, receive the another weight from the second register file, and compute a product of the another activation and the another weight.

[0226] Example 21 provides a DPUs, including a first PE, the first PE including a comparator to compare data elements received by the first PE; a second PE, the second PE including a multiplexer, a register file, and a multiplier to compute a product of a data element from the multiplexer and a data element from the register file; and a third PE, the third PE including a first register file, a second register file, and a multiplier to compute a product of a data element from the first register file and a data element from the second register file.

[0227] Example 22 provides the DPUs of example 21, in which the first PE, the second PE, and the third PE are to perform different types of neural network operations.

[0228] Example 23 provides the DPUs of example 22, in which the different types of neural network operations include layer normalization operation, SoftMax operation, and convolution.

[0229] Example 24 provides the DPUs of any one of examples 21-23, in which the data elements received by the first PE are in different channels of a neural network operation and have the same spatial position in the different channels.

[0230] Example 25 provides the DPUs of any one of examples 21-24, further including a comparator tree including a plurality of comparators, in which the comparator is a first comparator in a comparator tree, a second comparator in the comparator tree is to compare other data elements received by a fourth PE, and a third comparator in the comparator tree is to compare an output of the first comparator and an output of the second comparator.

[0231] Example 26 provides the DPUs of any one of examples 21-25, in which the comparator is to output a maximum value of the data elements in a maximum mode and is to output a minimum value of the data elements in a minimum mode, and the first PE further includes a multiplexer to select one of the maximum mode and the minimum mode.

[0232] Example 27 provides the DPUs of any one of examples 21-26, in which the first PE further includes a multiplier and an adder.

[0233] Example 28 provides the DPUs of any one of examples 21-27, in which the second PE further includes an adder to compute a sum of the product of the data element from the multiplexer and the data element from the register file and an additional product computed in the second PE.

[0234] Example 29 provides the DPUs of any one of examples 21-28, in which the data element from the register file in the second PE is an activation, and the data element from the multiplexer in the second PE is the activation.

[0235] Example 30 provides the DPUs of any one of examples 21-29, in which the data element from the register file in the second PE is an activation, and the data element from the multiplexer in the second PE is one.

[0236] Example 31 provides a method, including determining a type of a neural network operation in a neural network; selecting one or more PEs from a DPU based on the determined type of the neural network operation, the data processing unit including PEs configured to perform different types of neural network operations; generating aconfiguration parameter based on the determined type of the neural network operation; and providing the configuration parameter to the one or more PEs, the one or more PEs to perform the neural network operation using the configuration parameter.

[0237] Example 32 provides the method of example 31, in which the PEs are arranged in one or more columns, and the one or more PEs are arranged in a same column.

[0238] Example 33 provides the method of example 31 or 32, in which a selected PE includes a multiplexer, and the multiplexer is to select data from a plurality of inputs based on the configuration parameter.

[0239] Example 34 provides the method of example 33, in which the selected PE further includes a multiplier to compute a product of the data selected by the multiplexer and an activation.

[0240] Example 35 provides the method of example 34, in which the data selected by the multiplexer is the activation.

[0241] Example 36 provides the method of example 31 or 32, in which a selected PE includes a comparator, and the comparator is to output a maximum value or a minimum value of a plurality of inputs based on the configuration parameter.

[0242] Example 37 provides the method of example 36, in which the selected PE further includes a multiply-accumulate unit including a multiplier and an adder.

[0243] Example 38 provides the method of any one of examples 31-37, in which determining the type of the neural network operation in the neural network includes identifying one or more computations in the neural network operation,

[0244] Example 39 provides the method of example 38, in which selecting the one or more PEs from the data processing unit includes selecting the one or more PEs based on the one or more identified computations.

[0245] Example 40 provides the method of example 38 or 39, in which generating the configuration parameter includes generating the configuration parameter based on the one or more identified computations.

[0246] Example 41 provides one or more non-transitory computer-readable media storing instructions executable to perform operations, the operations including determining a type of a neural network operation in a neural network; selecting one or more PEs from a DPU based on the determined type of the neural network operation, the data processing unit including PEs configured to perform different types of neural network operations;generating a configuration parameter based on the determined type of the neural network operation; and providing the configuration parameter to the one or more PEs, the one or more PEs to perform the neural network operation using the configuration parameter.

[0247] Example 42 provides the one or more non-transitory computer-readable media of example 41, in which the PEs are arranged in one or more columns, and the one or more PEs are arranged in a same column.

[0248] Example 43 provides the one or more non-transitory computer-readable media of example 41 or 42, in which a selected PE includes a multiplexer, and the multiplexer is to select data from a plurality of inputs based on the configuration parameter.

[0249] Example 44 provides the one or more non-transitory computer-readable media of example 43, in which the selected PE further includes a multiplier to compute a product of the data selected by the multiplexer and an activation.

[0250] Example 45 provides the one or more non-transitory computer-readable media of example 44, in which the data selected by the multiplexer is the activation.

[0251] Example 46 provides the one or more non-transitory computer-readable media of example 41 or 42, in which a selected PE includes a comparator, and the comparator is to output a maximum value or a minimum value of a plurality of inputs based on the configuration parameter.

[0252] Example 47 provides the one or more non-transitory computer-readable media of example 46, in which the selected PE further includes a MAC unit including a multiplier and an adder.

[0253] Example 48 provides the one or more non-transitory computer-readable media of any one of examples 41-47, in which determining the type of the neural network operation in the neural network includes identifying one or more computations in the neural network operation,

[0254] Example 49 provides the one or more non-transitory computer-readable media of example 48, in which selecting the one or more PEs from the data processing unit includes selecting the one or more PEs based on the one or more identified computations.

[0255] Example 50 provides the one or more non-transitory computer-readable media of example 48 or 49, in which generating the configuration parameter includes generating the configuration parameter based on the one or more identified computations.

[0256] The above description of illustrated implementations of the disclosure, including what is described in the Abstract, is not intended to be exhaustive or to limit the disclosure to the precise forms disclosed. While specific implementations of, and examples for, the disclosure are described herein for illustrative purposes, various equivalent modifications are possible within the scope of the disclosure, as those skilled in the relevant art will recognize. These modifications may be made to the disclosure in light of the above detailed description.

Claims

Claims1. A processing element, comprising: a multiplexer to select, from a plurality of inputs, data for a neural network operation performed by the processing element, the plurality of inputs comprising an activation and a weight, wherein the multiplexer is to select different data for different neural network operations performed by the processing element; and a multiplier to compute a product of the activation and the selected data.

2. The processing element of claim 1, wherein the processing element further comprises an adder, and the adder is to compute a sum of the product and an additional product computed in the processing element.

3. The processing element of claim 2, wherein the neural network operation is an operation in a layer normalization operation of a neural network, the selected data is the weight, and the weight has a predetermined value of one.

4. The processing element of claim 3, wherein the additional product is computed from an additional activation, and the additional activation and the activations are in different channels of the layer normalization operation.

5. The processing element of any one of claims 2-4, wherein the processing element further comprises an accumulator to accumulate the sum and a sum computed by another processing element.

6. The processing element of any one of claims 1-5, wherein the neural network operation is an operation in a layer normalization operation of a neural network, and the selected data is the activation.

7. The processing element of any one of claims 1-6, wherein the neural network operation is a multiply-accumulate operation in a convolution of a neural network, the selected data is the weight, and a value of the weight is determined by training the neural network.

8. The processing element of any one of claims 1-7, wherein the processing element further comprises a register file to store the activation, and the multiplexer and the multiplier are to receive the activation from the register file.

9. The processing element of claim 8, wherein the processing element further comprises an additional register file to store the weight, and the multiplexer is to receive the weight from the additional register file.

10. The processing element of any one of claims 1-9, wherein the processing element is in a data processing unit, the data processing unit further comprises a processing element, and the processing element comprises: a first register file to store another activation, a second register file to store another weight, and a multiplier to receive the another activation from the first register file, receive the another weight from the second register file, and compute a product of the another activation and the another weight.

11. A processing element, comprising: a comparator to perform a comparison on data elements and to generate a first output indicating a result of the comparison; a multiply-accumulate (MAC) unit comprising a multiplier and an adder, the MAC to perform a MAC operation and to generate a second output indicating a result of the MAC operation; and a multiplexer coupled to the comparator and the MAC unit, the multiplexer to output the first output or the second output as an output of the processing element.

12. The processing element of claim 11, wherein the data elements are in different channels of a neural network operation, and the first output is a maximum value or minimum value of the data elements.

13. The processing element of claim 12, wherein the data elements have a same spatial position in the different channels.

14. The processing element of any one of claims 11-13, wherein the comparator has a maximum mode and a minimum mode, the first output is a maximum value of the data elements in the maximum mode, and the first output is a minimum value of the data elements in the minimum mode.

15. The processing element of claim 14, further comprising: an additional multiplexer coupled to the comparator, the additional multiplexer to select one of the maximum mode and the minimum mode based on a configuration parameter.

16. The processing element of any one of claims 11-15, wherein the comparator is a first comparator in a comparator tree comprising a plurality of comparators, a second comparator in the comparator tree is to compute a third output indicating a result of an additional comparison, and a third comparator in the comparator tree is to compute a fourth output indicating a result of comparing the first output with the third output.

17. The processing element of claim 16, wherein the additional comparison is performed by the second comparator on additional data elements, and the data element of the data element and the additional data elements are in different channels of a neural network operation.

18. The processing element of claim 17, wherein the data elements and the additional data elements have a same spatial position in the different channels.

19. The processing element of any one of claims 16-18, wherein the plurality of comparators is implemented in a plurality of processing elements.

20. The processing element of any one of claims 11-19, wherein the processing element is in a data processing unit, the data processing unit further comprises a processing element, and the processing element comprises: a first register file to store another activation, a second register file to store another weight, and a multiplier to receive the another activation from the first register file, receive the another weight from the second register file, and compute a product of the another activation and the another weight.

21. A data processing unit, comprising: a first processing element, the first processing element comprising a comparator to compare data elements received by the first processing element; a second processing element, the second processing element comprising a multiplexer, a register file, and a multiplier to compute a product of a data element from the multiplexer and a data element from the register file; and a third processing element, the third processing element comprising a first register file, a second register file, and a multiplier to compute a product of a data element from the first register file and a data element from the second register file.

22. The data processing unit of claim 21, wherein the first processing element, the second processing element, and the third processing element are to perform different types of neural network operations.

23. The data processing unit of claim 21 or 22, wherein the data elements received by the first processing element are in different channels of a neural network operation and have a same spatial position in the different channels.

24. The data processing unit of any one of claims 21-23, further comprising: a comparator tree comprising a plurality of comparators, wherein the comparator is a first comparator in a comparator tree, a second comparator in the comparator tree is to compare other data elements received by a fourthprocessing element, and a third comparator in the comparator tree is to compare an output of the first comparator and an output of the second comparator.

25. The data processing unit of any one of claims 21-24, wherein the comparator is to output a maximum value of the data elements in a maximum mode and is to output a minimum value of the data elements in a minimum mode, and the first processing element further comprises a multiplexer to select one of the maximum mode and the minimum mode.

26. The data processing unit of any one of claims 21-25, wherein the comparator is to output a maximum value of the data elements in a maximum mode and is to output a minimum value of the data elements in a minimum mode, and the first processing element further comprises a multiplexer to select one of the maximum mode and the minimum mode.

27. The data processing unit of any one of claims 21-26, wherein the first processing element further comprises a multiplier and an adder.

28. The data processing unit of any one of claims 21-27, wherein the second processing element further comprises an adder to compute a sum of the product of the data element from the multiplexer and the data element from the register file and an additional product computed in the second processing element.

29. The data processing unit of any one of claims 21-28, wherein the data element from the register file in the second processing element is an activation, and the data element from the multiplexer in the second processing element is the activation.

30. The data processing unit of any one of claims 21-29, wherein the data element from the register file in the second processing element is an activation, and the data element from the multiplexer in the second processing element is one.

31. A method, comprising:determining a type of a neural network operation in a neural network; selecting one or more processing elements from a data processing unit based on the determined type of the neural network operation, the data processing unit comprising processing elements configured to perform different types of neural network operations; generating a configuration parameter based on the determined type of the neural network operation; and providing the configuration parameter to the one or more processing elements, the one or more processing elements to perform the neural network operation using the configuration parameter.

32. The method of claim 31, wherein the processing elements are arranged in one or more columns, and the one or more processing elements are arranged in a same column.

33. The method of claim 31 or 32, wherein a selected processing element comprises a multiplexer, and the multiplexer is to select data from a plurality of inputs based on the configuration parameter.

34. The method of claim 33, wherein the selected processing element further comprises a multiplier to compute a product of the data selected by the multiplexer and an activation.

35. The method of claim 34, wherein the data selected by the multiplexer is the activation.

36. The method of claim 31 or 32, wherein a selected processing element comprises a comparator, and the comparator is to output a maximum value or a minimum value of a plurality of inputs based on the configuration parameter.

37. The method of claim 36, wherein the selected processing element further comprises a multiply-accumulate unit comprising a multiplier and an adder.

38. The method of any one of claims 31-37, wherein determining the type of the neural network operation in the neural network comprises: identifying one or more computations in the neural network operation,39. The method of claim 38, wherein selecting the one or more processing elements from the data processing unit comprises: selecting the one or more processing elements based on the one or more identified computations.

40. The method of claim 38 or 39, wherein generating the configuration parameter comprises: generating the configuration parameter based on the one or more identified computations.

41. One or more non-transitory computer-readable media storing instructions executable to perform operations, the operations comprising: determining a type of a neural network operation in a neural network; selecting one or more processing elements from a data processing unit based on the determined type of the neural network operation, the data processing unit comprising processing elements configured to perform different types of neural network operations; generating a configuration parameter based on the determined type of the neural network operation; and providing the configuration parameter to the one or more processing elements, the one or more processing elements to perform the neural network operation using the configuration parameter.

42. The one or more non-transitory computer-readable media of claim 41, wherein the processing elements are arranged in one or more columns, and the one or more processing elements are arranged in a same column.

43. The one or more non-transitory computer-readable media of claim 41 or 42, wherein a selected processing element comprises a multiplexer, and the multiplexer is to select data from a plurality of inputs based on the configuration parameter.

44. The one or more non-transitory computer-readable media of claim 43, wherein the selected processing element further comprises a multiplier to compute a product of the data selected by the multiplexer and an activation.

45. The one or more non-transitory computer-readable media of claim 44, wherein the data selected by the multiplexer is the activation.

46. The one or more non-transitory computer-readable media of claim 41 or 42, wherein a selected processing element comprises a comparator, and the comparator is to output a maximum value or a minimum value of a plurality of inputs based on the configuration parameter.

47. The one or more non-transitory computer-readable media of claim 46, wherein the selected processing element further comprises a multiply-accumulate unit comprising a multiplier and an adder.

48. The one or more non-transitory computer-readable media of any one of claims 41-47, wherein determining the type of the neural network operation in the neural network comprises: identifying one or more computations in the neural network operation,49. The one or more non-transitory computer-readable media of claim 48, wherein selecting the one or more processing elements from the data processing unit comprises: selecting the one or more processing elements based on the one or more identified computations.

50. The one or more non-transitory computer-readable media of claim 48 or 49, wherein generating the configuration parameter comprises: generating the configuration parameter based on the one or more identified computations.

Images